omv/systemd: how to start the network earlier?

  • Hello


    After updates and reboots, my OMV installation ceased to work. Not just OMV, but the whole box is inaccessible.


    The serial console shows that it is not actially broken. It could work just (almost) fine. Just a USB disk stopped working. The only problem is that the network does not come up, preventing fixing the problem and also using the non-OMV parts.


    I found that, in systemd, the networking.service depends on local-fs.target, which in turn depends on all the local filesystems, including those that are not necessary for networking and similar stuff, and only hold the payload data that lives in OMV.


    That may also contribute to the problem that there is no way to do any filesystem related maintenance without completely turning off the box.


    How can I make the box work (partially) again? Starting the network and the most basic services (ssh ntp rsyslog) on boot would be a perfect.


    That probably involves some systemd magic, but also to stop OMV from undoing it again.


    thanks

  • There is no fix. The USB drive does not work.


    Dealing with extra thumb drives, another hassle on top of it to get something to work.


    It should be just trivial to get the network up inside the existing installation. Just how to do this?

    • Offizieller Beitrag

    Boot from a thumbdrive and you have full access to the root filesystem and can fix or edit any file.


    Or remove the drive with the root filesystem and plug it in to some other box and edit or fix the setup from there. Then put it back again.


    If the USB drive is broken, why is it plugged in? And what does it have to do with networking? Seems very strange.


    Or just write it off and restore from backup or reinstall from scratch.

  • The box boots only from SD card. Not worth to explore other possibilities.


    The USB drive is plugged in because this way I can get into "maintenance mode" i.e. get a root shell on sserial console. That is better than thumb drive booting, and better than taking out the SD card with the root filesystem.


    Yes I agree it should have nothing to do with networking. The systemd local-fs.target depends on both the root fs disks that are necessary for everything else, and the OMV data disks which are not, but happen to be "local filesystems".


    If these were separate "targets"/thingies the problem would be solved. However, OMV must also be prevented from restoring the problem.


    Any of these "Boot from thumb drive"/"reinstall OMV from scratch" does not solve the underlying problem: Whenever something minor fails, it will still be arbitrarily escalated into full system failure. That is what I intend to solve.

    • Offizieller Beitrag

    Take out the SD card and plug it into a SD card reader. Then use any Linux computer (possibly temporary) to mount the card and get full and unlimited access to the root filesystem. And do whatever you want to do to the filesystem and files on it. Change fstab or whatever.


    When you have it fixed, start cloning the SD card before any major changes or updates. That way any root fs problem, major or minor, can be instantly fixed by swapping to a SD card known to hold a good configuration.


    That is how I do it, if the configuration is non-trivial. If it is trivial, I reinstall from scratch.


    Still don't understand why you insist on having a broken USB drive plugged in. What purpose does that serve? It is broken! What benefit does it give?

  • Take out the SD card -> the system is down, and troubleshooting is harder. For example, the systemd logs are unreadable. Every time there is a minor problem in the future.


    Have networking -> The system is up and fully functional except for OMV itself. Every time this happens in the future. Maybe even OMV might start. Time to fix it.


    However, this thread has now so many postings about justifying my question that I cannot expect any more hints toward an answer.


    And again: With the "broken" drive, maintenance mode becomes available. The system is live, just not properly usable.Without, the system is effectively dead.

  • Oh, and a question like "One disk died or became inaccessible. Now the box won't boot. How to fix?" is not there where it is expected. Fixing will take quite some time.


    Zitat

    When you have it fixed, start cloning the SD card before any major changes or updates. That way any root fs problem, major or minor, can be instantly fixed by swapping to a SD card known to hold a good configuration.

    Hardware is not guaranteed to break in reverse order of configuration changes.


    And this discussion is getting way off topic, if I may set the topic as per original question.


    Some note about how a solution must look like:


    Just changing config files like the local-fs.target will not be stable. That file is obviously maintained by OMV.

  • I can imagine cloning local-fs.target into something like really-local-fs.target, and removing OMV data disks from this. Then make networking depend just on really-local-fs.target. However, make sure that some network services (NFS etc.) again depend on the full local-fs.target.


    This to start networking independently.


    Otherwise, to fix the OMV installation, I edited /etc/openmediavault/config.xml, with OMV not running. (Also removed a line from /etc/fstab) I wonder how to "apply" these changes, or propagate to generated files. Documentation seems to have nothing about it. I guess it is meant to be manually rebuilt from scratch each time.

    • Offizieller Beitrag

    If you have this many problems, there is something wrong with your hardware.


    OMV mounts all filesystems nofail. So, unless your raid array fails (shouldn't be using raid with usb disks...), the system will still boot unless the OS drive fails. In that case, nothing helps.

    Otherwise, to fix the OMV installation, I edited /etc/openmediavault/config.xml, with OMV not running. (Also removed a line from /etc/fstab) I wonder how to "apply" these changes, or propagate to generated files. Documentation seems to have nothing about it. I guess it is meant to be manually rebuilt from scratch each time.

    Some fixes could be fixed by chrooting into the OMV install and then running omv-mkconf (omv 4.x) or omv-salt (omv 5.x). Otherwise, make the database and config change manually. But I'm not sure why you are worried about this. I have never had to put an OMV os disk in another system to make a database change to get it to boot. If it is just networking, connect a keyboard and screen and login locally.

    Just changing config files like the local-fs.target will not be stable. That file is obviously maintained by OMV.

    Nope, OMV is not maintaining local-fs.target. Even if it was, you could create an override file that would never be touched by OMV. But your reason for doing this is still very odd and not something I would ever do.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!