Recently, I took care of my NAS with an upgrade to OMV7 (from 6) and change of the SFP+ PCIe card for a better one. I'm unsure if that's the hardware or software change that caused this issue, but processes took way too much time to start than usual, triggering monitoring alerts.
The issue is rather a Linux/Systemd issue than an OpenMediaVault. However I didn't experience it on a manually installed system with which I swapped the interface on as well (Debian 12), so there might also be something from OMV side that could be improved.
After swapping network interfaces, I noticed that upon reboot, the monitoring complained about many services not starting properly on boot.
After some research, I found some interesting commands to run right after reboot, and once the boot is fully completed, over here: https://ubuntuforums.org/showthread.php?t=2490962
PiHole also has issues related to this: https://forums.raspberrypi.com/viewtopic.php?t=362055
Right during boot time, before OMV panel is accessible, you can run : systemctl list-jobs
root@nas:~# systemctl list-jobs
JOB UNIT TYPE STATE
124 smbd.service start waiting
125 collectd.service start waiting
109 network-online.target start waiting
178 openmediavault-beep-up.service start waiting
147 postfix.service start waiting
148 postfix@-.service start waiting
123 wsdd.service start waiting
171 samba-ad-dc.service start waiting
2 multi-user.target start waiting
126 rrdcached.service start waiting
163 nginx.service start waiting
136 rpc-statd-notify.service start waiting
1 graphical.target start waiting
103 getty.target start waiting
173 systemd-update-utmp-runlevel.service start waiting
113 systemd-networkd-wait-online.service start running
177 proftpd.service start waiting
108 openmediavault-issue.service start waiting
18 jobs listedAs you can see here, the systemd-networkd-wait-online.service is the only one running, and other ones are waiting for it to complete.
It would appear that because of the interface change, this task waits for an interface that doesn't exist anymore and therefore takes a (relative) tremendous amount of time to complete (around 2m30) and let other tasks go through. In the meantime, that causes the monitoring feature from OMV to generate alerts, that will be sent all at once, as soon as the mailing system is up.
After boot, you can also use the following commands to get some info about what takes time to start:
systemd-analyze critical-chain
root@nas:~# systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.
graphical.target @2min 3.037s
└─multi-user.target @2min 3.036s └─getty.target @2min 3.036s └─openmediavault-issue.service @2min 546ms +2.489s └─network-online.target @2min 506ms └─network.target @2.026s └─wpa_supplicant.service @2.009s +16ms └─dbus.service @1.994s +12ms └─basic.target @1.993s └─sockets.target @1.993s └─systemd-journald@netdata.socket @1.993s └─sysinit.target @1.992s └─systemd-resolved.service @1.934s +57ms └─systemd-tmpfiles-setup.service @1.926s +6ms └─local-fs.target @1.922s └─run-credentials-systemd\x2dtmpfiles\x2dsetup.service.mount @1.927s └─local-fs-pre.target @384ms └─lvm2-monitor.service @216ms +134ms └─systemd-journald.socket @212ms └─-.mount @168ms └─-.slice @168mYou have to read this output from bottom to top in order to have sequential order. Here on this output, you can see 'network-online.target' takes a lot of time while everything else is basically instantly started.
Other useful command (I don't have the "before" output anymore for this one) is:
systemd-analyze blame
Ultimately, you can confirm that this service is the issue by restarting it manually. If your command prompt stalls for minutes, then that is likely your issue. Command to restart the service is the following:
systemctl restart systemd-networkd-wait-online.serviceSo we've confirmed the problem, what's the fix?
Solution is to tell systemd-networkd-wait-online to wait for any interface to be up, rather than for whatever interface it remembers.
For that, edit the service start options with:
systemctl edit systemd-networkd-wait-online.serviceWe have to set the ExecStart variable empty first otherwise it won't work, which is why you can see 'ExecStart=' twice in the fix code. That is not a mistake, I tried without it, it doesn't work. Systemd mysteries.
Add this:
[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --anyMake sure you set it at the right location, between the beginning lines, otherwise it will be overwritten and won't work. (I tried too, didn't read properly the conf file at first)
So full beginning of file looks like this:
### Editing /etc/systemd/system/systemd-networkd-wait-online.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file
[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --any
### Lines below this comment will be discardedThen you can try to restart the service.
systemctl restart systemd-networkd-wait-online.serviceIf it takes less than a second, you're probably good to go. If not, then you need to search further.
Some say they have to specify the interface name for this to work. This is achieved with the following syntax:
ExecStart=/lib/systemd/systemd-networkd-wait-online --interface=eth0I hope it helps fellow OpenMediaVaulters that maybe had this issue but didn't bother fixing, and Googlers.
Also maybe our awesome OMV devs with their systemd knowledge have simple ideas to mitigate the issue natively.
All the best, and never forget: RAID is not a backup.
