Upgraded to 6.4.0 from 6.3.10 and having problems booting

  • I also seem to have the same issue with the same configuration changes pending as pplucky. However I am running OMV inside Proxmox. When running omv-salt deploy run hosts it runs fine. But running omv-salt deploy run systemd-networkd causes and error.


    When running omv-salt deploy run systemd-networkd the console shows the following error:

    After running omv-salt deploy run systemd-networkd the webinterface is down. Also the network interface is down.

    Running ip addr then gives me no ip address:

    Code
    2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_code1 state DOWN group default qlen 1000
        link/ether (hidden) brd (hidden)
        altname enp0s18 

    After rebooting I get back my IP address

    Code
    2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_code1 state DOWN group default qlen 1000
     link/ether (hidden) brd (hidden)
     altname enp0s18
     inet 192.168.1.x/24 brd 192.168.1.255 scope global dynamic ens18
        valid_lft 3268sec preferred_lft 3268sec


    I have also tried re-configuring my network device by running omv-firstaid. This also caused the server to hang and lose its IP address.


    I hope this helps in analyzing the issue.

    • Offizieller Beitrag

    I have the same situation after applying the update. Same error in the output of omv-salt deploy run systemd-networkd.

    Hmmm, maybe a bug in systemd-networkd. Can you please try


    Code
    # systemctl reset-failed systemd-networkd
    # omv-salt deploy run systemd-networkd

    Applying these two commands didn't help, the output is still the same error. In case it helps or I can provide more information to help.

    • Offizieller Beitrag

    I've installed the previous openmediavault 6.3.12 and can confirm the bug exists there, too. So it is not a OMV problem (or a new one which was introduced by 6.4.0), but a serious one in Debian.

    ok, I'll go back with a backup and reinstall when I have time. Thanks for the information and effort.

  • I had openmediavault 6.3.12 installed and it was not giving me a problem.

    A few days later I installed linux-image-6.1.0-0.deb11.7-amd64 also without a problem

    Today I installed the latest updates that contained:

    - The following NEW packages will be installed: libnss-resolve

    - The following packages will be upgraded: docker-ce docker-ce-cli openmediavault


    I now also notice that after the failed omv-salt deploy run systemd-networkd the contents of /etc/resolv.conf do not contain a nameserver.


    votdev: Am I correct that it seems to go wrong with the latest debian in combination with omv-salt deploy run systemd-networkd?




  • I think it is related to this:


    systemd-networkd and friends do not wait for driver to register interfaces · Issue #27822 · systemd/systemd
    systemd version the issue has been seen with 253 Used distribution Arch Linux Linux kernel version used 6.3.4-arch1-1 CPU architectures issue was seen on…
    github.com


    someone suggested this in the link above:


    A workaround for the issue is specifying required interfaces in the command line, e.g.

    Code
    systemd-networkd-wait-online -i eth0 -i eth1
    • Offizieller Beitrag

    I now also notice that after the failed omv-salt deploy run systemd-networkd the contents of /etc/resolv.conf do not contain a nameserver.omv-salt deploy run systemd-networkd?

    /etc/resolv.conf now points to /run/systemd/resolve/stub-resolv.conf instead of /run/systemd/resolve/resolv.conf. Name resolution is now done by systemd-resolved which is a proxy that is doing the DNS according to the configured DNS servers and other settings.


    votdev: Am I correct that it seems to go wrong with the latest debian in combination with omv-salt deploy run systemd-networkd?

    IMO there is a bug in systemd-networkd which causes the service to crash. I'll open a bug report at Debian and systemd this evening.

    • Offizieller Beitrag

    I've not really tested that, but it looks to me that this issue or something similar is causing the problems. I had a bond device that required the ens7 and ens8 network interfaces. After removing the files

    • 10-netplan-bond0.netdev
    • 10-netplan-bond0.network
    • 10-netplan-ens7.network
    • 10-netplan-ens8.network

    in /run/systemd/network/ i could restart systemd-networkd.service as often as i want without getting an error. After removingthe files there is only one ens6 interface that is configured. The mentioned systemd GH issue makes sense because the bond interface surely needs to wait for its children.


    Update: /lib/systemd/systemd-networkd-wait-online -i ens7 -i ens8 does not solve the problem.

  • In my configuration in folder /run/systemd/network/ there has been and is only 1 network device

    • 10-netplan-ens18.network

    This device (ens18) is the device that I expect


    After a reboot systemd-networkd is working again. I have checked that using systemctl status systemd-networkd. When it is running I can restart the service without a problem. However when the service has crashed (after omv-salt) a restart does not work, the service stays in a "starting" state.

    • Offizieller Beitrag

    mmmm .... I don't know how but my server has gotten out of this error in part, although there are still problems with the keys of some repositories when trying to omv-upgrade.

    I'll try to reproduce all my steps in case it helps, I have no idea what might have fixed it.


    This is the list of CLI applied commands that might be relevant.

    One of the first things I did was disable backports and apt clean in the omv-extras GUI

    Somewhere between these commands I made a change to the interface configuration in the GUI to force the reconfiguration, I simply changed one of the two DNS addresses from 1.0.0.1 to 1.1.1.1. Logically increased the list of modules pending to apply changes in salt.

    At some point, in the middle or so, I did a reboot.


    Nothing I tried worked so I took pictures of the GUI setup with the intention of reinstalling, went into edit and cancel on various setups, this shouldn't affect anything. But I did make a change, I had three NFS shared folders that I no longer used and I deleted them. I don't know if this may have affected.


    At this time the server was left running without anyone touching it. After an hour running without touching anything the first thing I did was apply GUI changes, just by trying again, and it worked without any problems. The changes have been applied, why? I have no idea, I haven't done anything else. Now it's working.

    The output of systemctl status systemd-networkd is good.


    The server is usable in this state but the output of omv-upgrade is not good.


    I don't know if I can provide any other information that might help.

    • Offizieller Beitrag

    In case it still works for someone. My server is ok apparently. I did the following:

    - I enabled backports

    - apt clean

    - omv-upgrade still gave me an error related to a firmware and I uninstalled it. apt-get purge firmware-ath9k-htc

    Everything is correct now.

  • In case it still works for someone. My server is ok apparently. I did the following:

    - I enabled backports

    - apt clean

    - omv-upgrade still gave me an error related to a firmware and I uninstalled it. apt-get purge firmware-ath9k-htc

    Everything is correct now.

    I have tried solving the issue by following your steps. For me this did not work. Systemd-networkd still crashes.


    Looking at the comments at https://github.com/systemd/systemd/issues/27854:

    • I noticed that the oldest related bugreport is from 2021, so the issue has been there already for a long time. Why does this issue only surface just now? Is it the latest Debian that got updated recently?
    • In later versions of systemd, not yet in stable debian, the issue seems resolved.

    Based on the steps of chente combined with the bugreport from votdev I was able to find a workaround for the issue:

    • Enable backports using omv-extras
    • run apt update to be sure
    • run apt install -t bullseye-backports systemd and when prompted keep the current non existent version of /etc/systemd/logind.conf.distrib
    • apply the configuration changes using the webinterface
    • Disable backports using omv-extras (to be sure no other backports are installed down the line)

    This works for me.

    • Offizieller Beitrag

    I have tried solving the issue by following your steps. For me this did not work. Systemd-networkd still crashes.


    Looking at the comments at https://github.com/systemd/systemd/issues/27854:

    • I noticed that the oldest related bugreport is from 2021, so the issue has been there already for a long time. Why does this issue only surface just now? Is it the latest Debian that got updated recently?
    • In later versions of systemd, not yet in stable debian, the issue seems resolved.

    Based on the steps of chente combined with the bugreport from votdev I was able to find a workaround for the issue:

    • Enable backports using omv-extras
    • run apt update to be sure
    • run apt install -t bullseye-backports systemd and when prompted keep the current non existent version of /etc/systemd/logind.conf.distrib
    • apply the configuration changes using the webinterface
    • Disable backports using omv-extras (to be sure no other backports are installed down the line)

    This works for me.

    I'm also wondering why the problem pops up now. The only relevant changes in 6.4.0 where symlinking /etc/resolv.conf-> /run/systemd/resolve/stub-resolv.conf instead of /run/systemd/resolve/resolv.conf and installing libnss-resolve.


    libnss-resolve should only interact with systemd-resolved and the symlink also only affect DNS resolution. systemd-networkd is not affected by these changes IMO. Maybe something in the kernel has been changed. I think there is a timing problem, systemd-networkd is not waiting long enough before the interfaces are coming up. But that's only an assumption.


    Hopefully some of the systemd developers are answering to my question so that the Debian maintainers can backport the relevant patches to Debian 11 stable.

    • Offizieller Beitrag

    This works for me.

    I'm glad my post helped you, I wasn't sure if it would help anyone. I'm no linux savant, I wish, my usual procedures are usually trial/error, but sometimes, just sometimes, they work :)

  • I'm also wondering why the problem pops up now. The only relevant changes in 6.4.0 where symlinking /etc/resolv.conf-> /run/systemd/resolve/stub-resolv.conf instead of /run/systemd/resolve/resolv.conf and installing libnss-resolve.


    libnss-resolve should only interact with systemd-resolved and the symlink also only affect DNS resolution. systemd-networkd is not affected by these changes IMO. Maybe something in the kernel has been changed. I think there is a timing problem, systemd-networkd is not waiting long enough before the interfaces are coming up. But that's only an assumption.

    You say that libnss-resolve should only interact with systemd-resolved. When I install the backport version of systemd I get the following notification:

    So systemd-resolved is not installed. Also apt-cache policy systemd-resolved is showing systemd-resolved as not installed

    Code
    root@openmediavault:~# apt-cache policy systemd-resolved
    systemd-resolved:
    Installed: (none)
    Candidate: 252.5-2~bpo11+1
    Version table:
    252.5-2~bpo11+1 100
    100 http://httpredir.debian.org/debian bullseye-backports/main amd64 Packages

    So... if libnss-resolve uses systemd-resolved and systemd-resolved is not installed it could be the root cause.


    It looks there is no version of systemd-resolved in the current debian version.


    When installing systemd-resolved I get:

    Einmal editiert, zuletzt von notdisclosed () aus folgendem Grund: Updated with result of installation of systemd-resolved

    • Offizieller Beitrag

    systemd-resolved is part of the systemd Debian package.

    It is possible that I am not looking at it correctly..., but systemd-resolved I think it is not here but it is in bullseye-backports, although only in the suggests category https://packages.debian.org/bullseye-backports/systemd

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!