Upgraded to 6.4.0 from 6.3.10 and having problems booting

Architekt · 28. Mai 2023

I was last on 6.3.10. I'm running this on 2 SSDs, an Intel i7-4790k, and an NVidia GeForce 2080 Super. I haven't had upgrade issues until this particular time, upgrading just now to 6.4.0. The first time I ran the update/upgrade from the web GUI, it downloaded and installed, and then asked me to apply the settings. I did, and then it sat there for over an hour and a half without any changes to the screen (i.e. not changing the status). I tried SSHing in (which I had enabled a while back and last tried it a few days ago) but not surprisingly I can't (guessing the service is down at this point). So then I tried turning off the power and restarting and left it for a bit (30m) but I wasn't able to connect to the IP (it's static, I assigned it via my router). Turned off the power again, and this time it at least booted and I could connect to the web gui but it still said I had to apply changes.

This time after hitting apply changes, it did the usual process where when it's done it reboots and you wait seeing the "Software failure. Press left mouse button to continue. The server is unavailable to handle this request right now." red message, like you do when it hasn't fully booted yet. Been sitting here for another 30ish minutes and nothing happens. I left click, it says "Loading..." then goes back to the red software failure screen.

Not sure what to try since I can't SSH in. I'd bring it upstairs and hook it up to my monitor but I broke my right arm so that's a no-go. I do have a super cheap monitor arriving tomorrow I'm going to hook up and see if I can't get more details, but until then was just wondering if anyone has had similar experiences or any suggestions.

If by the time I get my testing monitor tomorrow it's still in this state, I'll just restore from my last Clonezilla backup, but was hoping it's not in the meantime bricked, or that there's something I can do. Thanks for any input!

gderf · 28. Mai 2023

Zitat von Architekt

This time after hitting apply changes, it did the usual process where when it's done it reboots and you wait seeing the "Software failure. Press left mouse button to continue. The server is unavailable to handle this request right now." red message, like you do when it hasn't fully booted yet. Been sitting here for another 30ish minutes and nothing happens. I left click, it says "Loading..." then goes back to the red software failure screen.

Clear your browser cache or force reload the page.

donh · 28. Mai 2023

Try on the machine console. It may be picking up the wrong ip. There were some network changes.

Architekt · 28. Mai 2023

gderf: Already tried. I keep a Chrome install with nothing but the default install settings, no plugins as well for cases like these, also didn't work. Tried on my laptop, same issue even clearing cookies and restarting browser. Same for force reload.

donh: I have a static IP assigned to it. I put the MAC address in of my only ethernet card into my router so it's always the same. When this happens, I can't SSH into the box (most likely cause it hasn't booted to any sort of internet connectivity point or started the service).

I just now, for the 4th time, turned it off then back on and at least I can get it to boot and I can log in: But once logged in it keeps saying I need to apply my changes, which then puts me back into this cycle all over again.

Edit: Just realized I can see what needs to be applied. Currently it now has this much smaller list:

You must apply these changes in order for them to take effect.The following modules will be updated:

cron
hdparm
samba
systemd-networkd

Edit 2: I should say thanks for the prompt help. Text can sometimes come off as sounding arrogant. Anyway, I appreciate the suggestions, just in case that wasn't clear.

donh · 28. Mai 2023

Just guessing. Quote

donh: I have a static IP assigned to it. I put the MAC address in of my only ethernet card into my router so it's always the same. When this happens, I can't SSH into the box (most likely cause it hasn't booted to any sort of internet connectivity point or started the service).

If dhcp is not working for some reason you wont get the address you are expecting. The console is the way to check that. ip a

Architekt · 28. Mai 2023

I'm assuming by console you mean SSH via PuTTY (or whatever method you prefer)? If this is the case, I always access it via the static IP: 192.168.1.200. Since it seems to boot if I power it off then back on, just with the message of the above 4 updates needing to be applied, is there some sort of diagnostic I can do to I guess see what's happening when I hit apply? Like right now, since I powered it off and logged back in, I can SSH into my box so I'm wondering if there's a command I can use to follow along once I hit "apply" and see if I get any more details.

Architekt · 28. Mai 2023

Alright, so this time after a 5th boot, I SSH'd in and ran:

tail -f /var/log/syslog

THEN I hit apply. I get the following and it seems to just stop at the end there:

Code

root@omv:~# tail -f /var/log/syslog
May 28 13:16:12 omv kernel: [   23.383854] CIFS: VFS: The server doesn't seem to support them properly o                                                                                                           r the files might be on different servers (DFS)
May 28 13:16:12 omv kernel: [   23.383882] CIFS: VFS: Hardlinks will not be recognized on this mount. Co                                                                                                           nsider mounting with the "noserverino" option to silence this message.
May 28 13:16:22 omv systemd[1]: systemd-fsckd.service: Succeeded.
May 28 13:16:22 omv monit[782]: 'omv' Monit 5.27.2 started
May 28 13:16:22 omv monit[782]: Queued event file: unable to read event size -- end of file
May 28 13:16:22 omv monit[782]: Queued event file: unable to read event size -- end of file
May 28 13:16:22 omv monit[782]: Queued event file: unable to read event size -- end of file
May 28 13:16:24 omv systemd[1]: systemd-hostnamed.service: Succeeded.
May 28 13:16:25 omv runsvc.sh[833]: 2023-05-28 17:16:25Z: Agent reconnected.
May 28 13:16:25 omv runsvc.sh[833]: 2023-05-28 17:16:25Z: Listening for Jobs
May 28 13:16:46 omv crontab[3810]: (root) LIST (root)
May 28 13:16:50 omv systemd[1]: Started /bin/systemctl restart smbd.service.
May 28 13:16:50 omv systemd[1]: Stopping Samba SMB Daemon...
May 28 13:16:50 omv systemd[1]: Stopping Web Services Dynamic Discovery host daemon...
May 28 13:16:50 omv systemd[1]: smbd.service: Succeeded.
May 28 13:16:50 omv systemd[1]: Stopped Samba SMB Daemon.
May 28 13:16:50 omv systemd[1]: Starting Samba SMB Daemon...
May 28 13:16:50 omv smbd[3981]: [2023/05/28 13:16:50.567258,  0] ../../lib/util/become_daemon.c:135(daemon_ready)
May 28 13:16:50 omv smbd[3981]:   daemon_ready: daemon 'smbd' finished starting up and ready to serve connections
May 28 13:16:50 omv systemd[1]: Started Samba SMB Daemon.
May 28 13:16:50 omv systemd[1]: run-r191f223f45ce4ef2863c8ea77094ce62.scope: Succeeded.
May 28 13:16:50 omv systemd[1]: Started /bin/systemctl start wsdd.service.
May 28 13:16:51 omv systemd[1]: wsdd.service: Succeeded.
May 28 13:16:51 omv systemd[1]: Stopped Web Services Dynamic Discovery host daemon.
May 28 13:16:51 omv systemd[1]: Started Web Services Dynamic Discovery host daemon.
May 28 13:16:51 omv systemd[1]: run-r24169885d93b4873950446d83bae2360.scope: Succeeded.
May 28 13:16:51 omv wsdd.py[3997]: WARNING: no interface given, using all interfaces
May 28 13:16:53 omv systemd[1]: Reloading.
May 28 13:16:53 omv systemd[1]: systemd-networkd-wait-online.service: Succeeded.
May 28 13:16:53 omv systemd[1]: Stopped Wait for Network to be Configured.

Alles anzeigen

donh · 28. Mai 2023

If you can ssh it is getting the ip properly. By console I meant a monitor and keyboard connected to the hardware. Or a kvm or ipmi if you have them. You might try apt install -f and see if any errors show up that way.

If the web ui is working you could change a setting for something like smb enable or disable. That should force an update requiring apply in the yellow box. I think it can be done from cli too but I don't remember how to do it. A search might find that. The cli might give better errors.

donh · 28. Mai 2023

Try omv-firstaid to setup your network.

Architekt · 28. Mai 2023

Thanks for the tips. apt install -f didn't find anything to update/install. I'm going to revert to my last clonezilla backup and try again. This box just hosts Plex/Lidarr/Radarr/Sonarr so not much from a user-standpoint changes between my backups. I only do them before and after a system update (or if I make a lot of config changes to one of the above docker images which happens rarely.)

votdev · 29. Mai 2023

Try deploying the network configuration manually via CLI (keyboard and monitor connected is the best becasue SSH might loose connection). Run omv-salt deploy run hosts systemd-networkd. Make sure network manager is not responsible for the network configuration on your system. This is done by systemd (especially systemd-networkd and systemd-resolved) in OMV.

pplucky · 29. Mai 2023

Hello.

Having the same exact issue as OP Architekt, but as I don't have a wireless keyboard on hand, I tried running the suggestion above from votdev from SSH, and I ended up without connection to the server (as if I had applied the pending settings from UI).

Currently running 6.4.0-3 (Shaitan) on a RPi4, with fixed IP on my router and if I forcefully reboot the server, in the UI I get:

If I choose to apply the pending configuration changes, my system will hang again and lose network connection.

Tomorrow I should be able to have my wireless keyboard to try it directly via console. Will this eventually yield a different result or at least provide insights on what is causing this issue?

Is this a known issue resulting from this upgrade?

Btw, is there an easy way to revert the upgrade, or do I really need to restore last backup?

Thanks for all your help.

votdev · 29. Mai 2023

Zitat von pplucky

Is this a known issue resulting from this upgrade?

There shouldn't be any problems because OMV now installs and uses systemd as it should be. The correct way of installing and configuring systemd network fixes some DNS resolving issues that cause Salt to slow down. Now with the correct configuration of systemd-resolved this problem has been fixed beside the fact that the DNS related config was incorrect in previous versions. The main problem is that every Debian based distro OMV is running on cooks his own soup. The only real problem that comes to my mind is that network manager will fight agains systemd on some Debian derivates. In this case network manager needs to be disabled.

pplucky · 29. Mai 2023

Zitat von votdev

There shouldn't be any problems because OMV now installs and uses systemd as it should be. The correct way of installing and configuring systemd network fixes some DNS resolving issues that cause Salt to slow down. Now with the correct configuration of systemd-resolved this problem has been fixed beside the fact that the DNS related config was incorrect in previous versions. The main problem is that every Debian based distro OMV is running on cooks his own soup. The only real problem that comes to my mind is that network manager will fight agains systemd on some Debian derivates. In this case network manager needs to be disabled.

What exactly would you suggest doing on Raspbian GNU/Linux 11 (bullseye), in order to return OMV to a working state as it was before the last update? This, but directly in console to get any execution log?

votdev · 29. Mai 2023

Zitat von pplucky

What exactly would you suggest doing on Raspbian GNU/Linux 11 (bullseye), in order to return OMV to a working state as it was before the last update? This, but directly in console to get any execution log?

Sadly I have no Raspi, so I can not test anything nor know exactly what Raspi OS is installing and using to configure the network.

According to what I found on the inet the dhcpcd service is used.

Can you run the command systemctl status dhcpcd and post it here?

pplucky · 30. Mai 2023

Zitat von votdev

Sadly I have no Raspi, so I can not test anything nor know exactly what Raspi OS is installing and using to configure the network.

According to what I found on the inet the dhcpcd service is used.

Can you run the command systemctl status dhcpcd and post it here?

Code

Unit dhcpcd.service could not be found.

votdev · 30. Mai 2023

Zitat von pplucky
Code
Unit dhcpcd.service could not be found.

Can you tell how the network is configured on your system? Is it if-up-down or network manager?

empty555 · 30. Mai 2023

I have the exact same problem. My ip adress changes from it's fixed ip (192.168.0.x) to something like 172.18.0.1 after update.

OMV Is installed on proxmox so I can see the ip address changed...

pplucky · 30. Mai 2023

Zitat von votdev

Can you tell how the network is configured on your system? Is it if-up-down or network manager?

It seems to be ifupdown, as per below:

Code

pi@omv:~ $ apt-cache policy ifupdown
ifupdown:
  Installed: 0.8.36
  Candidate: 0.8.36
  Version table:
 *** 0.8.36 500
        500 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages
        100 /var/lib/dpkg/status

pi@omv:~ $ apt-cache policy network-manager
network-manager:
  Installed: (none)
  Candidate: 1.30.6-1+deb11u1
  Version table:
     1.42.4-1~bpo11+1 100
        100 http://httpredir.debian.org/debian bullseye-backports/main armhf Packages
     1.30.6-1+deb11u1 500
        500 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages

pi@omv:~ $ apt-cache policy dhcpcd5
dhcpcd5:
  Installed: (none)
  Candidate: 1:8.1.2-1+rpt9
  Version table:
     1:8.1.2-1+rpt9 500
        500 http://archive.raspberrypi.org/debian bullseye/main armhf Packages
     7.1.0-2 500
        500 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages

pi@omv:~ $ apt-cache policy dhcpcd
dhcpcd:
  Installed: (none)
  Candidate: 1:3.2.3-11
  Version table:
     1:3.2.3-11 500
        500 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages

pi@omv:~ $ apt-cache policy libnss-resolve
libnss-resolve:
  Installed: 247.3-7+rpi1+deb11u2
  Candidate: 247.3-7+rpi1+deb11u2
  Version table:
     252.5-2~bpo11+1 100
        100 http://httpredir.debian.org/debian bullseye-backports/main armhf Packages
 *** 247.3-7+rpi1+deb11u2 500
        500 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages
        100 /var/lib/dpkg/status

Alles anzeigen

Not absolutely sure if relevant but this is a fresh-installed Raspberry OS with OMV initially installed on a RPi3 with a dynamic IP address and then the SD card was moved to the final RPi4 with a different fixed IP address (fixed on router). To make it work, as per this post, I had to use console to run omv-firstaid to 'Configure network interface'. After that, I don't recall touching any network-related settings.

votdev · 30. Mai 2023

Zitat von pplucky

It seems to be ifupdown, as per below:

Code

pi@omv:~ $ apt-cache policy ifupdown
ifupdown:
  Installed: 0.8.36
  Candidate: 0.8.36
  Version table:
 *** 0.8.36 500
        500 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages
        100 /var/lib/dpkg/status

pi@omv:~ $ apt-cache policy network-manager
network-manager:
  Installed: (none)
  Candidate: 1.30.6-1+deb11u1
  Version table:
     1.42.4-1~bpo11+1 100
        100 http://httpredir.debian.org/debian bullseye-backports/main armhf Packages
     1.30.6-1+deb11u1 500
        500 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages

pi@omv:~ $ apt-cache policy dhcpcd5
dhcpcd5:
  Installed: (none)
  Candidate: 1:8.1.2-1+rpt9
  Version table:
     1:8.1.2-1+rpt9 500
        500 http://archive.raspberrypi.org/debian bullseye/main armhf Packages
     7.1.0-2 500
        500 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages

pi@omv:~ $ apt-cache policy dhcpcd
dhcpcd:
  Installed: (none)
  Candidate: 1:3.2.3-11
  Version table:
     1:3.2.3-11 500
        500 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages

pi@omv:~ $ apt-cache policy libnss-resolve
libnss-resolve:
  Installed: 247.3-7+rpi1+deb11u2
  Candidate: 247.3-7+rpi1+deb11u2
  Version table:
     252.5-2~bpo11+1 100
        100 http://httpredir.debian.org/debian bullseye-backports/main armhf Packages
 *** 247.3-7+rpi1+deb11u2 500
        500 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages
        100 /var/lib/dpkg/status

Alles anzeigen

Not absolutely sure if relevant but this is a fresh-installed Raspberry OS with OMV initially installed on a RPi3 with a dynamic IP address and then the SD card was moved to the final RPi4 with a different fixed IP address (fixed on router). To make it work, as per this post, I had to use console to run omv-firstaid to 'Configure network interface'. After that, I don't recall touching any network-related settings.

The ifupdown configuration files are disabled by OMV, you should see something like

Code

# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.

# interfaces(5) file used by ifup(8) and ifdown(8)
# Better use netplan.io or systemd-networkd to configure additional interface stanzas.

# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d

in /etc/network/interfaces. Are there any error messages when you run omv-salt deploy run hosts systemd-networkd from CLI. You should connect a keyboard and monitor to the system because SSH might get lost. When the command has succeeded, did you get the expected IP when you run ip addr? Does a reboot work? If possible, get the full syslog output of the reboot after applying the changes and if the network does not come up successfully. There must be something in the logs that might help to find the reason what happens.

Upgraded to 6.4.0 from 6.3.10 and having problems booting

Jetzt mitmachen!

Tags