Can't SSH into RPi server after OMV 6 to OMV 7 upgrade

After updating from OMV 6 to OMV 7 on my Raspberry Pi 4B (Raspberry Pi OS 32bit) using sudo omv-release-upgrade and running wget -O - https://github.com/OpenMediaVault-Plugin-Developers/installScript/raw/master/fix6to7upgrade | sudo bash, attempting to connect to the server using SSH results in the following error message: Permission denied, please try again.

I can log into the web GUI and I can also log into the account I'm using for the SSH connection when I connect the server to a monitor and keyboard. SSH is enabled in Services/SSH and the user is in the _ssh group. I have tried Putty, cmd and PowerShell (W10), none of which work. I have also tried changing the user's password using passwd, but this didn't help. There are no updates available for the system and I have done plenty of reboots since the issue arose. I haven't installed any plugins (except for omv-extras and any others which might have been included by default in the RPi install script).

I had also run into the issue where I couldn't access the web GUI with a 502 error (as described in OMV 6 to 7 upgrade.. Error 502 Bad gateway). I also couldn't establish an SSH connection at this point. The web GUI functionality was fixed after connecting the server to a monitor and keyboard and running the fix6to7upgrade script.

Any tips on what I might try to fix this would be appreciated.

Quote from ryecoaaron

Did you look at the Authentication log in the web interface under Diagnostics -> System Logs - > Logs?

Not before you suggested it, thanks. I can see the following messages regarding SSH (chronologically descending):

sshd[951]: Server listening on 0.0.0.0 port 22.
sshd[951]: Server listening on :: port 22.
login[1119]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/tty1 ruser= rhost= user=<root username>
login[1119]: FAILED LOGIN (1) on '/dev/tty1' FOR '<root username>', Authentication failure
sshd[1529]: User <root username> from <workstation local IP> not allowed because none of user's groups are listed in AllowGroups
sshd[1529]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<workstation local IP> user=<root username>
sshd[1529]: Failed password for invalid user <root username> from <workstation local IP> port 50667 ssh2
sshd[1529]: Connection reset by invalid user <root username> <workstation local IP> port 50667 [preauth]

Also potentially relevant are the following (happened after the above):

systemd-logind[...]: The system will power off at Mon 2024-07-15 13:36:18 CEST!
systemd-logind[...]: Creating /run/nologin, blocking further logins...
systemd-logind[...]: System shutdown has been cancelled

These are from a reboot I called and then cancelled when logged in with a monitor and keyboard.

Also, for some reason, the authentication logs stop after the cancelled shutdown, which was about 2 hours ago, even though I have attempted more SSH logins after that.

I think this might be the same issue as in Upgrade to OMV 6 -> 7 Failed: the user is in the _ssh group (note the underscore) but sshd_config allows the group ssh (no underscore). Would the appropriate step to fix this be to add the user to the ssh group, or should I change sshd_config to allow the _ssh group?

Anyone looking at this thread in the future might also be interested in reading Another "502 Bad Gateway" error after upgrading to OMV7 on Raspberry Pi.

Quote from ryecoaaron

You are trying to login as root. root does not need to be in _ssh.

Because this is a RPi, the root user's username is not root and they are not in the root user group (rather they are in the pi as well as the sudo groups). sshd_config allows the groups root and ssh, neither of which this user is in.

Quote from ryecoaaron

When you "cancelled" (can't really do that) the shutdown

Do you mean "can't" as in "it is technically not possible", or as in "it is highly not recommended"? Because I definitely was technically able to cancel the shutdown.

Quote from ryecoaaron

you stranded the /run/nologin file.

I have never not been able to log in locally (monitor + keyboard).

Quote from ryecoaaron

I would reboot the machine and try again.

I have done that multiple times (one more time as of writing this) without any luck.

Quote from ryecoaaron

I would also make sure that root is allowed to login in the ssh tab.

I can confirm that "Permit root login" is definitely enabled in Services -> SSH and that there are no pending configuration changes.

Are you sure that the issue doesn't lay with the user groups? One of the logged messages did say [...] not allowed because none of user's groups are listed in AllowGroups.

Quote from votdev

Run omv-salt deploy run ssh

Thank you, this was it! I don't understand the full extent of what this command did, but it did change the AllowGroups line in /etc/ssh/sshd_config from AllowGroups root ssh to AllowGroups root _ssh.

Quote from ryecoaaron

The install script tries to add the pi user to the _ssh group

The pi user was in the _ssh group (with the underscore), but the /etc/ssh/sshd_config file allowed SSH for the ssh group (no underscore). I don't know what the groups and configs were in OMV6, but here the problem seemed to be with the config file - maybe it was recreated into a default state during the upgrade? That's just guessing on my part though.

Quote from ryecoaaron

[Cancelling a shutdown] leaves the system in a bad state where many things are going to be broken.

I never knew that, I always assumed that the command "knew what it was doing" haha. Thanks!

Quote from ryecoaaron

I added to the fix6to7upgrade script to rebuild the ssh config file.

Thank you! Others have definitely encountered the same issue in Upgrade to OMV 6 -> 7 Failed, so hopefully it won't be a problem moving forward