SFTP-Plugin deactivates root-login -> intentionally?

Hi there,

I just installed 3 almost identical machines with OMV 6.0.24 amd64 (and added a couple of plugins, no drives yet, no shares, no data. Apart from the other plugins the system is as clean as an infants buttocks. Some errors occurred with plugins, but just the usual 500-error of nginx, and the "need to clear cache" thing, nothing that seemed to be a real error).

After installing the SFTP-Plugin I noticed the root-login had been disabled. I got the machines in the next room, so no big deal, but I wondered if that is intentional.

Just to test it I uninstalled the plugin on one of the machines, still no login.

I also tried what happens if I deactivate SSH root login, save that and then reactivate it (did that before I uninstalled SFTP). Also did not resolve it.

So for someone installing the plugin on a remote machine using SSH passwords possibly a bit of an issue.

Just wanted to let you know. Since it happened on all three machines I'm rather certain it's not an exceptional thing.

Interesting.

I am getting the login prompt still on 22 and it says

Quote

Permission denied, please try again.

Also I just checked, on 222 is no response, as it should be. Would be weird.

Quote from ryecoaaron

sftp plugin does not change anything with ssh. It only changes settings on the port it is listening on. Can you post your /etc/ssh/sshd_config and /etc/ssh/omv_sftp_config files when root login is disabled?

I'd love to, but right now I got trouble getting in...

I created another User, which I added to groups SSH (as well as sftp-access and omv-admin). Can't login with that one either...

So copy&paste is a bit inconvenient right now.

But I got the file in front of me, I'll quote what I think might be important and if that's not enough let me know what to look for.

/etc/ssh/sshd_config

StrictModes yes

AllowGroups root ssh

Port 22

PermitRootLogin yes

PasswordAuthentication yes

/etc/ssh/omv_sftp_config

StrictModes yes

Port 222

PermitRootLogin yes

This is weird.

My first thought was it was something dumb, like using the wrong login since it's three machines. But I made sure, the one I'm locally logged in is the same IP I used for SSH, locally password works, via SSH it doesn't.

Quote from ryecoaaron

sftp plugin does not change anything with ssh. It only changes settings on the port it is listening on.

Just in case it wasn't the SFTP-Plugin and the error is actually caused by another one (I might have just noticed it after the SFTP-Plugin-Install) here is a list of the other plugins I installed:

backup

borgbackup

cputemp

flashmemory

kvm

mergerfs

omvextras

remotemount

resetperms

rsnapshot

sftp

sharerootfs

snapraid

symlinks

(side note: sharerootfs is a dependency of one of the others? I do not recall installing it and... I'm a bit of a docu freak, I logged in a text file every plugin I installed, and that one isn't there... But it's installed on the other machines as well, so wasn't a misclick)

Also I installed a bunch of cli-tools which usually don't cause problems, but not that clean after all (just thought of those):

ncdu glances htop atop nmon bmon byobu rdfind tree tldr aria2 mosh speedtest-cli rclone autojump

Maybe that helps.

Just to be sure I just tried a local login with the second user I mentioned, no problem.

So both can login locally, both can't via SSH.

I also made sure it's not the keyboard and copy-pasted the password.

So unless I'm missing something: somewhere the login isn't permitted, but it is neither deactivated PasswordAuth nor PermitRootLogin.

I'm not aware of another switch for that...

Since I didn't have any better idea:

- there's no AllowUsers restriction etc. apart from the above mentioned AllowGroups

- restarting SSH --> no change

- turn off root login via webinterface -->

/etc/ssh/sshd_config RootLogin No (as it should)

--> exactely same behavior as before, root AND the non-root user can not login, same "permission denied" response. All on port 22.

- changing it back also no problem, but still same result, no login via SSH.

I thought this is just a small thing but I sort of am irritated now...

managed to do it via USB

sshd_config

Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
StrictModes yes
IgnoreRhosts yes
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
UsePAM yes
AllowGroups root ssh
AddressFamily any
Port 22
PermitRootLogin yes
AllowTcpForwarding no
Compression no
PasswordAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 /var/lib/openmediavault/ssh/authorized_keys/%u
PubkeyAuthentication yes

omv_sftp_config

Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
StrictModes yes
IgnoreRhosts yes
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
Subsystem sftp internal-sftp -l INFO
UsePAM yes
Port 222
PermitRootLogin yes
Compression yes
PasswordAuthentication yes
PubkeyAuthentication no
AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 /var/lib/openmediavault/ssh/authorized_keys/%u

AddressFamily any
Match Group sftp-access
  ChrootDirectory /sftp/%u
  X11Forwarding no
  AllowTcpForwarding no

Quote from ryecoaaron

With those configs, root should be able to login to port 22 and 222. Are you saying it is not allowed?

Yes, exactly. I'm getting "Permission denied, please try again."

Only on port 22 though, on port 222 I get a "connection refused".

Quote from ryecoaaron

Do you have fail2ban installed? The configs say it should allow root unless you have the wrong password or the service hasn't been restarted.

- no fail2ban, only the plugins listed above

- dashboard says SSH runs fine, also it does respond with "Permission denied" and I did restart SSH before and restarted the machine a couple of times by now

Quote

Have you looked at /var/log/auth.log when the connection is being rejected? I really don't know why your system is working this way.

Three systems. All the same behavior. I am sort of irritated as well...

I could install on another SSD to see if this can be replicated. But like I said - I did exactly the same thing on three machines, and the same result.

/var/log/auth.log

I just went through it, didn't know that one, thanks.

So I can see that the problems started after I installed the plugins (I had a list of plugins i would need and got through that list on all three machines simultaneously).

Now what is interesting:

Quote

Failed password for root from XXX.XXX.XXX.XXX port 49154 ssh2

Is the port normal?

Now in case you didn't notice what I wrote above I'd like to stress this again - I'm a documentation nerd. I got a hierarchical lists I'm working through in an outliner when I install OMV (so mostly copy and paste) and I installed quite a lot of them this way.

But if I were you I'd suspect user-error - which is why I tried to eliminate those above. But still - if I may have overlooked anything lets verify. However the only explanation I got at this point would be that the root password got changed somehow - which just can't happen...

To clarify:

- I use the same password to login locally just fine

- I used the exact same document to copy the password from before to login just fine, (before installing the plugins)

- now that password does not work, copied or typed by hand

- just to make sure I didn't make some stupid mistake without noticing (even though I guarantee I'm as precise about these things as it gets) I tried all three different passwords of all three machines - so one of them would have to be the one, even if I would have mixed things up - which I did not.

I got nothing.

Also as mentioned above I get the same result with another user, which I specifically created to test if that one can log in. With the password I created I can log in locally, not via SSH.

Edit:

just had an idea and changed the password of that user in the webinterface, to something very simple (xxx) - and now I can log in.

OK, I really wanna know if I can replicate this... I'll report back on that once I did all that on another machine.

In the mean time:

- resetting the password over cli locally is the simple solution for now I guess to get the machines back to work. If anyone should ever have an issue like this. Which is... so weird.

edit2:

I can't get my head around how I could log in as "kwon" locally, not via SSH but after resetting the password I suddenly can?!??

also I get at login via ssh:

Quote

Could not chdir to home directory /home/kwon: No such file or directory

Quote from ryecoaaron

Can you try a simple password without cut & paste? I have a dozen OMV boxes and use root on all of them. I have no idea how you are getting this issue.

I do too, and never had that issue before.

I can try that with one of those already installed that show the issue, but to be honest, I'd like to keep two of them as is for now, so I can investigate further if necessary in the logs.

And I'll install the third one on another SSD with - like you suggested - a simple password and try to recreate the issue. Since I have a log of everything I did in detail that should not be a problem.

No worries, thanks so far and have a good trip.

I had a bunch of other things to deal with myself, but now I can offer some further results and I'm quite irritated still and wonder what the heck I might do that is different since it does not seem to effect others:

- I had a completely different machine (different hardware) installed two weeks ago, exactly the same issue. In this case it was a bit of a problem, since it went to a friends place, so being able to log in remotely was kind of important. In the end I couldn't really fix it before we had to take the machine to his place. Here is my log of what I did:

1. install with freshly downloaded openmediavault_6.0.24-amd64

2. install bunch of cli-tools (I'll go into that later, not the issue even though I suspected those first)

3. installed plugins (backup, borg, diskstats, mergerfs, resetperms, rsnapshot, sftp, snapraid, symlinks, cputemp)

4. formated HDDs, created shared folders and activated SMART checks

5. backupjobs for rsync and rsnapshot

6. at that point I noticed the notifications aren't sent, I had that issue before, that after a while they suddenly appear. I guess this is another issue.

7. dashboard activated, set userpassword for webinterface, added two users.

After a restart I again could

- login to the webinterface with no issue

- login locally on the machine with root

- could NOT log in with SSH

- after changing the root-password locally it works again, until the next restart.

I originally thought it is related to the complexity of the password, with easier passwords it seemed the problem does not occur, but that was false. See below.

So the strongest indicator what somehow creates the problem is the restart. (??!??)

Now today I had time again to fiddle with the three machines I had the problem first with. And this is weird, stay with me:

I intentionally did only very rudimentary things and logged everything yet still I get the same problem:

1. install with freshly downloaded openmediavault_6.0.24-amd64 (on three identical machines, installed with three separate USB-thumbdrives, but created from the same ISO)

2. NO cli tools installed at all this time to be sure it's not them.

3. loaded updates via webinterface

4. plugins: backup, borg, rsnapshot - nothing else

5. settings: set timezone, logout to an hour, powerbutton to shutdown

6. restart

--> same fucking shit...

During the installation of updates I got this error which I'm pretty sure is not related:

Setting up Salt environment ...
[ERROR   ] Command '/usr/bin/patch' failed with return code: 1
[ERROR   ] stdout: patching file /tmp/__salt.tmp.lv7pru4c (read from /lib/python3/dist-packages/salt/fileserver/roots.py)
Reversed (or previously applied) patch detected!  Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file /tmp/__salt.tmp.cr2of4lk
[ERROR   ] retcode: 1

Solution / sort of work around:

because it seemed just unreal I tried again a couple of different ways to log in and varying ways to insert the password:

- remember, locally always no problem, only via SSH the login fails

- I safe the passwords in a passwordsafe / during installation in keepnote

a) if copy and paste from keepnote --> does not work after restart, but once changed the password locally it works again (????)

b) if instead of GUI for copy paste I use strg+shift+v -> same result, no login

c) if I type it in by hand --> no problem (which sucks, since I use rather complex passwords) but at least I can log in now

d) Now the funny bit: I type the password by hand in another line (directly below to make sure there's no typo), copy it --> NO login

e) AFTER I once had logged in by typing by hand THEN it works again with copy and paste, no matter of GUI or strg+shift+v

It took me a while to get through all the possible variants. And it fuckin sucked!!!

But I guess this is not a OMV issue. I'm using Debian Buster on the client, might be that one (unlikely, since it only occurs with OMV6 machines) or maybe some weird thing how debian 11 deals with copypasted passwords.

Does anyone of you have regular contact to someone in the debian-universe to investigate this further?

If nobody else has had this issue it may mean

a) everybody instantly switches over to keys once installed (unlikely)

b) everyone uses way to simple passwords and don't mind typing them (I hope this is unlikely but am not sure)

c) there's something else involved I haven't figured out yet (might very well be)

In any case, this was not fun. I'm gonna get some chocolate ice now. And rethink my choices...