Mounting existing EXT4 filesystem = unresponsive GUI, SSH fails, root errors

Good evening, I recently upgraded my server hardware and decided I would start with a fresh installation of OMV 6. I've overcome a few minor issues along the way, but I seem to be stuck at mounting the existing filesystems from my 10 data drives (all of which were created in previous versions of OMV).

I installed OMV6 with only the system drive plugged in, and since then, have been able to mount the filesystem on my secondary SSD. All my other drives are HDDs , and immediately upon mounting any of their filesystems, the GIU turns unresponsive with a "502 - Bad Gateway" error, and I can no longer SSH into the machine. If I manually shut the server down, log back in to the GUI and revert the changes, then everything is fine.

I'd appreciate any tips you could provide to get those filesystems working again!

Edit: If I already have established an SSH connection before I mount the filesystem and accept the changes, it will stay alive, but trying to use sudo or switch to root gives me an error message that the "effective UID is not 0." Initiating a new SSH connection once the 502 errors start are always closed unexpectedly.

Thanks greno, I'll give that a shot in a couple of hours, but the fact that SSH connections are rejected would lead me to believe there's more going on. Also, some additional info:

If I already have established an SSH connection before I mount the filesystem and accept the changes, it will stay alive, but trying to use sudo or switch to root gives me an error message that the "effective UID is not 0." Initiating a new SSH connection once the 502 errors start are always closed unexpectedly.

Zoki - Here is fstab prior to mounting a "troublesome" filesystem:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# systemd generates mount units based on this file, see systemd.mount(5).
# Please run 'systemctl daemon-reload' after making changes here.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sdb2 during installation
UUID=fc5fc7aa-d086-459a-af24-fb9181a4a157 /               ext4    errors=remount-ro 0       1
# /boot/efi was on /dev/sdb1 during installation
UUID=0F22-816F  /boot/efi       vfat    umask=0077      0       1
# swap was on /dev/sdb3 during installation
UUID=fe6ecd0b-f840-48de-89e5-33dcdbdc50d1 none            swap    sw              0       0
# >>> [openmediavault]
/dev/disk/by-uuid/e6462b5e-818f-47a3-a526-8577ce4566b6          /srv/dev-disk-by-uuid-e6462b5e-818f-47a3-a526-8577ce4566b6   ext4    defaults,nofail,user_xattr,jqfmt=vfsv0,acl      0 2
# <<< [openmediavault]

After mounting one, fstab looks identical, so it appears that the new filesystem is never committed. Since I started running into trouble, I have only been connecting a single drive at a time. Some more information that may or may not mean anything:

I'm using the Proxmox kernel. I have most of the drives (but not all) connected through an SAS expander, but all the drives have this issue, even the ones connected directly to a mobo SATA port. greno, using an Incognito window or new browser does not resolve the issue.

Zoki I just meant that after mounting the FS in the GUI and clicking to accept the changes, there is no update to fstab. Also, the new FS does not appear in /srv/.

I have not tried to manually mount anything, I'll look that up and give it a try.

Edit - could this have anything to do with the filesystem paths changing from when the OS was installed? In other words, OMV is installed on /dev/sdb*, and that probably keeps changing as I connect new drives to the motherboard.

Final edit - Mounting the drive manually actually does work (the drive is immediately accessible through the /srv/sda1 mount point I selected), but the FS is not displayed anywhere in the OMV GUI. Also., after rebooting, the /srv/sda1 mount point is empty when viewing in the terminal. Mount shows that the mount point is no longer in use:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=16301552k,nr_inodes=4075388,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3266592k,mode=755,inode64)
/dev/sdc2 on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19992)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/sdc1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
tmpfs on /tmp type tmpfs (rw,relatime,inode64)
/dev/sdb1 on /srv/dev-disk-by-uuid-e6462b5e-818f-47a3-a526-8577ce4566b6 type ext4 (rw,relatime,jqfmt=vfsv0)
/dev/sdc2 on /var/folder2ram/var/log type ext4 (rw,relatime,errors=remount-ro)
folder2ram on /var/log type tmpfs (rw,relatime,inode64)
/dev/sdc2 on /var/folder2ram/var/tmp type ext4 (rw,relatime,errors=remount-ro)
folder2ram on /var/tmp type tmpfs (rw,relatime,inode64)
/dev/sdc2 on /var/folder2ram/var/lib/openmediavault/rrd type ext4 (rw,relatime,errors=remount-ro)
folder2ram on /var/lib/openmediavault/rrd type tmpfs (rw,relatime,inode64)
/dev/sdc2 on /var/folder2ram/var/spool type ext4 (rw,relatime,errors=remount-ro)
folder2ram on /var/spool type tmpfs (rw,relatime,inode64)
/dev/sdc2 on /var/folder2ram/var/lib/rrdcached type ext4 (rw,relatime,errors=remount-ro)
folder2ram on /var/lib/rrdcached type tmpfs (rw,relatime,inode64)
/dev/sdc2 on /var/folder2ram/var/lib/monit type ext4 (rw,relatime,errors=remount-ro)
folder2ram on /var/lib/monit type tmpfs (rw,relatime,inode64)
/dev/sdc2 on /var/folder2ram/var/cache/samba type ext4 (rw,relatime,errors=remount-ro)
folder2ram on /var/cache/samba type tmpfs (rw,relatime,inode64)

Thanks for the heads up that mount is only temporary and does not persist through a reboot. I am open for suggestions on where to go from here.

One of the drives is brand new with no data on it, so I tried a few things since there was no risk of data loss. First, the drive itself was visible in the drives section of the GUI, so I tried to create a new filesystem on that drive, and it didn't show up in the drop-down menu. I assume this is because the existing FS was being detected. I then wiped the drive and created a new filesystem directly in OMV6 (this was much faster than OMV5 on a 14TB drive, by the way). This newly created filesystem could also NOT be mounted. I experienced the exact same issues as all the others.

See the output below (shortened for sanity). Afterwards, I rebooted and tried to mount again, same issue.

omv-salt stage run prepare
<snip>

Summary for debian
------------
Succeeded: 6 (changed=5)
Failed:    0
------------
Total states run:     6
Total run time:  16.266 s

omv-salt stage run deploy
<snip>

Summary for debian
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:  31.393 s

Thanks votdev, I checked the log and these are all the entries from the time I rebooted until mounting the filesystem in the GUI and before I accepted the changes. Nothing here stands out to my eyes: https://pastebin.com/6cyfDV4k.

After clicking the check box and confirming the changes, resulting in the errors described earlier, a great deal of the end of the log is actually gone completely. The timestamp for the latest entries is a full 5 hours earlier than the previous log: https://pastebin.com/1KwKLXrq.

(Edit - See below, not fixed as I had hoped) Since I had some time to kill and nothing to lose, I did a fresh installation of OMV 6. I followed almost the exact same process, but this time, I was able to mount all my filesystems without issue. Either the whole thing was a fluke, or one one of the following things is what caused the error (I didn't do any of these before mounting the filesystems, unlike the first time when I experienced all the problems):

1. changing my kernal to Proxmox and removing the non-Proxmox kernel
2. Installing omv-extras and the following plugins: flashmemory, mergerfs, resetperms, snapraid, and symlinks

Edit - Well, now I am unable to gain access to the GUI (Error 500 - Internal Server Error, Failed to Connect to Socket). This time I installed omv-extras and all the plugins listed above AFTER everything was mounted. I have no evidence to support this, but I feel like it may be flashmemory. I noticed that it was not running (red status on the dashboard), realized I never rebooted after installing, so I rebooted to see if the service would run. Immediately I was faced with this new issue.

I found this thread which sounded similar, and tried the command that was suggested there:

dpkg --configure -a
dpkg: error: unable to access the dpkg database directory /var/lib/dpkg: Read-only file system

And then, to test this, did the following:

mkdir test
mkdir: cannot create directory ‘test’: Read-only file system

So, somehow my root filesystem has been turned read-only. Thoughts?

Well my disk was giving some errors regarding the superblock having an invalid journal and a corrupt partition table, so I used GParted to wipe the OS drive and install OMV6 once again. This time I did everything EXCEPT install the flashmemory plugin and have had no issues whatsoever. I think this is the likely culprit by process of elimination. Thanks for spending so much time working through this with me.

ryecoaaron, any idea how flashmemory would render my root drive read-only?

Quote from ryecoaaron

It definitely didn't. flashmemory only copies files between a tmpfs bind mount and a mounted filesystem. This happens *after* the filesystems are mounted. Now, if you system is low on memory and the copy from the mounted filesystem to the bind mount fills the tmpfs mount, it could cause problems. The sync folder2ram does between tmpfs and the mounted filesystem at shutdown is very important. So, bad things can happen if this never happens. But it is hard to say while your system is mounted read only. You are using mergerfs as well. If a filesystem wasn't ready when the pool was to be assembled, the system could possibly be mounted read only. kern.log or messages in /var/log/ might be good to look at.

Thanks for chiming in. While I had the mergerfs plugin installed, I hadn't actually created a pool with it yet as all the filesystems that were going to be used in the pool weren't able to be mounted without the issues I was running into as discussed in the thread.

Ultimately, I was able to get my setup to work fine just by avoiding the flashmemory plugin (after 3 fresh installs using it that all failed), so I have to imagine it's somehow involved. As long as nobody else is having issues, maybe it was a fluke or an issue with my system drive, who knows...