OOPS! Can't boot after unionfs changes

1. offizieller Beitrag

    I operate mostly in the GUI of OMV, but understand some concepts of linux, so an answer doesn't have to be hand-holding (but don't intentionally obfuscate, ya sadist!). I have a basic question with an elaborate backstory.


    The basic question: When editing the parameters/arguments of a unionfs (mergerfs?), where is that information stored? How do I go about editing that from CLI? You can stop here, you don't have to read the below.....


    Elaboration: I went and pseudo-blindly followed some internet advise that suggested adding


    x-systemd.requires=/srv/dev-disk-by-label-[label]


    to the unionfs options (irrelevant details why below). To the unionfs plugin's one merged filesystem, I added 3 of these so that (as I assumed) my three disks in the merged fs would have their mount points available before continuing with the rest of the boot process. My system disk is NOT included in this. I added exactly:


    x-systemd.requires=/srv/dev-disk-by-label-backup,x-systemd.requires=/srv/dev-disk-by-label-Disk1,x-systemd.requires=/srv/dev-disk-by-label-backup,x-systemd.requires=/srv/dev-disk-by-label-Disk2,x-systemd.requires=/srv/dev-disk-by-label-backup,x-systemd.requires=/srv/dev-disk-by-label-Z


    Aaaand after a reboot, OMV webgui will not launch. From the CLI, if I try to log into the omv admin user, I get:


    No directory, logging in with HOME=/

    This account is currently not available


    Checking the log, it looks like it sees the drives:


    OMV systemd[1]: Mounting /srv/dev-disk-by-label-Z...

    -- The start-up result is done.


    (for each of the drives)

    sidenote, relevant line:

    OMV Kernel: EXT4-fs (sdb): mounted filesystem with ordered data mode.  Opts: user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,ac1


    The first failure I see here is for each non-system disk, it's looking for aquota.group and aquota.user and failing. Dunno what this is, but i assume its not necessary. Including just in case.


    As it starts to bring up interfaces and whatnot, I get a few more failures: "Cleanup monit state file", "Enable File System Quotas", "Update UTMP about System Boot/Shutdown", but thing is, I don't know what's expected here. I've never bothered to look at the logs when it worked!


    How do I undo what I did? I just want to get back to those parameters and remove that bit that I added. I don't care about the why, I just wanna boot back up and return to my original migraine of an issue.


    Below are just general, disorganized notes and errors.


    anacron service fails

    liblogging-stdlog "read-only file system"

    fan.controlservice "failed to start fan speed regulator" god this pisses me off so much


    First, i think unique, failure is samba:

    OMV sh[619]: /sbin/dhclient-script: 46: /etc/hdcp/dhclient-enter-hooks.d/samba: cannot create /etc/samba/dhcp.conf.new: Read-only file system

    OMV systemd[1]: failed to start Samba SMB Daemon


    ...and a whole bunch of other complaints: "Failed to start Generate the prelogin message" "Failed to start Samba NMB Daemon." "rrdcached plugin: Failed to connect to RRDCacheD at unix:/var/run/rrdcached.sock: Unable to connect to rrdcached: No such file or directory (status=2) with a giant failure later:


    OMV systemd[1]: Failed to start Docker Application Container Engine.

    and

    Failed to start A high performance web server and a reverse proxy server.


    pretty much everything points to "Read-only file system"


    The "why" was because I was having issues with permissions on the mount points in my dockers. I pointed one container to the /srv/[mergerfsid] path and it worked great. I pointed another to the same, and it couldn't create a folder. I had already tried a lot of other standard troubleshooting steps. This was a trial-and-error attempt. I had assumed "well since my system drive isn't a part of these, it won't prevent the system from booting if this argument is wrong!" Wow. Wow. just.... wow.

    I can't tell you how to cleanup or undo what you did, but that should be possible. Trivial if you made backups of your system disk before making changes.


    As for me, I don't use either of the OMV mergerfs plugins preferring to configure it by hand in the /etc/fstab file but OUTSIDE THE # >>> [openmediavault] # <<< [openmediavault] stanza.


    These are mine and haven't needed to be changed in any way, even when I need to add new disks. I'm currently at seven merged data disks. My docker stuff on another disk doesn't need to be in a merged pool so it isn't, and neither are my two SnapRAID parity disks.



    From my /etc/fstab file:

    Code
    # >>> [sftp-mergerfs]

/srv/dev-disk-by-label-d*/multimedia-content-d*/movies /srv/dev-disk-by-label-d1/sftp/outgoing/movies fuse.mergerfs noforget,nonempty,defaults,allow_other,use_ino,func.getattr=newest,category.create=lfs,moveonenospc=true,minfreespace=110G,hard_remove,fsname=mergerfs-movies 0 0

/srv/dev-disk-by-label-d*/multimedia-content-d*/music /srv/dev-disk-by-label-d1/sftp/outgoing/music fuse.mergerfs noforget,nonempty,defaults,allow_other,use_ino,func.getattr=newest,category.create=lfs,moveonenospc=true,minfreespace=110G,hard_remove,fsname=mergerfs-music 0 0

/srv/dev-disk-by-label-d*/multimedia-content-d*/tv-series /srv/dev-disk-by-label-d1/sftp/outgoing/tv-series fuse.mergerfs noforget,nonempty,defaults,allow_other,use_ino,func.getattr=newest,category.create=lfs,moveonenospc=true,minfreespace=110G,hard_remove,fsname=mergerfs-tv-series 0 0


# <<< [sftp-mergerfs]

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

    Danke 1

    Thanks gderf.


    I did find what I had changed in /etc/fstab. So thank you for mentioning that. I had to use mount -o remount,rw to allow editing (and copying for backup). Edited out what was added for that mergerfs, saved, restarted, back in business. to my original migraine. that simple comment helped me immensely, I owe you a beer (can you PM me a paypal/patreon/buymeacoffee name pls?)


    Anyway continuing the conversation.... your example fstab raises some other questions i'm dying to understand:

    You have 3 separate ...fused...mergerfs.... OK hang on, what do you CALL the merged filesystems?


    Whatever you call them, you have three? One for each /outgoing/movies, /outgoing/music, /outgoing/tv-series? Why not do what I did and make one for /outgoing? What differences occur there?

    I just call them merges. But they have defined mergerfs names of mergerfs-tv-series, mergerfs-music, and mergerfs-movies.They appear that way in a df -h listing in the shell.


    The reason I have three separate merges is that I also point my Plex Media Server at the same individual merges. Plex is geared to having libraries based on the type of media you have. So I have a separate library for each type - Movies, Music, TV-Series. If I had only one merge containing all of it, Plex would not be able to properly deal with it.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

    I see. I accomplished that by setting up shared folders from the merge. The only problem was I couldn't use /sharedfolder/ for docker mount points; i had to point to /srv/[fuse id]/[media folder], but for some reason i couldn't get permissions correct and the containers, even though running with PGID/PUID of a group/user with full rwe access, couldn't create folders... That's part of why I was trying to use that x-systemd.requires=


    Speaking of, I found what I did wrong. It's in my original post. I had copy/pasted code into notepad, edited, and copy/pasted twice more, to edit twice more.... anyway three times i had x-systemd.requires=/srv/dev-disk-by-label-backup which is a disk that doesn't even exist. Careless. This does spawn a new question. I'm having a hard time interpreting exactly what x-systemd.requires affects... If one of my drives fails, and is referenced with this, will my system fail to boot like this?

    • Offizieller Beitrag
    Zitat von Headache

    I see. I accomplished that by setting up shared folders from the merge. The only problem was I couldn't use /sharedfolder/ for docker mount points; i had to point to /srv/[fuse id]/[media folder], but for some reason i couldn't get permissions correct and the containers, even though running with PGID/PUID of a group/user with full rwe access, couldn't create folders... That's part of why I was trying to use that x-systemd.requires=

    You can't use the fuse/mergerfs mount point for docker containers particularly the /configs, easiest way around it is to a share on single drive within the mount, unless you have a spare drive which is what I am using

    Zitat von geaves

    You can't use the fuse/mergerfs mount point for docker containers particularly the /configs, easiest way around it is to a share on single drive within the mount, unless you have a spare drive which is what I am using

    Really?! Well .... AHHH that explains it. Though I'm not doing my /configs on the merge - those are separate on their own drive.

    Zitat von Headache

    Speaking of, I found what I did wrong. It's in my original post. I had copy/pasted code into notepad, edited, and copy/pasted twice more, to edit twice more.... anyway three times i had x-systemd.requires=/srv/dev-disk-by-label-backup which is a disk that doesn't even exist.

    Don't use Windows Notepad for this. It uses incompatible with Linux CRLFs. If you have to use a Windows program, use Wordpad.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!

Benutzerkonto erstellen Anmelden