MergerFS creates fstab trouble - stops OMV from booting with root rw

Just had an interesting experience, while configuring MergerFS and then testing it on reboot, I found my system to no longer mount root read-write. Swap wouldn't mount either and that was my first warning sign as the kernel finished up initializing.

I noticed that /etc/fstab.bak existed from MergerFS backing it up prior to modifying it, so I copied the current fstab to fstab.mergerfs and then cp'd fstab.bak back. I rebooted and the system worked fine. I then copied fstab.mergfs back to fstab and ran mount -av which insanely reported that the filesystem in question mounted successfully (mount -fav also actually executes MergerFS mount, so it doesn't fake it like it's supposed to). A reboot yielded failure again and still requires a mount -o remount,rw / to be able to modify anything on the filesystem.

The line that MergerFS added to my /etc/fstab is:

/mnt/bay_1:/mnt/bay_2:/mnt/bay_3:/mnt/bay_4:/mnt/bay_5:/mnt/bay_6        /srv/1fd5adb1-3581-4d14-9856-5c288aa699ff    fuse.mergerfs    defaults,allow_other,cache.files=off,use_ino,category.create=mfs,minfreespace=275G,fsname=tank:1fd5adb1-3581-4d14-9856-5c288aa699ff,x-systemd.requires=/mnt/bay_1,x-systemd.requires=/mnt/bay_2,x-systemd.requires=/mnt/bay_3,x-systemd.requires=/mnt/bay_4,x-systemd.requires=/mnt/bay_5,x-systemd.requires=/mnt/bay_6    0 0

I'm going to break this down into segments where each space has a new line for readability:

<file system>   ::  /mnt/bay_1:
                    /mnt/bay_2:
                    /mnt/bay_3:
                    /mnt/bay_4:
                    /mnt/bay_5:
                    /mnt/bay_6        

<mount point>   :: /srv/1fd5adb1-3581-4d14-9856-5c288aa699ff    

<type>          :: fuse.mergerfs    

<options>       :: defaults,
                   allow_other,
                   cache.files=off,
                   use_ino,
                   category.create=mfs,
                   minfreespace=275G,
                   fsname=tank:1fd5adb1-3581-4d14-9856-5c288aa699ff,
                   x-systemd.requires=/mnt/bay_1,
                   x-systemd.requires=/mnt/bay_2,
                   x-systemd.requires=/mnt/bay_3,
                   x-systemd.requires=/mnt/bay_4,
                   x-systemd.requires=/mnt/bay_5,
                   x-systemd.requires=/mnt/bay_6    

<dump>          :: 0 

<pass>          :: 0

The GUI seems to show nothing remarkable:

Any thoughts on what the problem is?

I'm still digging through this, but I thought I should at least document this so far and in case I don't find anything someone else may have some experience with this situation.

Since this fstab entry seems to actually work just fine AFTER system startup, so I can't help but wonder:

1. Is a dependency such as fuse or another not starting early enough to allow /etc/fstab mounting to work? and/or
2. Can/should we trigger the mount from a script instead of putting this in fstab? It seems like making some kind of /etc/init.d startup script and then triggering it at the right time (perhaps just after fuse's dependencies are met?) might be a better way to fire off MergerFS mount points...

What do you think?

SOLVED: Adding _netdev to the options in /etc/fstab appears to give ZFS and/or fuse (not sure what service or module was not loading quickly enough) enough time to be ready to receive the mount.

I also slightly cleaned up the drive mountpoints all into a singular /mnt/tank_drives parent folder, so the command is now:

/mnt/tank_drives/*        /srv/1fd5adb1-3581-4d14-9856-5c288aa699ff    fuse.mergerfs    _netdev,defaults,allow_other,cache.files=off,use_ino,category.create=mfs,minfreespace=275G,fsname=tank:1fd5adb1-3581-4d14-9856-5c288aa699ff,x-systemd.requires=/mnt/bay_1,x-systemd.requires=/mnt/bay_2,x-systemd.requires=/mnt/bay_3,x-systemd.requires=/mnt/bay_4,x-systemd.requires=/mnt/bay_5,x-systemd.requires=/mnt/bay_6    0 0

Expanded:

<file system>   :: /mnt/tank_drives/*

<mount point>   :: /srv/1fd5adb1-3581-4d14-9856-5c288aa699ff

<type>          :: fuse.mergerfs

<options>       :: _netdev,
                   defaults,
                   allow_other,
                   cache.files=off,
                   use_ino,
                   category.create=mfs,
                   minfreespace=275G,
                   fsname=tank:1fd5adb1-3581-4d14-9856-5c288aa699ff,
                   x-systemd.requires=/mnt/bay_1,
                   x-systemd.requires=/mnt/bay_2,
                   x-systemd.requires=/mnt/bay_3,
                   x-systemd.requires=/mnt/bay_4,
                   x-systemd.requires=/mnt/bay_5,
                   x-systemd.requires=/mnt/bay_6

<dump>          :: 0

<pass>          :: 0

Any way to clean up those last parts for x-systemd.requires?

Thus far MergerFS has required me to reset /etc/fstab each time I make mount changes for external drives I'm attaching for initial population. It's not the end of the world, and I'm liking the compression vs using Ext4, but I've obviously hit some bugs and we'll need to fix them at some point.

Hey ryecoaaron - again, thanks and I understand. You might be right, or there might be other things (bugs) going on, absolutely. As you may have seen, I marked this thread as `resolved` previously because we can live with this in terms of at least now I understand how to work with / live with it presently by the manual tweaks as needed. Suboptimal, sure, but it get's us from A to B albeit a bit circuitously.

I have everything working and it's all live now. I've transferred nearly all of the entire massive store that I'm working with and things are relatively nice so far. The experience is not as "refined" as some other options out there, BUT the increased flexibility is to die for.

I look forward to both your work and hearing from you in any way you feel I may when you'd like some feedback or otherwise.