Severe issue with openmediavault-mergerfsfolders

auanasgheps · 25. Februar 2020

Whilst I was setting up my new OMV system, I wanted to give a try to openmediavault-mergerfsfolder, instead of using the "old" openmediavault-unionfilesystems.
Well, it's been a nightmare: it crippled my OMV installation twice before I realised it.

Disclaimer: I don't have many details and I don't want to replicate the issue because I already wasted a lot of time, but I thought was still worth reporting the issue.
OMV: 5.3.3-1
openmediavault-mergerfsfolder: 5.0.1

I installed OMV along with many plugins: backup, usbbackup, flashmemory, snapraid, resetperms, diskstats, wakealarm and mergerfsfolder.
I didn't configure most of them and didn't make any weird change to the system.

I configured mergerfsfolder to merge two main folders from different drives. Yes, I used /srv/dev-by-disk-by-label-XXX .
It seemed to work fine, I could see the new filestystem, browse it via WinSCP, mount it in SMB and do all sort of things.
Until I rebooted the system.

I could not login to OMV UI - I think the error was very similar to this thread
Any command issued via shell such as omv-update or other simple tasks resulted in "Read Only File System"
I was able to remount the root FS and login to WebUI issuing the command mount -o remount / but the issue was still there If I rebooted again, services like SMB would not start (RED status in Dashboard)

I thought was my fault. Before rebooting I also updated from 5.3.2-1 to 5.3.3-1, so I thought this could have caused the issue.

I had to start all over again, because I hadn't made a backup yet ( ),
This time I configured everything but mergerfsfolder, and made a backup first. Rebooted, everything went fine, so the backup was healthy.
I configured mergerfsfolder and as before I could use correctly the merged FS.
Rebooted and bang! The same issue.

I managed to remount the root fs via the command above, install systemrescuecd and boot to it. By the way, it does not start ssh automatically (nothing was listening on port 22 but the system was connected correctly as I could ping it)

I restored the backup, removed mergerfsfolder and installed unionfilesystems.

I configured unionfilesystems, rebooted and everything has been working great since then.

Sorry for the rant, I hope this will help.

ryecoaaron · 26. Februar 2020

I hate to say it but that doesn't really help. I would need to see the exact setup and resulting fstab entry. The new plugin does allow you pool any folders which can be dangerous if done incorrectly. But believe it or not, both plugins use the same backend. And I did a shit load of testing when writing the plugin without this issue. So, this must be something specific to your setup.

I will look at the systemrescuecd issue but it is getting harder and harder to maintain all of the pre-configured settings when systemrescuecd (and clonezille) keep changing significantly. Sometimes the old options are not possible anymore. If I don't update the version, people complain it is too old.

ryecoaaron · 26. Februar 2020

omv-extras 5.2.5 fixes the systemrescuecd ssh issue. sshd was running but systemrescuecd 6.0.4 added a firewall. I added nofirewall to the boot arguments and it works as expected. Just click install systemrescuecd again.

auanasgheps · 26. Februar 2020

Zitat von ryecoaaron

omv-extras 5.2.5 fixes the systemrescuecd ssh issue. sshd was running but systemrescuecd 6.0.4 added a firewall. I added nofirewall to the boot arguments and it works as expected. Just click install systemrescuecd again.

Thanks for updating systemrescuedcd. Probably you should not update that component, it's just a rescue CD
Regarding the mergefolders issue, I had the feeling this wasn't enough.
If I will play with it in the future I will let you know.

ryecoaaron · 26. Februar 2020

Zitat von thedarkness

If I will play with it in the future I will let you know.

You don't want to post your setup and fstab?

auanasgheps · 26. Februar 2020

Zitat von ryecoaaron

You don't want to post your setup and fstab?

Sure I can, but what is the point of doing it now when the system is up and running and using the "older" plugin?

ryecoaaron · 26. Februar 2020

Zitat von thedarkness

what is the point of doing it now when the system is up and running and using the "older" plugin?

None. Just trying to help. Wasn't sure if you knew what you put in the fields and/or had cut&paste the fstab to compare differences.

auanasgheps · 27. Februar 2020

Zitat von ryecoaaron

None. Just trying to help. Wasn't sure if you knew what you put in the fields and/or had cut&paste the fstab to compare differences.

Thanks for you effort, always appreciated. I understand at this point we can only make speculations. Unfortunately I didn't know I had to check fstab otherwise I would have saved such file.

Well, below you can find my current fstab (working). Things to note:
- Made changes as instructed by the flashmemory plugin (updated root partition properties and disabled swap partition)
- Removed "noexec" property from the nvme partition which runs Docker

The bind I tried to made with mergefsfolder was slightly different. I entered the following paths, because I wanted the exact folders to be merged.I added them as the GUI instructed, one line per folder
/srv/dev-disk-by-label-HDD2/HDD2/data
/srv/dev-disk-by-label-HDD2/HDD3/media
I cant' remember if there was a slash / at the end of the folders. Could its presence/absence have caused the problem?

Code

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sdd1 during installation
UUID=05e735e2-df32-45df-996b-1d3ba752a19c / ext4 noatime,nodiratime,errors=remount-ro 0 1
# swap was on /dev/sdd5 during installation
#UUID=1b60aa3a-7b3d-4812-bea5-c687f368537e none            swap    sw              0       0
# >>> [openmediavault]
/dev/disk/by-label/HDD3		/srv/dev-disk-by-label-HDD3	ext4	defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl	0 2
/dev/disk/by-label/HDD2		/srv/dev-disk-by-label-HDD2	ext4	defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl	0 2
/dev/disk/by-label/nvme		/srv/dev-disk-by-label-nvme	ext4	defaults,nofail,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl	0 2
/dev/disk/by-label/HDD1		/srv/dev-disk-by-label-HDD1	ext4	defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl	0 2
/srv/dev-disk-by-label-HDD2:/srv/dev-disk-by-label-HDD3		/srv/fb4ab0da-2b79-4740-834a-7435c0123e28	fuse.mergerfs	defaults,allow_other,cache.files=off,use_ino,category.create=mfs,minfreespace=200G,x-systemd.requires=/srv/dev-disk-by-label-HDD2,x-systemd.requires=/srv/dev-disk-by-label-HDD3	0 0
# <<< [openmediavault]

Alles anzeigen

ryecoaaron · 27. Februar 2020

Zitat von thedarkness

Could its presence/absence have caused the problem?

No. But I was able to recreate this using the subdirectories. Testing now.

ryecoaaron · 27. Februar 2020

So, the problem is caused by the x-systemd.requires= options that the plugin adds. These are added on both plugins and works just fine on both plugins. The problem happens when you use a subdirectory on a mountpoint because the requires is set to the full path when it should just be the filesystem's mountpoint. I will have to change the code to get the filesystem's path instead of the paths specified.

ryecoaaron · 27. Februar 2020

This problem with sub-directory branches is fixed in 5.0.2 in the repo now.

auanasgheps · 27. Februar 2020

Bingo, so you've found the issue! I'm glad my incident has helped.

As a side note, is there a way to customize the virtual filesystem name created by the plugin? It's ugly but most importantly long, I have to copy and paste it in every configuration (such as Docker files), would be handy if the user could customize it.

ryecoaaron · 27. Februar 2020

Zitat von thedarkness

As a side note, is there a way to customize the virtual filesystem name created by the plugin?

On the unionfilesystem plugin, you can add the option fsname=SOMEUNIQUENAME in the options box. On the mergerfsfolders plugin, it is already setting it to POOLNAME:POOLUUID

Zitat von thedarkness

It's ugly but most importantly long, I have to copy and paste it in every configuration (such as Docker files), would be handy if the user could customize it.

But I think you are actually referring to the mount point of /srv/LONG_UUID_STRING? If so, you can't customize that. You can make a symlink though using the symlink plugin or just ln -s.

nicktids · 2. März 2020

Hi

I am also mounting folders in mergefs

Code

/srv/dev-disk-by-label-8tb/Movies
/srv/dev-disk-by-label-4tb/Movies

and then I create a shared folder with root / and I can samba share that out but when I try and share via NFS I get the below issue. Now is this a Merge FS issue or a NFS issue.

Thought I would post here as it is very similar to the issue above with folders.

running 5.3.5-1 and mergefs is mergefsfolder 5.0.2

Code

Error #0:
OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; omv-salt deploy run fstab 2>&1' with exit code '1': /usr/lib/python3/dist-packages/salt/utils/decorators/signature.py:31: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
  *salt.utils.args.get_function_argspec(original_function)
debian:
----------
          ID: create_filesystem_mountpoint_b9b4b8b4-e247-4a98-8576-bfe00246b5a2
    Function: file.accumulated
      Result: True
     Comment: Accumulator create_filesystem_mountpoint_b9b4b8b4-e247-4a98-8576-bfe00246b5a2 for file /etc/fstab was charged by text
     Started: 13:18:45.113131
    Duration: 0.573 ms
     Changes:
----------
          ID: mount_filesystem_mountpoint_b9b4b8b4-e247-4a98-8576-bfe00246b5a2
    Function: mount.mounted
        Name: /srv/dev-disk-by-label-4tb
      Result: True
     Comment: Target was already mounted
     Started: 13:18:45.114134
    Duration: 116.031 ms
     Changes:
              ----------
              umount:
                  Forced remount because options (acl) changed
----------
          ID: create_filesystem_mountpoint_38ea0888-848d-4a71-bbbd-bde0f456cf7c
    Function: file.accumulated
      Result: True
     Comment: Accumulator create_filesystem_mountpoint_38ea0888-848d-4a71-bbbd-bde0f456cf7c for file /etc/fstab was charged by text
     Started: 13:18:45.230356
    Duration: 0.825 ms
     Changes:
----------
          ID: mount_filesystem_mountpoint_38ea0888-848d-4a71-bbbd-bde0f456cf7c
    Function: mount.mounted
        Name: /srv/dev-disk-by-label-8tb
      Result: True
     Comment: Target was already mounted
     Started: 13:18:45.231317
    Duration: 43.773 ms
     Changes:
              ----------
              umount:
                  Forced remount because options (acl) changed
----------
          ID: create_filesystem_mountpoint_7db54a46-9f39-4b95-a5e6-bbd05f9ded81
    Function: file.accumulated
      Result: True
     Comment: Accumulator create_filesystem_mountpoint_7db54a46-9f39-4b95-a5e6-bbd05f9ded81 for file /etc/fstab was charged by text
     Started: 13:18:45.275277
    Duration: 0.873 ms
     Changes:
----------
          ID: mount_filesystem_mountpoint_7db54a46-9f39-4b95-a5e6-bbd05f9ded81
    Function: mount.mounted
        Name: /srv/dev-disk-by-label-3tb
      Result: True
     Comment:
     Started: 13:18:45.276233
    Duration: 37.461 ms
     Changes:
----------
          ID: create_mergerfsfolder_mountpoint_0ad3c219-d7c7-489d-b7b4-9923bdaab224
    Function: file.accumulated
      Result: True
     Comment: Accumulator create_mergerfsfolder_mountpoint_0ad3c219-d7c7-489d-b7b4-9923bdaab224 for file /etc/fstab was charged by text
     Started: 13:18:45.313928
    Duration: 0.78 ms
     Changes:
----------
          ID: mount_filesystem_mountpoint_0ad3c219-d7c7-489d-b7b4-9923bdaab224
    Function: mount.mounted
        Name: /srv/7f59ac98-4af5-43e4-b9bd-d63c8b7e7973
      Result: False
     Comment: Unable to unmount /srv/7f59ac98-4af5-43e4-b9bd-d63c8b7e7973: umount: /srv/7f59ac98-4af5-43e4-b9bd-d63c8b7e7973: target is busy..
     Started: 13:18:45.314793
    Duration: 27.425 ms
     Changes:
              ----------
              umount:
                  Forced unmount and mount because options (cache.files=off) changed
----------
          ID: create_bind_mountpoint_36cb74e8-a79b-448c-b82d-e1ee51bd2d81
    Function: file.accumulated
      Result: True
     Comment: Accumulator create_bind_mountpoint_36cb74e8-a79b-448c-b82d-e1ee51bd2d81 for file /etc/fstab was charged by text
     Started: 13:18:45.342507
    Duration: 0.805 ms
     Changes:
----------
          ID: mount_bind_mountpoint_36cb74e8-a79b-448c-b82d-e1ee51bd2d81
    Function: mount.mounted
        Name: /export/8tb
      Result: True
     Comment: Target was already mounted
     Started: 13:18:45.343410
    Duration: 17.427 ms
     Changes:
----------
          ID: create_bind_mountpoint_7b0c467a-ecee-44f0-a7af-30893a67c277
    Function: file.accumulated
      Result: True
     Comment: Accumulator create_bind_mountpoint_7b0c467a-ecee-44f0-a7af-30893a67c277 for file /etc/fstab was charged by text
     Started: 13:18:45.361019
    Duration: 0.763 ms
     Changes:
----------
          ID: mount_bind_mountpoint_7b0c467a-ecee-44f0-a7af-30893a67c277
    Function: mount.mounted
        Name: /export/Vault_Movies
      Result: True
     Comment: Target was already mounted
     Started: 13:18:45.361863
    Duration: 16.772 ms
     Changes:
----------
          ID: append_fstab_entries
    Function: file.blockreplace
        Name: /etc/fstab
      Result: True
     Comment: No changes needed to be made
     Started: 13:18:45.380263
    Duration: 2.708 ms
     Changes:


Summary for debian
-------------
Succeeded: 12 (changed=3)
Failed:     1
-------------
Total states run:     13
Total run time:  266.216 ms in /usr/share/php/openmediavault/system/process.inc:182
Stack trace:
#0 /usr/share/php/openmediavault/engine/module/serviceabstract.inc(60): OMV\System\Process->execute()
#1 /usr/share/openmediavault/engined/rpc/config.inc(167): OMV\Engine\Module\ServiceAbstract->deploy()
#2 [internal function]: Engined\Rpc\Config->applyChanges(Array, Array)
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#4 /usr/share/php/openmediavault/rpc/serviceabstract.inc(149): OMV\Rpc\ServiceAbstract->callMethod('applyChanges', Array, Array)
#5 /usr/share/php/openmediavault/rpc/serviceabstract.inc(588): OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}('/tmp/bgstatus9Y...', '/tmp/bgoutputI3...')
#6 /usr/share/php/openmediavault/rpc/serviceabstract.inc(159): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure))
#7 /usr/share/openmediavault/engined/rpc/config.inc(189): OMV\Rpc\ServiceAbstract->callMethodBg('applyChanges', Array, Array)
#8 [internal function]: Engined\Rpc\Config->applyChangesBg(Array, Array)
#9 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#10 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('applyChangesBg', Array, Array)
#11 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Config', 'applyChangesBg', Array, Array, 1)
#12 {main}

Alles anzeigen

ryecoaaron · 2. März 2020

Zitat von nicktids

Comment: Unable to unmount /srv/7f59ac98-4af5-43e4-b9bd-d63c8b7e7973: umount: /srv/7f59ac98-4af5-43e4-b9bd-d63c8b7e7973: target is busy..

Something is using the filesystem when it tries to unmount. I would reboot.

Jetzt mitmachen!