omv-extras plugins - porting progress to OMV 6.x (done)

NozomiYuki · Dec 15th 2021

Quote from chente

Regarding this, maybe this thread will be useful for you RE: Easy Active Directory integration of OMV 5 with PBIS Open

Cool thanks for that! I actually went searching through these forums about a week ago and didn't come across that one so I'll definitely look into it tomorrow. Brief skim through it has given me a little motivation at least!
I think my biggest annoyance with how I have it set up in OMV5 currently with sssd was android devices weren't authenticating against it to get into SMB shares because android didn't support the auth method or something dumb which I gave up on trying to fix long ago cause whenever I did fix it, it broke the shares for windows clients 😠 lol
That was like a year ago so I'm pretty sure I just gave up and told myself that I would wait until OMV6 came out and then look into reconfiguring it differently to avoid that problem

chente · Dec 15th 2021

Quote from NozomiYuki

I actually went searching through these forums about a week ago and didn't come across that one

Yes, unfortunately the forum search engine is not very efficient. Sometimes it is easier to find the threads with google.

Quote from NozomiYuki

Brief skim through it has given me a little motivation at least!

It would be very useful if you post your solutions later.

ryecoaaron · Dec 15th 2021

Quote from NozomiYuki

When trying to create a shared folder I encounter the following error:

The backend is probably doing the wrong "thing" when an entry is in fstab vs systemd. I will look at that.

ryecoaaron · Dec 15th 2021

openmediavault-mergerfs 6.0.12 is in the repo now. It should fix the sharedfolder uuid error.

NozomiYuki · Dec 16th 2021

Is there any way to have omv/the mergerfs plugin handle already in-use mounts more gracefully?

I had 1 test mount configured (Test01) and I created some shared folders on it which are in use by SMB/NFS/TFTP.
When trying to add more mergerfs mounts, they get successfully created / edited but omv wants to unmount it to apply the changes which leads to an error if any mount is already in use

SMB was definitely a big culprit here, TFTP as well. I imagine docker would be the worst one of the lot if you have containers that use a mergerfs share (which I don't have any set up just yet on the test VM). So basically I had to stop the TFTP service / disable the "busy" shares (or just disable smb to make sure any open files were closed) and apply the changes in OMV and check lsof /srv/mergerfs/Test01 was empty before adding mergerfs mounts and applying the changes again.

Maybe we could have it apply the systemd changes but ignore restarting the already in use mounts in a way that gets around this error. This way the changes are applied, we just need to alert the user that as the mount is busy, it couldn't be unmounted, so a reboot would be required before we see the applied change? Or restart the services (which may not always work depending on different user configurations) Probably a better way to handle it than that but either way, needed to annoy ryecoaaron some more with yet another mergerfs related thing 😉😆

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; omv-salt deploy run --no-color mergerfs 2>&1' with exit code '1': debian:
----------
ID: configure_pool_dir
Function: file.directory
Name: /srv/mergerfs
Result: True
Comment: The directory /srv/mergerfs is in the correct state
Started: 16:44:37.719494
Duration: 12.424 ms
Changes:
----------
ID: remove_mergerfs_mount_files_/etc/systemd/system
Function: module.run
Result: True
Comment: file.find: ['/etc/systemd/system/srv-mergerfs-Backups.mount', '/etc/systemd/system/srv-mergerfs-Downloads.mount', '/etc/systemd/system/srv-mergerfs-Media.mount', '/etc/systemd/system/srv-mergerfs-Test01.mount', '/etc/systemd/system/srv-mergerfs-Union.mount']
Started: 16:44:37.734330
Duration: 10.742 ms
Changes:
----------
file.find:
- /etc/systemd/system/srv-mergerfs-Backups.mount
- /etc/systemd/system/srv-mergerfs-Downloads.mount
- /etc/systemd/system/srv-mergerfs-Media.mount
- /etc/systemd/system/srv-mergerfs-Test01.mount
- /etc/systemd/system/srv-mergerfs-Union.mount
----------
ID: configure_mergerfs_Test01
Function: file.managed
Name: /etc/systemd/system/srv-mergerfs-Test01.mount
Result: True
Comment: File /etc/systemd/system/srv-mergerfs-Test01.mount updated
Started: 16:44:37.745432
Duration: 257.176 ms
Changes:
----------
diff:
New file
mode:
0644
----------
ID: systemd-reload_Test01
Function: cmd.run
Name: systemctl daemon-reload
Result: True
Comment: Command "systemctl daemon-reload" run
Started: 16:44:38.011501
Duration: 1294.956 ms
Changes:
----------
pid:
33719
retcode:
0
stderr:
stdout:
----------
ID: enable_Test01_mergerfs
Function: service.enabled
Name: srv-mergerfs-Test01.mount
Result: True
Comment: Service srv-mergerfs-Test01.mount is already enabled, and is in the desired state
Started: 16:44:39.409053
Duration: 107.021 ms
Changes:
----------
ID: restart_Test01_mergerfs
Function: cmd.run
Name: systemctl restart srv-mergerfs-Test01.mount
Result: False
Comment: Command "systemctl restart srv-mergerfs-Test01.mount" run
Started: 16:44:39.517427
Duration: 93.166 ms
Changes:
----------
pid:
33746
retcode:
1
stderr:
Job failed. See "journalctl -xe" for details.


OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; omv-salt deploy run --no-color mergerfs 2>&1' with exit code '1': debian:
----------
ID: configure_pool_dir
Function: file.directory
Name: /srv/mergerfs
Result: True
Comment: The directory /srv/mergerfs is in the correct state
Started: 16:44:37.719494
Duration: 12.424 ms
Changes:
----------
ID: remove_mergerfs_mount_files_/etc/systemd/system
Function: module.run
Result: True
Comment: file.find: ['/etc/systemd/system/srv-mergerfs-Backups.mount', '/etc/systemd/system/srv-mergerfs-Downloads.mount', '/etc/systemd/system/srv-mergerfs-Media.mount', '/etc/systemd/system/srv-mergerfs-Test01.mount', '/etc/systemd/system/srv-mergerfs-Union.mount']
Started: 16:44:37.734330
Duration: 10.742 ms
Changes:
----------
file.find:
- /etc/systemd/system/srv-mergerfs-Backups.mount
- /etc/systemd/system/srv-mergerfs-Downloads.mount
- /etc/systemd/system/srv-mergerfs-Media.mount
- /etc/systemd/system/srv-mergerfs-Test01.mount
- /etc/systemd/system/srv-mergerfs-Union.mount
----------
ID: configure_mergerfs_Test01
Function: file.managed
Name: /etc/systemd/system/srv-mergerfs-Test01.mount
Result: True
Comment: File /etc/systemd/system/srv-mergerfs-Test01.mount updated
Started: 16:44:37.745432
Duration: 257.176 ms
Changes:
----------
diff:
New file
mode:
0644
----------
ID: systemd-reload_Test01
Function: cmd.run
Name: systemctl daemon-reload
Result: True
Comment: Command "systemctl daemon-reload" run
Started: 16:44:38.011501
Duration: 1294.956 ms
Changes:
----------
pid:
33719
retcode:
0
stderr:
stdout:
----------
ID: enable_Test01_mergerfs
Function: service.enabled
Name: srv-mergerfs-Test01.mount
Result: True
Comment: Service srv-mergerfs-Test01.mount is already enabled, and is in the desired state
Started: 16:44:39.409053
Duration: 107.021 ms
Changes:
----------
ID: restart_Test01_mergerfs
Function: cmd.run
Name: systemctl restart srv-mergerfs-Test01.mount
Result: False
Comment: Command "systemctl restart srv-mergerfs-Test01.mount" run
Started: 16:44:39.517427
Duration: 93.166 ms
Changes:
----------
pid:
33746
retcode:
1
stderr:
Job failed. See "journalctl -xe" for details.


Summary for debian
-------------
Succeeded: 21 (changed=16)
Failed:     1
-------------
Total states run:     22
Total run time:    6.797 s in /usr/share/php/openmediavault/system/process.inc:196
Stack trace:
#0 /usr/share/php/openmediavault/engine/module/serviceabstract.inc(62): OMV\System\Process->execute()
#1 /usr/share/openmediavault/engined/rpc/config.inc(167): OMV\Engine\Module\ServiceAbstract->deploy()
#2 [internal function]: Engined\Rpc\Config->applyChanges(Array, Array)
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#4 /usr/share/php/openmediavault/rpc/serviceabstract.inc(149): OMV\Rpc\ServiceAbstract->callMethod('applyChanges', Array, Array)
#5 /usr/share/php/openmediavault/rpc/serviceabstract.inc(588): OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}('/tmp/bgstatusxQ...', '/tmp/bgoutput8h...')
#6 /usr/share/php/openmediavault/rpc/serviceabstract.inc(159): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure))
#7 /usr/share/openmediavault/engined/rpc/config.inc(189): OMV\Rpc\ServiceAbstract->callMethodBg('applyChanges', Array, Array)
#8 [internal function]: Engined\Rpc\Config->applyChangesBg(Array, Array)
#9 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#10 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('applyChangesBg', Array, Array)
#11 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Config', 'applyChangesBg', Array, Array, 1)
#12 {main}

Display More

(I stripped out some stuff, full pastebin here: https://pastebin.com/1UCJApin)

ryecoaaron · Dec 16th 2021

Why did I port this plugin..............

No output needed. I know what it is doing.

The plugin restarts the systemd mounts (not fstab) to avoid a reboot. On my servers, I create pools and never touch the plugin again. So, this isn't really an issue for me (evidently I use a server unlike anyone else in the world). I will have to try to figure out in salt code what pool needs to be restarted and leave the others alone. I really don't like this but... This will be the last change I make in 2021. I need a break from OMV plugins.

noiro · Dec 16th 2021

Quote from ryecoaaron

Why did I port this plugin..............

cause you love us ?

NozomiYuki · Dec 16th 2021

Quote from ryecoaaron

I need a break from OMV plugins.

Haha, I don't blame you! Thank you for putting in the effort that you have though!

Quote from ryecoaaron

On my servers, I create pools and never touch the plugin again.

This is typically exactly what I do as well, it's exceptionally rare for me to ever update or even add new mounts, just happened to have occurred while trying to cover testing everything before I migrate from omv5 to 6!

ryecoaaron · Dec 16th 2021

mergerfs 6.0.13 is in the repo. It will not restart the pool unless the pool is not mounted. I also added a button to restart the pool. So, if you edit a pool, you can choose to restart it to get the changes instead of rebooting.

ajaja · Dec 17th 2021

Quote from ryecoaaron

openmediavault-mergerfs - done

So I am in Wisconsin for a while and am back at it. I did a fresh install of 6.0.5-3 (Shaitan) on my server. I want to merge the file systems of a few of my drives as I've done before. I just don't understand how to do it with the current plug-in interface.

Do I place something like this in 'Paths' field:
/dev/disk/by-uuid/ffd38b91-901e-47e5-a388-e58632a7b2c9-part1

/dev/disk/by-uuid/f41eded5-63d6-4b9c-8bfc-2a909e6331ec-part1

noiro · Dec 17th 2021

Quote from ajaja

So I am in Wisconsin for a while and am back at it. I did a fresh install of 6.0.5-3 (Shaitan) on my server. I want to merge the file systems of a few of my drives as I've done before. I just don't understand how to do it with the current plug-in interface.

Do I place something like this in 'Paths' field:
/dev/disk/by-uuid/ffd38b91-901e-47e5-a388-e58632a7b2c9-part1
/dev/disk/by-uuid/f41eded5-63d6-4b9c-8bfc-2a909e6331ec-part1

Display More

Yes.

exactly like that. you can even select from the pull down menu.

ryecoaaron · Dec 17th 2021

Quote from ajaja

So I am in Wisconsin for a while

Me too : ) Where at?

molnart · Dec 18th 2021

after restarting my server again my mergerfs volume failed to mount with error Couldn't extract an UUID from the provided path '/srv/mergerfs/Data_all'.. Adding the nonempty option solved it. Is there a reason why not to enable it by default?

ryecoaaron · Dec 19th 2021

Quote from molnart

after restarting my server again my mergerfs volume failed to mount with error Couldn't extract an UUID from the provided path '/srv/mergerfs/Data_all'.. Adding the nonempty option solved it. Is there a reason why not to enable it by default?

nonempty is dangerous because you can fill the root filesystem. You can add it to your options but it should not be a default. The "couldn't extract uuid" error is just a web interface error because the filesystem was not mounted. You could use the restart button to fix it. As for why it didn't mount on boot, hard to say but that is what I was hoping to fix with systemd mount files. The created mount file must not have the correct dependency for your system and/or you have a drive that takes longer than expected to mount.

ryecoaaron · Dec 19th 2021

openmediavault-downloader 6.0 in the repo.

raulfg3 · Dec 19th 2021

Quote from ryecoaaron

openmediavault-downloader 6.0 in the repo.

thanks a lot

NozomiYuki · Dec 20th 2021

So I've officially switched my main VM to OMV6, everything has been pretty flawless. Decided to just use winbind for the domain for now, drawbacks are worth it for less hassle till I decide what to do with it.
In the process of switching over I only have a single gripe, my ZFS pools add about 90 seconds to the boot process.
It was noticeable on OMV5 as well but seems to be a even slower with OMV6.
I noticed that it was definitely not this slow when I was just using 3x32GB virtual disks for ZFS in the test VM's
It's not a huge deal as OMV doesn't get rebooted very often but other VM's do rely on the ZFS storage shared via OMV

Any tips on making this faster?
Not sure how relevant this info is but if there's any other info that's needed to find the source of the slowness, I'm happy to provide

VM/Hardware Details:

8 CPU cores / 32GB of RAM assigned from a dual CPU Xeon E5-2650 v2 / 256GB RAM host

2x Dell MD1200 enclosures attached with a Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2

I don't think the hardware has anything to do with it as back when I had OMV5 running in a separate dedicated machine that had 16 disks split into 2 ZFS pools, it had the same delay as when I migrated it into the rackmount servers (I seem to recall always waiting on the "mounting zfs from cache file" or whatever it was along those lines a fair bit whenever it had to boot)

Code

zpool status
pool: LocalFS
state: ONLINE
scan: scrub repaired 0B in 01:25:43 with 0 errors on Sun Oct 10 01:49:44 2021
config:


NAME        STATE     READ WRITE CKSUM
LocalFS     ONLINE       0     0     0
sdc       ONLINE       0     0     0


errors: No known data errors


pool: ZFS01
state: ONLINE
scan: scrub repaired 0B in 1 days 11:52:54 with 0 errors on Mon Oct 11 12:16:57 2021
config:


NAME                        STATE     READ WRITE CKSUM
ZFS01                       ONLINE       0     0     0
raidz1-0                  ONLINE       0     0     0
scsi-35000cca03ecffa00  ONLINE       0     0     0
scsi-35000cca03ecfc098  ONLINE       0     0     0
scsi-35000cca03eccabac  ONLINE       0     0     0
scsi-35000cca03ecca25c  ONLINE       0     0     0
scsi-35000cca03ecb8ad4  ONLINE       0     0     0
scsi-35000cca03ecdceb4  ONLINE       0     0     0


errors: No known data errors


pool: ZFS02
state: ONLINE
scan: scrub repaired 0B in 1 days 01:01:26 with 0 errors on Mon Oct 11 01:25:32 2021
config:


NAME                        STATE     READ WRITE CKSUM
ZFS02                       ONLINE       0     0     0
raidz1-0                  ONLINE       0     0     0
scsi-35000c50033f05f9b  ONLINE       0     0     0
scsi-35000c50033e1adc3  ONLINE       0     0     0
scsi-35000c50033f127af  ONLINE       0     0     0
scsi-35000c50033ee9aef  ONLINE       0     0     0
scsi-35000c50033d3ec2b  ONLINE       0     0     0
scsi-35000c50033ee91d7  ONLINE       0     0     0


errors: No known data errors


pool: ZFS03
state: ONLINE
scan: scrub repaired 0B in 2 days 00:06:51 with 0 errors on Tue Oct 12 00:31:00 2021
config:


NAME                        STATE     READ WRITE CKSUM
ZFS03                       ONLINE       0     0     0
raidz2-0                  ONLINE       0     0     0
scsi-35000c500835dbb83  ONLINE       0     0     0
scsi-35000c500835d26af  ONLINE       0     0     0
scsi-35000c500835d87eb  ONLINE       0     0     0
scsi-35000c500835d8c5f  ONLINE       0     0     0
scsi-35000c500835d8d53  ONLINE       0     0     0
scsi-35000c500835db66b  ONLINE       0     0     0


errors: No known data errors


pool: ZFS04
state: ONLINE
scan: scrub repaired 0B in 2 days 00:30:12 with 0 errors on Tue Oct 12 00:54:25 2021
config:


NAME                        STATE     READ WRITE CKSUM
ZFS04                       ONLINE       0     0     0
raidz2-0                  ONLINE       0     0     0
scsi-35000c500835dcb67  ONLINE       0     0     0
scsi-35000c500835d898b  ONLINE       0     0     0
scsi-35000c500835d95a7  ONLINE       0     0     0
scsi-35000c500835d2a0b  ONLINE       0     0     0
scsi-35000c500835d56d7  ONLINE       0     0     0
scsi-35000c500835dc943  ONLINE       0     0     0


errors: No known data errors

Display More

Code

zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
LocalFS   119G  41.6G  77.4G        -         -    29%    34%  1.00x    ONLINE  -
ZFS01    16.4T  14.2T  2.18T        -         -    16%    86%  1.00x    ONLINE  -
ZFS02    10.9T  7.89T  3.01T        -         -    19%    72%  1.00x    ONLINE  -
ZFS03    32.7T  23.4T  9.32T        -         -     2%    71%  1.00x    ONLINE  -
ZFS04    32.7T  23.4T  9.35T        -         -     2%    71%  1.00x    ONLINE  -

Code

zfs list
NAME                     USED  AVAIL     REFER  MOUNTPOINT
LocalFS                 41.6G  73.7G     34.6G  /mnt/LocalFS
LocalFS/.cache            48K  73.7G       24K  /mnt/LocalFS/.cache
LocalFS/.cache/GDrive     24K  73.7G       24K  /mnt/RemoteFS/.cache/GDrive
LocalFS/Docker          6.90G  73.7G     6.90G  -
ZFS01                   11.8T  1.68T     11.7T  /mnt/ZFS01
ZFS01/Backups           1.35G  1.68T     1.35G  /mnt/ZFS01/Backups
ZFS01/GlusterFS         36.5K  1.68T     36.5K  /mnt/RemoteFS/GFSVol01
ZFS01/Virtual.Machines  71.1G  1.68T     71.1G  /mnt/ZFS01/Virtual.Machines
ZFS02                   6.55T  2.37T     6.09T  /mnt/ZFS02
ZFS02/Backups            353G  2.37T      353G  /mnt/ZFS02/Backups
ZFS02/Users              117G   383G      117G  /mnt/Users
ZFS02/Users/Guest       36.5K  50.0G     36.5K  /mnt/Users/Guest
ZFS03                   15.6T  6.08T     15.6T  /mnt/ZFS03
ZFS03/Backups           1.40G  6.08T     1.40G  /mnt/ZFS03/Backups
ZFS04                   15.6T  6.10T     13.7T  /mnt/ZFS04
ZFS04/Game.Servers      2.49G  6.10T     2.49G  /mnt/ZFS04/Game.Servers
ZFS04/Resources          133G  6.10T      133G  /mnt/ZFS04/Resources
ZFS04/Users/Private     1.69T  6.10T     1.69T  /mnt/Users/Private

Display More

donh · Dec 20th 2021

Try systemd-analyze blame to see what is taking long to boot. There are other parameters in the man page too.

NozomiYuki · Dec 20th 2021

Sorry if I wasn't clear, I already know exactly what is causing it which is ZFS. Specifically the initial ZFS import at boot (zfs-import-cache.service) taking around 90 seconds
The part I'm not sure on is the why (seems due to the amount of disks I have as it's a lot faster with 3 disks vs almost 30) and if anything can be done to make this process quicker

ryecoaaron · Dec 20th 2021

Even though I said I wasn't going to change anything, I like to spend my day off working on plugins (/sarcasm). openmediavault-mergerfs 6.0.14 in repo. Adds filesystem selector since this feature is easier than adding paths I guess. So, you can use filesystems, shared folders, and/or paths now.

Version	7.1.0-2 (Sandworm)
Processor	AMD EPYC 7302P 16-Core Processor
Kernel	Linux 6.1.15-1-pve
Hardware	Dell R7515

Participate now!