Filesystem failed on a disk from SnapRaid array / can't reach Filesystem to prepare drive to fix it

bermuda · May 9th 2025

Hello everyone,

I got an email notification about a missing mountpoint. The drive is a data-member of an SnapRaid array (8 data disks, 2 parity disks). All the data disks from that array are also in mergerfs. Since the mount of the mergerfs fails with "Couldn't extract an UUID from the provided path '/srv/ec947406-ea1a-4fc1-a176-628af4dca90c'." I am unable to get into the "filesystem" configuration and cannot create an new disk for the array to fix the array.

The disk seems to be OK. Only the filesystem is gone and the GPT-partition tabe too. I upgraded to the latest OMV a week ago. Can't say if that is somehow related. I startet with OMV5 and upgraded to OMV6 some time ago. Had no issues with any of my disks. SMART is not showing anything suspicious. That's just FYI. I want to restore the failed drive, but I can't reach filesystem configuration, due to the mentioned error.

gderf · Friday, 5:22 pm

Try removing the failed disk from the mergerfs pool. Then restart the pool or reboot.

bermuda · Friday, 5:40 pm

Hello,

thanks, I tried it, but got another error about the mountpoint still not being able to be mounted. The changes are not being applied. I still see the yellow banner above.

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LC_ALL=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color mergerfs 2>&1' with exit code '1': debian:
----------
ID: configure_pool_dir
Function: file.directory
Name: /srv/mergerfs
Result: True
Comment: The directory /srv/mergerfs is in the correct state
Started: 17:34:51.685566
Duration: 4.05 ms
Changes:
----------
ID: remove_mergerfs_mount_files_/etc/systemd/system
Function: module.run
Result: True
Comment: file.find: []
Started: 17:34:51.690034
Duration: 2.675 ms
Changes:
----------
file.find:
----------
ID: systemd_remove_dead_symlinks
Function: cmd.run
Name: find /etc/systemd/system/multi-user.target.wants -xtype l -print -delete
Result: True
Comment: Command "find /etc/systemd/system/multi-user.target.wants -xtype l -print -delete" run
Started: 17:34:51.693156
Duration: 2.931 ms
Changes:
----------
pid:
144451
retcode:
0
stderr:
stdout:
----------
ID: configure_mergerfs_storage
Function: file.managed
Name: /etc/systemd/system/srv-ec947406\x2dea1a\x2d4fc1\x2da176\x2d628af4dca90c.mount
Result: True
Comment: File /etc/systemd/system/srv-ec947406\x2dea1a\x2d4fc1\x2da176\x2d628af4dca90c.mount is in the correct state
Started: 17:34:51.696171
Duration: 125.539 ms
Changes:
----------
ID: systemd-reload_storage
Function: cmd.run
Name: systemctl daemon-reload
Result: True
Comment: Command "systemctl daemon-reload" run
Started: 17:34:51.821817
Duration: 385.604 ms
Changes:
----------
pid:
144488
retcode:
0
stderr:
stdout:
----------
ID: enable_storage_mergerfs
Function: service.enabled
Name: srv-ec947406\x2dea1a\x2d4fc1\x2da176\x2d628af4dca90c.mount
Result: True
Comment: Service srv-ec947406\x2dea1a\x2d4fc1\x2da176\x2d628af4dca90c.mount is already enabled, and is in the desired state
Started: 17:34:52.892862
Duration: 16.506 ms
Changes:
----------
ID: restart_storage_mergerfs
Function: cmd.run
Name: systemctl restart srv-ec947406\x2dea1a\x2d4fc1\x2da176\x2d628af4dca90c.mount
Result: False
Comment: Command "systemctl restart srv-ec947406\x2dea1a\x2d4fc1\x2da176\x2d628af4dca90c.mount" run
Started: 17:34:52.909484
Duration: 6.295 ms
Changes:
----------
pid:
144548
retcode:
5
stderr:
Failed to restart srv-ec947406x2dea1ax2d4fc1x2da176x2d628af4dca90c.mount: Unit srv-ec947406x2dea1ax2d4fc1x2da176x2d628af4dca90c.mount not found.
stdout:


Summary for debian
------------
Succeeded: 6 (changed=4)
Failed:    1
------------
Total states run:     7
Total run time: 543.600 ms
[ERROR   ] Command 'systemctl' failed with return code: 5
[ERROR   ] stderr: Failed to restart srv-ec947406x2dea1ax2d4fc1x2da176x2d628af4dca90c.mount: Unit srv-ec947406x2dea1ax2d4fc1x2da176x2d628af4dca90c.mount not found.
[ERROR   ] retcode: 5
[ERROR   ] {'pid': 144548, 'retcode': 5, 'stdout': '', 'stderr': 'Failed to restart srv-ec947406x2dea1ax2d4fc1x2da176x2d628af4dca90c.mount: Unit srv-ec947406x2dea1ax2d4fc1x2da176x2d628af4dca90c.mount not found.'}


OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LC_ALL=C.UTF-8; export LANGUAGE=; omv-salt deploy run --no-color mergerfs 2>&1' with exit code '1': debian:
----------
ID: configure_pool_dir
Function: file.directory
Name: /srv/mergerfs
Result: True
Comment: The directory /srv/mergerfs is in the correct state
Started: 17:34:51.685566
Duration: 4.05 ms
Changes:
----------
ID: remove_mergerfs_mount_files_/etc/systemd/system
Function: module.run
Result: True
Comment: file.find: []
Started: 17:34:51.690034
Duration: 2.675 ms
Changes:
----------
file.find:
----------
ID: systemd_remove_dead_symlinks
Function: cmd.run
Name: find /etc/systemd/system/multi-user.target.wants -xtype l -print -delete
Result: True
Comment: Command "find /etc/systemd/system/multi-user.target.wants -xtype l -print -delete" run
Started: 17:34:51.693156
Duration: 2.931 ms
Changes:
----------
pid:
144451
retcode:
0
stderr:
stdout:
----------
ID: configure_mergerfs_storage
Function: file.managed
Name: /etc/systemd/system/srv-ec947406\x2dea1a\x2d4fc1\x2da176\x2d628af4dca90c.mount
Result: True
Comment: File /etc/systemd/system/srv-ec947406\x2dea1a\x2d4fc1\x2da176\x2d628af4dca90c.mount is in the correct state
Started: 17:34:51.696171
Duration: 125.539 ms
Changes:
----------
ID: systemd-reload_storage
Function: cmd.run
Name: systemctl daemon-reload
Result: True
Comment: Command "systemctl daemon-reload" run
Started: 17:34:51.821817
Duration: 385.604 ms
Changes:
----------
pid:
144488
retcode:
0
stderr:
stdout:
----------
ID: enable_storage_mergerfs
Function: service.enabled
Name: srv-ec947406\x2dea1a\x2d4fc1\x2da176\x2d628af4dca90c.mount
Result: True
Comment: Service srv-ec947406\x2dea1a\x2d4fc1\x2da176\x2d628af4dca90c.mount is already enabled, and is in the desired state
Started: 17:34:52.892862
Duration: 16.506 ms
Changes:
----------
ID: restart_storage_mergerfs
Function: cmd.run
Name: systemctl restart srv-ec947406\x2dea1a\x2d4fc1\x2da176\x2d628af4dca90c.mount
Result: False
Comment: Command "systemctl restart srv-ec947406\x2dea1a\x2d4fc1\x2da176\x2d628af4dca90c.mount" run
Started: 17:34:52.909484
Duration: 6.295 ms
Changes:
----------
pid:
144548
retcode:
5
stderr:
Failed to restart srv-ec947406x2dea1ax2d4fc1x2da176x2d628af4dca90c.mount: Unit srv-ec947406x2dea1ax2d4fc1x2da176x2d628af4dca90c.mount not found.
stdout:


Summary for debian
------------
Succeeded: 6 (changed=4)
Failed:    1
------------
Total states run:     7
Total run time: 543.600 ms
[ERROR   ] Command 'systemctl' failed with return code: 5
[ERROR   ] stderr: Failed to restart srv-ec947406x2dea1ax2d4fc1x2da176x2d628af4dca90c.mount: Unit srv-ec947406x2dea1ax2d4fc1x2da176x2d628af4dca90c.mount not found.
[ERROR   ] retcode: 5
[ERROR   ] {'pid': 144548, 'retcode': 5, 'stdout': '', 'stderr': 'Failed to restart srv-ec947406x2dea1ax2d4fc1x2da176x2d628af4dca90c.mount: Unit srv-ec947406x2dea1ax2d4fc1x2da176x2d628af4dca90c.mount not found.'} in /usr/share/php/openmediavault/system/process.inc:247
Stack trace:
#0 /usr/share/php/openmediavault/engine/module/serviceabstract.inc(62): OMV\System\Process->execute()
#1 /usr/share/openmediavault/engined/rpc/config.inc(187): OMV\Engine\Module\ServiceAbstract->deploy()
#2 [internal function]: Engined\Rpc\Config->applyChanges()
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(124): call_user_func_array()
#4 /usr/share/php/openmediavault/rpc/serviceabstract.inc(155): OMV\Rpc\ServiceAbstract->callMethod()
#5 /usr/share/php/openmediavault/rpc/serviceabstract.inc(628): OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}()
#6 /usr/share/php/openmediavault/rpc/serviceabstract.inc(152): OMV\Rpc\ServiceAbstract->execBgProc()
#7 /usr/share/openmediavault/engined/rpc/config.inc(208): OMV\Rpc\ServiceAbstract->callMethodBg()
#8 [internal function]: Engined\Rpc\Config->applyChangesBg()
#9 /usr/share/php/openmediavault/rpc/serviceabstract.inc(124): call_user_func_array()
#10 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod()
#11 /usr/sbin/omv-engined(544): OMV\Rpc\Rpc::call()
#12 {main}

Display More

This is the output from the ls command from the shell:

Code

drwxr-xr-x 2 root root 4096  3. Mai 18:28  sockets.target.wants
-rw-r--r-- 1 root root  874  9. Mai 17:04 'srv-ec947406\x2dea1a\x2d4fc1\x2da176\x2d628af4dca90c.mount'
lrwxrwxrwx 1 root root   31 17. Apr 2020   sshd.service -> /lib/systemd/system/ssh.service

The name of that file seems rather strange. The error message says, it can't find that file.

The mountpoint under /srv looks like this:

Code

drwxr-xr-x 7 root root    4096  3. Mai 18:30 dev-disk-by-label-appdata
drwxr-xr-x 4 root root    4096  3. Mai 18:30 dev-disk-by-label-Disk.A.P1
drwxr-xr-x 4 root root    4096  3. Mai 18:30 dev-disk-by-label-Disk.A.P2
drwxr-xr-x 4 root root    4096  3. Mai 18:30 dev-disk-by-label-Disk.A.P3
drwxrwxrwx 2 root root    4096 29. Apr 2020  dev-disk-by-label-Disk.A.P4
drwxr-xr-x 4 root root    4096  3. Mai 18:30 dev-disk-by-label-Disk.B.P1
drwxr-xr-x 4 root root    4096  3. Mai 18:30 dev-disk-by-label-Disk.B.P2
drwxr-xr-x 4 root root    4096  3. Mai 18:30 dev-disk-by-label-Disk.B.P3
drwxr-xr-x 4 root root    4096  3. Mai 18:30 dev-disk-by-label-Disk.B.P4
drwxr-xr-x 3 root root    4096  3. Mai 18:30 dev-disk-by-label-Disk.SATA.2
drwxr-xr-x 3 root root    4096  3. Mai 18:30 dev-disk-by-label-Disk.SATA.3
drwxrwxrwx 2 root root    4096 29. Apr 2020  ec947406-ea1a-4fc1-a176-628af4dca90c
drwxr-xr-x 2 ftp  nogroup 4096 29. Aug 2020  ftp
drwxrwxrwx 2 root root    4096 16. Apr 2024  mergerfs
drwxr-xr-x 3 root root    4096  3. Mai 18:50 pillar
drwxr-xr-x 7 root root    4096  3. Mai 18:50 salt

Display More

This seems at first glance like wrong coding. The dashes - are replaced with \x2d and later the backslash \ is gone (see line 86 in the first code-block).

bermuda · Saturday, 1:35 pm

Hello everyone,

I went on and deleted my shares from mergerfs and after that I also deleted the mergerfs-pool which gave me back the possibility to use filesystem configuration again. So I went on and created a new disk. But the OMV7 snapraid plugin docs seem outdated. There is no fix menu entry in the tools-menu, instead there are fix by drive and fix missing. I was unable to find something about which one to use.

Is someone in here able to tell which one it should be?

crashtest · Saturday, 3:26 pm

I'm not sure, exactly, what you mean.
Under, SnapRAID, Drives, "Fix" is in the menu.

The plugin document outlines how to do -> a "Fix" restore.

bermuda · Saturday, 3:42 pm

Thanks. My menu is different. I'm currently running an audit (also taken from tools menu) and have to wait for it to end. Then I will check again why my menu looks different. I was also under "Services > SnapRAID > Drives". I could see all the drives being listed there with their name and check-marks for data and parity.

crashtest · Saturday, 5:38 pm

Quote from bermuda

My menu is different.

The only explanation I can come up with is, perhaps, your install needs to be updated.

"OR"

You've upgraded from OMV6 to OMV7 without running the 6 to 7 upgrade script. (The script fixes a few things.)

bermuda · Saturday, 5:58 pm

Hello again,

just finished the audit. All data is recoverable. I checked it again. The fix is not in "Drives" but in "Arrays". So something has to be updated I guess. Like I said, I'm on OMV7.

gderf · Saturday, 6:15 pm

Check your version of openmediavault-snapraid plugin. Current is 7.0.13 and has fix under the Arrays menu.

bermuda · Saturday, 6:32 pm

Thanks,

I have chosen the fix by drive from the Drives menu. The process is running and the data is being restored on the drive.

Is there a way to get the old "dev-disk-by-label" link? The new created drive has the now default uuid format.

That is great for machines but not so good for humans. As I see it, I should create that manually, but will it be usable in/by the UI?

crashtest · Saturday, 6:47 pm

Do you have amd64 architecture? My screenshot was from an R-PI4's SnapRAID array. That may be the difference. "Arrays" would not be possible or, at least, highly unlikely on an SBC.

Still, I don't see multiple fix commands in your menu. Fix, within an individual array, would work the same way.

bermuda · Saturday, 6:47 pm

Quote from crashtest

The only explanation I can come up with is, perhaps, your install needs to be updated.
"OR"
You've upgraded from OMV6 to OMV7 without running the 6 to 7 upgrade script. (The script fixes a few things.)

Yes, I did the upgrade as described in this forum. The script wasn't mentioned. There was only a fix which should be used if there were errors during the upgrade process. Mine went with no errors. I only had to manually install openmediavault-md package, as I really do use a software raid level 1 for my system disk.

bermuda · Saturday, 6:59 pm

Quote from crashtest

Do you have amd64 architecture? My screenshot was from an R-PI4's SnapRAID array. That may be the difference. "Arrays" would not be possible or, at least, highly unlikely on an SBC.

Still, I don't see multiple fix commands in your menu. Fix, within an individual array, would work the same way.

Yes, I run my server on regular intel hardware. It's way easier to add/change hardware or replace something when it breaks. Back then when I considered to switch to OMV I also designed my system purely on an PI4. But in the end I leaned towards intel as the performance to cost ratio in my opinion was in favor to that platform. Certain Intel hardware has also good power saving capabilities if run on minimalist hardware. I use an i7-7700T. It barley goes above room temperature with 2 big fans slowly blowing air out of the case. But it has enough power to run ollama AI-models on iGPU if I need it. It's a nice addition to nextcloud-applications and other stuff that has an AI-interface.

gderf · Saturday, 8:45 pm

Quote from bermuda

Is there a way to get the old "dev-disk-by-label" link? The new created drive has the now default uuid format.
That is great for machines but not so good for humans. As I see it, I should create that manually, but will it be usable in/by the UI?

It is not possible to revert OMV to use dev-disk-by-label for any newly added drives. You can create a symlink to the new UUID mountpoint and the symlink can be whatever you want it to be.

bermuda · Sunday, 6:32 am

Quote from gderf

It is not possible to revert OMV to use dev-disk-by-label for any newly added drives. You can create a symlink to the new UUID mountpoint and the symlink can be whatever you want it to be.

Hello,

big thanks again. I will manage that. I still haven't figured out what whiped my disk. But thanks to snapraid I was able to recover from that

UPDATE:
I was able to switch back to dev-disk-by-label, so it is in fact doable, but I know of course that this is not seen as the right procedure since the switch to uuid. So, I won't make it public, but I'm glad it still works and it wasn't that much of a hassle anyway.

Mergerfs-pool now also looks better after I recreated it and it is not referenced by uuid anymore, so I don't have any issues with coding, like the one in post #3.

Filesystem failed on a disk from SnapRaid array / can't reach Filesystem to prepare drive to fix it

macom May 9th 2025

Participate now!

Tags