strange filesystem notifications after drive replacement

BernH · Mar 15th 2026

Yesterday I had to replace a drive in my RAID5 due to some smart errors I didn't like to see. After doing it, I have been getting filesystem monitoring notifications stating the following:

Service: mountpoint_srv_dev-disk-by-uuid-4d26adbd-029a-4d74-a81f-c69bf193a099

Event: Status failed

Description: status failed (32) -- /srv/dev-disk-by-uuid-4d26adbd-029a-4d74-a81f-c69bf193a099 is not a mountpoint

and:

Service: filesystem_srv_dev-disk-by-uuid-4d26adbd-029a-4d74-a81f-c69bf193a099

Event: Does not exist

Description: unable to read filesystem '/srv/dev-disk-by-uuid-4d26adbd-029a-4d74-a81f-c69bf193a099' state

This triggered the monitoring system to: restart

This mountpoint is indeed the one used by the RAID5 and it does exist.

This only started after replacing the drive.

Any help or ideas are welcome.

macom · Mar 16th 2026

You could try to redeploy the configuration of monit with

omv-salt deploy run monit

If that does not solve the issue, check the configuration file

/etc/monit/conf.d/openmediavault-filesystem.conf

Did you check, if /srv/dev-disk-by-uuid-4d26adbd-029a-4d74-a81f-c69bf193a099 is actually a mountpoint and not only a directory. Could the mountpoint of the filesystem on the RAID5 be another one (now)?

BernH · Mar 16th 2026

Quote from macom

You could try to redeploy the configuration of monit with
omv-salt deploy run monit

If that does not solve the issue, check the configuration file
/etc/monit/conf.d/openmediavault-filesystem.conf

Did you check, if /srv/dev-disk-by-uuid-4d26adbd-029a-4d74-a81f-c69bf193a099 is actually a mountpoint and not only a directory. Could the mountpoint of the filesystem on the RAID5 be another one (now)?

Display More

Thanks,

I'll try the monit related things you mentioned.

The mountpoint is correct. I can see all the correct data content of the array there, and if the array is unmounted there is no content visible there.

votdev · Mar 16th 2026

Please post the following output to get a clear overview:

Bash

# blkid
# mount
# cat /etc/fstab

BernH · Mar 16th 2026

I just did the monit re-deploy and a reboot.

The array seems to have mounted where it should, but I think it was slow to mount as my nextcloud lxc tried to start before the the mount and could not run because it couldn't find the data directory I use as a passthrough filesystem to it.

It's rather perplexing.

Here are the outputs. The mount one I had to attach as a text files because it is too big.

Code: blkid

root@nas:~# blkid
/dev/sdf: UUID="9a94e451-1ea3-0c84-2f04-05aa2f4a75bc" UUID_SUB="31bd8004-db89-c1e3-2e81-b40743646e8a" LABEL="nas:NASR5" TYPE="linux_raid_member"
/dev/sdd: UUID="65c691a0-703e-45a6-7704-87884d714f01" UUID_SUB="5f5a5c6d-88cb-e25d-08fe-092e83c4c187" LABEL="nas:0" TYPE="linux_raid_member"
/dev/md127: LABEL="NASR5" UUID="4d26adbd-029a-4d74-a81f-c69bf193a099" BLOCK_SIZE="4096" TYPE="xfs"
/dev/sdb: UUID="65c691a0-703e-45a6-7704-87884d714f01" UUID_SUB="67b189c3-f97e-da1c-fa64-77a1eb7d2dab" LABEL="nas:0" TYPE="linux_raid_member"
/dev/sdk1: UUID="773f7d89-fa39-4e5c-91ed-7d4263a8115a" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="b11ded33-7567-42ac-a734-f81443d3b6b4"
/dev/sdi: UUID="9a94e451-1ea3-0c84-2f04-05aa2f4a75bc" UUID_SUB="a892c8f6-f816-929a-3aa2-f04bdfccecdb" LABEL="nas:NASR5" TYPE="linux_raid_member"
/dev/md0: UUID="0e13d684-5e3e-41e0-a4fa-d1be722413dd" BLOCK_SIZE="4096" TYPE="xfs"
/dev/sdg: UUID="9a94e451-1ea3-0c84-2f04-05aa2f4a75bc" UUID_SUB="bbcdd57f-b1d7-ce2b-bc66-4e21359d6778" LABEL="nas:NASR5" TYPE="linux_raid_member"
/dev/sde: UUID="9a94e451-1ea3-0c84-2f04-05aa2f4a75bc" UUID_SUB="c092fabf-7168-ad3c-f5db-6fe8b7117fdd" LABEL="nas:NASR5" TYPE="linux_raid_member"
/dev/sdc1: UUID="d69073e2-e8f5-4605-aaa2-eb2c662b20b4" BLOCK_SIZE="4096" TYPE="xfs" PARTUUID="a92fc2b0-bfd0-4c5b-a08b-882fcfc0a86f"
/dev/sda2: UUID="c6a6b7b4-8054-4bb5-8633-e4b0df974502" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="9f14a38f-5a6a-4b09-993d-39bc0300984f"
/dev/sda3: UUID="e5f57fbb-cd35-4457-bd60-fad84de417f9" TYPE="swap" PARTUUID="377a8430-a0ff-4800-b95e-9a3180d4c4d2"
/dev/sda1: UUID="18BD-5C9D" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="f36dce67-0b7c-475f-8638-621aa07e12b6"
/dev/sdj: UUID="9a94e451-1ea3-0c84-2f04-05aa2f4a75bc" UUID_SUB="5f0327f2-219a-8d86-7045-8976ec626e81" LABEL="nas:NASR5" TYPE="linux_raid_member"
/dev/sdh: UUID="9a94e451-1ea3-0c84-2f04-05aa2f4a75bc" UUID_SUB="74c3f1fd-6177-67b4-65d3-5a4593a1ab3e" LABEL="nas:NASR5" TYPE="linux_raid_member"

Display More

Code: fstab

root@nas:~# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# systemd generates mount units based on this file, see systemd.mount(5).
# Please run 'systemctl daemon-reload' after making changes here.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda2 during installation
UUID=c6a6b7b4-8054-4bb5-8633-e4b0df974502 /               ext4    noatime,errors=remount-ro 0       1
# /boot/efi was on /dev/sda1 during installation
UUID=18BD-5C9D  /boot/efi       vfat    umask=0077      0       1
# swap was on /dev/sda3 during installation
UUID=e5f57fbb-cd35-4457-bd60-fad84de417f9 none            swap    sw              0       0
# >>> [openmediavault]
/dev/disk/by-uuid/4d26adbd-029a-4d74-a81f-c69bf193a099          /srv/dev-disk-by-uuid-4d26adbd-029a-4d74-a81f-c69bf193a099      xfs     defaults,nofail,usrquota,grpquota,inode64   0 2
/dev/disk/by-uuid/0e13d684-5e3e-41e0-a4fa-d1be722413dd          /srv/dev-disk-by-uuid-0e13d684-5e3e-41e0-a4fa-d1be722413dd      xfs     defaults,nofail,usrquota,grpquota,inode64   0 2
/dev/disk/by-uuid/d69073e2-e8f5-4605-aaa2-eb2c662b20b4          /srv/dev-disk-by-uuid-d69073e2-e8f5-4605-aaa2-eb2c662b20b4      xfs     defaults,nofail,usrquota,grpquota,inode64   0 2
# <<< [openmediavault]

Display More

votdev · Mar 16th 2026

Hmmm, looks good IMHO. The fact that you get the monit notification is that the filesystem was not available at the time monit executed the check. When the check fails 2 times, monit is remounting the filesystem. That might explain why the file system is now shown as mounted.

BernH · Mar 16th 2026

Quote from votdev

Hmmm, looks good IMHO. The fact that you get the monit notification is that the filesystem was not available at the time monit executed the check. When the check fails 2 times, monit is remounting the filesystem. That might explain why the file system is now shown as mounted.

Yeah, it's a little strange. At seems as though the array assembly is slow when starting the system up. I have had to replace drives before without a problem. This time however, something is a bit strange.

I will have to do a bit of digging into mdadm I guess to see if I can find any problems with the assembly, or perhaps a omv-salt deploy run mdadm incase something is a bit "out of sync" for some reason

BernH · Mar 16th 2026

UPDATE

I ran omv-salt deploy run mdadm to ensure the mdadm config was correct. I also reseated the data cables to the drives, just in case they were causing a problem.

The array now assembles quickly and is mounted before docker and kvm starts, and I am getting no more mdadm/mountpoint errors on boot. The filesystem passthrough to kvm still seems to not be passed through before my nextcloud lxc starts, so this particular quirk is still there, and is likely related to a more recent kvm/lib-virt update, but I can't really pinpoint the time this may have started, as my systyem is only shutdown/rebooted when needed.

As a workaround, I set up a bash script on an on boot cron job in the lxc to check for the existence of a .ncdata file I created in the nextcloud data directory, and reboot the lxc if the files does not exist.

strange filesystem notifications after drive replacement

BernH Mar 16th 2026

BernH Mar 16th 2026

Participate now!