File System won't mount

Hi,

First post on here, so hopefully someone will be able to help. I know very little about Linus commands etc, so be gentle.

I'm running OMV7, have been running fine for a few months now, but today I noticed that I couldn't access my share (via PLEX) and that the system was running at 40-50% CPU utilisation since midnight. I rebooted and now my File System (RAID 5 Share BTRFS) won't mount.

Devices are x4 disks all showing as Good under SMART

File System: /dev/md0. BTRFS Reference (Ticked), Status (Online). But not mounted and can't mount.

I've tried a number of commands I've seen online but the following is what's shown up:

root@omvnas:~# lsusb

Bus 003 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub

Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 002 Device 002: ID 03f0:7029 HP, Inc Virtual Keyboard

Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

root@omvnas:~# blkid

/dev/sdd: UUID="f28ecc4b-6a6a-5e55-77e4-a570d2f42678" UUID_SUB="4b986b8c-6db3-f2c2-de95-6d5a4585f3e3" LABEL="omvnas:0" TYPE="linux_raid_member"

/dev/sdb: UUID="f28ecc4b-6a6a-5e55-77e4-a570d2f42678" UUID_SUB="4cfa9164-e711-d0ff-b789-afa00baa40fa" LABEL="omvnas:0" TYPE="linux_raid_member"

/dev/md0: UUID="908419ac-9660-4a30-956b-5f6848e46dfe" UUID_SUB="cd927f09-6b64-44bc-9310-790b0f1bdbdf" BLOCK_SIZE="4096" TYPE="btrfs"

/dev/sde5: UUID="b1578c80-3cf4-46e8-9222-8240b29b80ac" TYPE="swap" PARTUUID="b7d44676-05"

/dev/sde1: UUID="0d13b6be-275c-4357-9677-ca3f920bdd8e" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="b7d44676-01"

/dev/sdc: UUID="f28ecc4b-6a6a-5e55-77e4-a570d2f42678" UUID_SUB="a62d3a3f-4a0f-c4df-74dc-0c143bc7d156" LABEL="omvnas:0" TYPE="linux_raid_member"

/dev/sda: UUID="f28ecc4b-6a6a-5e55-77e4-a570d2f42678" UUID_SUB="d18baccb-5219-12df-9f90-5fb5cd57b648" LABEL="omvnas:0" TYPE="linux_raid_member"

root@omvnas:~# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS

sda 8:0 0 3.6T 0 disk

└─md0 9:0 0 10.9T 0 raid5

sdb 8:16 0 3.6T 0 disk

└─md0 9:0 0 10.9T 0 raid5

sdc 8:32 0 3.6T 0 disk

└─md0 9:0 0 10.9T 0 raid5

sdd 8:48 0 3.6T 0 disk

└─md0 9:0 0 10.9T 0 raid5

sde 8:64 0 232.9G 0 disk

├─sde1 8:65 0 231.9G 0 part /

├─sde2 8:66 0 1K 0 part

└─sde5 8:69 0 976M 0 part [SWAP]

root@omvnas:~# mount -a

mount: /srv/dev-disk-by-uuid-908419ac-9660-4a30-956b-5f6848e46dfe: wrong fs type, bad option, bad superblock on /dev/md0, missing codepage or helper program, or other error.

dmesg(1) may have more information after failed mount system call.

root@omvnas:~# sudo fdisk -l

Disk /dev/sde: 232.89 GiB, 250059350016 bytes, 488397168 sectors

Disk model: VB0250EAVER

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0xb7d44676

Device Boot Start End Sectors Size Id Type

/dev/sde1 * 2048 486395903 486393856 231.9G 83 Linux

/dev/sde2 486397950 488396799 1998850 976M 5 Extended

/dev/sde5 486397952 488396799 1998848 976M 82 Linux swap / Solaris

Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: WDC WD40EFRX-68W

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/sdd: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: WDC WD40EFRX-68N

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/sdc: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: WDC WD40EFRX-68W

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: WDC WD40EFRX-68W

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/md0: 10.92 TiB, 12001954234368 bytes, 23441316864 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 524288 bytes / 1572864 bytes

root@omvnas:~# omv-salt deploy run fstab

debian:

----------

    ID: create_filesystem_mountpoint_1777e36c-edf0-4835-8382-9ab5f95d74ea

    Function: file.accumulated

    Result: True

     Comment: Accumulator create_filesystem_mountpoint_1777e36c-edf0-4835-8382-9ab5f95d74ea for file /etc/fstab was charged by text

     Started: 15:22:22.184474

    Duration: 0.709 ms

Changes:

----------

    ID: mount_filesystem_mountpoint_1777e36c-edf0-4835-8382-9ab5f95d74ea

    Function: mount.mounted

    Name: /srv/dev-disk-by-uuid-908419ac-9660-4a30-956b-5f6848e46dfe

    Result: False

     Comment: mount: /srv/dev-disk-by-uuid-908419ac-9660-4a30-956b-5f6848e46dfe: wrong fs type, bad option, bad superblock on /dev/md0, missing codepage or helper program, or other error.

dmesg(1) may have more information after failed mount system call.

     Started: 15:22:22.185740

    Duration: 45.821 ms

Changes:

----------

    ID: append_fstab_entries

    Function: file.blockreplace

    Name: /etc/fstab

    Result: True

     Comment: No changes needed to be made

     Started: 15:22:22.232344

    Duration: 5.749 ms

Changes:

----------

    ID: fstab_systemctl_daemon_reload

    Function: module.run

    Result: True

     Comment: State was not run because none of the onchanges reqs changed

     Started: 15:22:22.239443

    Duration: 0.007 ms

Changes:

Summary for debian

------------

Succeeded: 3

Failed: 1

------------

Total states run: 4

Total run time: 52.286 ms

[ERROR ] Command 'mount' failed with return code: 32

[ERROR ] stderr: mount: /srv/dev-disk-by-uuid-908419ac-9660-4a30-956b-5f6848e46dfe: wrong fs type, bad option, bad superblock on /dev/md0, missing codepage or helper program, or other error.

dmesg(1) may have more information after failed mount system call.

[ERROR ] retcode: 32

[ERROR ] mount: /srv/dev-disk-by-uuid-908419ac-9660-4a30-956b-5f6848e46dfe: wrong fs type, bad option, bad superblock on /dev/md0, missing codepage or helper program, or other error.

dmesg(1) may have more information after failed mount system call.

root@omvnas:~# mdadm --detail /dev/md0

/dev/md0:

Version : 1.2

Creation Time : Sat Nov 30 17:25:08 2024

Raid Level : raid5

Array Size : 11720658432 (10.92 TiB 12.00 TB)

Used Dev Size : 3906886144 (3.64 TiB 4.00 TB)

Raid Devices : 4

Total Devices : 4

Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Wed Jan 1 05:31:14 2025

State : clean

Active Devices : 4

Working Devices : 4

Failed Devices : 0

Spare Devices : 0

Layout : left-symmetric

Chunk Size : 512K

Consistency Policy : bitmap

Name : omvnas:0 (local to host omvnas)

UUID : f28ecc4b:6a6a5e55:77e4a570:d2f42678

Events : 15731

Number Major Minor RaidDevice State

0 8 0 0 active sync /dev/sda

1 8 32 1 active sync /dev/sdc

2 8 16 2 active sync /dev/sdb

3 8 48 3 active sync /dev/sdd

Any help is massively appreciated

Tony

Hi Krisbee,

Thanks for the response, I didn't setup email notifications as I was having issues with that and my email smtp. I do have data backup yes,

Please find outputs below, journalclt said command not found though. (Edited, realised this should have been journalctl, now rerun)

Quote from Krisbee

Batfink_UK Just to say, the output from CLI commands is best posted between code marks </>

From what I can make out of your info the MD RAID5 is intact, active and showing all 4 devices in the array. So the problem is with your BTRFS filesystem. If it's refusing to mount then you cannot check the FS status until it is.

When setting up your OMV did you set up email notifications? These can warn you of accumulated errors on a BTRFS system?

Do you have backups, be warned that RAID is not a substitute for proper backups. Using BTRFS on top of MD RAID means you lose the BTRFS capacity to recover from certain error conditions.

Can you please post ( between code marks </>) the output of the following CLI commands:

cat /etc/fstab

# /etc/fstab: static file system information.

#

# Use 'blkid' to print the universally unique identifier for a

# device; this may be used with UUID= as a more robust way to name devices

# that works even if disks are added and removed. See fstab(5).

#

# systemd generates mount units based on this file, see systemd.mount(5).

# Please run 'systemctl daemon-reload' after making changes here.

#

# <file system> <mount point> <type> <options> <dump> <pass>

# / was on /dev/sdb1 during installation

UUID=0d13b6be-275c-4357-9677-ca3f920bdd8e / ext4 errors=remount-ro 0 1

# swap was on /dev/sdb5 during installation

UUID=b1578c80-3cf4-46e8-9222-8240b29b80ac none swap sw 0 0

/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0

# >>> [openmediavault]

/dev/disk/by-uuid/908419ac-9660-4a30-956b-5f6848e46dfe /srv/dev-disk-by-uuid-908419ac-9660-4a30-956b-5f6848e46dfe btrfdefaults,nofail 0 2

# <<< [openmediavault]

findmnt  --real

TARGET SOURCE FSTYPE OPTIONS

/ /dev/sde1 ext4 rw,relatime,errors=remount-ro

journalclt -b g btrfs

Failed to add match 'g': Invalid argument

Display More

Sorry just realised what you meant by </>

cat /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# systemd generates mount units based on this file, see systemd.mount(5).
# Please run 'systemctl daemon-reload' after making changes here.
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sdb1 during installation
UUID=0d13b6be-275c-4357-9677-ca3f920bdd8e / ext4 errors=remount-ro 0 1
# swap was on /dev/sdb5 during installation
UUID=b1578c80-3cf4-46e8-9222-8240b29b80ac none swap sw 0 0
/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0
# >>> [openmediavault]
/dev/disk/by-uuid/908419ac-9660-4a30-956b-5f6848e46dfe /srv/dev-disk-by-uuid-908419ac-9660-4a30-956b-5f6848e46dfe btrfdefaults,nofail 0 2
# <<< [openmediavault]

findmnt --real

TARGET SOURCE    FSTYPE OPTIONS
/      /dev/sde1 ext4   rw,relatime,errors=remount-ro

journalctl -b g btrfs

Failed to add match 'g': Invalid argument

Quote from Krisbee

That' s journalclt -b -g btrfs  as in my correction above.

Krisbee I just get the following:

root@omvnas:~# journalclt -b -g btrfs
-bash: journalclt: command not found

Quote from Krisbee

Would you believe it? I seem to have fat figured that not once by twice . I guess it never dawned on you that I mistyped it.

Code

journalctl -b -g btrfs

While you're at the CLI you can add this one too:

btrfs check /deb/md0

Krisbee Ah that got the following:

root@omvnas:~# journalctl -b -g btrfs
Jan 01 20:42:01 omvnas kernel: Btrfs loaded, crc32c=crc32c-intel, zoned=yes, fsverity=yes
Jan 01 20:42:01 omvnas kernel: BTRFS: device fsid 908419ac-9660-4a30-956b-5f6848e46dfe devid 1 transid 15939 /dev/md0 scanned by btrfs (254)
Jan 01 20:42:24 omvnas kernel: BTRFS info (device md0): first mount of filesystem 908419ac-9660-4a30-956b-5f6848e46dfe
Jan 01 20:42:24 omvnas kernel: BTRFS info (device md0): using crc32c (crc32c-intel) checksum algorithm
Jan 01 20:42:24 omvnas kernel: BTRFS info (device md0): using free space tree
Jan 01 20:42:24 omvnas kernel: BTRFS error (device md0): bad tree block start, mirror 1 want 24461312 have 23412736
Jan 01 20:42:24 omvnas kernel: BTRFS error (device md0): bad tree block start, mirror 2 want 24461312 have 23412736
Jan 01 20:42:24 omvnas kernel: BTRFS error (device md0): failed to read chunk root
Jan 01 20:42:24 omvnas kernel: BTRFS error (device md0): open_ctree failed

check /deb/md0 ( Krisbee & Soma) gets the following:

root@omvnas:~# btrfs check /dev/md0
Opening filesystem to check...
bad tree block 24461312, bytenr mismatch, want=24461312, have=23412736
ERROR: cannot read chunk root
ERROR: cannot open file system

Quote from Krisbee

Batfink_UK Let's see if I can type something without error. The fact that you listed the output of lusb makes me wonder if your HDDs are connected via USB. I hope that's not the case.

What's your next move? This depends how complete and recent your backups are. If a restore from backup means you will lose little or no data, the a full restore to a new BTRFS FS might be the quickest route. Otherwise, it's a case of attempting to mount the BTRFS FS read only and then back up the data before starting over. ( See the 1st reply by Chris Murphy on this thread: https://discussion.fedoraproje…superblock-btrfs/114445/2 and use the first set of mount read-only options. )

Personally, I'd restore from backup but would seriously consider using a BTRFS RAID1 profile and not BTRFS on top of MD RAID5. If your HDDs are connected by USB, then you shouldn't be using RAID at all.

Krisbee HDDs are not connected via USB, they're internally connected via a LSI9211-8I HBA. Backup is recent, so I hopefully wouldn't be missing too many things. I wouldn't mind trying to BTRFS as read only so I can validate I'm not missing too much before rebuilding, so I'll take a look at the link.

As for RAID1 that would be preferable, but I've got approx 9.5TB of data which just fits on the 10.9TB RAID5, I just don't have enough storage for RAID1 unfortunately.

Quote from Krisbee

Batfink_UK I signed off last night b4 following up on previous comments. It's possible the command btrfs rescue chunk-recover /dev/md0 may be useful but would probably take many hrs to run on a 10TB filesystem. While there's nothing to stop the use of BTRFS on top of another block device layer, BTRFS won't be able to repair data on a "single profile". BTRFS is also designed with survival from power failures/ungraceful shutdowns in mind and there should be no "partial writes", but does the MD RAID layer guarantee this?

With your storage constraints, a RAIDZ1 zfs pool is an alternative.

Hi Krisbee,

Thanks for all of your advice, but unfortunately nothing worked.

I have no idea what caused this to happen (which is still a concern for me), but in the end I just rebuilt the whole nas, new OMV7 install, wiped the old drives and setup as ZFS RAIDZ1 Pools instead, so we'll see what happens.

One thing that really confused me before rebuilding was that I noticed my security cams shared folder was still showing on my desktop and my cameras were still sending/storing footage to it. This was to the drive that was unable to be mounted, so I have absolutely no idea what was going on with the system.

Quote from Krisbee

Batfink_UK

Sending data to a shared folder on your OMV NAS when the filesystem is not mounted ( as confirmed at the OMV CLI ) is not possible, you cannot write to an unmounted FS. That really makes no sense.

I'm not surprised the BTRFS fs refused to mount. The two error lines re: mirror 1 & mirror 2 above can only be referring to part of the BTRFS metadata as the data has a "single" profile. Metadata corruption typically means restore from backup.

As to cause, by a total re-install you lost the chance to sift through the system journal for any clues. I don't know the details of your build, but your

LSI9211-8I HBA temps need monitoring. Those cards were designed for forced cooling in rack servers, not your typical tower case, so heat might have been a factor to cause a glitch. COW FS are more sensitive to hardware troubles than say EXT4.

Make sure you enable ZED under email notifications. One weakness of BTRFS is the lack of a daemon monitor for alerts of failed drives. etc. ZFS has a comprehensive set of alerts to warn of problems. If your familiar with ZFS, the you'll know to make good use of datasets and snapshots.

Display More

Krisbee

I now have monitoring enabled and have enabled ZED via email notifications.

As for the LSI19211-8I, I'm using an old server I had, HP ML110 G7 which has forced air channelled over the PCIe slot that card is mounted to, so hopefully it should be staying cool enough.

Do you know if there's a way I can monitor temps for the card via command line?

Thanks Tony

Quote from Krisbee

@Natfink_UK Do SAS2008 cards have temp sensors? I'll leave that one with you as I seem to remember they don't, but could be wrong.

Krisbee I think you're right and they don't. I'll just make sure I keep a regular backup (luckily like before as it doesn't look like I lost anything), then recover again if it goes wrong again. The only things to have changed recently are the HBA, so I'd swap that and rebuild again, as there's never been issues before.

Thanks for all of your assistance.