Missing file system (raid5) after power loss

Oberon · Nov 14th 2023

Hey guys so I had a power loss today, actually likely 2-3 in a span of half an hour if that's relevant. I usually leave my NAS alone, just login if I get a SMART error or once a month to perform updates.

So OMV is starting fine as far as I can tell, but in the Software RAID section there's nothing, the file system associated to that raid array is "missing". Drives are all visible in the Drives section.

I use 6 4tiB drives, mix of WD, Toshiba and Seagate.

Any help greatly appreciated.

cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]

unused devices: <none>

blkid

/dev/sda1: UUID="A9E2-A40E" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="8f08deeb-a2c9-476d-8807-e3033e7fd223"

/dev/sda2: UUID="5a65c3f3-36a6-470f-bff3-0b29dfbcabd2" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="5fbb5628-3caf-4472-aade-fe58ff57321b"

/dev/sdd: UUID="4211a61c-1dcb-3783-1228-0f01549d9cde" UUID_SUB="14daa08e-c7ef-72ba-8677-d6b157f65836" LABEL="Athena:0" TYPE="linux_raid_member"

/dev/sdc: UUID="4211a61c-1dcb-3783-1228-0f01549d9cde" UUID_SUB="7a276e9e-44f6-150b-8537-63f6784a6048" LABEL="Athena:0" TYPE="linux_raid_member"

/dev/sdg: UUID="4211a61c-1dcb-3783-1228-0f01549d9cde" UUID_SUB="3ce1eccd-bbed-e913-64f0-58a881049919" LABEL="Athena:0" TYPE="linux_raid_member"

/dev/sdf: UUID="4211a61c-1dcb-3783-1228-0f01549d9cde" UUID_SUB="80aadf8f-55f8-2b14-8511-0921fa05a680" LABEL="Athena:0" TYPE="linux_raid_member"

/dev/sde: UUID="4211a61c-1dcb-3783-1228-0f01549d9cde" UUID_SUB="63c9a77b-2027-84a5-057c-759b37c6c2fc" LABEL="Athena:0" TYPE="linux_raid_member"

/dev/sda3: PARTUUID="b0610092-bab0-4090-a201-51659aae791e"

/dev/sdb1: PARTUUID="7349b940-bc1f-4ff5-9965-c1d78d80b692"

fdisk -l | grep "Disk "

Disk /dev/sda: 111.79 GiB, 120034123776 bytes, 234441648 sectors

Disk model: KINGSTON SV300S3

Disk identifier: C3D7740C-FB5D-4AEF-B616-CE49BBD08854

Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: WDC WD40EFRX-68N

Disk identifier: 6C8E2A46-9F5B-4FA2-8201-D8EB607F82BC

Disk /dev/sdd: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: TOSHIBA HDWD240

Disk /dev/sdc: 3.64 TiB, 4000753476096 bytes, 7813971633 sectors

Disk model: WDC WD40EZRX-00S

Disk /dev/sdg: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: ST4000DM004-2CV1

Disk /dev/sdf: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: TOSHIBA MD04ACA4

Disk /dev/sde: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: WDC WD40EFRX-68N

cat /etc/mdadm/mdadm.conf

# This file is auto-generated by openmediavault (https://www.openmediavault.org)

# WARNING: Do not edit this file, your changes will get lost.

# mdadm.conf

#

# Please refer to mdadm.conf(5) for information about this file.

#

# by default, scan all partitions (/proc/partitions) for MD superblocks.

# alternatively, specify devices to scan, using wildcards if desired.

# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.

# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is

# used if no RAID devices are configured.

DEVICE partitions

# auto-create devices with Debian standard permissions

CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system

HOMEHOST <system>

# definitions of existing MD arrays

ARRAY /dev/md/Athena:0 metadata=1.2 name=Athena:0 UUID=4211a61c:1dcb3783:12280f01:549d9cde

MAILADDR root

mdadm --detail --scan --verbose

Nothing after I hit enter in the terminal

geaves · Nov 15th 2023

Well there's nothing to work with, there should be some output from cat /proc/mdstat and mdadm --detail and there isn't, but there could be some corruption to the omv device.

First try simply restarting the server from OMV's GUI, check raid management and cat /proc/mdstat if it's still in the same state (you're looking for some output) then ->

Disconnect all the drives and reinstall on a another drive/usb flash drive, set up and update OMV, shut down and connect the data drives and check again

Oberon · Nov 15th 2023

Thank I did reboot the server a couple of times before to no avail, will still try again, doesn't take long.

If nothing new, I'll re-install following your instructions.

Oberon · Nov 26th 2023

Ok so I had focus on other things but I'm back trying to sort this raid array out.

Rebooting didn't change the output from the commands or what is visible in the web gui.

Quote from geaves

Disconnect all the drives and reinstall on a another drive/usb flash drive, set up and update OMV, shut down and connect the data drives and check again

After doing this, I'm getting an output to mdadm

mdadm --detail --scan --verbose

INACTIVE-ARRAY /dev/md127 num-devices=5 metadata=1.2 name=Athena:0 UUID=4211a61c:1dcb3783:12280f01:549d9cde

devices=/dev/sdc,/dev/sdd,/dev/sde,/dev/sdf,/dev/sdg

It says I have 5 drives in this inactive array but it's a raid5 with 6 drives, all 6 are connected as can be seen below and the array wasn't degraded before the power loss.

For good measure, I ran the other commands again :

cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]

md127 : inactive sdc[0](S) sdg[6](S) sdd[5](S) sdf[7](S) sde[4](S)

19534417592 blocks super 1.2

unused devices: <none>

blkid

/dev/sdc: UUID="4211a61c-1dcb-3783-1228-0f01549d9cde" UUID_SUB="7a276e9e-44f6-150b-8537-63f6784a6048" LABEL="Athena:0" TYPE="linux_raid_member"

/dev/sde: UUID="4211a61c-1dcb-3783-1228-0f01549d9cde" UUID_SUB="63c9a77b-2027-84a5-057c-759b37c6c2fc" LABEL="Athena:0" TYPE="linux_raid_member"

/dev/sdd: UUID="4211a61c-1dcb-3783-1228-0f01549d9cde" UUID_SUB="14daa08e-c7ef-72ba-8677-d6b157f65836" LABEL="Athena:0" TYPE="linux_raid_member"

/dev/sda1: UUID="F9A5-C588" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="fb813cbb-2e0d-4594-9ba5-f7660f0fed7c"

/dev/sda2: UUID="e10b24bd-a91b-4571-a4b7-587f2f57744e" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="6a00e573-ceae-4484-b766-43fda099ea55"

/dev/sda3: UUID="88c23d4c-914c-4f47-8de0-922d2caad798" TYPE="swap" PARTUUID="a96c33be-ecbb-40c0-94d0-ace3603cba0e"

/dev/sdf: UUID="4211a61c-1dcb-3783-1228-0f01549d9cde" UUID_SUB="80aadf8f-55f8-2b14-8511-0921fa05a680" LABEL="Athena:0" TYPE="linux_raid_member"

/dev/sdg: UUID="4211a61c-1dcb-3783-1228-0f01549d9cde" UUID_SUB="3ce1eccd-bbed-e913-64f0-58a881049919" LABEL="Athena:0" TYPE="linux_raid_member"

/dev/sdb1: PARTUUID="7349b940-bc1f-4ff5-9965-c1d78d80b692"

fdisk -l | grep "Disk "

Disk /dev/sdc: 3.64 TiB, 4000753476096 bytes, 7813971633 sectors

Disk model: WDC WD40EZRX-00S

Disk /dev/sde: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: WDC WD40EFRX-68N

Disk /dev/sdd: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: TOSHIBA HDWD240

Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: WDC WD40EFRX-68N

Disk identifier: 6C8E2A46-9F5B-4FA2-8201-D8EB607F82BC

Disk /dev/sda: 238.47 GiB, 256060514304 bytes, 500118192 sectors

Disk model: Crucial_CT256MX1

Disk identifier: 0FE33E33-AB4B-4362-B1FD-1563CC19FFB2

Disk /dev/sdf: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: TOSHIBA MD04ACA4

Disk /dev/sdg: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors

Disk model: ST4000DM004-2CV1

cat /etc/mdadm/mdadm.conf

# mdadm.conf

#

# !NB! Run update-initramfs -u after updating this file.

# !NB! This will ensure that initramfs has an uptodate copy.

#

# Please refer to mdadm.conf(5) for information about this file.

#

# by default (built-in), scan all partitions (/proc/partitions) and all

# containers for MD superblocks. alternatively, specify devices to scan, using

# wildcards if desired.

#DEVICE partitions containers

# automatically tag new arrays as belonging to the local system

HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts

MAILADDR root

# definitions of existing MD arrays

# This configuration was auto-generated on Mon, 24 Jul 2023 10:14:16 +0000 by mkconf

Oberon · Nov 26th 2023

Update

After looking at the many similar issues I am assuming the array is viable but for one disk (sdb) for some reason.

So I mdadm --stop /dev/md127 then mdadm --assemble --force --verbose /dev/md127 /dev/sd[cdefg] which returned this :

Code

mdadm: looking for devices for /dev/md127
mdadm: /dev/sdc is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdd is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sde is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdf is identified as a member of /dev/md127, slot 5.
mdadm: /dev/sdg is identified as a member of /dev/md127, slot 2.
mdadm: forcing event count in /dev/sdf(5) from 1688456 up to 3032476
mdadm: clearing FAULTY flag for device 3 in /dev/md127 for /dev/sdf
mdadm: Marking array /dev/md127 as 'clean'
mdadm: added /dev/sdd to /dev/md127 as 1
mdadm: added /dev/sdg to /dev/md127 as 2
mdadm: no uptodate device for slot 3 of /dev/md127
mdadm: added /dev/sde to /dev/md127 as 4
mdadm: added /dev/sdf to /dev/md127 as 5
mdadm: added /dev/sdc to /dev/md127 as 0
mdadm: /dev/md127 has been started with 5 drives (out of 6).

Display More

mdadm --detail /dev/md127 returns

Code

/dev/md127:
           Version : 1.2
     Creation Time : Mon Jun 15 23:49:17 2015
        Raid Level : raid5
        Array Size : 19534272000 (18629.33 GiB 20003.09 GB)
     Used Dev Size : 3906854400 (3725.87 GiB 4000.62 GB)
      Raid Devices : 6
     Total Devices : 5
       Persistence : Superblock is persistent

       Update Time : Tue Nov 14 19:03:04 2023
             State : clean, degraded
    Active Devices : 5
   Working Devices : 5
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : resync

              Name : Athena:0
              UUID : 4211a61c:1dcb3783:12280f01:549d9cde
            Events : 3032476

    Number   Major   Minor   RaidDevice State
       0       8       32        0      active sync   /dev/sdc
       5       8       48        1      active sync   /dev/sdd
       6       8       96        2      active sync   /dev/sdg
       -       0        0        3      removed
       4       8       64        4      active sync   /dev/sde
       7       8       80        5      active sync   /dev/sdf

Display More

And I can now see the array as degraded in OMV's web gui.

I cannot recover the array using sdb as it is, I had to wipe it. It is now recovering, fingers crossed.

If I've missed an important step or there's something else I should check, I would be grateful to get pointed in the right direction ! If not, thanks for the help, I'd have fiddled with my borked system disk for a while before installing on a new one (still investigating whether that first disk is really bad or not).

geaves · Nov 27th 2023

Quote from Oberon

If I've missed an important step or there's something else I should check,

No, that looks fine

Missing file system (raid5) after power loss

ryecoaaron Nov 14th 2023

Participate now!

Similar Threads

RAID 5 missing after power loss

Raid + Filesystem missing after failed drive