RAID Array Disappeared

  • Hello,


    I had a RAID 6 array with 6 drives sd[bcdefg]


    I added 3 drives and tried to create a new stripe to temporarily move the data too. When I tried to create a shared folder on the new striped array I got a huge error in the UI and everything took a crap. When I rebooted all raid volumes were gone, including the original good volume.


    The filesystems are "missing"


    For some reason md127 says it's now a raid0 array.


    Examining the disks they are all RAID6 except sdg is RAID5 now? I have no idea why...


    They say there are 5 raid devices, when there should be 6.


    sdg seems to be royally borked and for some reason was used when I tried to create the new stripe.


    root@omv:~# mdadm --examine /dev/sdb
    /dev/sdb:
    Magic : a92b4efc
    Version : 1.2
    Feature Map : 0x1
    Array UUID : 519165ce:33fd48e1:1b948ddf:db836d5a
    Name : omv:volume1
    Creation Time : Sat Aug3 11:00:10 2019
    Raid Level : raid6
    Raid Devices : 5


    Avail Dev Size : 19532611584 (9313.88 GiB 10000.70 GB)
    Array Size : 29298917376 (27941.63 GiB 30002.09 GB)
    Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262056 sectors, after=0 sectors
    State : clean
    Device UUID : a2f6bbb2:9ee0f2ac:cd2d287d:ac583392


    Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Oct 18 16:17:54 2019
    Bad Block Log : 512 entries available at offset 72 sectors
    Checksum : 856b67ab - correct
    Events : 47453


    Layout : left-symmetric
    Chunk Size : 512K


    Device Role : Active device 3
    Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
    root@omv:~# mdadm --examine /dev/sdc
    /dev/sdc:
    Magic : a92b4efc
    Version : 1.2
    Feature Map : 0x1
    Array UUID : 519165ce:33fd48e1:1b948ddf:db836d5a
    Name : omv:volume1
    Creation Time : Sat Aug3 11:00:10 2019
    Raid Level : raid6
    Raid Devices : 5


    Avail Dev Size : 19532611584 (9313.88 GiB 10000.70 GB)
    Array Size : 29298917376 (27941.63 GiB 30002.09 GB)
    Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262056 sectors, after=0 sectors
    State : clean
    Device UUID : 05301c7f:e0707c69:8d798f65:a1a62730


    Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Oct 18 16:17:54 2019
    Bad Block Log : 512 entries available at offset 72 sectors
    Checksum : 94b0cb05 - correct
    Events : 47453


    Layout : left-symmetric
    Chunk Size : 512K


    Device Role : Active device 4
    Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
    root@omv:~# mdadm --examine /dev/sdd
    /dev/sdd:
    Magic : a92b4efc
    Version : 1.2
    Feature Map : 0x1
    Array UUID : 519165ce:33fd48e1:1b948ddf:db836d5a
    Name : omv:volume1
    Creation Time : Sat Aug3 11:00:10 2019
    Raid Level : raid6
    Raid Devices : 5


    Avail Dev Size : 19532611584 (9313.88 GiB 10000.70 GB)
    Array Size : 29298917376 (27941.63 GiB 30002.09 GB)
    Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262056 sectors, after=0 sectors
    State : clean
    Device UUID : b14f7e66:ba66e835:d8082671:c8e7f874


    Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Oct 18 16:17:54 2019
    Bad Block Log : 512 entries available at offset 72 sectors
    Checksum : 98e6b0f9 - correct
    Events : 47453


    Layout : left-symmetric
    Chunk Size : 512K


    Device Role : Active device 0
    Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
    root@omv:~# mdadm --examine /dev/sde
    /dev/sde:
    Magic : a92b4efc
    Version : 1.2
    Feature Map : 0x1
    Array UUID : 519165ce:33fd48e1:1b948ddf:db836d5a
    Name : omv:volume1
    Creation Time : Sat Aug3 11:00:10 2019
    Raid Level : raid6
    Raid Devices : 5


    Avail Dev Size : 19532611584 (9313.88 GiB 10000.70 GB)
    Array Size : 29298917376 (27941.63 GiB 30002.09 GB)
    Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262056 sectors, after=0 sectors
    State : clean
    Device UUID : 8cc20a5d:ea1725c0:c7f24137:4805728a


    Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Oct 18 16:17:54 2019
    Bad Block Log : 512 entries available at offset 72 sectors
    Checksum : f544dc74 - correct
    Events : 47453


    Layout : left-symmetric
    Chunk Size : 512K


    Device Role : Active device 1
    Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
    root@omv:~# mdadm --examine /dev/sdf
    /dev/sdf:
    Magic : a92b4efc
    Version : 1.2
    Feature Map : 0x1
    Array UUID : 519165ce:33fd48e1:1b948ddf:db836d5a
    Name : omv:volume1
    Creation Time : Sat Aug3 11:00:10 2019
    Raid Level : raid6
    Raid Devices : 5


    Avail Dev Size : 19532611584 (9313.88 GiB 10000.70 GB)
    Array Size : 29298917376 (27941.63 GiB 30002.09 GB)
    Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262056 sectors, after=0 sectors
    State : clean
    Device UUID : 9dc32668:bfb9214e:e8a5b48e:680ff584


    Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Oct 18 16:17:54 2019
    Bad Block Log : 512 entries available at offset 72 sectors
    Checksum : e0533c9c - correct
    Events : 47453


    Layout : left-symmetric
    Chunk Size : 512K


    Device Role : Active device 2
    Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
    root@omv:~# mdadm --examine /dev/sdg
    /dev/sdg:
    Magic : a92b4efc
    Version : 1.2
    Feature Map : 0x1
    Array UUID : beceed91:d117821d:129ca027:896c33f8
    Name : omv:volume1
    Creation Time : Wed Aug1 15:04:05 2018
    Raid Level : raid5
    Raid Devices : 4


    Avail Dev Size : 19532614656 (9313.88 GiB 10000.70 GB)
    Array Size : 29298917376 (27941.63 GiB 30002.09 GB)
    Used Dev Size : 19532611584 (9313.88 GiB 10000.70 GB)
    Data Offset : 259072 sectors
    Super Offset : 8 sectors
    Unused Space : before=258984 sectors, after=3072 sectors
    State : clean
    Device UUID : 82292aed:dec222ca:ea7f0498:f5101e92


    Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Jul 31 15:13:54 2019
    Bad Block Log : 512 entries available at offset 72 sectors
    Checksum : 7f99f59f - correct
    Events : 114145


    Layout : left-symmetric
    Chunk Size : 512K


    Device Role : Active device 3
    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing).



    Can i safely rebuild this array? I wasn't even touching the original array. :( I'm a little concerned as /dev/md127 says RAID0 now too. I assume to rebuild it right, this should be raid6 with total devices 6 but I don't mind creating md0 with raid6 and 5 devices in a degraded state if it means getting the array back temporarily..



    maybe:


    mdadm --assemble /dev/md0 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf


    then add /dev/sdg after?

    Proxmox Cluster with OMV 5x on KVM
    OMV 5.0.10-1
    Containers, Containers, Containers

    Edited 7 times, last by seed ().

  • So I was able to get the array back but with only 5 drives. mdadm says it's clean but that isnt true as it should be 6 drives in the array. This was a major blow up and almost catastrophic as I was moving drives from the NAS to the new OMV arrays, where my backups were! oof. :)


    Will save the data to another store and not try to add a new array with new disks on this setup until after it's offloaded to the NAS

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!