RAID Array Disappeared

    • OMV 5.x (beta)
    • RAID Array Disappeared

      Hello,

      I had a RAID 6 array with 6 drives sd[bcdefg]

      I added 3 drives and tried to create a new stripe to temporarily move the data too. When I tried to create a shared folder on the new striped array I got a huge error in the UI and everything took a crap. When I rebooted all raid volumes were gone, including the original good volume.

      The filesystems are "missing"

      For some reason md127 says it's now a raid0 array.

      Examining the disks they are all RAID6 except sdg is RAID5 now? I have no idea why...

      They say there are 5 raid devices, when there should be 6.

      sdg seems to be royally borked and for some reason was used when I tried to create the new stripe.

      root@omv:~# mdadm --examine /dev/sdb
      /dev/sdb:
      Magic : a92b4efc
      Version : 1.2
      Feature Map : 0x1
      Array UUID : 519165ce:33fd48e1:1b948ddf:db836d5a
      Name : omv:volume1
      Creation Time : Sat Aug3 11:00:10 2019
      Raid Level : raid6
      Raid Devices : 5

      Avail Dev Size : 19532611584 (9313.88 GiB 10000.70 GB)
      Array Size : 29298917376 (27941.63 GiB 30002.09 GB)
      Data Offset : 262144 sectors
      Super Offset : 8 sectors
      Unused Space : before=262056 sectors, after=0 sectors
      State : clean
      Device UUID : a2f6bbb2:9ee0f2ac:cd2d287d:ac583392

      Internal Bitmap : 8 sectors from superblock
      Update Time : Fri Oct 18 16:17:54 2019
      Bad Block Log : 512 entries available at offset 72 sectors
      Checksum : 856b67ab - correct
      Events : 47453

      Layout : left-symmetric
      Chunk Size : 512K

      Device Role : Active device 3
      Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
      root@omv:~# mdadm --examine /dev/sdc
      /dev/sdc:
      Magic : a92b4efc
      Version : 1.2
      Feature Map : 0x1
      Array UUID : 519165ce:33fd48e1:1b948ddf:db836d5a
      Name : omv:volume1
      Creation Time : Sat Aug3 11:00:10 2019
      Raid Level : raid6
      Raid Devices : 5

      Avail Dev Size : 19532611584 (9313.88 GiB 10000.70 GB)
      Array Size : 29298917376 (27941.63 GiB 30002.09 GB)
      Data Offset : 262144 sectors
      Super Offset : 8 sectors
      Unused Space : before=262056 sectors, after=0 sectors
      State : clean
      Device UUID : 05301c7f:e0707c69:8d798f65:a1a62730

      Internal Bitmap : 8 sectors from superblock
      Update Time : Fri Oct 18 16:17:54 2019
      Bad Block Log : 512 entries available at offset 72 sectors
      Checksum : 94b0cb05 - correct
      Events : 47453

      Layout : left-symmetric
      Chunk Size : 512K

      Device Role : Active device 4
      Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
      root@omv:~# mdadm --examine /dev/sdd
      /dev/sdd:
      Magic : a92b4efc
      Version : 1.2
      Feature Map : 0x1
      Array UUID : 519165ce:33fd48e1:1b948ddf:db836d5a
      Name : omv:volume1
      Creation Time : Sat Aug3 11:00:10 2019
      Raid Level : raid6
      Raid Devices : 5

      Avail Dev Size : 19532611584 (9313.88 GiB 10000.70 GB)
      Array Size : 29298917376 (27941.63 GiB 30002.09 GB)
      Data Offset : 262144 sectors
      Super Offset : 8 sectors
      Unused Space : before=262056 sectors, after=0 sectors
      State : clean
      Device UUID : b14f7e66:ba66e835:d8082671:c8e7f874

      Internal Bitmap : 8 sectors from superblock
      Update Time : Fri Oct 18 16:17:54 2019
      Bad Block Log : 512 entries available at offset 72 sectors
      Checksum : 98e6b0f9 - correct
      Events : 47453

      Layout : left-symmetric
      Chunk Size : 512K

      Device Role : Active device 0
      Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
      root@omv:~# mdadm --examine /dev/sde
      /dev/sde:
      Magic : a92b4efc
      Version : 1.2
      Feature Map : 0x1
      Array UUID : 519165ce:33fd48e1:1b948ddf:db836d5a
      Name : omv:volume1
      Creation Time : Sat Aug3 11:00:10 2019
      Raid Level : raid6
      Raid Devices : 5

      Avail Dev Size : 19532611584 (9313.88 GiB 10000.70 GB)
      Array Size : 29298917376 (27941.63 GiB 30002.09 GB)
      Data Offset : 262144 sectors
      Super Offset : 8 sectors
      Unused Space : before=262056 sectors, after=0 sectors
      State : clean
      Device UUID : 8cc20a5d:ea1725c0:c7f24137:4805728a

      Internal Bitmap : 8 sectors from superblock
      Update Time : Fri Oct 18 16:17:54 2019
      Bad Block Log : 512 entries available at offset 72 sectors
      Checksum : f544dc74 - correct
      Events : 47453

      Layout : left-symmetric
      Chunk Size : 512K

      Device Role : Active device 1
      Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
      root@omv:~# mdadm --examine /dev/sdf
      /dev/sdf:
      Magic : a92b4efc
      Version : 1.2
      Feature Map : 0x1
      Array UUID : 519165ce:33fd48e1:1b948ddf:db836d5a
      Name : omv:volume1
      Creation Time : Sat Aug3 11:00:10 2019
      Raid Level : raid6
      Raid Devices : 5

      Avail Dev Size : 19532611584 (9313.88 GiB 10000.70 GB)
      Array Size : 29298917376 (27941.63 GiB 30002.09 GB)
      Data Offset : 262144 sectors
      Super Offset : 8 sectors
      Unused Space : before=262056 sectors, after=0 sectors
      State : clean
      Device UUID : 9dc32668:bfb9214e:e8a5b48e:680ff584

      Internal Bitmap : 8 sectors from superblock
      Update Time : Fri Oct 18 16:17:54 2019
      Bad Block Log : 512 entries available at offset 72 sectors
      Checksum : e0533c9c - correct
      Events : 47453

      Layout : left-symmetric
      Chunk Size : 512K

      Device Role : Active device 2
      Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
      root@omv:~# mdadm --examine /dev/sdg
      /dev/sdg:
      Magic : a92b4efc
      Version : 1.2
      Feature Map : 0x1
      Array UUID : beceed91:d117821d:129ca027:896c33f8
      Name : omv:volume1
      Creation Time : Wed Aug1 15:04:05 2018
      Raid Level : raid5
      Raid Devices : 4

      Avail Dev Size : 19532614656 (9313.88 GiB 10000.70 GB)
      Array Size : 29298917376 (27941.63 GiB 30002.09 GB)
      Used Dev Size : 19532611584 (9313.88 GiB 10000.70 GB)
      Data Offset : 259072 sectors
      Super Offset : 8 sectors
      Unused Space : before=258984 sectors, after=3072 sectors
      State : clean
      Device UUID : 82292aed:dec222ca:ea7f0498:f5101e92

      Internal Bitmap : 8 sectors from superblock
      Update Time : Wed Jul 31 15:13:54 2019
      Bad Block Log : 512 entries available at offset 72 sectors
      Checksum : 7f99f59f - correct
      Events : 114145

      Layout : left-symmetric
      Chunk Size : 512K

      Device Role : Active device 3
      Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing).

      Source Code

      1. cat /proc/mdstat
      2. md127 : inactive sdg[3](S)
      3. 9766307328 blocks super 1.2
      4. /dev/md127:
      5. Version : 1.2
      6. Raid Level : raid0
      7. Total Devices : 1
      8. Persistence : Superblock is persistent
      9. State : inactive
      10. Working Devices : 1
      11. Name : omv:volume1
      12. UUID : beceed91:d117821d:129ca027:896c33f8
      13. Events : 114145
      14. Number Major Minor RaidDevice
      15. - 8 96 - /dev/sdg
      16. /dev/sdg:
      17. Magic : a92b4efc
      18. Version : 1.2
      19. Feature Map : 0x1
      20. Array UUID : beceed91:d117821d:129ca027:896c33f8
      21. Name : omv:volume1
      22. Creation Time : Wed Aug 1 15:04:05 2018
      23. Raid Level : raid5
      24. Raid Devices : 4
      25. Avail Dev Size : 19532614656 (9313.88 GiB 10000.70 GB)
      26. Array Size : 29298917376 (27941.63 GiB 30002.09 GB)
      27. Used Dev Size : 19532611584 (9313.88 GiB 10000.70 GB)
      28. Data Offset : 259072 sectors
      29. Super Offset : 8 sectors
      30. Unused Space : before=258984 sectors, after=3072 sectors
      31. State : clean
      32. Device UUID : 82292aed:dec222ca:ea7f0498:f5101e92
      33. Internal Bitmap : 8 sectors from superblock
      34. Update Time : Wed Jul 31 15:13:54 2019
      35. Bad Block Log : 512 entries available at offset 72 sectors
      36. Checksum : 7f99f59f - correct
      37. Events : 114145
      38. Layout : left-symmetric
      39. Chunk Size : 512K
      40. Device Role : Active device 3
      41. Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
      Display All


      Can i safely rebuild this array? I wasn't even touching the original array. :( I'm a little concerned as /dev/md127 says RAID0 now too. I assume to rebuild it right, this should be raid6 with total devices 6 but I don't mind creating md0 with raid6 and 5 devices in a degraded state if it means getting the array back temporarily..


      maybe:

      mdadm --assemble /dev/md0 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf

      then add /dev/sdg after?
      NAS: 1x 120GB SSD (system) - i5 8600 - ASROCK - 9200-16e
      OMV 5.0.10-1
      Containers, Containers, Containers

      The post was edited 7 times, last by seed ().

    • So I was able to get the array back but with only 5 drives. mdadm says it's clean but that isnt true as it should be 6 drives in the array. This was a major blow up and almost catastrophic as I was moving drives from the NAS to the new OMV arrays, where my backups were! oof. :)

      Will save the data to another store and not try to add a new array with new disks on this setup until after it's offloaded to the NAS
      NAS: 1x 120GB SSD (system) - i5 8600 - ASROCK - 9200-16e
      OMV 5.0.10-1
      Containers, Containers, Containers