RAID6 disappeared, could be assembled back together, left with drive removed and State : clean, degraded, resyncing

preexo · 19. Januar 2022

Hi,

I lost my RAID weirdly. It was suddenly just gone.

I did the following things, output below in the code block

1. logged in and checked the status: inactive

2. tried to run the raid -> mdadm: failed to start array /dev/md/Storage: Input/output error

3. rebooted

4. tried to run the raid -> mdadm: failed to start array /dev/md/Storage: Input/output error

5. described array -> State : active, degraded, Not Started

6. stopped raid

7. assembled with the drives that were visible

8. RAID returned as State : clean, degraded, resyncing

9. Tried to add the remaining drive back in without success

Questions and ask for help and advise, thank you so much in advance for any pointers:

* What happened? Did my drive maybe just die?

* Why is the raid rebuilding, I would expect it just be degraded, but the rebuilding is worrying me

* What do I do next, I have new drives here, but what's the best order to proceed safely to add a new drive back in and bring it back to clean state?

Thank you so much for the support, I have been reading here for many years and always found help without need to start a new thread. Hoping someone can advise on next steps or pointers. Many thanks!

Code

root@msa-nas3:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : inactive sda[6](S) sdg[3](S) sdf[8](S) sdc[7](S) sdd[0](S)
      29301957560 blocks super 1.2
       
unused devices: <none>
root@msa-nas3:~# mdadm -D /dev/md127 
/dev/md127:
        Version : 1.2
     Raid Level : raid0
  Total Devices : 5
    Persistence : Superblock is persistent

          State : inactive

           Name : msa-nas1:Storage
           UUID : 583eb285:b11c7310:f7ed4333:ab40c2f4
         Events : 210539

    Number   Major   Minor   RaidDevice

       -       8       32        -        /dev/sdc
       -       8        0        -        /dev/sda
       -       8       80        -        /dev/sdf
       -       8       48        -        /dev/sdd
       -       8       96        -        /dev/sdg
root@msa-nas3:~# mdadm --run /dev/md127 
mdadm: failed to start array /dev/md/Storage: Input/output error

root@msa-nas3:~# mdadm --run /dev/md127 
mdadm: failed to start array /dev/md/Storage: Input/output error
root@msa-nas3:~# mdadm -D /dev/md127 
/dev/md127:
        Version : 1.2
  Creation Time : Fri Jul 10 17:57:27 2015
     Raid Level : raid6
  Used Dev Size : 18446744073709551615
   Raid Devices : 6
  Total Devices : 5
    Persistence : Superblock is persistent

    Update Time : Wed Jan 19 04:23:31 2022
          State : active, degraded, Not Started 
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : msa-nas1:Storage
           UUID : 583eb285:b11c7310:f7ed4333:ab40c2f4
         Events : 210539

    Number   Major   Minor   RaidDevice State
       0       8       48        0      active sync   /dev/sdd
       -       0        0        1      removed
       8       8       80        2      active sync   /dev/sdf
       3       8       96        3      active sync   /dev/sdg
       6       8        0        4      active sync   /dev/sda
       7       8       32        5      active sync   /dev/sdc
root@msa-nas3:~# mdadm --misc -v --stop /dev/md127 
mdadm: stopped /dev/md127
root@msa-nas3:~# mdadm --assemble --verbose --force /dev/md127 /dev/sd[acdfg]
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda is identified as a member of /dev/md127, slot 4.
mdadm: /dev/sdc is identified as a member of /dev/md127, slot 5.
mdadm: /dev/sdd is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdf is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdg is identified as a member of /dev/md127, slot 3.
mdadm: Marking array /dev/md127 as 'clean'
mdadm: no uptodate device for slot 1 of /dev/md127
mdadm: added /dev/sdf to /dev/md127 as 2
mdadm: added /dev/sdg to /dev/md127 as 3
mdadm: added /dev/sda to /dev/md127 as 4
mdadm: added /dev/sdc to /dev/md127 as 5
mdadm: added /dev/sdd to /dev/md127 as 0
mdadm: /dev/md127 has been started with 5 drives (out of 6).

root@msa-nas3:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : active raid6 sdd[0] sdc[7] sda[6] sdg[3] sdf[8]
      23441563648 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/5] [U_UUUU]
      [>....................]  resync =  1.0% (61922612/5860390912) finish=1071.5min speed=90185K/sec
      
unused devices: <none>
root@msa-nas3:~# mdadm --manage '/dev/md127' --add /dev/sde
mdadm: add new device failed for /dev/sde as 9: Invalid argument

root@msa-nas3:/var/log# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : active raid6 sdd[0] sdc[7] sda[6] sdg[3] sdf[8]
      23441563648 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/5] [U_UUUU]
      [>....................]  resync =  1.7% (101869152/5860390912) finish=1053.2min speed=91123K/sec
      
unused devices: <none>
root@msa-nas3:/var/log# blkid
/dev/sda: UUID="583eb285-b11c-7310-f7ed-4333ab40c2f4" UUID_SUB="0ffe155c-5753-7301-86d8-4cb1ce034d8b" LABEL="msa-nas1:Storage" TYPE="linux_raid_member"
/dev/sdb1: UUID="92acfec9-634b-498c-8ed7-6544e2496ad1" TYPE="ext4" PARTUUID="be691291-01"
/dev/sdb5: UUID="29c57066-b3f1-4139-b40b-82ce56276d48" TYPE="swap" PARTUUID="be691291-05"
/dev/sdc: UUID="583eb285-b11c-7310-f7ed-4333ab40c2f4" UUID_SUB="61b1a63f-bc27-7f3e-ea71-574f28d525d1" LABEL="msa-nas1:Storage" TYPE="linux_raid_member"
/dev/sdd: UUID="583eb285-b11c-7310-f7ed-4333ab40c2f4" UUID_SUB="987e845a-2107-9998-8728-dee68f024d54" LABEL="msa-nas1:Storage" TYPE="linux_raid_member"
/dev/sdf: UUID="583eb285-b11c-7310-f7ed-4333ab40c2f4" UUID_SUB="5732e866-dd1f-92a6-24e5-aea08deb75d0" LABEL="msa-nas1:Storage" TYPE="linux_raid_member"
/dev/sdg: UUID="583eb285-b11c-7310-f7ed-4333ab40c2f4" UUID_SUB="93ac7076-0e90-66f3-b46e-10f4b2dc9fc0" LABEL="msa-nas1:Storage" TYPE="linux_raid_member"
/dev/md127: LABEL="storage" UUID="81873555-ec62-4a18-a6c5-bc112658ff5f" TYPE="ext4"
root@msa-nas3:/var/log# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : active raid6 sdd[0] sdc[7] sda[6] sdg[3] sdf[8]
      23441563648 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/5] [U_UUUU]
      [>....................]  resync =  2.0% (121132856/5860390912) finish=1065.3min speed=89785K/sec
      
unused devices: <none>

root@msa-nas3:/var/log# mdadm --zero-superblock /dev/sde
mdadm: Unrecognised md component device - /dev/sde
root@msa-nas3:/var/log# mdadm -E /dev/sde
mdadm: No md superblock detected on /dev/sde.
root@msa-nas3:/var/log# mdadm --add /dev/md127 /dev/sde
mdadm: add new device failed for /dev/sde as 9: Invalid argument

root@msa-nas3:/var/log# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : active raid6 sdd[0] sdc[7] sda[6] sdg[3] sdf[8]
      23441563648 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/5] [U_UUUU]
      [>....................]  resync =  2.9% (174862408/5860390912) finish=1040.3min speed=91082K/sec
      
unused devices: <none>
root@msa-nas3:/var/log# mdadm -D /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Fri Jul 10 17:57:27 2015
     Raid Level : raid6
     Array Size : 23441563648 (22355.62 GiB 24004.16 GB)
  Used Dev Size : 5860390912 (5588.90 GiB 6001.04 GB)
   Raid Devices : 6
  Total Devices : 5
    Persistence : Superblock is persistent

    Update Time : Wed Jan 19 20:44:19 2022
          State : clean, degraded, resyncing 
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

  Resync Status : 3% complete

           Name : msa-nas1:Storage
           UUID : 583eb285:b11c7310:f7ed4333:ab40c2f4
         Events : 210611

    Number   Major   Minor   RaidDevice State
       0       8       48        0      active sync   /dev/sdd
       -       0        0        1      removed
       8       8       80        2      active sync   /dev/sdf
       3       8       96        3      active sync   /dev/sdg
       6       8        0        4      active sync   /dev/sda
       7       8       32        5      active sync   /dev/sdc

root@msa-nas3:/var/log# mdadm --detail /dev/md127 
/dev/md127:
        Version : 1.2
  Creation Time : Fri Jul 10 17:57:27 2015
     Raid Level : raid6
     Array Size : 23441563648 (22355.62 GiB 24004.16 GB)
  Used Dev Size : 5860390912 (5588.90 GiB 6001.04 GB)
   Raid Devices : 6
  Total Devices : 5
    Persistence : Superblock is persistent

    Update Time : Wed Jan 19 20:45:32 2022
          State : clean, degraded, resyncing 
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

  Resync Status : 3% complete

           Name : msa-nas1:Storage
           UUID : 583eb285:b11c7310:f7ed4333:ab40c2f4
         Events : 210613

    Number   Major   Minor   RaidDevice State
       0       8       48        0      active sync   /dev/sdd
       -       0        0        1      removed
       8       8       80        2      active sync   /dev/sdf
       3       8       96        3      active sync   /dev/sdg
       6       8        0        4      active sync   /dev/sda
       7       8       32        5      active sync   /dev/sdc

Alles anzeigen

geaves · 19. Januar 2022

Wait until the resync has finished and come back, you should do nothing until that's complete

preexo · 19. Januar 2022

Alright, I expected that as well. Currently just waiting, likely for another 14 hours, then I'll return with the state here. Thank you!

edit: I may be asking something very obvious - but the resync was unexpected to me. Why does it resync on a degraded state?

geaves · 19. Januar 2022

Zitat von preexo

but the resync was unexpected to me. Why does it resync on a degraded state

With raid6 you have parity as your first post was somewhat confusing, whilst you appeared to start in the right direction with software raid you have to wait and confirm that each step has completed, basically what it's doing is checking/resyncing your data.

The problem is the more you stress it whilst it's working the worst things can become, you were throwing so many commands at it I'm surprised it didn't bork completely

RAID6 disappeared, could be assembled back together, left with drive removed and State : clean, degraded, resyncing

Jetzt mitmachen!

Tags