Raid 5 array missing after failed rebuild

geaves · 20. März 2020

Zitat von zachlow

sde is showing SMART errors.

As this is V2 if it shows any bad sectors i.e. in the GUI there's a red dot against that drive then that drive needs replacing.

So with that information follow what I'm suggesting and hopefully the array will come back up

zachlow · 20. März 2020

I pulled the SATA cable for sde and turned the server back on. Now the array isn't showing.

geaves · 20. März 2020

Zitat von zachlow

I pulled the SATA cable for sde and turned the server back on. Now the array isn't showing.

Ok run some of the commands cat /proc/mdstat blkid cat /etc/mdadm/mdadm.conf

zachlow · 20. März 2020

root@NAS:~# cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4]

md0 : inactive sdb[1] sdf[7](S) sdd[5] sde[6] sdc[2]

14650677560 blocks super 1.2

unused devices: <none>

root@NAS:~# blkid

/dev/sda1: UUID="f7394bdd-cf50-47f0-99c1-58780a3d5c86" TYPE="ext4"

/dev/sda5: UUID="82b05653-a32e-44e3-a814-917fbcae3d43" TYPE="swap"

/dev/sdc: UUID="9a74f8dd-30a9-5450-999e-44c3e36af552" UUID_SUB="aba0a944-bfd2-23ab-43cb-4863269e6b20" LABEL="NAS:NASvol1" TYPE="linux_raid_member"

/dev/sdd: UUID="9a74f8dd-30a9-5450-999e-44c3e36af552" UUID_SUB="6966fbb8-8745-f6ad-f456-218fb598202a" LABEL="NAS:NASvol1" TYPE="linux_raid_member"

/dev/sdf: UUID="9a74f8dd-30a9-5450-999e-44c3e36af552" UUID_SUB="3780c559-3151-073e-86ea-bd4f0df7d245" LABEL="NAS:NASvol1" TYPE="linux_raid_member"

/dev/sdb: UUID="9a74f8dd-30a9-5450-999e-44c3e36af552" UUID_SUB="22007f01-db25-482f-048a-9ecb92b0b230" LABEL="NAS:NASvol1" TYPE="linux_raid_member"

/dev/sde: UUID="9a74f8dd-30a9-5450-999e-44c3e36af552" UUID_SUB="b0c703e2-312e-2851-1c94-2bd4b04fd988" LABEL="NAS:NASvol1" TYPE="linux_raid_member"

root@NAS:~# cat /etc/mdadm/mdadm.conf

# mdadm.conf

#

# Please refer to mdadm.conf(5) for information about this file.

#

# by default, scan all partitions (/proc/partitions) for MD superblocks.

# alternatively, specify devices to scan, using wildcards if desired.

# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.

# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is

# used if no RAID devices are configured.

DEVICE partitions

# auto-create devices with Debian standard permissions

CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system

HOMEHOST <system>

# definitions of existing MD arrays

ARRAY /dev/md0 metadata=1.2 spares=1 name=NAS:NASvol1 UUID=9a74f8dd:30a95450:999e44c3:e36af552

# instruct the monitoring daemon where to send mail alerts

MAILADDR zachlow77@gmail.com

MAILFROM root

geaves · 20. März 2020

That should now stop as it's showing inactive,

mdadm --stop /dev/md0 mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcdef]

zachlow · 20. März 2020

root@NAS:~# mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcdef]

mdadm: looking for devices for /dev/md0

mdadm: /dev/sdb is identified as a member of /dev/md0, slot 1.

mdadm: /dev/sdc is identified as a member of /dev/md0, slot 2.

mdadm: /dev/sdd is identified as a member of /dev/md0, slot 4.

mdadm: /dev/sde is identified as a member of /dev/md0, slot 3.

mdadm: /dev/sdf is identified as a member of /dev/md0, slot -1.

mdadm: no uptodate device for slot 0 of /dev/md0

mdadm: added /dev/sdc to /dev/md0 as 2

mdadm: added /dev/sde to /dev/md0 as 3

mdadm: added /dev/sdd to /dev/md0 as 4

mdadm: no uptodate device for slot 5 of /dev/md0

mdadm: added /dev/sdf to /dev/md0 as -1

mdadm: added /dev/sdb to /dev/md0 as 1

mdadm: /dev/md0 assembled from 4 drives and 1 spare - not enough to start the array.

root@NAS:~#

geaves · 20. März 2020

Back to where we were before without shutting down can you plug that other drive back in and run blkid

zachlow · 20. März 2020

root@NAS:~# blkid

/dev/sda1: UUID="f7394bdd-cf50-47f0-99c1-58780a3d5c86" TYPE="ext4"

/dev/sda5: UUID="82b05653-a32e-44e3-a814-917fbcae3d43" TYPE="swap"

/dev/sdc: UUID="9a74f8dd-30a9-5450-999e-44c3e36af552" UUID_SUB="aba0a944-bfd2-23ab-43cb-4863269e6b20" LABEL="NAS:NASvol1" TYPE="linux_raid_member"

/dev/sdd: UUID="9a74f8dd-30a9-5450-999e-44c3e36af552" UUID_SUB="6966fbb8-8745-f6ad-f456-218fb598202a" LABEL="NAS:NASvol1" TYPE="linux_raid_member"

/dev/sdf: UUID="9a74f8dd-30a9-5450-999e-44c3e36af552" UUID_SUB="3780c559-3151-073e-86ea-bd4f0df7d245" LABEL="NAS:NASvol1" TYPE="linux_raid_member"

/dev/sdb: UUID="9a74f8dd-30a9-5450-999e-44c3e36af552" UUID_SUB="22007f01-db25-482f-048a-9ecb92b0b230" LABEL="NAS:NASvol1" TYPE="linux_raid_member"

/dev/sde: UUID="9a74f8dd-30a9-5450-999e-44c3e36af552" UUID_SUB="b0c703e2-312e-2851-1c94-2bd4b04fd988" LABEL="NAS:NASvol1" TYPE="linux_raid_member"

/dev/sdg: UUID="9a74f8dd-30a9-5450-999e-44c3e36af552" UUID_SUB="954f5e15-55db-8cea-c7fe-bd99df8e2957" LABEL="NAS:NASvol1" TYPE="linux_raid_member"

geaves · 20. März 2020

That sees the six drives, but I'm now guessing that the array is active cat /proc/mdstat

zachlow · 20. März 2020

root@NAS:~# cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4]

md0 : inactive sdb[1](S) sdf[7](S) sdd[5](S) sde[6](S) sdc[2](S)

14650677560 blocks super 1.2

unused devices: <none>

geaves · 20. März 2020

Interesting ok, assuming that sdf in slot -1 is the culprit, lets try this,

mdadm --stop /dev/md0

mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcdeg]

zachlow · 20. März 2020

root@NAS:~# mdadm --assemble --force --verbose /dev/md0 /dev/sd[bcdeg]

mdadm: looking for devices for /dev/md0

mdadm: /dev/sdb is identified as a member of /dev/md0, slot 1.

mdadm: /dev/sdc is identified as a member of /dev/md0, slot 2.

mdadm: /dev/sdd is identified as a member of /dev/md0, slot 4.

mdadm: /dev/sde is identified as a member of /dev/md0, slot 3.

mdadm: /dev/sdg is identified as a member of /dev/md0, slot 5.

mdadm: forcing event count in /dev/sdg(5) from 30697 upto 32014

mdadm: clearing FAULTY flag for device 4 in /dev/md0 for /dev/sdg

mdadm: Marking array /dev/md0 as 'clean'

mdadm: no uptodate device for slot 0 of /dev/md0

mdadm: added /dev/sdc to /dev/md0 as 2

mdadm: added /dev/sde to /dev/md0 as 3

mdadm: added /dev/sdd to /dev/md0 as 4

mdadm: added /dev/sdg to /dev/md0 as 5

mdadm: added /dev/sdb to /dev/md0 as 1

mdadm: /dev/md0 assembled from 5 drives - not enough to start the array.

geaves · 20. März 2020

At this present moment I don't know, either sdf or sdg or both are faulty and will not allow the array to start, and it doesn't like the fact there is no device in slot 0.

From the above something is missing on one of those drives or it's related to the m'board connections, the madam conf is wrong but that can be corrected.

zachlow · 20. März 2020

If we can't get it to work I'm alright with that. I've kinda accepted the fact that my data is gone at this point.

geaves · 20. März 2020

Zitat von zachlow

If we can't get it to work I'm alright with that. I've kinda accepted the fact that my data is gone at this point.

There is one option this would be sh!t or bust

zachlow · 20. März 2020

Im game.

geaves · 20. März 2020

Ok make sure the array is stopped mdadm --stop /dev/md0

mdadm --create --assume-clean --level=raid5 --raid-devices=6 /dev/md0 /dev/sd[bcdefg]

zachlow · 20. März 2020

root@NAS:~# mdadm --create --assume-clean --level=raid5 --raid-devices=6 /dev/md0 /dev/sd[bcdefg]

mdadm: /dev/sdb appears to be part of a raid array: