How to recover a "clean, degraded" RAID after cables problems of one of the 4 disks in RAID 10?

zerozenit · 7. August 2023

Hi everyone,

after i had some problems with the cables to one of the 4 drives in Raid 10 (unable to open ATA device), i fixed the cable connections and the drive restarted and is visible again (/dev/sde). From the OMV interface the disk is now visible again in Storage -> Disks, and also in S.M.A.R.T., but in Software RAID it is not listed among the devices and the status is "clean, degraded". Do I have to swipe the disk and then recover or is it better to follow another procedure?

Thank you.

Code

cat /proc/mdstat
Personalities : [raid10] [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] 
md0 : active raid10 sdb[1] sda[0] sdd[2]
      23437508608 blocks super 1.2 512K chunks 2 near-copies [4/3] [UUU_]
      bitmap: 79/175 pages [316KB], 65536KB chunk

unused devices: <none>

Code

blkid
/dev/sdb: UUID="8b767a7d-c52c-068d-c04f-1a3cfd8d4c5f" UUID_SUB="a6bb8aa8-4e9b-7f90-b105-45a9301acbce" LABEL="pandora:Raid4x12TBWdRed" TYPE="linux_raid_member"
/dev/sde: UUID="8b767a7d-c52c-068d-c04f-1a3cfd8d4c5f" UUID_SUB="2c68c265-01f7-dd1b-ffff-6d28eb140780" LABEL="pandora:Raid4x12TBWdRed" TYPE="linux_raid_member"
/dev/sdd: UUID="8b767a7d-c52c-068d-c04f-1a3cfd8d4c5f" UUID_SUB="6c9c5433-6838-c39f-abfa-7807205a3238" LABEL="pandora:Raid4x12TBWdRed" TYPE="linux_raid_member"
/dev/sda: UUID="8b767a7d-c52c-068d-c04f-1a3cfd8d4c5f" UUID_SUB="3904f2f1-fe1f-bde3-a965-d9dbe0074f66" LABEL="pandora:Raid4x12TBWdRed" TYPE="linux_raid_member"
/dev/sdc1: UUID="2218-DC43" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="09f69470-ba7b-4b6b-9456-c09f4c6ad2ee"
/dev/sdc2: UUID="87bfca96-9bee-4725-ae79-d8d7893d5a49" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="3c45a8f0-3106-4ba8-89bc-b15d22e81144"
/dev/md0: LABEL="REDRAID4X12" UUID="5fd65f52-b922-45e3-a940-eb7c75460446" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sdc3: PARTUUID="fda4b444-cf82-4ae8-b916-01b8244acee3"

Code

fdisk -l | grep "Disk "
Disk /dev/sdb: 10.91 TiB, 12000138625024 bytes, 23437770752 sectors
Disk model: WDC WD120EFAX-68
Disk /dev/sde: 10.91 TiB, 12000138625024 bytes, 23437770752 sectors
Disk model: WDC WD120EFAX-68
Disk /dev/sdd: 10.91 TiB, 12000138625024 bytes, 23437770752 sectors
Disk model: WDC WD120EFAX-68
Disk /dev/sda: 10.91 TiB, 12000138625024 bytes, 23437770752 sectors
Disk model: WDC WD120EFAX-68
Disk /dev/sdc: 28.64 GiB, 30752636928 bytes, 60063744 sectors
Disk model: Ultra Fit       
Disk identifier: 51328880-3F36-4C4F-A18D-76E5CF56DD7D
Disk /dev/md0: 21.83 TiB, 24000008814592 bytes, 46875017216 sectors

Alles anzeigen

Code

cat /etc/mdadm/mdadm.conf
# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR ...@...com
MAILFROM root

# definitions of existing MD arrays
ARRAY /dev/md0 metadata=1.2 name=pandora:Raid4x12TBWdRed UUID=8b767a7d:c52c068d:c04f1a3c:fd8d4c5f

Alles anzeigen

Code

mdadm --detail --scan --verbose
ARRAY /dev/md0 level=raid10 num-devices=4 metadata=1.2 name=pandora:Raid4x12TBWdRed UUID=8b767a7d:c52c068d:c04f1a3c:fd8d4c5f
   devices=/dev/sda,/dev/sdb,/dev/sdd

Code

           Version : 1.2
     Creation Time : Thu Mar 26 17:58:31 2020
        Raid Level : raid10
        Array Size : 23437508608 (22351.75 GiB 24000.01 GB)
     Used Dev Size : 11718754304 (11175.88 GiB 12000.00 GB)
      Raid Devices : 4
     Total Devices : 3
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Mon Aug  7 12:13:00 2023
             State : clean, degraded
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 0

            Layout : near=2
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : pandora:Raid4x12TBWdRed  (local to host pandora)
              UUID : 8b767a7d:c52c068d:c04f1a3c:fd8d4c5f
            Events : 9110920

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync set-A   /dev/sda
       1       8       16        1      active sync set-B   /dev/sdb
       2       8       48        2      active sync set-A   /dev/sdd
       -       0        0        3      removed

Alles anzeigen

geaves · 7. August 2023

Zitat von zerozenit

Do I have to swipe the disk and then recover

That would be the correct procedure

zerozenit · 9. August 2023

Zitat von geaves

That would be the correct procedure

I performed the procedure and now the RAID is clean again and everything works. But I get this mail from mdadm:

Code

This is an automatically generated mail message from mdadm
running on pandora

A SparesMissing event had been detected on md device /dev/md0.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid10] [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] 
md0 : active raid10 sde[4] sdb[1] sda[0] sdd[2]
     23437508608 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
     bitmap: 2/175 pages [8KB], 65536KB chunk

unused devices: <none>

Alles anzeigen

It is not clear if everything is ok and how to stop receiving these messages. Thank you.

geaves · 9. August 2023

Zitat von zerozenit

It is not clear if everything is ok

Everything is fine, but the email is odd; post the output of cat /etc/mdadm/mdadm.conf sometimes a reboot can help, it could be that mdadm has detected a missing drive but hasn't caught up with the change.

zerozenit · 9. August 2023

Zitat von geaves

Everything is fine, but the email is odd; post the output of cat /etc/mdadm/mdadm.conf sometimes a reboot can help, it could be that mdadm has detected a missing drive but hasn't caught up with the change.

Here below:

Code

cat /etc/mdadm/mdadm.conf
# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR xxx@xxx.com
MAILFROM root

# definitions of existing MD arrays
ARRAY /dev/md0 metadata=1.2 spares=1 name=pandora:Raid4x12TBWdRed UUID=8b767a7d:c52c068d:c04f1a3c:fd8d4c5f

Alles anzeigen

Code

mdadm --detail --verbose /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Thu Mar 26 17:58:31 2020
        Raid Level : raid10
        Array Size : 23437508608 (22351.75 GiB 24000.01 GB)
     Used Dev Size : 11718754304 (11175.88 GiB 12000.00 GB)
      Raid Devices : 4
     Total Devices : 4
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Wed Aug  9 19:43:38 2023
             State : clean 
    Active Devices : 4
   Working Devices : 4
    Failed Devices : 0
     Spare Devices : 0

            Layout : near=2
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : pandora:Raid4x12TBWdRed  (local to host pandora)
              UUID : 8b767a7d:c52c068d:c04f1a3c:fd8d4c5f
            Events : 9111434

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync set-A   /dev/sda
       1       8       16        1      active sync set-B   /dev/sdb
       2       8       48        2      active sync set-A   /dev/sdd
       4       8       64        3      active sync set-B   /dev/sde

Alles anzeigen

geaves · 9. August 2023

Zitat von zerozenit

Here below:

If you look at the mdadm.conf file it has spares=1 that's why you're getting the email, best way to solve, edit the file and remove spares=1, save the file.

Seen this before on here and this was the solution.

How to recover a "clean, degraded" RAID after cables problems of one of the 4 disks in RAID 10?

Jetzt mitmachen!

Tags