Lost RAID6

tibodreep · Jan 14th 2023

Hi everyone and excuse for my poor english, i will try my best.

I loose my RAID6 in OMV. I don't know how and when, sorry.

It seems that the sde disk is missing in the cluster, which is well recognized under omv, but the raid is not available...

As request by moderator :

Code

root@NAS-Clauce:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : inactive sdb[5](S) sdg[4](S) sdf[3](S) sdc[0](S) sdd[1](S)
      9766912440 blocks super 1.2

unused devices: <none>
root@NAS-Clauce:~# blkid
/dev/sdb: UUID="e2c2cb4c-3016-aaf3-85eb-945434697d09" UUID_SUB="30619ac9-18c1-234f-bab7-20a703821dfd" LABEL="NAS-Clauce.local:nAsRaid6" TYPE="linux_raid_member"
/dev/sda1: UUID="c9d17701-fd97-441d-8cee-dd016bbf61d6" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="cbe3b062-01"
/dev/sda5: UUID="3d257171-4b7a-4340-93bb-984a6dff68b0" TYPE="swap" PARTUUID="cbe3b062-05"
/dev/sdc: UUID="e2c2cb4c-3016-aaf3-85eb-945434697d09" UUID_SUB="9a8b8a48-3f16-a7ea-3ba4-ce98d62c82c2" LABEL="NAS-Clauce.local:nAsRaid6" TYPE="linux_raid_member"
/dev/sdd: UUID="e2c2cb4c-3016-aaf3-85eb-945434697d09" UUID_SUB="0a8d8147-3bf0-390f-40a0-616322cab317" LABEL="NAS-Clauce.local:nAsRaid6" TYPE="linux_raid_member"
/dev/sdf: UUID="e2c2cb4c-3016-aaf3-85eb-945434697d09" UUID_SUB="5a9a2cbe-f33d-9c52-b4b2-b1e009620b7f" LABEL="NAS-Clauce.local:nAsRaid6" TYPE="linux_raid_member"
/dev/sdg: UUID="e2c2cb4c-3016-aaf3-85eb-945434697d09" UUID_SUB="f928ff46-faf9-37ea-9651-7aa2f85740bf" LABEL="NAS-Clauce.local:nAsRaid6" TYPE="linux_raid_member"

root@NAS-Clauce:~# cat /etc/mdadm/mdadm.conf
# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR fournier.thibaut@gmail.com
MAILFROM root

# definitions of existing MD arrays
ARRAY /dev/md/NAS-Clauce.local:nAsRaid6 metadata=1.2 name=NAS-Clauce.local:nAsRaid6 UUID=e2c2cb4c:3016aaf3:85eb9454:34697d09

root@NAS-Clauce:~# mdadm --detail --scan --verbose
INACTIVE-ARRAY /dev/md127 num-devices=5 metadata=1.2 name=NAS-Clauce.local:nAsRaid6 UUID=e2c2cb4c:3016aaf3:85eb9454:34697d09
   devices=/dev/sdb,/dev/sdc,/dev/sdd,/dev/sdf,/dev/sdg

Display More

Someone can help me ?

Maybe the problem has been dealt with in another topic but it's impossible for me to know if it's exactly the same problem.

Thx a lot in advance, i'm completely lost !!!

tibodreep · Jan 14th 2023

I can provide you with the following additional information :

Code

root@NAS-Clauce:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : inactive sdf[3](S) sdd[1](S) sdg[4](S) sdc[0](S) sdb[5](S)      9766912440 blocks super 1.2
unused devices: <none>
root@NAS-Clauce:~# fdisk -l
Disk /dev/sda: 111,79 GiB, 120034123776 bytes, 234441648 sectors
Disk model: CT120BX500SSD1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xcbe3b062
Device     Boot     Start       End   Sectors   Size Id Type
/dev/sda1  *         2048 232441855 232439808 110,8G 83 Linux
/dev/sda2       232443902 234440703   1996802   975M  5 Extended
/dev/sda5       232443904 234440703   1996800   975M 82 Linux swap / Solaris
Disk /dev/sdg: 1,82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: ST2000DL003-9VT1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdc: 1,82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: ST2000DL003-9VT1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sdf: 1,82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WDC WD20PURX-64P
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdb: 1,82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WDC WD20EFRX-68E
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdd: 1,82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WDC WD20EFRX-68E
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sde: 1,82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WDC WD20PURX-64P
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
root@NAS-Clauce:~# lsblk -o size,name,fstype,label,mountpoint  SIZE NAME   FSTYPE            LABEL                     MOUNTPOINT
111,8G sda
110,8G ├─sda1 ext4                                        /    1K ├─sda2  975M └─sda5 swap                                        [SWAP]  1,8T sdb    linux_raid_member NAS-Clauce.local:nAsRaid6  1,8T sdc    linux_raid_member NAS-Clauce.local:nAsRaid6  1,8T sdd    linux_raid_member NAS-Clauce.local:nAsRaid6  1,8T sde  1,8T sdf    linux_raid_member NAS-Clauce.local:nAsRaid6  1,8T sdg    linux_raid_member NAS-Clauce.local:nAsRaid6
root@NAS-Clauce:~# mdadm --detail /dev/md127
/dev/md127:           Version : 1.2        Raid Level : raid0     Total Devices : 5       Persistence : Superblock is persistent             State : inactive   Working Devices : 5              Name : NAS-Clauce.local:nAsRaid6              UUID : e2c2cb4c:3016aaf3:85eb9454:34697d09            Events : 76091    Number   Major   Minor   RaidDevice       -       8       32        -        /dev/sdc       -       8       80        -        /dev/sdf       -       8       48        -        /dev/sdd       -       8       16        -        /dev/sdb       -       8       96        -        /dev/sdg
root@NAS-Clauce:~# mdadm --examine --scan
ARRAY /dev/md/nAsRaid6  metadata=1.2 UUID=e2c2cb4c:3016aaf3:85eb9454:34697d09 name=NAS-Clauce.local:nAsRaid6
root@NAS-Clauce:~# mdadm --examine /dev/sdb
mdadm --examine /dev/sdc
mdadm --examine /dev/sdd
mdadm --examine /dev/sde
mdadm --examine /dev/sdf
mdadm --examine /dev/sdg
/dev/sdb:          Magic : a92b4efc        Version : 1.2    Feature Map : 0x1     Array UUID : e2c2cb4c:3016aaf3:85eb9454:34697d09           Name : NAS-Clauce.local:nAsRaid6  Creation Time : Thu Sep 10 00:54:20 2020     Raid Level : raid6   Raid Devices : 6
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)     Array Size : 7813529600 (7451.56 GiB 8001.05 GB)  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)    Data Offset : 264192 sectors   Super Offset : 8 sectors   Unused Space : before=264112 sectors, after=176 sectors          State : clean    Device UUID : 30619ac9:18c1234f:bab720a7:03821dfd
Internal Bitmap : 8 sectors from superblock    Update Time : Wed Nov 23 18:00:46 2022  Bad Block Log : 512 entries available at offset 16 sectors       Checksum : ec46047a - correct         Events : 76091         Layout : left-symmetric     Chunk Size : 512K   Device Role : Active device 5   Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:          Magic : a92b4efc        Version : 1.2    Feature Map : 0x1     Array UUID : e2c2cb4c:3016aaf3:85eb9454:34697d09           Name : NAS-Clauce.local:nAsRaid6  Creation Time : Thu Sep 10 00:54:20 2020     Raid Level : raid6   Raid Devices : 6
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)     Array Size : 7813529600 (7451.56 GiB 8001.05 GB)  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)    Data Offset : 264192 sectors   Super Offset : 8 sectors   Unused Space : before=264112 sectors, after=176 sectors          State : clean    Device UUID : 9a8b8a48:3f16a7ea:3ba4ce98:d62c82c2
Internal Bitmap : 8 sectors from superblock    Update Time : Wed Nov 23 18:00:46 2022  Bad Block Log : 512 entries available at offset 16 sectors       Checksum : bdcc1b5a - correct         Events : 76091         Layout : left-symmetric     Chunk Size : 512K   Device Role : Active device 0   Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:          Magic : a92b4efc        Version : 1.2    Feature Map : 0x1     Array UUID : e2c2cb4c:3016aaf3:85eb9454:34697d09           Name : NAS-Clauce.local:nAsRaid6  Creation Time : Thu Sep 10 00:54:20 2020     Raid Level : raid6   Raid Devices : 6
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)     Array Size : 7813529600 (7451.56 GiB 8001.05 GB)  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)    Data Offset : 264192 sectors   Super Offset : 8 sectors   Unused Space : before=264112 sectors, after=176 sectors          State : clean    Device UUID : 0a8d8147:3bf0390f:40a06163:22cab317
Internal Bitmap : 8 sectors from superblock    Update Time : Wed Nov 23 18:00:46 2022  Bad Block Log : 512 entries available at offset 16 sectors       Checksum : 11a9017 - correct         Events : 76091         Layout : left-symmetric     Chunk Size : 512K   Device Role : Active device 1   Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
mdadm: No md superblock detected on /dev/sde.
/dev/sdf:          Magic : a92b4efc        Version : 1.2    Feature Map : 0x1     Array UUID : e2c2cb4c:3016aaf3:85eb9454:34697d09           Name : NAS-Clauce.local:nAsRaid6  Creation Time : Thu Sep 10 00:54:20 2020     Raid Level : raid6   Raid Devices : 6
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)     Array Size : 7813529600 (7451.56 GiB 8001.05 GB)  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)    Data Offset : 264192 sectors   Super Offset : 8 sectors   Unused Space : before=264112 sectors, after=176 sectors          State : clean    Device UUID : 5a9a2cbe:f33d9c52:b4b2b1e0:09620b7f
Internal Bitmap : 8 sectors from superblock    Update Time : Wed Nov 23 18:00:46 2022  Bad Block Log : 512 entries available at offset 16 sectors       Checksum : 9fcf957d - correct         Events : 76091         Layout : left-symmetric     Chunk Size : 512K   Device Role : Active device 3   Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg:          Magic : a92b4efc        Version : 1.2    Feature Map : 0x1     Array UUID : e2c2cb4c:3016aaf3:85eb9454:34697d09           Name : NAS-Clauce.local:nAsRaid6  Creation Time : Thu Sep 10 00:54:20 2020     Raid Level : raid6   Raid Devices : 6
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)     Array Size : 7813529600 (7451.56 GiB 8001.05 GB)  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)    Data Offset : 264192 sectors   Super Offset : 8 sectors   Unused Space : before=264112 sectors, after=176 sectors          State : clean    Device UUID : f928ff46:faf937ea:96517aa2:f85740bf
Internal Bitmap : 8 sectors from superblock    Update Time : Wed Nov 23 18:00:46 2022  Bad Block Log : 512 entries available at offset 16 sectors       Checksum : c23b74f5 - correct         Events : 76091         Layout : left-symmetric     Chunk Size : 512K   Device Role : Active device 4   Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)

Display More

geaves · Jan 14th 2023

The array is inactive, whilst /dev/sde is displayed in fdisk it is not displayed in blkid post the output of the following in separate code boxes please;

mdadm --detail /dev/md127

mdadm --examine /dev/sde

what actually happened for the array to 'disappear'

tibodreep · Jan 14th 2023

Thank you very much to take a look on my problem !!!

Code

root@NAS:~# mdadm --detail /dev/md127
/dev/md127:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 5
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 5

              Name : NAS-Clauce.local:nAsRaid6
              UUID : e2c2cb4c:3016aaf3:85eb9454:34697d09
            Events : 76091

    Number   Major   Minor   RaidDevice

       -       8       32        -        /dev/sdc
       -       8       80        -        /dev/sdf
       -       8       48        -        /dev/sdd
       -       8       16        -        /dev/sdb
       -       8       96        -        /dev/sdg

Display More

the sde disk is missing

Code

root@NAS:~# mdadm --examine /dev/sde
mdadm: No md superblock detected on /dev/sde.

The smart info for sde disk is bad. I think there is a problem with that.

I would like to restore my RAID and after change this disk to a better.

But now, i don't know how to not lost everything...

geaves · Jan 14th 2023

To get the array back up;

mdadm --stop /dev/md127

mdadm --assemble --force --verbose /dev/md127 /dev/sd[bdcfg]

tibodreep · Jan 14th 2023

I know it's not a single backup.

I had mostly my video library on it.

If I lose it, it's the game...

Code

root@NAS:~# mdadm --stop /dev/md127
mdadm: stopped /dev/md127
root@NAS:~# mdadm --assemble --force --verbose /dev/md127 /dev/sd[bdcfg]
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdb is identified as a member of /dev/md127, slot 5.
mdadm: /dev/sdc is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdd is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdf is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdg is identified as a member of /dev/md127, slot 4.
mdadm: added /dev/sdd to /dev/md127 as 1
mdadm: no uptodate device for slot 2 of /dev/md127
mdadm: added /dev/sdf to /dev/md127 as 3
mdadm: added /dev/sdg to /dev/md127 as 4
mdadm: added /dev/sdb to /dev/md127 as 5
mdadm: added /dev/sdc to /dev/md127 as 0
mdadm: /dev/md127 has been started with 5 drives (out of 6).

Display More

geaves · Jan 14th 2023

What's the output of;

cat /proc/mdstat

tibodreep · Jan 14th 2023

tibodreep · Jan 14th 2023

Code

root@NAS:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : active (auto-read-only) raid6 sdc[0] sdb[5] sdg[4] sdf[3] sdd[1]
      7813529600 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/5] [UU_UUU]
      bitmap: 0/15 pages [0KB], 65536KB chunk

unused devices: <none>

sde is in OMV disk management

geaves · Jan 14th 2023

The array is in auto-read-only do the following;

mdadm --readwrite /dev/md127

Quote from tibodreep

sde is in OMV disk management

It will be that section lists all devices omv locates that are connected to the system, so a drive can have SMART errors and still be displayed there

tibodreep · Jan 14th 2023

Quote from geaves

The array is in auto-read-only do the following;

mdadm --readwrite /dev/md127
It will be that section lists all devices omv locates that are connected to the system, so a drive can have SMART errors and still be displayed there

Done for readwrite.
Ok for the other information. So i will replace the bad hdd, say me when... i have one in stock

tibodreep · Jan 14th 2023

Code

root@NAS:~# mdadm --readwrite /dev/md127
root@NAS:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : active raid6 sdc[0] sdb[5] sdg[4] sdf[3] sdd[1]
      7813529600 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/5] [UU_UUU]
      bitmap: 0/15 pages [0KB], 65536KB chunk

unused devices: <none>

geaves · Jan 14th 2023

Quote from tibodreep

So i will replace the bad hdd

Wipe the drive before you add it to the array

tibodreep · Jan 14th 2023

Quote from geaves

Wipe the drive before you add it to the array

wipe in progress. 4 hours to the end.

next step ?

geaves · Jan 14th 2023

Quote from tibodreep

next step

Raid Management -> Recover (I think recover is the icon that looks like a bag with a cross in it) then on the next screen select the new drive and click save

TBH, I haven't tested much config in OMV6 on my OMV6 VM test rig to familiarise myself with the changes to the Raid set up

tibodreep · Jan 14th 2023

ok, so :

1. wipe

2. replace the bad disk

3. recover raid

recover raid by web interface or in command line ?

geaves · Jan 14th 2023

Quote from tibodreep

recover raid by web interface

Easier by the OMV's GUI than cli

tibodreep · Jan 14th 2023

ok, when it will be done, i come back to give the results !

very, very thanks you !!!

tibodreep · Jan 15th 2023

Hi,

Wipe done but can't mount New HDD and not ask to add in RAID...

Another help plz ?

geaves · Jan 15th 2023

Quote from tibodreep

but can't mount New HDD

You don't mount it, you just add it to the array using recover in raid management

Is the drive new or repurposed?

Lost RAID6

Participate now!

Similar Threads

Raid 5 Array missing after power cut

RAID6 not showing up in GUI anymore, rebuilding tutorials fail

missing RAID 5

RAID is dead ...

deleted raid6, try to reconstruct it

RAID6 disappeared, could be assembled back together, left with drive removed and State : clean, degraded, resyncing