Degraded Array in raid1 after shutdown

mat-m · Feb 20th 2022

Hello,

I have this config in my NAS:

OMV 5.6.22 on 32GB USB stick
2 SSD of ~ 230GB in RAID0
One RAID1 array with a 3rd SSD of ~ 470GB and the RAID0 array

There was a power loss, but the UPS kept it cool. I triggered the shutdown through power button, and had the beep to confirm the soft shutdown triggered.

Once started, I received an email with subject "DegradedArray event on /dev/md0:grange"

/dev/md0 is still mounted and readable

It seems I didn't loose data, though I'm using only ~ 40GB.

IIUC, from mdstat:

md127 is missing from md0
sdb is "disabled" from md127

I don't know what to do to restore a clean state.

SMART tests look OK:

blkid

Code

# blkid
/dev/sr0: UUID="2007-09-30-21-03-00-0" LABEL="Photos_2006_2007" TYPE="iso9660"
/dev/sdc: UUID="bb6dc4fa-6340-95e9-e456-8765c5bcf9ab" UUID_SUB="57e1c067-8015-5683-724f-8dc116859fcf" LABEL="grange:Deux230" TYPE="linux_raid_member"
/dev/sdb: UUID="bb6dc4fa-6340-95e9-e456-8765c5bcf9ab" UUID_SUB="52c5b38d-1598-419c-e5d7-84dcdc2e5dd9" LABEL="grange:Deux230" TYPE="linux_raid_member"
/dev/md127: UUID="b55fa23a-352e-6aa8-d591-105992535c4a" UUID_SUB="4df40837-4ae0-4b0d-d7d6-b363bb2554aa" LABEL="grange:0" TYPE="linux_raid_member"
/dev/sda: UUID="b55fa23a-352e-6aa8-d591-105992535c4a" UUID_SUB="a647c19e-4599-f2d1-9c89-695f0addfdd6" LABEL="grange:0" TYPE="linux_raid_member"
/dev/md0: LABEL="data" UUID="b207ff0e-7941-4359-89a4-3415d0928de3" TYPE="ext4"
/dev/sdd1: UUID="5a787299-6e09-4939-9b4a-7765bcd5c689" TYPE="ext4" PARTUUID="a2b1c244-01"
/dev/sdd5: UUID="7eae7069-fbbc-4a85-a085-a530abcdada9" TYPE="swap" PARTUUID="a2b1c244-05"

fdisk -l | grep "Disk "

Code

# fdisk -l | grep "Disk "
Disk /dev/sdc: 238,5 GiB, 256060514304 bytes, 500118192 sectors
Disk model: SAMSUNG SSD PM83
Disk /dev/sdb: 232,9 GiB, 250059350016 bytes, 488397168 sectors
Disk model: CT250MX500SSD1  
Disk /dev/sda: 465,8 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 860 
Disk /dev/md127: 471,1 GiB, 505848791040 bytes, 987985920 sectors
Disk /dev/md0: 465,7 GiB, 499972571136 bytes, 976508928 sectors
Disk /dev/sdd: 29,7 GiB, 31914983424 bytes, 62333952 sectors
Disk model: STORAGE DEVICE  
Disk identifier: 0xa2b1c244

Display More

/proc/mdstat

Code

# cat /proc/mdstat
Personalities : [raid0] [raid1] [linear] [multipath] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda[1]
      488254464 blocks super 1.2 [2/1] [_U]
      bitmap: 2/4 pages [8KB], 65536KB chunk

md127 : active raid0 sdc[1] sdb[0]
      493992960 blocks super 1.2 512k chunks

/etc/mdadm/mdadm.conf

Code

# cat /etc/mdadm/mdadm.conf
# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR ****@*******
MAILFROM root

# definitions of existing MD arrays
ARRAY /dev/md/grange:Deux230 metadata=1.2 name=grange:Deux230 UUID=bb6dc4fa:634095e9:e4568765:c5bcf9ab
ARRAY /dev/md0 metadata=1.2 name=grange:0 UUID=b55fa23a:352e6aa8:d5911059:92535c4a

Display More

mdadm --detail --scan --verbose

Code

# mdadm --detail --scan --verbose
ARRAY /dev/md/grange:Deux230 level=raid0 num-devices=2 metadata=1.2 name=grange:Deux230 UUID=bb6dc4fa:634095e9:e4568765:c5bcf9ab
   devices=/dev/sdb,/dev/sdc
ARRAY /dev/md0 level=raid1 num-devices=2 metadata=1.2 name=grange:0 UUID=b55fa23a:352e6aa8:d5911059:92535c4a
   devices=/dev/sda

geaves · Feb 20th 2022

I'm about to shut down but looking through the output I can't see anything wrong, added to that I've never seen anyone with this setup before I'm going to have to test this in a vm tomorrow.

Quote from mat-m

md127 is missing from md0

It would be if one of the drives in that Raid0 died or went offline the Raid would be toast, so you would have to recreate it then re add it back to the Raid1 (md0), but the output, as I said does not suggest that.

mat-m · Mar 13th 2022

Hello geaves

Quote from geaves

, but the output, as I said does not suggest that.

Did you manage to test it as you wish ?

Going back from holidays., I booted it up, and issue rose again.

Code

# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Jan 14 22:50:27 2021
Raid Level : raid1
Array Size : 488254464 (465.64 GiB 499.97 GB)
Used Dev Size : 488254464 (465.64 GiB 499.97 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent


Intent Bitmap : Internal


Update Time : Sun Mar 13 17:07:33 2022
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0


Consistency Policy : bitmap


Name : grange:0  (local to host grange)
UUID : b55fa23a:352e6aa8:d5911059:92535c4a
Events : 34426


Number   Major   Minor   RaidDevice State
-       0        0        0      removed
1       8        0        1      active sync   /dev/sda

Display More

and examining both volumes lead to:

Code

root@grange:~# mdadm --examine /dev/md127
/dev/md127:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : b55fa23a:352e6aa8:d5911059:92535c4a
Name : grange:0  (local to host grange)
Creation Time : Thu Jan 14 22:50:27 2021
Raid Level : raid1
Raid Devices : 2


Avail Dev Size : 987721728 (470.98 GiB 505.71 GB)
Array Size : 488254464 (465.64 GiB 499.97 GB)
Used Dev Size : 976508928 (465.64 GiB 499.97 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=11212800 sectors
State : clean
Device UUID : 4df40837:4ae04b0d:d7d6b363:bb2554aa


Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb  7 09:09:01 2022
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : 57bfceee - correct
Events : 704




Device Role : Active device 0
Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

root@grange:~# mdadm --examine /dev/sda
/dev/sda:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : b55fa23a:352e6aa8:d5911059:92535c4a
Name : grange:0  (local to host grange)
Creation Time : Thu Jan 14 22:50:27 2021
Raid Level : raid1
Raid Devices : 2


Avail Dev Size : 976508976 (465.64 GiB 499.97 GB)
Array Size : 488254464 (465.64 GiB 499.97 GB)
Used Dev Size : 976508928 (465.64 GiB 499.97 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=48 sectors
State : clean
Device UUID : a647c19e:4599f2d1:9c89695f:0addfdd6


Internal Bitmap : 8 sectors from superblock
Update Time : Sun Mar 13 17:14:26 2022
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : ac01f614 - correct
Events : 34440




Device Role : Active device 1
Array State : .A ('A' == active, '.' == missing, 'R' == replacing)

Display More

md127 does not know it's not active in the raid1.

mdadm --stop and --assemble seems dangerous and overkill to me.
Maybe I can try --re-add ?

Backup is done, ready to test

geaves · Mar 13th 2022

Quote from mat-m

Did you manage to test it as you wish

Didn't get the opportunity, but as I said from your first post the output from each cat /proc/mdstat shows both the arrays as active, but the output from the post above of --detail /md0 shows 2 devices but a total of 1. That means it still cannot locate /md127.

TBH, this is first time I have ever come across this and to create it you would have done this from the cli, the setup is just weird!! Using mergerfs along with a backup would have been a better option.

But to emphasise what I said previously if one single drive within a Raid 0 fails, is disabled, loses it's connection the array is toast, you can't bring it back, the only way forward is to recreate it then re add it to the Raid 1. You can try re add if you want, or assemble if you've backed up your data you've nothing to lose.

mat-m · Mar 14th 2022

That was it.

Code

# mdadm /dev/md0 --add /dev/md127
mdadm: re-added /dev/md127

That looked good

Code

# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Jan 14 22:50:27 2021
Raid Level : raid1
Array Size : 488254464 (465.64 GiB 499.97 GB)
Used Dev Size : 488254464 (465.64 GiB 499.97 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent


Intent Bitmap : Internal


Update Time : Mon Mar 14 23:09:00 2022
State : clean, degraded, recovering
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1


Consistency Policy : bitmap


Rebuild Status : 50% complete


Name : grange:0  (local to host grange)
UUID : b55fa23a:352e6aa8:d5911059:92535c4a
Events : 36636


Number   Major   Minor   RaidDevice State
0       9      127        0      spare rebuilding   /dev/md/grange:Deux230
1       8        0        1      active sync   /dev/sda

Display More

And now State is clean.

Back to business, now

geaves: your insights helped, thank you !

Degraded Array in raid1 after shutdown

mat-m Mar 14th 2022

Participate now!

Tags