missing RAID 5

arkansis · 20. September 2022

Hello !

I'm panicked, I just got an email alert from my nas OMV 6.

I no longer see my RAID 5 and data.

The email message says:

Code

This is an automatically generated mail message from mdadm

running on NASOMV

A Fail event had been detected on md device /dev/md0.

It could be related to component device /dev/sdd.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]

md0 : active raid5 sdb[1](F) sda[0](F) sdc[2](F)

29298914304 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/0] [____]

bitmap: 0/73 pages [0KB], 65536KB chunk

unused devices: <none>

Alles anzeigen

Code

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices: <none>

Code

# blkid
/dev/nvme0n1p1: LABEL="512Go" UUID="519ef04c-2f30-4eb3-8925-68ec3fe13e34" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="86b061fb-c9da-47cb-871b-86c8385c13bb"
/dev/nvme1n1p1: UUID="e714394a-0aea-4eff-84d7-d709ef085358" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="21023d25-01"
/dev/nvme1n1p5: UUID="5fc35304-0437-4676-894d-3b54e8907af0" TYPE="swap" PARTUUID="21023d25-05"

Code

# fdisk -l | grep "Disk "
Disk /dev/nvme0n1: 476,94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: NE-512
Disk identifier: E13E89A7-826A-4E01-BDE3-582E078A123A
Disk /dev/nvme1n1: 119,24 GiB, 128035676160 bytes, 250069680 sectors
Disk model: NE-128
Disk identifier: 0x21023d25

Code

# cat /etc/mdadm/mdadm.conf
# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR gxxxxxxxxx@xxxxxxx.com
MAILFROM root

# definitions of existing MD arrays
ARRAY /dev/md0 metadata=1.2 name=NASOMV:RAID5 UUID=d9dad86e:51cbe65e:5b4d88af:fa593c45

Alles anzeigen

SMART Logs before failure:

Code

20/09/2022 11:40:25
Device: /dev/disk/by-id/nvme-NE-512_9800619606032, state written to /var/lib/smartmontools/smartd.NE_512-9800619606032-n1.nvme.state
20/09/2022 11:40:25
Device: /dev/disk/by-id/nvme-NE-128_9800624803340, state written to /var/lib/smartmontools/smartd.NE_128-9800624803340-n1.nvme.state
20/09/2022 11:40:24
smartd 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-18-amd64] (local build)
20/09/2022 11:40:24
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
20/09/2022 11:40:24
Opened configuration file /etc/smartd.conf
20/09/2022 11:40:24
Configuration file /etc/smartd.conf parsed.
20/09/2022 11:40:24
Device: /dev/disk/by-id/nvme-NE-128_9800624803340, opened
20/09/2022 11:40:24
Device: /dev/disk/by-id/nvme-NE-128_9800624803340, NE-128, S/N:9800624803340, FW:S1111H0L, NSID:1, 128 GB
20/09/2022 11:40:24
Device: /dev/disk/by-id/nvme-NE-128_9800624803340, is SMART capable. Adding to "monitor" list.
20/09/2022 11:40:24
Device: /dev/disk/by-id/nvme-NE-128_9800624803340, state read from /var/lib/smartmontools/smartd.NE_128-9800624803340-n1.nvme.state
20/09/2022 11:40:24
Device: /dev/disk/by-id/nvme-NE-512_9800619606032, opened
20/09/2022 11:40:24
Device: /dev/disk/by-id/nvme-NE-512_9800619606032, is SMART capable. Adding to "monitor" list.
20/09/2022 11:40:24
Device: /dev/disk/by-id/nvme-NE-512_9800619606032, NE-512, S/N:9800619606032, FW:S1111A0L, NSID:1, 512 GB
20/09/2022 11:40:24
Device: /dev/disk/by-id/ata-WDC_WD101EFAX-68LDBN0_VCG7U31N, unable to autodetect device type
20/09/2022 11:40:24
Device: /dev/disk/by-id/ata-WDC_WD101EFAX-68LDBN0_VCG7U31N, not available
20/09/2022 11:40:24
Device: /dev/disk/by-id/ata-WDC_WD101EFAX-68LDBN0_VCG7VADN, unable to autodetect device type
20/09/2022 11:40:24
Device: /dev/disk/by-id/ata-WDC_WD101EFAX-68LDBN0_VCG7VADN, not available
20/09/2022 11:40:24
Device: /dev/disk/by-id/ata-WDC_WD101EFAX-68LDBN0_VCG7W4EN, unable to autodetect device type
20/09/2022 11:40:24
Device: /dev/disk/by-id/ata-WDC_WD101EFAX-68LDBN0_VCG7W4EN, not available
20/09/2022 11:40:24
Device: /dev/disk/by-id/ata-WDC_WD101EFAX-68LDBN0_VCG7W6LN, unable to autodetect device type
20/09/2022 11:40:24
Device: /dev/disk/by-id/ata-WDC_WD101EFAX-68LDBN0_VCG7W6LN, not available
20/09/2022 11:40:24
Monitoring 0 ATA/SATA, 0 SCSI/SAS and 2 NVMe devices
20/09/2022 11:40:24
Device: /dev/disk/by-id/nvme-NE-512_9800619606032, state read from /var/lib/smartmontools/smartd.NE_512-9800619606032-n1.nvme.state
20/09/2022 11:39:58
Device: /dev/disk/by-id/nvme-NE-512_9800619606032, state written to /var/lib/smartmontools/smartd.NE_512-9800619606032-n1.nvme.state
20/09/2022 11:39:58
smartd received signal 15: Terminated
20/09/2022 11:39:58
Device: /dev/disk/by-id/nvme-NE-128_9800624803340, state written to /var/lib/smartmontools/smartd.NE_128-9800624803340-n1.nvme.state
20/09/2022 11:39:58
Device: /dev/disk/by-id/ata-WDC_WD101EFAX-68LDBN0_VCG7U31N [SAT], state written to /var/lib/smartmontools/smartd.WDC_WD101EFAX_68LDBN0-VCG7U31N.ata.state
20/09/2022 11:39:58
Device: /dev/disk/by-id/ata-WDC_WD101EFAX-68LDBN0_VCG7VADN [SAT], state written to /var/lib/smartmontools/smartd.WDC_WD101EFAX_68LDBN0-VCG7VADN.ata.state
20/09/2022 11:39:58
Device: /dev/disk/by-id/ata-WDC_WD101EFAX-68LDBN0_VCG7W4EN [SAT], state written to /var/lib/smartmontools/smartd.WDC_WD101EFAX_68LDBN0-VCG7W4EN.ata.state
20/09/2022 11:39:58
Device: /dev/disk/by-id/ata-WDC_WD101EFAX-68LDBN0_VCG7W6LN [SAT], state written to /var/lib/smartmontools/smartd.WDC_WD101EFAX_68LDBN0-VCG7W6LN.ata.state 
20/09/2022 11:39:58
smartd is exiting (exit status 0)

Alles anzeigen

What is the problem ?

Can someone help me ?

Do I have to shutdown my server remotely ?

Lot of thanks !

geaves · 20. September 2022

Zitat von arkansis

What is the problem

The problem is the output is not seeing any of the drives + the email places (F) next to /dev/sd[abc] this would suggest that these drives have failed

Try the output mdadm --query --detail /dev/md0

arkansis · 20. September 2022

Zitat von geaves

The problem is the output is not seeing any of the drives + the email places (F) next to /dev/sd[abc] this would suggest that these drives have failed

Try the output mdadm --query --detail /dev/md0

Code

# mdadm --query --detail /dev/md0
mdadm: cannot open /dev/md0: No such file or directory

So my disks are dead and my data lost ?

It's incredible, all the material is not even 2 years old...

geaves · 20. September 2022

Zitat von arkansis

mdadm: cannot open /dev/md0: No such file or directory

That's not good at this moment I've no idea how to proceed, the blkid output shows no drives as does fdisk, in the GUI are the drives displayed in storage -> disks

arkansis · 20. September 2022

Zitat von geaves

That's not good at this moment I've no idea how to proceed, the blkid output shows no drives as does fdisk, in the GUI are the drives displayed in storage -> disks

geaves · 20. September 2022

Well the pictures tell the story the drives are missing, OMV can't see them so until you can resolve that you can't sort the array

arkansis · 20. September 2022

Zitat von geaves

Well the pictures tell the story the drives are missing, OMV can't see them so until you can resolve that you can't sort the array

Okay, thanks.

Is it possible that the problem comes from the electronic card that connects the hard drives to the chassis ?

I can understand on 4 hard drives having a failure, but that the 4 hard drives are no longer seen by the system at all is not normal.

The chassis is a chinese case like a QNAP with standard ITX format.

crashtest · 20. September 2022

(With apologies to geaves for butting in.)

Zitat von arkansis

the electronic card that connects the hard drives to the chassis ?

??? Do you have an add-on JBOD or RAID card?
(Usually, for up to four drives, hard drives are plugged directly into the motherboard.)

Zitat von arkansis

The chassis is chinese like a QNAP with standard ITX format.

Did the chassis come with a power supply? Did you check the 12V supply

geaves · 20. September 2022

I would say the possibility is the m'board, if the board has a bios are the drives seen/located in there, if they are then it's probably related to your OMV system being corrupted, if they can't be seen then it's quite possibly m'board related.

arkansis · 20. September 2022

Zitat von crashtest

(With apologies to geaves for butting in.)
??? Do you have an add-on JBOD or RAID card?
(Usually, for up to four drives, hard drives are plugged directly into the motherboard.)
Did the chassis come with a power supply? Did you check the 12V supply

The sata cables are connected to the motherboard and on the other side, to a PCB card which is powered by the power supply (molex connector) and the hard disks are connected and powered by this PCB card.

I haven't RAID card. It's just a sata motherboard RAID.

Yes, the chassis come with a power supply (1U factor brand Enhance).

arkansis · 20. September 2022

Zitat von geaves

I would say the possibility is the m'board, if the board has a bios are the drives seen/located in there, if they are then it's probably related to your OMV system being corrupted, if they can't be seen then it's quite possibly m'board related.

Ok, I'll look in the bios if the hard drives are detected. If they are detected in the bios, but not in OMV, what should I do?

I so hope it's the power supply or the PCB that's the problem.

geaves · 20. September 2022

IF they are seen in the bios and to do that you will have to restart the server, that reboot may bring them back up for OMV to see them.

arkansis · 20. September 2022

This is the PCB card:

arkansis · 20. September 2022

Zitat von geaves

IF they are seen in the bios and to do that you will have to restart the server, that reboot may bring them back up for OMV to see them.

I will try that. Thanks a lot !

arkansis · 20. September 2022

Good news !

The nas was plugged into a first price UPS back-up power from amazon.

It was the UPS back-up power that no longer worked well and the hard drives could no longer start.

By plugging the nas directly into a power outlet, the hard drives are recognized in the bios, the RAID5 is detected in the admin GUI and all the data is present from Windows !

I am relieved !

Thanks a lot for your help !

missing RAID 5

geaves 20. September 2022

Jetzt mitmachen!

Ähnliche Themen

RAID and disks are missing

Cannot get Raid 5 to work in OMV6

OMV missing fille system after restart - missing array

Disk bad. File system MISSING - How to proceed to recover RAID 5 Array/File System?

Design question about using OMV VM to create RAID via passthrough disks.

Tags