Raid5 Missing, File System Missing

QPlus7 · 15. August 2017

Hello, all.

I added a new drive (6 x 1Tb total drives now), expanded my array and then resized the file system. I lost network connectivity during the resize and had to reboot. Now it appears I've lost my array. Below is the output based on ryecoaaron request in his sticky thread:

Code

root@omv:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : inactive sdd[0] sdb[5] sda[3] sdf[2] sde[1]
4883157560 blocks super 1.2


unused devices: <none>

Code

root@omv:~# blkid
/dev/sdd: UUID="c628d2a9-0132-5816-afb8-3a2e73031b40" UUID_SUB="e072462a-09bb-7769-b0cd-88f58f9bb5ca" LABEL="omv:R5Array" TYPE="linux_raid_member"
/dev/sdb: UUID="c628d2a9-0132-5816-afb8-3a2e73031b40" UUID_SUB="b0bd798e-80b4-db90-4e35-5d832793fe30" LABEL="omv:R5Array" TYPE="linux_raid_member"
/dev/sde: UUID="c628d2a9-0132-5816-afb8-3a2e73031b40" UUID_SUB="ea552547-e7ab-f516-5b71-5b6a5234da89" LABEL="omv:R5Array" TYPE="linux_raid_member"
/dev/sdg1: UUID="1d42ce8e-7dca-4183-8723-926eafc7c182" TYPE="ext4"
/dev/sdg5: UUID="7bbc0b57-05ef-4b1b-8d19-2e1a6c71d8b5" TYPE="swap"
/dev/sdf: UUID="c628d2a9-0132-5816-afb8-3a2e73031b40" UUID_SUB="bbf79cbf-98e4-2acb-37f9-2301d9fe9c1d" LABEL="omv:R5Array" TYPE="linux_raid_member"
/dev/sda: UUID="c628d2a9-0132-5816-afb8-3a2e73031b40" UUID_SUB="f092b520-1794-41f6-5ca7-33251544ddd9" LABEL="omv:R5Array" TYPE="linux_raid_member"

Code

root@omv:~# fdisk -l | grep "Disk "
Disk /dev/sda doesn't contain a valid partition table


WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdd doesn't contain a valid partition table
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sde doesn't contain a valid partition table
Disk /dev/sdf doesn't contain a valid partition table
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
Disk identifier: 0x00000000
Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
Disk identifier: 0x00000000
Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
Disk identifier: 0x00000000
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
Disk identifier: 0x00000000
Disk /dev/sde: 1000.2 GB, 1000204886016 bytes
Disk identifier: 0x00000000
Disk /dev/sdf: 1000.2 GB, 1000204886016 bytes
Disk identifier: 0x00000000
Disk /dev/sdg: 500.1 GB, 500107862016 bytes
Disk identifier: 0x000ab530

Alles anzeigen

Code

root@omv:~# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#


# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions


# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes


# automatically tag new arrays as belonging to the local system
HOMEHOST <system>


# definitions of existing MD arrays
ARRAY /dev/md/R5Array metadata=1.2 name=omv:R5Array UUID=c628d2a9:01325816:afb83a2e:73031b40

Alles anzeigen

Code

root@omv:~# mdadm --detail --scan --verbose
mdadm: cannot open /dev/md/R5Array: No such file or directory

I've rebooted a few times but do not see the array in the GUI. See attachments for more details on Physical Disks, Raid Management, and File System screenshots.

Any help is GREATLY appreciated.

QPlus7 · 15. August 2017

Well, a little more digging and I came up with this:

Code

root@omv:/# mdadm --stop /dev/md127
mdadm: stopped /dev/md127

Code

root@omv:/# mdadm --assemble --force --verbose /dev/md0 /dev/sd[abcdef]
mdadm: looking for devices for /dev/md0
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdc has no superblock - assembly aborted

Code

root@omv:/# mdadm --assemble --force --verbose /dev/md0 /dev/sd[abdef]
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdf is identified as a member of /dev/md0, slot 2.
mdadm: Marking array /dev/md0 as 'clean'
mdadm:/dev/md0 has an active reshape - checking if critical section needs to be restored
mdadm: added /dev/sde to /dev/md0 as 1
mdadm: added /dev/sdf to /dev/md0 as 2
mdadm: added /dev/sda to /dev/md0 as 3
mdadm: added /dev/sdb to /dev/md0 as 4
mdadm: no uptodate device for slot 5 of /dev/md0
mdadm: added /dev/sdd to /dev/md0 as 0
mdadm: /dev/md0 has been started with 5 drives (out of 6).

Alles anzeigen

My array showed up and I tried to mount the file system. The system has now locked up and cat /proc/mdstat doesn't return anything.

I'm going to leave it overnight and check it again the morning.

QPlus7 · 15. August 2017

It has been over 10 hours and no changes. Is this normal during a resize?

QPlus7 · 16. August 2017

I restarted the box and went through the process in post #2. I now see the array in the GUI and the file system shows unmounted.

If I click mount, how long should it take? I tried it again and it locks up and "cat /proc/mdstat" doesn't show anything and clicking on RAID Management or File System in the GUI shows a loading button and nothing else.

What would be the best way to proceed from here?

ryecoaaron · 16. August 2017

That is not a good sign that cat /proc/mdstat doesn't return anything. You will through the right steps to fix it. Sounds like the system isn't stable or a drive is failing. Anything in dmesg?

I would ask yourself two questions:
1 - Do you really need raid? Raid isn't backup and there are other ways to pool drives.
2 - Can you back up the array? Raid isn't backup.

QPlus7 · 18. August 2017

Zitat von ryecoaaron

That is not a good sign that cat /proc/mdstat doesn't return anything. You will through the right steps to fix it. Sounds like the system isn't stable or a drive is failing. Anything in dmesg?

I would ask yourself two questions:
1 - Do you really need raid? Raid isn't backup and there are other ways to pool drives.
2 - Can you back up the array? Raid isn't backup.

Thanks for the reply!

1) Going forward, I don't care what the pooling method is but for now I need to recover the array/data if possible
2) I'm aware a RAID doesn't provide backups but I was in the process of consolidating everything so I could take a backup. At this point, I have none, unfortunately.

I've made some progress in troubleshooting, I think.

I put the 2 drives that were in after I expanded and reshaped the array. The following information is with these 2 drives in:

Code

#############################################################################################
root@omv:~# mdadm --examine /dev/sd*
/dev/sda:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : c628d2a9:01325816:afb83a2e:73031b40
Name : omv:R5Array  (local to host omv)
Creation Time : Tue Jan  6 12:00:20 2015
Raid Level : raid5
Raid Devices : 6


Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
Array Size : 4883156480 (4656.94 GiB 5000.35 GB)
Used Dev Size : 1953262592 (931.39 GiB 1000.07 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : active
Device UUID : f092b520:179441f6:5ca73325:1544ddd9


Reshape pos'n : 577431040 (550.68 GiB 591.29 GB)
Delta Devices : 1 (5->6)


Update Time : Thu Aug 17 01:44:30 2017
Checksum : 76b3597c - correct
Events : 970956


Layout : left-symmetric
Chunk Size : 512K


Device Role : Active device 3
Array State : AAAAA. ('A' == active, '.' == missing)
/dev/sdb:
MBR Magic : aa55
Partition[0] :   1953525167 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sdb1.
mdadm: No md superblock detected on /dev/sdb2.
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : c628d2a9:01325816:afb83a2e:73031b40
Name : omv:R5Array  (local to host omv)
Creation Time : Tue Jan  6 12:00:20 2015
Raid Level : raid5
Raid Devices : 6


Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
Array Size : 4883156480 (4656.94 GiB 5000.35 GB)
Used Dev Size : 1953262592 (931.39 GiB 1000.07 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : active
Device UUID : b0bd798e:80b4db90:4e355d83:2793fe30


Reshape pos'n : 577431040 (550.68 GiB 591.29 GB)
Delta Devices : 1 (5->6)


Update Time : Thu Aug 17 01:44:30 2017
Checksum : 345c81ab - correct
Events : 970956


Layout : left-symmetric
Chunk Size : 512K


Device Role : Active device 4
Array State : AAAAA. ('A' == active, '.' == missing)
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : c628d2a9:01325816:afb83a2e:73031b40
Name : omv:R5Array  (local to host omv)
Creation Time : Tue Jan  6 12:00:20 2015
Raid Level : raid5
Raid Devices : 6


Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
Array Size : 4883156480 (4656.94 GiB 5000.35 GB)
Used Dev Size : 1953262592 (931.39 GiB 1000.07 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : active
Device UUID : e072462a:09bb7769:b0cd88f5:8f9bb5ca


Reshape pos'n : 577431040 (550.68 GiB 591.29 GB)
Delta Devices : 1 (5->6)


Update Time : Thu Aug 17 01:44:30 2017
Checksum : b4a7de29 - correct
Events : 970956


Layout : left-symmetric
Chunk Size : 512K


Device Role : Active device 0
Array State : AAAAA. ('A' == active, '.' == missing)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : c628d2a9:01325816:afb83a2e:73031b40
Name : omv:R5Array  (local to host omv)
Creation Time : Tue Jan  6 12:00:20 2015
Raid Level : raid5
Raid Devices : 6


Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
Array Size : 4883156480 (4656.94 GiB 5000.35 GB)
Used Dev Size : 1953262592 (931.39 GiB 1000.07 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : active
Device UUID : ea552547:e7abf516:5b715b6a:5234da89


Reshape pos'n : 577431040 (550.68 GiB 591.29 GB)
Delta Devices : 1 (5->6)


Update Time : Thu Aug 17 01:44:30 2017
Checksum : b2fbee7f - correct
Events : 970956


Layout : left-symmetric
Chunk Size : 512K


Device Role : Active device 1
Array State : AAAAA. ('A' == active, '.' == missing)
/dev/sdf:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : c628d2a9:01325816:afb83a2e:73031b40
Name : omv:R5Array  (local to host omv)
Creation Time : Tue Jan  6 12:00:20 2015
Raid Level : raid5
Raid Devices : 6


Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
Array Size : 4883156480 (4656.94 GiB 5000.35 GB)
Used Dev Size : 1953262592 (931.39 GiB 1000.07 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : active
Device UUID : bbf79cbf:98e42acb:37f92301:d9fe9c1d


Reshape pos'n : 577431040 (550.68 GiB 591.29 GB)
Delta Devices : 1 (5->6)


Update Time : Thu Aug 17 01:44:30 2017
Checksum : a3422e6 - correct
Events : 970956


Layout : left-symmetric
Chunk Size : 512K


Device Role : Active device 2
Array State : AAAAA. ('A' == active, '.' == missing)
/dev/sdg:
MBR Magic : aa55
Partition[0] :    962410496 sectors at         2048 (type 83)
Partition[1] :     14356482 sectors at    962414590 (type 05)
mdadm: No md superblock detected on /dev/sdg1.
/dev/sdg2:
MBR Magic : aa55
Partition[0] :     14356480 sectors at            2 (type 82)
mdadm: No md superblock detected on /dev/sdg5.
#############################################################################################


#############################################################################################
/etc/mdadm/mdadm.conf


root@omv:~# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#


# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions


# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes


# automatically tag new arrays as belonging to the local system
HOMEHOST <system>


# definitions of existing MD arrays
ARRAY /dev/md/R5Array metadata=1.2 name=omv:R5Array UUID=c628d2a9:01325816:afb83a2e:73031b40
#############################################################################################


#############################################################################################
cat /proc/mdstat
root@omv:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : inactive sdd[0] sdc[5] sda[3] sdf[2] sde[1]
4883157560 blocks super 1.2


unused devices: <none>
root@omv:~#
#############################################################################################


#############################################################################################
mdadm --assemble --force --verbose /dev/md0 /dev/sd[abcdef]


root@omv:~# mdadm --assemble --force --verbose /dev/md0 /dev/sd[abcdef]
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: /dev/sdb has no superblock - assembly aborted
root@omv:~#
#############################################################################################

Alles anzeigen

QPlus7 · 18. August 2017

I bought a new 1TB drive and put it in along with the original drives and the previously "new" drive that was in. This info is from that setup:

Code

#############################################################################################
mdadm --examine /dev/sd*


Chunk Size : 512K


Device Role : Active device 1
Array State : AAAAA. ('A' == active, '.' == missing)
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : c628d2a9:01325816:afb83a2e:73031b40
Name : omv:R5Array  (local to host omv)
Creation Time : Tue Jan  6 12:00:20 2015
Raid Level : raid5
Raid Devices : 6


Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
Array Size : 4883156480 (4656.94 GiB 5000.35 GB)
Used Dev Size : 1953262592 (931.39 GiB 1000.07 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : active
Device UUID : bbf79cbf:98e42acb:37f92301:d9fe9c1d


Reshape pos'n : 577431040 (550.68 GiB 591.29 GB)
Delta Devices : 1 (5->6)


Update Time : Thu Aug 17 01:44:30 2017
Checksum : a3422e6 - correct
Events : 970956


Layout : left-symmetric
Chunk Size : 512K


Device Role : Active device 2
Array State : AAAAA. ('A' == active, '.' == missing)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : c628d2a9:01325816:afb83a2e:73031b40
Name : omv:R5Array  (local to host omv)
Creation Time : Tue Jan  6 12:00:20 2015
Raid Level : raid5
Raid Devices : 6


Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
Array Size : 4883156480 (4656.94 GiB 5000.35 GB)
Used Dev Size : 1953262592 (931.39 GiB 1000.07 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : active
Device UUID : f092b520:179441f6:5ca73325:1544ddd9


Reshape pos'n : 577431040 (550.68 GiB 591.29 GB)
Delta Devices : 1 (5->6)


Update Time : Thu Aug 17 01:44:30 2017
Checksum : 76b3597c - correct
Events : 970956


Layout : left-symmetric
Chunk Size : 512K


Device Role : Active device 3
Array State : AAAAA. ('A' == active, '.' == missing)
/dev/sdf:
MBR Magic : aa55
Partition[0] :    962410496 sectors at         2048 (type 83)
Partition[1] :     14356482 sectors at    962414590 (type 05)
mdadm: No md superblock detected on /dev/sdf1.
/dev/sdf2:
MBR Magic : aa55
Partition[0] :     14356480 sectors at            2 (type 82)
mdadm: No md superblock detected on /dev/sdf5.
#############################################################################################


#############################################################################################
/etc/mdadm/mdadm.conf


# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#


# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions


# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes


# automatically tag new arrays as belonging to the local system
HOMEHOST <system>


# definitions of existing MD arrays
ARRAY /dev/md/R5Array metadata=1.2 name=omv:R5Array UUID=c628d2a9:01325816:afb83a2e:73031b40
#############################################################################################


#############################################################################################
cat /proc/mdstat


Personalities : [raid6] [raid5] [raid4]
md127 : inactive sdb[0] sda[5] sde[3] sdd[2] sdc[1]
4883157560 blocks super 1.2


unused devices: <none>
#############################################################################################


#############################################################################################
mdadm --assemble --force --verbose /dev/md0 /dev/sd[abcdef]


root@omv:~# mdadm --assemble --force --verbose /dev/md0 /dev/sd[abcdef]
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda is busy - skipping
mdadm: /dev/sdb is busy - skipping
mdadm: /dev/sdc is busy - skipping
mdadm: /dev/sdd is busy - skipping
mdadm: /dev/sde is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdf
mdadm: /dev/sdf has no superblock - assembly aborted
#############################################################################################

Alles anzeigen

I think my array is still in tact. I'm a bit confused on the naming though: is it /dev/md0, /dev/md127, or /dev/R5Array?

At this point, I'm getting desperate. I'm willing to work with someone remotely and pay for their time.

QPlus7 · 18. August 2017

Attached is the dmesg output.

QPlus7 · 18. August 2017

This is probably my latest update for the night as I think I've made progress and the array might be reshaping right now.

I removed the 6th disk I added and ran the following:

Code

root@omv:~#  mdadm --stop /dev/md127
mdadm: stopped /dev/md127
root@omv:~# mdadm --assemble --force /dev/md127 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde
mdadm: Marking array /dev/md127 as 'clean'
mdadm: /dev/md127 has been started with 5 drives (out of 6).




root@omv:~# sudo mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Tue Jan  6 12:00:20 2015
Raid Level : raid5
Array Size : 3906525184 (3725.55 GiB 4000.28 GB)
Used Dev Size : 976631296 (931.39 GiB 1000.07 GB)
Raid Devices : 6
Total Devices : 5
Persistence : Superblock is persistent


Update Time : Thu Aug 17 01:44:30 2017
State : clean, degraded
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0


Layout : left-symmetric
Chunk Size : 512K


Delta Devices : 1, (5->6)


Name : omv:R5Array  (local to host omv)
UUID : c628d2a9:01325816:afb83a2e:73031b40
Events : 970956


Number   Major   Minor   RaidDevice State
0       8       32        0      active sync   /dev/sdc
1       8       48        1      active sync   /dev/sdd
2       8       64        2      active sync   /dev/sde
3       8        0        3      active sync   /dev/sda
5       8       16        4      active sync   /dev/sdb
5       0        0        5      removed

Alles anzeigen

Following that, I did this:

Code

mkdir -p /mnt/md127


mount /dev/md127 /mnt/md127

At this point my console just has a blinking cursor as if it was doing something.

I logged in to another session and ran cat /proc/mdstat but get the blinking cursor again.

Before I tried to mount the array, I saw it mounted in the GUI when I clicked on RAID Management and it showed "clean,degraded".

After I mounted it, I get "Loading..." when I click on RAID Management and File System.

I'm hoping the array is reshaping and will just take some time but I'm not sure how I can confirm that other than just waiting it out.

Attached is the latest dmesg.

QPlus7 · 18. August 2017

Ok, maybe it wasn't my last post - sorry. Just thought of something...

Is it possible that the reshaping never finished [Reshape pos'n : 577431040 (550.68 GiB 591.29 GB)]? I'm thinking if I power down, plug in the 6th drive, boot the machine, stop md127, and then run mdadm --assemble --force /dev/md127 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde that might do the trick?

QPlus7 · 19. August 2017

Well, I did just that. Once I added the new drive and added it to the array it started to resync. Once the resync finished it started to recover. After about 18 hours total, my array is now online and I have access to all my data!

First order of business, transfer everything to backblaze and a 6TB drive!

After that, I'll start my research into other alternatives.

Hope this helps someone in the future!

crashtest · 21. August 2017

Zitat von QPlus7

First order of business, transfer everything to backblaze and a 6TB drive!

In a word, "smart".

I've noticed that Home and small business RAID users usually fall into one of two different camps.
1. Those who have never had a problem, or minor problems, that they easily recovered from (using, perhaps, an online hot spare). They love RAID and promote it.
2. Those who had a major problem, where they lost their entire array. These folks, almost without fail, "had an array" in the past.

(There's a 3rd group who actually backup their arrays and are ready for a full array failure, but among home and small business users, they seem to be exceedingly rare.)

With the sizes of drives that are available these days (up to 8TB), I don't understand why NAS users would feel the need to pool disks. Does administering a NAS become easier, somehow, with a common mount point? If it does, things become a bit more complicated and inconvenient when the inevitable physical disk problem crops up, in the pool.
________________________________

In a JBOD config - it's easy enough to divide up data folders, in a logical manner, over different physical drives. When a NAS puts shares out to the network, the source physical drive is irrelevant.

Further, it's easy enough to Rsync network shares / folders to a local destination or a remote server - or even entire drives - without risking the quarks of running a RAID1 broken mirror. Rsync provides true backup, versus the false sense of security that users believe they're getting with RAID.

Jetzt mitmachen!