RAID 5 array has vaporized! Advice needed please

IamTed · 12. April 2015

Good morning everyone,
Yesterday we were watching stuff from my OMV box and playback stopped. I found that Plex had pegged the CPU use at 100% and wasn't giving up, so I instructed the box to reboot through the WebGUI. When the box came back on line, the RAID was gone. Not even degraded, but gone. In the Physical disks tab, I can see all of the drives, but under the RAID management, nothing. There are six 2Tb drives in the box and from what I had remembered, I had one setup as a hot swap and the other five as the array, but I can't remember if I had taken that sixth drive and changed it from a hot swap and added it to the RAID as an active drive. Seeing that everything with this box was running well, I haven't logged into it for a while.

Here is what I get from cat /proc/mdstat

Code

Personalities :
unused devices: <none>

When I do a fdisk -l, this is what I get back:

Code

Disk /dev/sda: 73.3 GB, 73295462400 bytes
255 heads, 63 sectors/track, 8910 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b0f3f


   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        8544    68628480   83  Linux
/dev/sda2            8545        8911     2946049    5  Extended
/dev/sda5            8545        8911     2946048   82  Linux swap / Solaris


Disk /dev/sdb: 2000.3 GB, 2000263577600 bytes
255 heads, 63 sectors/track, 243184 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sdb doesn't contain a valid partition table


Disk /dev/sdc: 2000.3 GB, 2000263577600 bytes
255 heads, 63 sectors/track, 243184 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sdc doesn't contain a valid partition table


Disk /dev/sdd: 2000.3 GB, 2000263577600 bytes
255 heads, 63 sectors/track, 243184 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sdd doesn't contain a valid partition table


Disk /dev/sde: 2000.3 GB, 2000263577600 bytes
255 heads, 63 sectors/track, 243184 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sde doesn't contain a valid partition table


Disk /dev/sdf: 2000.3 GB, 2000263577600 bytes
255 heads, 63 sectors/track, 243184 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sdf doesn't contain a valid partition table


Disk /dev/sdg: 2000.3 GB, 2000263577600 bytes
255 heads, 63 sectors/track, 243184 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sdg doesn't contain a valid partition table

Alles anzeigen

When I issue a blkid, I get this back:

Code

/dev/sda1: UUID="412ed2e2-1c83-4c2f-93f1-6a36aa8fbd7d" TYPE="ext4"
/dev/sda5: UUID="b3b5c740-6e4a-43d6-b695-dd59743fd5cd" TYPE="swap"
/dev/sdc: UUID="3e952187-f4e8-e08a-19b7-63a4cdc912c7" LABEL="OMV2:OMV" TYPE="linux_raid_member"
/dev/sdd: UUID="3e952187-f4e8-e08a-19b7-63a4cdc912c7" LABEL="OMV2:OMV" TYPE="linux_raid_member"
/dev/sde: UUID="3e952187-f4e8-e08a-19b7-63a4cdc912c7" LABEL="OMV2:OMV" TYPE="linux_raid_member"
/dev/sdf: UUID="3e952187-f4e8-e08a-19b7-63a4cdc912c7" LABEL="OMV2:OMV" TYPE="linux_raid_member"
/dev/sdg: UUID="3e952187-f4e8-e08a-19b7-63a4cdc912c7" LABEL="OMV2:OMV" TYPE="linux_raid_member"

While I have seen a few posts that are similar to my issue, I'm not sure enough to start trying things and find out that they will cause me to permanently loose everything. (That is, I'm while I can't get to my data right now, I'm hoping it is still there.)
The system is running 0.5.53 and my system hard drive is a 80Gb mechanical HDD.
If anyone can assist, that would be great and I will be very grateful.
Thanks,
Ted

IamTed · 12. April 2015

I am wondering if my system drive is flaking out. I noticed that the syslog has been cleared and when I issue cat /etc/mdadm/mdadm.conf this is what I get back:

Code

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#


# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions


# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes


# automatically tag new arrays as belonging to the local system
HOMEHOST <system>


# definitions of existing MD arrays

Alles anzeigen

Am I thinking in the right direction?

IamTed · 16. April 2015

Well, I'm back up and running. I am guessing that the mdadm.conf was the problem. I pulled the RAID drives, swapped out the sys drive with a spare and reinstalled OMV.
When I was done, I could see the RAID. I was clean and degraded. I added the spare drive that I thought was already setup as a hot swap and it started to rebuild. Here is the details:

Code

mdadm --detail /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Sun Jan  1 13:44:03 2006
     Raid Level : raid5
     Array Size : 7813523456 (7451.56 GiB 8001.05 GB)
  Used Dev Size : 1953380864 (1862.89 GiB 2000.26 GB)
   Raid Devices : 5
  Total Devices : 5
    Persistence : Superblock is persistent


    Update Time : Wed Apr 15 18:31:11 2015
          State : active, degraded, recovering
 Active Devices : 4
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 1


         Layout : left-symmetric
     Chunk Size : 512K


 Rebuild Status : 29% complete


           Name : OMV2:OMV
           UUID : 3e952187:f4e8e08a:19b763a4:cdc912c7
         Events : 1803823


    Number   Major   Minor   RaidDevice State
       5       8       64        0      active sync   /dev/sde
       6       8       80        1      active sync   /dev/sdf
       2       8       32        2      active sync   /dev/sdc
       3       8       48        3      active sync   /dev/sdd
       7       8       16        4      spare rebuilding   /dev/sdb

Alles anzeigen

What I'm wondering now, is why sdg dropped out of the RAID. Looking at this, it looks like it thinks that it should be still there.

Code

mdadm --examine /dev/sdg
/dev/sdg:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 3e952187:f4e8e08a:19b763a4:cdc912c7
           Name : OMV2:OMV
  Creation Time : Sun Jan  1 13:44:03 2006
     Raid Level : raid5
   Raid Devices : 5


 Avail Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
     Array Size : 15627046912 (7451.56 GiB 8001.05 GB)
  Used Dev Size : 3906761728 (1862.89 GiB 2000.26 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 44029c1f:b0a408e3:6d6d35c2:b5660be0


    Update Time : Sun Mar  1 06:44:42 2015
       Checksum : 614b8b1f - correct
         Events : 153067


         Layout : left-symmetric
     Chunk Size : 512K


   Device Role : Active device 4
   Array State : AAAAA ('A' == active, '.' == missing)

Alles anzeigen

BTW, here is the output from BLKID.

Code

blkid
/dev/sda1: UUID="d81eacfe-439c-4e12-bbb2-a933e69d4dfa" TYPE="ext4"
/dev/sda5: UUID="6e724718-95ae-4e0c-9e17-a469c4a7627e" TYPE="swap"
/dev/sdc: UUID="3e952187-f4e8-e08a-19b7-63a4cdc912c7" LABEL="OMV2:OMV" TYPE="linux_raid_member"
/dev/sdd: UUID="3e952187-f4e8-e08a-19b7-63a4cdc912c7" LABEL="OMV2:OMV" TYPE="linux_raid_member"
/dev/sde: UUID="3e952187-f4e8-e08a-19b7-63a4cdc912c7" LABEL="OMV2:OMV" TYPE="linux_raid_member"
/dev/sdf: UUID="3e952187-f4e8-e08a-19b7-63a4cdc912c7" LABEL="OMV2:OMV" TYPE="linux_raid_member"
/dev/md127: LABEL="OMV" UUID="13a7164c-7be5-49e9-ab63-d704f96f890e" TYPE="ext4"
/dev/sdg: UUID="3e952187-f4e8-e08a-19b7-63a4cdc912c7" LABEL="OMV2:OMV" TYPE="linux_raid_member"
/dev/sdb: UUID="3e952187-f4e8-e08a-19b7-63a4cdc912c7" LABEL="OMV2:OMV" TYPE="linux_raid_member"

So, should I just wipe sdg and add it to the pool or should I worry that it is a flaky drive?
Thanks!

RAID 5 array has vaporized! Advice needed please

Jetzt mitmachen!

Tags