RAID Missing After Re-Boot

piscdi · Apr 8th 2021

Hi guys,

First time poster and self-professed moron; I’m not bright enough to know what I don’t know. After a year and a half of flawless operation, my RAID went missing after a reboot.

I tried some of the ‘generic’ recommendations offered here but stopped short of applying any “forced” instructions (I may be a moron, but I’m a cautious moron). I have not rebooted since I found the array missing.

All my critical files were backed up and are safe, but I have about 14 tbs of media (all legal) hanging in the balance. If the RAID can be restored – fantastic; if not – a big hit, but not a life altering event.

Per ryecoaaron’s info request for degraded or missing RAID issues, I use 4 WD Red 6TB NAS Internal Hard Drives and the results of the requested inquiries are as follows:

Code

root@Server:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices: <none>

Code

root@Server:~# blkid
/dev/sdf1: UUID="b8f86e19-3cb3-4d0a-b1e2-623620314887" TYPE="ext4" PARTUUID="79c1501c-01"
/dev/sdf5: UUID="a631ab49-ab21-4169-8352-aa1829c8a95b" TYPE="swap" PARTUUID="79c1501c-05"
/dev/sdb1: LABEL="BackUp" UUID="9238bbb9-e494-487d-941e-234cad83a670" TYPE="ext4" PARTUUID="d6e47150-672f-4fb8-a57d-72c6ff0ca4ae"
/dev/sde: UUID="98379905-d139-d263-d58d-5eb3893ba95b" UUID_SUB="97feb0f7-c46c-0e05-4b6f-4c40a9448f9f" LABEL="Server:Raid1" TYPE="linux_raid_member"
/dev/sdc: UUID="98379905-d139-d263-d58d-5eb3893ba95b" UUID_SUB="98a8cd6c-cb21-5f16-8540-aa6c88960541" LABEL="Server:Raid1" TYPE="linux_raid_member"
/dev/sda: UUID="98379905-d139-d263-d58d-5eb3893ba95b" UUID_SUB="f3a4ee0f-ed2a-bf8a-ead3-034b9f975d88" LABEL="Server:Raid1" TYPE="linux_raid_member"
/dev/sdd: UUID="98379905-d139-d263-d58d-5eb3893ba95b" UUID_SUB="df979bad-92c3-ac42-f3e5-512838996555" LABEL="Server:Raid1" TYPE="linux_raid_member"

Code

root@Server:~# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# definitions of existing MD arrays
ARRAY /dev/md0 metadata=1.2 name=Server:Raid1 UUID=98379905:d139d263:d58d5eb3:893ba95b

Display More

Code

root@Server:~# mdadm --detail --scan --verbose
root@Server:~#

Thank you in advance for any help and/or condolences you may offer.

geaves · Apr 8th 2021

Quote from piscdi

condolences you may offer

it doesnae look good Jim, some of the output from the dilithium chamber is simply non existent!!

What type of raid is it?

output of mdadm --detail /dev/md0 and mdadm --examine /dev/sda

piscdi · Apr 8th 2021

Thanks for responding; it is, er' used to be RAID 5

geaves · Apr 8th 2021

Quote from piscdi

Thanks for responding; it is, er' used to be RAID 5

If it 'used to be' Raid5 then I would expect some output from /proc/mdstat and mdadm --detail --scan, what's the output of what I asked

piscdi · Apr 8th 2021

I'm guessing that this can't be good:

Code

root@Server:~#  /proc/mdstat and mdadm --detail --scan
-bash: /proc/mdstat: Permission denied
root@Server:~# root@Server:~#  /proc/mdstat and mdadm --detail --scan
-bash: root@Server:~#: command not found

piscdi · Apr 8th 2021

Sorry, I warned you I was a moron. Here's what you asked for in the first response:

Code

root@Server:~# mdadm --detail /dev/md0 and mdadm --examine /dev/sda
mdadm: --examine/-E cannot be given with -D

piscdi · Apr 8th 2021

Code

root@Server:~# mdadm --detail /dev/md0
mdadm: cannot open /dev/md0: No such file or directory

Code

root@Server:~# mdadm --examine /dev/sda
/dev/sda:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 98379905:d139d263:d58d5eb3:893ba95b
           Name : Server:Raid1  (local to host Server)
  Creation Time : Mon Nov 25 18:05:25 2019
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 11720783024 (5588.90 GiB 6001.04 GB)
     Array Size : 17581174272 (16766.71 GiB 18003.12 GB)
  Used Dev Size : 11720782848 (5588.90 GiB 6001.04 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=176 sectors
          State : clean
    Device UUID : f3a4ee0f:ed2abf8a:ead3034b:9f975d88

Internal Bitmap : 8 sectors from superblock
    Update Time : Thu Jun 18 19:32:29 2020
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : bbd177a9 - correct
         Events : 11265

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

Display More

geaves · Apr 8th 2021

Good and bad news, post the output of each of the drives in the array

mdadm --examine /dev/sdc

mdadm --examine /dev/sdd

mdadm --examine /dev/sde

piscdi · Apr 8th 2021

Thank you for your patience, geaves.

Code

root@Server:~# mdadm --examine /dev/sdc
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 98379905:d139d263:d58d5eb3:893ba95b
           Name : Server:Raid1  (local to host Server)
  Creation Time : Mon Nov 25 18:05:25 2019
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 11720783024 (5588.90 GiB 6001.04 GB)
     Array Size : 17581174272 (16766.71 GiB 18003.12 GB)
  Used Dev Size : 11720782848 (5588.90 GiB 6001.04 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=176 sectors
          State : clean
    Device UUID : 98a8cd6c:cb215f16:8540aa6c:88960541

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Apr  5 21:06:23 2021
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 80233a6e - correct
         Events : 183873

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : .A.A ('A' == active, '.' == missing, 'R' == replacing)
root@Server:~# root@Server:~# mdadm --examine /dev/sdc
root@Server:~#
-bash: root@Server:~#: command not found
root@Server:~# /dev/sdc:
-bash: /dev/sdc:: No such file or directory
root@Server:~#           Magic : a92b4efc
-bash: Magic: command not found
root@Server:~#         Version : 1.2
-bash: Version: command not found
root@Server:~#     Feature Map : 0x1
-bash: Feature: command not found
root@Server:~#      Array UUID : 98379905:d139d263:d58d5eb3:893ba95b
-bash: Array: command not found
root@Server:~#            Name : Server:Raid1  (local to host Server)
-bash: syntax error near unexpected token `('
root@Server:~#   Creation Time : Mon Nov 25 18:05:25 2019
-bash: Creation: command not found
root@Server:~#      Raid Level : raid5
-bash: Raid: command not found
root@Server:~#    Raid Devices : 4
-bash: Raid: command not found
root@Server:~#
root@Server:~#  Avail Dev Size : 11720783024 (5588.90 GiB 6001.04 GB)
-bash: syntax error near unexpected token `('
root@Server:~#      Array Size : 17581174272 (16766.71 GiB 18003.12 GB)
-bash: syntax error near unexpected token `('
root@Server:~#   Used Dev Size : 11720782848 (5588.90 GiB 6001.04 GB)
-bash: syntax error near unexpected token `('
root@Server:~#     Data Offset : 262144 sectors
-bash: Data: command not found
root@Server:~#    Super Offset : 8 sectors
-bash: Super: command not found
root@Server:~#    Unused Space : before=262056 sectors, after=176 sectors
-bash: Unused: command not found
root@Server:~#           State : clean
-bash: State: command not found
root@Server:~#     Device UUID : 98a8cd6c:cb215f16:8540aa6c:88960541
-bash: Device: command not found
root@Server:~#
root@Server:~# Internal Bitmap : 8 sectors from superblock
-bash: Internal: command not found
root@Server:~#     Update Time : Mon Apr  5 21:06:23 2021
-bash: Update: command not found
root@Server:~#   Bad Block Log : 512 entries available at offset 72 sectors
-bash: Bad: command not found
root@Server:~#        Checksum : 80233a6e - correct
-bash: Checksum: command not found
root@Server:~#          Events : 183873
-bash: Events: command not found
root@Server:~#
root@Server:~#          Layout : left-symmetric
-bash: Layout: command not found
root@Server:~#      Chunk Size : 512K
-bash: Chunk: command not found
root@Server:~#
root@Server:~#    Device Role : Active device 1
-bash: Device: command not found
root@Server:~#    Array State : .A.A ('A' == active, '.' == missing, 'R' == replacing)
-bash: syntax error near unexpected token `('
root@Server:~# root@Server:~#
-bash: root@Server:~#: command not found

Display More

Code

root@Server:~# mdadm --examine /dev/sdd
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 98379905:d139d263:d58d5eb3:893ba95b
           Name : Server:Raid1  (local to host Server)
  Creation Time : Mon Nov 25 18:05:25 2019
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 11720783024 (5588.90 GiB 6001.04 GB)
     Array Size : 17581174272 (16766.71 GiB 18003.12 GB)
  Used Dev Size : 11720782848 (5588.90 GiB 6001.04 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=176 sectors
          State : clean
    Device UUID : df979bad:92c3ac42:f3e55128:38996555

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Apr  5 06:29:06 2021
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : bd44a5fc - correct
         Events : 183867

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : .AAA ('A' == active, '.' == missing, 'R' == replacing)

Display More

Code

root@Server:~# mdadm --examine /dev/sde
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 98379905:d139d263:d58d5eb3:893ba95b
           Name : Server:Raid1  (local to host Server)
  Creation Time : Mon Nov 25 18:05:25 2019
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 11720783024 (5588.90 GiB 6001.04 GB)
     Array Size : 17581174272 (16766.71 GiB 18003.12 GB)
  Used Dev Size : 11720782848 (5588.90 GiB 6001.04 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=176 sectors
          State : clean
    Device UUID : 97feb0f7:c46c0e05:4b6f4c40:a9448f9f

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Apr  5 21:06:23 2021
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 2be1b850 - correct
         Events : 183873

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : .A.A ('A' == active, '.' == missing, 'R' == replacing)

Display More

piscdi · Apr 8th 2021

Here's the fourth drive info:

Code

root@Server:~# mdadm --examine /dev/sda
/dev/sda:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 98379905:d139d263:d58d5eb3:893ba95b
           Name : Server:Raid1  (local to host Server)
  Creation Time : Mon Nov 25 18:05:25 2019
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 11720783024 (5588.90 GiB 6001.04 GB)
     Array Size : 17581174272 (16766.71 GiB 18003.12 GB)
  Used Dev Size : 11720782848 (5588.90 GiB 6001.04 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=176 sectors
          State : clean
    Device UUID : f3a4ee0f:ed2abf8a:ead3034b:9f975d88

Internal Bitmap : 8 sectors from superblock
    Update Time : Thu Jun 18 19:32:29 2020
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : bbd177a9 - correct
         Events : 11265

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

Display More

geaves · Apr 8th 2021

According to the drives they are reporting correctly;

1) mdadm --stop /dev/md0 that may return an error of not being found so proceed to next

2) mdadm --assemble --force --verbose /dev/md0 /dev/sd[acde] the array should hopefully rebuild.

I'm about to go out for about an hour, if the above works DO NOT REBOOT report back.

It should display rebuilding in the GUI.

piscdi · Apr 8th 2021

I just re-ran your first command for the first drive (sdc) and got this:

Apparently even my copy and paste 'skills' are now suspect

Code

root@Server:~# mdadm --examine /dev/sdc
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 98379905:d139d263:d58d5eb3:893ba95b
           Name : Server:Raid1  (local to host Server)
  Creation Time : Mon Nov 25 18:05:25 2019
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 11720783024 (5588.90 GiB 6001.04 GB)
     Array Size : 17581174272 (16766.71 GiB 18003.12 GB)
  Used Dev Size : 11720782848 (5588.90 GiB 6001.04 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=176 sectors
          State : clean
    Device UUID : 98a8cd6c:cb215f16:8540aa6c:88960541

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Apr  5 21:06:23 2021
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 80233a6e - correct
         Events : 183873

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : .A.A ('A' == active, '.' == missing, 'R' == replacing)

Display More

piscdi · Apr 8th 2021

Response in transit. Thank you so much; I will try that and report back in an hour or so.

piscdi · Apr 8th 2021

Hi geaves,

This looks promising, I think. GUI shows array as mounted but degraded.

Code

root@Server:~# mdadm --stop /dev/md0
mdadm: error opening /dev/md0: No such file or directory
root@Server:~# mdadm --assemble --force --verbose /dev/md0 /dev/sd[acde]
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 3.
mdadm: forcing event count in /dev/sdd(2) from 183867 upto 183873
mdadm: clearing FAULTY flag for device 2 in /dev/md0 for /dev/sdd
mdadm: Marking array /dev/md0 as 'clean'
mdadm: added /dev/sda to /dev/md0 as 0 (possibly out of date)
mdadm: added /dev/sdd to /dev/md0 as 2
mdadm: added /dev/sde to /dev/md0 as 3
mdadm: added /dev/sdc to /dev/md0 as 1
mdadm: /dev/md0 has been started with 3 drives (out of 4).
root@Server:~#

Display More

geaves · Apr 8th 2021

That's a good start, to get sda back into the array the easiest way to be sure due to the out of date error,

Storage -> Disks select /dev/sda click wipe on the menu, then click short, then OK when it's finished

Raid Management -> Recover, from the dialog box select /dev/sda, the array will now add the drive and rebuild, WAIT for it to complete

DO NOT REBOOT

Then run omv-mkconf mdadm

Then update-initramfs -u

piscdi · Apr 8th 2021

Roger that. Wipe the errant disc, recover the RAID, wait to finish and run the commands (AND NO RE-BOOTING). I'm taking my time, so I'll be a few minutes.

piscdi · Apr 8th 2021

Not to be obtuse, but my options at wipe are 'secure' and 'quick'. By 'short' did you mean quick or did spell check change it?

I'm sure it really doesn't matter, but this close to the finish line, I really don't want to take chances.

geaves · Apr 8th 2021

Quote from piscdi

I'm taking my time, so I'll be a few minutes.

it will be more than a few minutes for the array to rebuild ----------- hours more like

piscdi · Apr 8th 2021

piscdi · Apr 8th 2021

Hi geaves,

I opted for the quick format and the rebuild is at 2%. I know its getting late there, so I wanted to thank you for all your help. I will comply with your instructions and report back tomorrow.

Again, I can't thank you enough. Have a nice evening.

Participate now!