My Raspberry Pi crashed, then my RAID was gone at reboot.

I guess I was asking my Pi to do just a little much. I have omv_3.0.51_rpi2_rpi3.img installed on a Raspberry Pi 3. It's not pure omv_3.0.51_rpi2_rpi3.img as I've updates on it a few times.

Using two external Seagate STEL5000600 drives, I built a RAID1 and started moving all my media to the system. It's been running great for a few weeks, and has appeared to be rock solid. Most of what I use it for is a SMB server to provide shared storage between my systems. Tonight I took it to a new level, and set up an rsync job to backup one of my remote servers. The rsync of 10G wasn't an issue for it, but when i was running a tar/gz to snapshot that rsync, the pi crashed. It felt hot to the touch. I let it cool down, and when it rebooted my RAID1 was nowhere to be seen. The Pi sees the drives, and the partitions, but I can't get a RAID out of it.

As requested in the sticky thread "Degraded or missing raid array questions":

# cat /proc/mdstat
Personalities : 
unused devices: <none>

# blkid
/dev/mmcblk0: PTUUID="000b5098" PTTYPE="dos"
/dev/mmcblk0p1: SEC_TYPE="msdos" LABEL="boot" UUID="7D5C-A285" TYPE="vfat" PARTUUID="000b5098-01"
/dev/mmcblk0p2: LABEL="omv" UUID="5d18be51-3217-4679-9c72-a54e0fc53d6b" TYPE="ext4" PARTUUID="000b5098-02"
/dev/mmcblk0p3: UUID="fa36508a-b3c4-4499-b30a-711dd5994225" TYPE="ext4" PARTUUID="000b5098-03"
/dev/sdb1: PARTLABEL="Microsoft reserved partition" PARTUUID="90ad1b6d-4dda-49fb-9f51-cbc47c97efac"
/dev/sdb2: PARTLABEL="Basic data partition" PARTUUID="62d5ea9e-c7df-447e-93e6-fef40ec53719"
/dev/sda1: PARTLABEL="Microsoft reserved partition" PARTUUID="ac7b8699-c6a8-4cf1-b8c7-f72b3408f6e4"
/dev/sda2: PARTLABEL="Basic data partition" PARTUUID="75e67b85-d287-4c4e-9648-d7e5bdb823c7"

# fdisk -l


Disk /dev/mmcblk0: 7.4 GiB, 7948206080 bytes, 15523840 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000b5098


Device         Boot   Start      End Sectors  Size Id Type
/dev/mmcblk0p1         8192   122879  114688   56M  c W95 FAT32 (LBA)
/dev/mmcblk0p2       122880  7028735 6905856  3.3G 83 Linux
/dev/mmcblk0p3      7028736 15523839 8495104  4.1G 83 Linux


Disk /dev/sdb: 4.6 TiB, 5000981077504 bytes, 9767541167 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 40977E8B-402D-4A17-84BB-3CC3250B82DE


Device      Start        End    Sectors  Size Type
/dev/sdb1      34     262177     262144  128M Microsoft reserved
/dev/sdb2  264192 9767540735 9767276544  4.6T Microsoft basic data


Partition 2 does not start on physical sector boundary.




Disk /dev/sda: 4.6 TiB, 5000981077504 bytes, 9767541167 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 94E4F535-0E49-4955-8B14-7897E08EA6E7


Device      Start        End    Sectors  Size Type
/dev/sda1      34     262177     262144  128M Microsoft reserved
/dev/sda2  264192 9767540735 9767276544  4.6T Microsoft basic data


Partition 2 does not start on physical sector boundary.

# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions
# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# definitions of existing MD arrays
ARRAY /dev/md0 metadata=1.2 name=raspberrypi:RaspiRaid101 UUID=0c362bdf:6ccb7735:9eb90f3b:7be0a626
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
MAILFROM root

# mdadm --detail --scan --verbose

When I used gdisk to look at the partition tables earlier it said that my checksum was incorrect, and that I needed to rebuild from the backup copy. I did that, but there was no change.

Thoughts, suggestions?

Progress

I ran this command:

mdadm --create /dev/md0 --assume-clean --level=1 --raid-devices=2 /dev/sda /dev/sdb

The raid1 was rebuild, and I could mount it again. Whew! Right?

And then I rebooted the pi, just to make sure the fix was persistent. The raid1 is gone again. No /dev/md0

I needed to update /etc/mdadm/mdadm.confwith the new UUID

But... The RAID still isn't there on a reboot. It does come back if I run: mdadm --assemble --scan --verbose
So, progress. I didn't need to recreate the drive, but I'd like it to be available at boot, right?

A friend sent me this reference: http://projpi.com/diy-home-pro…raid-array-with-usb-hdds/
Apparently the Pi has an issue sometimes with booting faster than USB drives become available.

I added rootdelay=5 to the end of the boot command in /boot/cmdline.txt

My md raid is back at reboot!!!

I hope you don't mind my using the openmediavault forum as a sounding board while I worked through what was really a md issue.
I'll leave this here in case it can help anyone else.

1) Fair enough, but then why have the RAID functionality in the RPI image of openmediavault?

2) I did! When I first plugged the drives in openmediavault raid creation tools didn't even see the drives. I had to 'dd if=/dev/zero of=dev/sda bs=8k count=10000' and then I could create the array. I think those partitions being there are a result of me trying to fix the array by following gdisk's suggestions and rebuilding the partition table from the backup. fidsk and gdisk are back to saying the "primary GPT table is corrupt" so I think running the create stomped on those erroneous partition entries.

3) Needless to say, I didn't put in my post every little thing I tried. I tried a half dozen variations of --assemble --force commands, all with the result of 'missing superblock'. The only way I could re-create that was with --create. Let me tell you, I was not feeling confident when I issued that command.