My Raspberry Pi crashed, then my RAID was gone at reboot.

    • OMV 3.x
    • Resolved
    • My Raspberry Pi crashed, then my RAID was gone at reboot.

      I guess I was asking my Pi to do just a little much. I have omv_3.0.51_rpi2_rpi3.img installed on a Raspberry Pi 3. It's not pure omv_3.0.51_rpi2_rpi3.img as I've updates on it a few times.

      Using two external Seagate STEL5000600 drives, I built a RAID1 and started moving all my media to the system. It's been running great for a few weeks, and has appeared to be rock solid. Most of what I use it for is a SMB server to provide shared storage between my systems. Tonight I took it to a new level, and set up an rsync job to backup one of my remote servers. The rsync of 10G wasn't an issue for it, but when i was running a tar/gz to snapshot that rsync, the pi crashed. It felt hot to the touch. I let it cool down, and when it rebooted my RAID1 was nowhere to be seen. The Pi sees the drives, and the partitions, but I can't get a RAID out of it.

      As requested in the sticky thread "Degraded or missing raid array questions":

      Source Code

      1. # cat /proc/mdstat
      2. Personalities :
      3. unused devices: <none>






      Source Code

      1. # blkid
      2. /dev/mmcblk0: PTUUID="000b5098" PTTYPE="dos"
      3. /dev/mmcblk0p1: SEC_TYPE="msdos" LABEL="boot" UUID="7D5C-A285" TYPE="vfat" PARTUUID="000b5098-01"
      4. /dev/mmcblk0p2: LABEL="omv" UUID="5d18be51-3217-4679-9c72-a54e0fc53d6b" TYPE="ext4" PARTUUID="000b5098-02"
      5. /dev/mmcblk0p3: UUID="fa36508a-b3c4-4499-b30a-711dd5994225" TYPE="ext4" PARTUUID="000b5098-03"
      6. /dev/sdb1: PARTLABEL="Microsoft reserved partition" PARTUUID="90ad1b6d-4dda-49fb-9f51-cbc47c97efac"
      7. /dev/sdb2: PARTLABEL="Basic data partition" PARTUUID="62d5ea9e-c7df-447e-93e6-fef40ec53719"
      8. /dev/sda1: PARTLABEL="Microsoft reserved partition" PARTUUID="ac7b8699-c6a8-4cf1-b8c7-f72b3408f6e4"
      9. /dev/sda2: PARTLABEL="Basic data partition" PARTUUID="75e67b85-d287-4c4e-9648-d7e5bdb823c7"



      Source Code

      1. # fdisk -l
      2. Disk /dev/mmcblk0: 7.4 GiB, 7948206080 bytes, 15523840 sectors
      3. Units: sectors of 1 * 512 = 512 bytes
      4. Sector size (logical/physical): 512 bytes / 512 bytes
      5. I/O size (minimum/optimal): 512 bytes / 512 bytes
      6. Disklabel type: dos
      7. Disk identifier: 0x000b5098
      8. Device Boot Start End Sectors Size Id Type
      9. /dev/mmcblk0p1 8192 122879 114688 56M c W95 FAT32 (LBA)
      10. /dev/mmcblk0p2 122880 7028735 6905856 3.3G 83 Linux
      11. /dev/mmcblk0p3 7028736 15523839 8495104 4.1G 83 Linux
      12. Disk /dev/sdb: 4.6 TiB, 5000981077504 bytes, 9767541167 sectors
      13. Units: sectors of 1 * 512 = 512 bytes
      14. Sector size (logical/physical): 512 bytes / 4096 bytes
      15. I/O size (minimum/optimal): 4096 bytes / 4096 bytes
      16. Disklabel type: gpt
      17. Disk identifier: 40977E8B-402D-4A17-84BB-3CC3250B82DE
      18. Device Start End Sectors Size Type
      19. /dev/sdb1 34 262177 262144 128M Microsoft reserved
      20. /dev/sdb2 264192 9767540735 9767276544 4.6T Microsoft basic data
      21. Partition 2 does not start on physical sector boundary.
      22. Disk /dev/sda: 4.6 TiB, 5000981077504 bytes, 9767541167 sectors
      23. Units: sectors of 1 * 512 = 512 bytes
      24. Sector size (logical/physical): 512 bytes / 4096 bytes
      25. I/O size (minimum/optimal): 4096 bytes / 4096 bytes
      26. Disklabel type: gpt
      27. Disk identifier: 94E4F535-0E49-4955-8B14-7897E08EA6E7
      28. Device Start End Sectors Size Type
      29. /dev/sda1 34 262177 262144 128M Microsoft reserved
      30. /dev/sda2 264192 9767540735 9767276544 4.6T Microsoft basic data
      31. Partition 2 does not start on physical sector boundary.
      Display All

      Source Code

      1. # cat /etc/mdadm/mdadm.conf
      2. # mdadm.conf
      3. #
      4. # Please refer to mdadm.conf(5) for information about this file.
      5. #
      6. # by default, scan all partitions (/proc/partitions) for MD superblocks.
      7. # alternatively, specify devices to scan, using wildcards if desired.
      8. # Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
      9. # To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
      10. # used if no RAID devices are configured.
      11. DEVICE partitions
      12. # auto-create devices with Debian standard permissions
      13. CREATE owner=root group=disk mode=0660 auto=yes
      14. # automatically tag new arrays as belonging to the local system
      15. HOMEHOST <system>
      16. # definitions of existing MD arrays
      17. ARRAY /dev/md0 metadata=1.2 name=raspberrypi:RaspiRaid101 UUID=0c362bdf:6ccb7735:9eb90f3b:7be0a626
      18. # instruct the monitoring daemon where to send mail alerts
      19. MAILADDR root
      20. MAILFROM root
      Display All



      Source Code

      1. # mdadm --detail --scan --verbose


      When I used gdisk to look at the partition tables earlier it said that my checksum was incorrect, and that I needed to rebuild from the backup copy. I did that, but there was no change.

      Thoughts, suggestions?
    • Progress

      I ran this command:

      Source Code

      1. mdadm --create /dev/md0 --assume-clean --level=1 --raid-devices=2 /dev/sda /dev/sdb

      The raid1 was rebuild, and I could mount it again. Whew! Right?

      And then I rebooted the pi, just to make sure the fix was persistent. The raid1 is gone again. No /dev/md0

      I needed to update /etc/mdadm/mdadm.confwith the new UUID

      But... The RAID still isn't there on a reboot. It does come back if I run: mdadm --assemble --scan --verbose
      So, progress. I didn't need to recreate the drive, but I'd like it to be available at boot, right?

      A friend sent me this reference: projpi.com/diy-home-projects-w…raid-array-with-usb-hdds/
      Apparently the Pi has an issue sometimes with booting faster than USB drives become available.

      I added rootdelay=5 to the end of the boot command in /boot/cmdline.txt

      My md raid is back at reboot!!!

      I hope you don't mind my using the openmediavault forum as a sounding board while I worked through what was really a md issue. :)
      I'll leave this here in case it can help anyone else.

      The post was edited 2 times, last by ChrisKnight: Solving my own problems.. ().

    • Just a few comments...

      1 - I recommend against using raid on an RPi. Too many bad things have happen probably because of the usb issues.

      2 - I would've wiped those drives before creating the array. You still have microsoft partitions on them.

      3 - Using the --create command on an existing array is very dangerous. I would say half the time, people lose data doing that. I would have used --assemble --force.

      4 - Glad it is working for you and it is always good to hear a positive story. Maybe it will help others.
      omv 4.1.15 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • 1) Fair enough, but then why have the RAID functionality in the RPI image of openmediavault?

      2) I did! When I first plugged the drives in openmediavault raid creation tools didn't even see the drives. I had to 'dd if=/dev/zero of=dev/sda bs=8k count=10000' and then I could create the array. I think those partitions being there are a result of me trying to fix the array by following gdisk's suggestions and rebuilding the partition table from the backup. fidsk and gdisk are back to saying the "primary GPT table is corrupt" so I think running the create stomped on those erroneous partition entries.

      3) Needless to say, I didn't put in my post every little thing I tried. I tried a half dozen variations of --assemble --force commands, all with the result of 'missing superblock'. The only way I could re-create that was with --create. Let me tell you, I was not feeling confident when I issued that command.
    • ChrisKnight wrote:

      but then why have the RAID functionality in the RPI image of openmediavault?
      Because I don't want to create/maintain a custom OMV package. And it is just a suggestion. raid works on the rpi but it is slow and has issues. I don't like raid on usb devices on any platform.

      ChrisKnight wrote:

      I was not feeling confident when I issued that command.
      I never feel confident when I tell people to use that command (happens very rarely).
      omv 4.1.15 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!