Raid missing after restart

    • OMV 5.x (beta)
    • Resolved
    • Raid missing after restart

      Hi,

      i've restarted my omv 5 after updating it to 5.0.8.1 and configuring Plesk within Portainer.
      After restart my raid is not found anymore.
      The disks are still displayed within omv. The raid disappeared.
      Raid5 consits out of 4 disks 2.73TiB.

      What would be the next step to avoid data loss/recovery.
      Sincerly
      Chris

      cat /proc/mdstat

      Display Spoiler

      Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
      md127 : inactive sdd[1](S) sde[2](S) sdc[0](S)
      8790406536 blocks super 1.2

      unused devices: <none>




      blkid
      Display Spoiler

      /dev/sda1: UUID="ea27481b-8a34-417e-975a-3cfea6d23262" TYPE="ext4" PARTUUID="676bb235-01"
      /dev/sda5: UUID="587c3195-5b41-4013-a64a-12181686b83f" TYPE="swap" PARTUUID="676bb235-05"
      /dev/sdc: UUID="2333f40c-fe53-ef98-0f57-7f54059abd6b" UUID_SUB="5521a62c-3ebb-34ba-1006-886dda6d5291" LABEL="Openmediavault:HDD" TYPE="linux_raid_member"
      /dev/sde: UUID="2333f40c-fe53-ef98-0f57-7f54059abd6b" UUID_SUB="1a0dbe29-b168-6f90-2cc1-1c53b4d1f592" LABEL="Openmediavault:HDD" TYPE="linux_raid_member"
      /dev/sdd: UUID="2333f40c-fe53-ef98-0f57-7f54059abd6b" UUID_SUB="cb9cde4e-78b9-807c-b0cd-223dc68c602d" LABEL="Openmediavault:HDD" TYPE="linux_raid_member"
      /dev/sdb1: PARTUUID="cb32c8be-13a6-453f-a70f-3e9b89f5b0ed"


      fdisk -l | grep "Disk "
      Display Spoiler

      Disk /dev/sda: 465.8 GiB, 500107862016 bytes, 976773168 sectors
      Disk model: ST9500420AS
      Disk identifier: 0x676bb235
      Disk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
      Disk model: WDC WD30EZRX-00M
      Disk identifier: E2AF84A6-0745-447C-823A-374C7CBB1F59
      Disk /dev/sdc: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
      Disk model: WDC WD30EFRX-68E
      Disk /dev/sde: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
      Disk model: WDC WD30EFRX-68E
      Disk /dev/sdd: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
      Disk model: WDC WD30EFRX-68E


      cat /etc/mdadm/mdadm.con
      Display Spoiler

      # mdadm.conf
      #
      # !NB! Run update-initramfs -u after updating this file.
      # !NB! This will ensure that initramfs has an uptodate copy.
      #
      # Please refer to mdadm.conf(5) for information about this file.
      #

      # by default (built-in), scan all partitions (/proc/partitions) and all
      # containers for MD superblocks. alternatively, specify devices to scan, using
      # wildcards if desired.
      #DEVICE partitions containers

      # automatically tag new arrays as belonging to the local system
      HOMEHOST <system>

      # instruct the monitoring daemon where to send mail alerts
      MAILADDR root

      # definitions of existing MD arrays

      # This configuration was auto-generated on Fri, 16 Aug 2019 07:12:16 +0000 by mkconf


      mdadm --detail --scan --verbose
      Display Spoiler

      INACTIVE-ARRAY /dev/md127 num-devices=3 metadata=1.2 name=Openmediavault:HDD UUID=2333f40c:fe53ef98:0f577f54:059abd6b
      devices=/dev/sdc,/dev/sdd,/dev/sde
    • The raid is actually inactive, which would cause it not to show in the gui, the output from mdstat shows 3 out of 4 drives, as does blkid, what is also inexplicable is there is no reference to the array in mdadm.conf.

      For whatever reason /dev/sdb is missing from the array hence the output from mdstat and blkid, but the drive shows up under fdisk.

      First off try mdadm --assemble --verbose --force /dev/md127 /dev/sd[bcde] and post the output, this may return an error on /dev/sdb missing superblock.
      Raid is not a backup! Would you go skydiving without a parachute?
    • hi,

      as u said the array is back clean and degraded and visible withing omv webclient. Shares are also working.
      See beloew:

      root@Openmediavault:~# mdadm --assemble --verbose --force /dev/md127 /dev/sd[bcd]
      Display Spoiler

      mdadm: looking for devices for /dev/md127
      mdadm: /dev/sdb is identified as a member of /dev/md127, slot 0.
      mdadm: /dev/sdc is identified as a member of /dev/md127, slot 1.
      mdadm: /dev/sdd is identified as a member of /dev/md127, slot 2.
      mdadm: added /dev/sdc to /dev/md127 as 1
      mdadm: added /dev/sdd to /dev/md127 as 2
      mdadm: no uptodate device for slot 3 of /dev/md127
      mdadm: added /dev/sdb to /dev/md127 as 0
      mdadm: /dev/md127 has been started with 3 drives (out of 4).




      I've not restartet omv so far.
      So what are the next steps to readd the missing drive ?
      i've done the raid status commands and again for a full status for you:

      root@Openmediavault:~# cat /proc/mdstat
      Display Spoiler

      Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
      md127 : active raid5 sdb[0] sdd[2] sdc[1]
      8790405120 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]

      unused devices: <none>


      root@Openmediavault:~# blkid
      Display Spoiler

      /dev/sdb: UUID="2333f40c-fe53-ef98-0f57-7f54059abd6b" UUID_SUB="5521a62c-3ebb-34ba-1006-886dda6d5291" LABEL="Openmediavault:HDD" TYPE="linux_raid_member"
      /dev/sdd: UUID="2333f40c-fe53-ef98-0f57-7f54059abd6b" UUID_SUB="1a0dbe29-b168-6f90-2cc1-1c53b4d1f592" LABEL="Openmediavault:HDD" TYPE="linux_raid_member"
      /dev/sdc: UUID="2333f40c-fe53-ef98-0f57-7f54059abd6b" UUID_SUB="cb9cde4e-78b9-807c-b0cd-223dc68c602d" LABEL="Openmediavault:HDD" TYPE="linux_raid_member"
      /dev/sde1: UUID="ea27481b-8a34-417e-975a-3cfea6d23262" TYPE="ext4" PARTUUID="676bb235-01"
      /dev/sde5: UUID="587c3195-5b41-4013-a64a-12181686b83f" TYPE="swap" PARTUUID="676bb235-05"
      /dev/md127: LABEL="HDD" UUID="d82c0f35-c597-4012-8d18-c73fa9312358" TYPE="ext4"
      /dev/sda1: PARTUUID="cb32c8be-13a6-453f-a70f-3e9b89f5b0ed"


      root@Openmediavault:~# fdisk -l | grep "Disk "
      Display Spoiler

      Disk /dev/sda: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
      Disk model: WDC WD30EZRX-00M
      Disk identifier: E2AF84A6-0745-447C-823A-374C7CBB1F59
      Disk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
      Disk model: WDC WD30EFRX-68E
      Disk /dev/sdd: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
      Disk model: WDC WD30EFRX-68E
      Disk /dev/sdc: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
      Disk model: WDC WD30EFRX-68E
      Disk /dev/sde: 465.8 GiB, 500107862016 bytes, 976773168 sectors
      Disk model: ST9500420AS
      Disk identifier: 0x676bb235
      Disk /dev/md127: 8.2 TiB, 9001374842880 bytes, 17580810240 sectors


      root@Openmediavault:~# cat /etc/mdadm/mdadm.conf
      Display Spoiler

      # mdadm.conf
      #
      # !NB! Run update-initramfs -u after updating this file.
      # !NB! This will ensure that initramfs has an uptodate copy.
      #
      # Please refer to mdadm.conf(5) for information about this file.
      #

      # by default (built-in), scan all partitions (/proc/partitions) and all
      # containers for MD superblocks. alternatively, specify devices to scan, using
      # wildcards if desired.
      #DEVICE partitions containers

      # automatically tag new arrays as belonging to the local system
      HOMEHOST <system>

      # instruct the monitoring daemon where to send mail alerts
      MAILADDR root

      # definitions of existing MD arrays

      # This configuration was auto-generated on Fri, 16 Aug 2019 07:12:16 +0000 by mkconf


      root@Openmediavault:~# mdadm --detail --scan --verbose
      Display Spoiler

      ARRAY /dev/md127 level=raid5 num-devices=4 metadata=1.2 name=Openmediavault:HDD UUID=2333f40c:fe53ef98:0f577f54:059abd6b
      devices=/dev/sdb,/dev/sdc,/dev/sdd
    • Hi,

      i've used the mentioned command with sda instead of sde.
      sde is my boot disk.

      Error while doing first command. Thought the reason is because its missing anyway.
      so i've done the the second command and it seems to be working.


      Display Spoiler

      root@Openmediavault:~# mdadm --zero-superblock /dev/sda
      mdadm: Unrecognised md component device - /dev/sda
      root@Openmediavault:~# mdadm --add /dev/md127 /dev/sda
      mdadm: added /dev/sda


      Thats the result out of radi menu item within omv webclient
      Display Spoiler

      Version : 1.2
      Creation Time : Wed Aug 26 19:54:51 2015
      Raid Level : raid5
      Array Size : 8790405120 (8383.18 GiB 9001.37 GB)
      Used Dev Size : 2930135040 (2794.39 GiB 3000.46 GB)
      Raid Devices : 4
      Total Devices : 4
      Persistence : Superblock is persistent

      Update Time : Wed Sep 18 21:47:34 2019
      State : clean, degraded, recovering
      Active Devices : 3
      Working Devices : 4
      Failed Devices : 0
      Spare Devices : 1

      Layout : left-symmetric
      Chunk Size : 512K

      Consistency Policy : resync

      Rebuild Status : 0% complete

      Name : Openmediavault:HDD (local to host Openmediavault)
      UUID : 2333f40c:fe53ef98:0f577f54:059abd6b
      Events : 7430

      Number Major Minor RaidDevice State
      0 8 16 0 active sync /dev/sdb
      1 8 32 1 active sync /dev/sdc
      2 8 48 2 active sync /dev/sdd
      4 8 0 3 spare rebuilding /dev/sda
    • Lif3styl3 wrote:

      Do u have any clou how the raid is lost?
      None as 5 is beta it could be anything, I'm still not sure why mdadm.conf contains no raid arrays, but neither does mine and my 5 set up is just a test.

      Ok I've just run a command on my test machine and my mdadm conf is populated, what's the output of your cat /etc/mdadm/mdadm.conf
      Raid is not a backup! Would you go skydiving without a parachute?
    • Lif3styl3 wrote:

      hopefully the raid is already there after next reboot without any entry within that file.
      Don't reboot yet, wait for the rebuild then run the following omv-salt deploy run mdadm wait for it to complete there appears to be a delay of up to 50 secs then check the mdadm.conf again and under # definitions of existing MD arrays your array should be displayed.
      Raid is not a backup! Would you go skydiving without a parachute?
    • Hi,

      because I do not know what to do the whole day, i've ported my raid to a new system.
      That system has got a backplane.

      After connecting the disks to newly installed omv5 system the same error happend.
      I've got excatly the same result as described in my first post. The same disk left the array and the disks left within the array were busy.

      I've now done the same steps as described above any everything is fine now. But why ?!