OMV 2.0 - Raid 6 (6 x 4TB) FAILED state - Need some serious help

  • So this afternoon I get emails from OMV (which I did not see until 1/2 hour ago) stating that a drive failed, then another email another drive, then another email another drive.. basically got emails identifying 6 drive failures. The email I got just before these emails was an email from OMV telling me that my resource limit had been exceeded.


    Anyhow, fast forward to this evening. I see emails, I log on using OMV and ssh. OMV shows me raid md0 Clean, FAILED and the volume no longer shows up in the volume management area.


    ssh shows me /dev/md0 is stll mounted on the filesytem and I can see some dirs and files. Seems fine, but I don't really want anything writing to the filesystem at this point so I reboot expecting things to clean themselves up because there is a very low likelihood that all the drives failed at once (and the /dev/md0 was accessible no problem). I don't have any hot or cold spares.


    after reboot, raid isn't starting (not surprised) but I now am left with this:




    No drives are reporting any kind of hardware errors or anything like that and all show up fine in the Intel raid bios screen.



    Here is the other information:





    Question is, where do I go from here? How can I re-add the drives in a way that tells the RAID they are the original drives all good? Because I can't add them in a way where it tries to rebuild because I don't have enough drives for a rebuild.


    Help is appreciated.


    Cheers


    Update #1


    I also did this based on an article somewhere and I think that his is also a good sign I could assemble the raid again? I just have no experience with this type of recovery so I need some advice before proceeding:



    • Offizieller Beitrag

    wow, nobody has any ideas on how to recover this?

    I seem to be the only one who answers these posts and I am tired of trying to figure out why mdadm raid arrays don't start...


    mdadm --stop /dev/md0
    mdadm --assemble --verbose --force /dev/md0 /dev/[cdefgh]
    update-initramfs -u

    omv 7.0-32 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.9 | compose 7.0.9 | cputemp 7.0 | mergerfs 7.0.3


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Well then I must say not only do I appreciate your response in general, but doubly-so due to the fact that mdadm is a source of frustration. I really thought going software raid for me was a better choice than having to rely on a proprietary hardware raid solution in the event of failures. I sure hope that these things don't occur enough for me to be concerned about (or they are easily recovered).


    I will be trying this as soon as I get home from work and will post the results.


    Thanks Again.

    • Offizieller Beitrag

    I used mdadm for years on multiple systems and never had an issue. BUT, I never turn my systems off or spin down the drives or put the system to sleep. I also connected all of my drives via sata. Not sure if you do any of that.


    As for hardware raid, as long as you use a common raid card and never cheap motherboard raid, it is better than software raid.

    omv 7.0-32 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.9 | compose 7.0.9 | cputemp 7.0 | mergerfs 7.0.3


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • I made a DIY NAS. I used the Silverstone DS 8 bay enclosure with a high end ASUS mobo and an Intel sata card x8 and 6x 4TB WD reds. It's my home media machine. Never shutdown , connected to dedicated UPS. As for spin down, I thought I had set that somewhere when it was setup a year ago, but now I cannot be sure nor do I recall how you do it. My NAS is pretty active constantly, but that is not to say it has no dead time so that's possible I guess. that NAS never sleeps either.


    Now it is suspicious that a minute before these registered as failed, my box reported a high resource utilization.




    Resource limit matched Service localhost


    Date: Sat, 29 Oct 2016 14:41:39
    Action: alert
    Host: CHOMEOMV.local
    Description: loadavg(5min) of 4.5 matches resource limit [loadavg(5min)>4.0]




    Then the an email for each of the 6 drives in the array with the essentially same email time stamp



    Date: Sat, 29 Oct 2016 14:42:48 -0300 (ADT)


    This is an automatically generated mail message from mdadm
    running on CHOMEOMV


    A FailSpare event had been detected on md device /dev/md0.


    It could be related to component device /dev/sda.


    Faithfully yours, etc.


    P.S. The /proc/mdstat file currently contains the following:


    Personalities : [raid6] [raid5] [raid4]
    md0 : active raid6 sda[6](F) sdf[5](F) sde[4](F) sdd[3](F) sdc[2](F) sdb[1](F)
    15623215104 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/0] [______]
    unused devices: <none>

  • I seem to be the only one who answers these posts and I am tired of trying to figure out why mdadm raid arrays don't start...
    mdadm --stop /dev/md0
    mdadm --assemble --verbose --force /dev/md0 /dev/sd[cdefgh]
    update-initramfs -u


    Well, I wish I could buy you a <insert your beverage of choice>.


    Thanks for taking the time to help me out. I really appreciate it.



    • Offizieller Beitrag

    Looks like it is working. cat /proc/mdstat will tell you for sure.

    omv 7.0-32 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.9 | compose 7.0.9 | cputemp 7.0 | mergerfs 7.0.3


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Looks like it is working. cat /proc/mdstat will tell you for sure.


    Gotcha. This looks good.


    Code
    root@CHOMEOMV:/tmp# cat /proc/mdstat
    Personalities : [raid6] [raid5] [raid4]
    md0 : active raid6 sda[6] sdf[5] sde[4] sdd[3] sdc[2] sdb[1]
          15623215104 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/6] [UUUUUU]
    
    
    unused devices: <none>
    • Offizieller Beitrag

    Is there any FAQ or something else on MDADM usage ? i haven't found one (at least not user friendly enough) .

    https://raid.wiki.kernel.org/index.php/RAID_setup

    omv 7.0-32 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.9 | compose 7.0.9 | cputemp 7.0 | mergerfs 7.0.3


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!