OMV Raid Missing after reboot

1st Official Post

    Hi,


    on OMV Arrakis I have two RAIDs, one consisting of 2 WD Red 4TB drives and another consisting of 4 WD Red 12TB drives. Since I was not using the Raid 2x4 I turned off OMV and disconnected the power to the two disks, after which I restarted the system. After the restart, the 4x12 Raid was missing. So I turned off and reconnected the two discs of the Raid 2x4 and restarted but the Raid 4x12 is always missing.




    How can I solve?


    Thank you very much.


    zerozenit

    OMV 6.9.16-1 (Shaitan) - Debian 11 (Bullseye) - Linux 6.1.0-0.deb11.21-amd64

    OMV Plugins: backup 6.1.1 | compose 6.11.3 | cputemp 6.1.3 | flashmemory 6.2 | ftp 6.0.7-1 | kernel 6.4.10 | nut 6.0.7-1 | omvextrasorg 6.3.6 | resetperms 6.0.3 | sharerootfs 6.0.3-1

    ASRock J5005-ITX - 16GB DDR4 - 4x WD RED 12TB (Raid10), 2x WD RED 4TB (Raid1) [OFF], Boot from SanDisk Ultra Fit Flash Drive 32GB - Fractal Design Node 304

    • Official Post
    Quote from zerozenit

    So I turned off and reconnected the two discs of the Raid 2x4 and restarted but the Raid 4x12 is always missing.

    Interesting, odd behaviour;


    Your mirror is active but in auto-read-only, this -> mdadm --readwrite /dev/md1 should correct that.


    /dev/md0 the 4x12TB is inactive so you'll have to stop it before reassembling;


    mdadm --stop /dev/md0 wait for confirmation before continuing


    mdadm --assemble --force --verbose /dev/md0 /dev/sd[abfg]


    An array usually only becomes inactive if there has been a power failure, the system has not been shut down gracefully or a drive has been removed for replacement.


    Before you ask I have no idea what has caused this odd behaviour, but it could be hardware related.

    Raid is not a backup! Would you go skydiving without a parachute?


    OMV 6x amd64 running on an HP N54L Microserver

    Like 1

    Hi geaves,


    Thanks, you were very helpful, I reassembled and this is the output:

    What do you advise me to do? The removed sdg disk appears to be in good condition.

    Also I would like to turn off md1 and remove (physically) the power to the two 4TB disks, what is the correct procedure to do this?

    Thank you

    OMV 6.9.16-1 (Shaitan) - Debian 11 (Bullseye) - Linux 6.1.0-0.deb11.21-amd64

    OMV Plugins: backup 6.1.1 | compose 6.11.3 | cputemp 6.1.3 | flashmemory 6.2 | ftp 6.0.7-1 | kernel 6.4.10 | nut 6.0.7-1 | omvextrasorg 6.3.6 | resetperms 6.0.3 | sharerootfs 6.0.3-1

    ASRock J5005-ITX - 16GB DDR4 - 4x WD RED 12TB (Raid10), 2x WD RED 4TB (Raid1) [OFF], Boot from SanDisk Ultra Fit Flash Drive 32GB - Fractal Design Node 304

    • Official Post
    Quote from zerozenit

    Also I would like to turn off md1 and remove (physically) the power to the two 4TB disks, what is the correct procedure to do this


    What do you advise me to do? The removed sdg disk appears to be in good condition.

    To remove md1;


    Storage Manangement -> File Systems select the array md1 on the menu click Unmount then Apply when it appears

    Raid Management from the menu select the array md1 on the menu click Delete then Apply when it appears


    That should remove md1 from your system and you can remove the power from the drives, or go to Storage -> Disks select each drive in turn from the array and wipe them, then remove the power.


    Please ensure you select the correct array to be removed


    This line mdadm: added /dev/sdg to /dev/md0 as 3 (possibly out of date) from the output that usually suggests a missing superblock, run mdadm --detail /dev/md0 at the end of that output it should display that drive as removed, but show the other 3 as active sync.

    If that's the case go to Storage -> Disks select the drive sdg click wipe on the menu and select short, when complete, Raid Management select the array md0, on the menu click recover, from the dialog box select the drive you've just wiped click OK and the raid should rebuild. Due to the size of the drives this will take some time.


    Going back to your first post and re reading, removing the power from md1 then re starting the server threw a curve ball at mdadm, it had no idea where that array had gone and therefore started the errors you now see.


    BTW in case any of this goes pear shaped do you have a backup (pear shaped = goes wrong)

    Hi geaves,


    i rebuilt the raid, the process ran smoothly and the raid md0 was finally clean again. After following your instructions for md1 I turned off the server and removed the power to the two md1 disks.


    I thought it was all fixed, but when I went to reboot the server I noticed that problems booting, it seems there are problems starting the file system of md0, I attach the screenshot.



    I am really sad and afraid for my data, I hope you can help me solve this situation too.


    Thank you.

    OMV 6.9.16-1 (Shaitan) - Debian 11 (Bullseye) - Linux 6.1.0-0.deb11.21-amd64

    OMV Plugins: backup 6.1.1 | compose 6.11.3 | cputemp 6.1.3 | flashmemory 6.2 | ftp 6.0.7-1 | kernel 6.4.10 | nut 6.0.7-1 | omvextrasorg 6.3.6 | resetperms 6.0.3 | sharerootfs 6.0.3-1

    ASRock J5005-ITX - 16GB DDR4 - 4x WD RED 12TB (Raid10), 2x WD RED 4TB (Raid1) [OFF], Boot from SanDisk Ultra Fit Flash Drive 32GB - Fractal Design Node 304

  • zerozenit

    Added the Label OMV 4.x
    • Official Post
    Quote from zerozenit

    I am really sad and afraid for my data

    You should be, it will either recover or you will lose the lot :(


    The error states that it failed to complete a file system check on that array, I'm assuming although not completely sure that led to the other dependency errors.

    How to check or look for the cause is in the actual error, the same goes for the emergency mode error, Control-D will get you to a login where you would login in as root then run joutnalctl -xb and look for errors. You could just try to reboot from that login and see if the system corrects itself.

    Other options are

    booting from a systemrescuecd

    disconnect the array and install OMV5 on another USB Flash Drive, then connect the array once the system is set up


    My concern here is that there could one possibly two failing drives which is preventing fsck from running, you have no way of knowing until you can locate the specific error.


    Your new problem is somewhat out of my comfort zone as I have never had this problem, but if this gets sorted you may want to consider another option other than Raid5 with 12TB drives!

    Quote from geaves

    You should be, it will either recover or you will lose the lot :(

    Hi geaves,


    I fixed the file system problems using the manual command a couple of times:

    Code
    fsck /dev/md0

    All the important data is still there, I only had problems with some Docker containers, but I quickly rebuilt them.

    Now everything runs very well and the system seems very stable.

    Many thanks for your fruitful help, I really appreciate it! :)

    OMV 6.9.16-1 (Shaitan) - Debian 11 (Bullseye) - Linux 6.1.0-0.deb11.21-amd64

    OMV Plugins: backup 6.1.1 | compose 6.11.3 | cputemp 6.1.3 | flashmemory 6.2 | ftp 6.0.7-1 | kernel 6.4.10 | nut 6.0.7-1 | omvextrasorg 6.3.6 | resetperms 6.0.3 | sharerootfs 6.0.3-1

    ASRock J5005-ITX - 16GB DDR4 - 4x WD RED 12TB (Raid10), 2x WD RED 4TB (Raid1) [OFF], Boot from SanDisk Ultra Fit Flash Drive 32GB - Fractal Design Node 304

  • zerozenit

    Added the Label resolved

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login