RAID1 disk failed after power outage. File system unavailable.

  • I set up OMV over the summer to get all our family photos off facebook. I attached 2 1TB hard drives to my RPi3b+, and set them up to mirror each other (RAID 1). Shortly after that we had a power outage, and I was unable to access OMV from my computer. I wasn't terribly worried at the time, but after investigating, it looks like the disk does not show up under "Physical Disks" in the OMV GUI (edit: it shows up now that I've plugged that disk directly into the pi and not the powered USB hub). When I go to "File Systems", I see the file system, but OMV says it is not mounted and is "missing". I set the drives up to mirror each other specifically in the event a drive dies (they both power on and spin up when plugged into the pi, and when I plug the problematic disk into a manjaro machine, it can see it with the correct name, but cannot mount it). I've read through forum questions including this one: Missing Disk and this one: Software to read data from disk if OMV crash but nothing seems to answer the question of HOW to recover the data once one of the drives goes bad.


    If anyone has any word of encouragement or a guide that will help, I would be extremely grateful. I've been at this for hours, and my wife is holed up in the bedroom crying her eyes out over the potential lost photos of our family.

  • I attached 2 1TB hard drives to my RPi3b+, and set them up to mirror each other (RAID 1)

    Using a Raid option on USB drives was removed in OMV4 for the reason you are experiencing at the moment, I'm guessing you set this up via the cli after reading something on the 'net'

    I set the drives up to mirror each other specifically in the event a drive dies

    This is the normal misnomer that end users believe a Raid is for

    When I go to "File Systems", I see the file system, but OMV says it is not mounted and is "missing"

    :/ the drive is part of an array you can't mount it on it's own


    Technically your Raid should still be accessible as one drive is working, therefore the data should be there, shh into the Pi and post the output of the following;

    blkid

    cat /proc/mdstat

    if you use the Pi user then you will probably have to add sudo in front of those

    Raid is not a backup! Would you go skydiving without a parachute?


    OMV 6x amd64 running on an HP N54L Microserver

  • Thank you for responding! I used putty to ssh into the pi and logged in as the user, then ran sudo blkid with the following result:

    Code
    Jonathan@raspberrypi:/$ sudo blkid
    /dev/mmcblk0p1: LABEL="boot" UUID="70CD-BC89" TYPE="vfat" PARTUUID="c7ed32fe-01"
    /dev/mmcblk0p2: UUID="8a9074c8-46fe-4807-8dc9-8ab1cb959010" TYPE="ext4" PARTUUID="c7ed32fe-02"
    /dev/sda: UUID="d6e65450-f0ef-37f9-8706-cf2144a5e381" UUID_SUB="6f5c3497-a2aa-e8c0-6077-c35f49e6f49e" LABEL="raspberrypi:TheCloud" TYPE="linux_raid_member"
    /dev/sdb: UUID="d6e65450-f0ef-37f9-8706-cf2144a5e381" UUID_SUB="902f5e94-06a3-8c90-2342-7305a38ac2ef" LABEL="raspberrypi:TheCloud" TYPE="linux_raid_member"
    /dev/mmcblk0: PTUUID="c7ed32fe" PTTYPE="dos"

    The next command only produces this:

    Code
    Jonathan@raspberrypi:/$ sudo cat /proc/mdstat
    Personalities :
    unused devices: <none>



    Technically your Raid should still be accessible as one drive is working

    It seems now that maybe both drives are "working". I'm not entirely sure. When I check physical devices in the GUI, I see both drives:

    This is what I see in file systems though:


    Thank you for all your help!

  • Jonathan@raspberrypi:/$ sudo cat /proc/mdstat
    Personalities :
    unused devices: <none>

    This is not good, that tells me there are no raid arrays within your set up!! although the two drives in question are showing as inux_raid_member


    post the output of:


    cat /mdadm/mdadm.conf

    cat /etc/fstab

    Raid is not a backup! Would you go skydiving without a parachute?


    OMV 6x amd64 running on an HP N54L Microserver

  • Probably also not good...

    Code
    Jonathan@raspberrypi:/$ sudo cat /mdam/mdam.conf
    cat: /mdam/mdam.conf: No such file or directory
    Code
    Jonathan@raspberrypi:/$ sudo cat /etc/fstab
    proc            /proc           proc    defaults          0       0
    /dev/mmcblk0p1  /boot           vfat    defaults          0       2
    /dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
    # a swapfile is not a swap partition, no line here
    #   use  dphys-swapfile swap[on|off]  for that
    tmpfs           /tmp            tmpfs   defaults        0       0
    # >>> [openmediavault]
    /dev/disk/by-id/md-name-raspberrypi:TheCloud /srv/dev-disk-by-id-md-name-raspberrypi-TheCloud ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
    # <<< [openmediavault]

    When I initially set this up, I used the RAID Management panel in the GUI and selected "Mirror" in the level.

  • Probably also not good...

    No, I'm not going to ask you how you did this, the only shining light is the fstab information.

    When I initially set this up, I used the RAID Management panel in the GUI and selected "Mirror" in the level

    The use of USB drives for Raid was removed in OMV4, and has never been reinstated, what version of OMV are you using


    Ok try this


    mdadm --assemble --force --verbose /dev/md0 /dev/sd[ab]

    Raid is not a backup! Would you go skydiving without a parachute?


    OMV 6x amd64 running on an HP N54L Microserver

  • The use of USB drives for Raid was removed in OMV4, and has never been reinstated, what version of OMV are you using

    Looks like the version is 3.0.100

    Ok try this


    mdadm --assemble --force --verbose /dev/md0 /dev/sd[ab]

    This could be a problem... doesn't look like mdam is installed, and when I try to use apt update, it appears my version of raspbian is no longer supported, and I can't retrieve the packages. I haven't been a Linux user since college (~2007), so a lot of this is going over my head. I tried searching around to replace the repositories, but am not really having any luck.


    Is there any possibility of pulling the working drive from the pi and accessing it via a windows or linux machine?

  • Is there any possibility of pulling the working drive from the pi and accessing it via a windows or linux machine

    No because they are part of a raid, only choice here is to get another sd card and follow the instructions here as your drives have a linux raid signature on them there is a good chance of recovery. If you do this do not have the drives plugged in until the system is set up, BTW do you know what version the Pi is, OMV5 will only work on a 2B or above.


    Another alternative is to use a workstation or laptop, but install a blank drive to install OMV on to then connect the usb drives.

    Raid is not a backup! Would you go skydiving without a parachute?


    OMV 6x amd64 running on an HP N54L Microserver

  • No because they are part of a raid, only choice here is to get another sd card and follow the instructions here as your drives have a linux raid signature on them there is a good chance of recovery.

    Sorry for being so ignorant about all this... is this as simple as "plug and play"? If I put OMV 5 on a new card, pop it into the pi, and start things up, should I be able to "just access" the drives given I have my passwords? Would things potentially be quicker installing OMV5 on a virtual machine and passing the drives through windows?

    BTW do you know what version the Pi is, OMV5 will only work on a 2B or above.

    This pi is a 3B+

    I bought it on a whim years and years ago at a MicroCenter, but never had a project I was really interested in until I'd heard about OMV.

  • Would things potentially be quicker installing OMV5 on a virtual machine and passing the drives through windows

    :/ I have tested OMV using windows hyper-v on W10 and it does work, I know you can pass through drives, or rather select a drive when adding it to the hyper-v guest OS, but I have never attempted using usb drives, might be worth a shot.


    HOWEVER, initially try this with one drive only, if that works and you can see it storage -> disks that's a start, shutdown OMV add the other, don't forget when you plug those into your windows machine it will want you to format them, obviously this would be a bad idea :)


    If you get OMV in hyper-v, then update it, set your time zone etc, you'd be surprised what doesn't work if you don't.


    TBH I like your plan, better than getting another sd card, if it works you'll need ssh access to OMV, also once you have both drives visible in OMV reboot, it just might find the raid signatures on the drives.


    If this doesn't work then it's the RPi approach on a new SD

    Raid is not a backup! Would you go skydiving without a parachute?


    OMV 6x amd64 running on an HP N54L Microserver

  • I know you can pass through drives, or rather select a drive when adding it to the hyper-v guest OS, but I have never attempted using usb drives, might be worth a shot.

    I feel like I'm finally getting somewhere! It took a bit of time and effort, but i got hyper-v working, installed OMV 5 on it, brought it online, and connected the first drive to the VM:

    forum.openmediavault.org/wsc/index.php?attachment/22693/

    As you can see, the expansion drive is recognized by the VM.


    After that I tried your suggestion of shutting down OMV, plugging in the second drive, and booting up again but encountered all sorts of Hyper-V errors. Eventually I just deleted the VM and started over, and now I have both drives showing up:

    At this point I'm a little apprehensive about what to do. The one labeled "cloud" looks like I can mount it in the file systems utility:

    I'm not really sure what this will do though. The other two drives do not show a mount or unmount option available.

    I'm seeing this in the RAID Management settings where on the pi I saw nothing:

  • Well done, this looks more than promising :) at least I now know you can use usb drives in a hyper-v

    At this point I'm a little apprehensive about what to do. The one labeled "cloud" looks like I can mount it in the file systems utility

    Do so, that's your Raid file system

    The other two drives do not show a mount or unmount option available

    They won't as they are your Raid set up and get 'presented' to the file system as 'one'

    I'm seeing this in the RAID Management settings where on the pi I saw nothing

    That's correct, due to the signatures on the drives (blkid) mdadm in OMV has detected them and presented the array to the system, the array is clean meaning there are no missing drives, and it displays the type (level).


    If that mounts and there is no reason why it shouldn't you have two choices, a) setup your shared folders again and then add them to smb, b) install winscp on your windows machine and use sftp to browse the file system of OMV and get to your data that way.


    Then, if the above works you have some decisions to make as to how you want to proceed, for what you are doing the RPi is OK, but using single external drives is not a good idea you really need a single storage unit. If you're prone to power outages a UPS is a must have if your Pi is always on.

    Raid is not a backup! Would you go skydiving without a parachute?


    OMV 6x amd64 running on an HP N54L Microserver

  • at least I now know you can use usb drives in a hyper-v

    :D hey now cut me some slack! This was actually my first time even trying a VM. Clearly I have a long road ahead. After a few bumps along the way, I was able to set up a user, but when I try to rebuild my shared folders, I have no option for a device. I'll click the dropdown and it doesn't even give me blank options. Just nothing. I thought "no big deal, I'll use winscp instead". I'm able to use the user credentials to get in (which don't work if I try to access the drive over network files in explorer), but when I navigate to the drive I get a "no such file or directory" error. Here's what it looks like in OMV:
    (you can see the device is called dev/md127)

    And here's the winscp error:


    That's the same problem I was getting when I tried getting into the file system on the pi using winscp.


    It does look like RAID Management sees the device:

    If you're prone to power outages a UPS is a must have if your Pi is always on.

    We probably get one outage per year... this year's was particularly bad and knocked us out for 3 full days in the heat of the summer. You're right though: if I plan on saving this data correctly, I should be willing to cough up the dough.

  • And here's the winscp error:

    :) you'll find the array under /srv check there and see if the data is there


    hey now cut me some slack! This was actually my first time even trying a VM

    ^^ it can be tough learning curve


    Hm your power outages aren't that bad, we very rarely get them, hence I don't use a UPS, you might also consider using an alternative to two single drives, something like this but you don't use the raid option, there are better ways.

    Raid is not a backup! Would you go skydiving without a parachute?


    OMV 6x amd64 running on an HP N54L Microserver

  • you'll find the array under /srv check there and see if the data is there

    It's there! Thank you for saving my marriage ^^


    I really appreciate all the guidance you gave me here and all the help you've given others on the forums (pretty much every question I had was answered by you in either this thread or another that I looked up). Hopefully this thread can help other doofuses like me who installed an older version of OMV and thought RAID was a good way to backup files (now I know it's like skydiving without a parachute).

  • geaves

    Added the Label resolved

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!