Lost RAID, Lost Disk after Shut down.

  • Hello all,


    Fist, I'm still a nubie so please bare with me if I seem slow understanding what you are saying.


    The other day we had some pretty intense storms come through my area so I shut down the server in hopes of preventing issues with it. My server is hooked to a APU and there were no power surges nor were there any brown-outs or a power loss. After the storms passed, I started up the server and noticed PLEX was having issues. I figured it was something minor and went to bed with hopes to get to it later. When I finally did get back to it, the Plex server couldn't be found. Thinking this was still a Plex issue I did some searching and found that that if there was a Plex update that could cause the issue. So I re-installed Plex. I then went to populate my movies, I noticed all my movies etc were missing. Still thinking this was a Plex issue I broke out my backup and started to load my movies from the back up. Basically hooked up the USB device and used windows to copy from the storage device to the movie folders on the server...until I got to all the kids movies. I then got the error "Not enough space." I was like WTF no way I've used 10 TB of space, I then went back to OMV to look for errors and noticed my raid 5 was gone.


    I continued to look at the settings and config in OMV and noticed that I'm missing a disk. This was not my first thought because I have a 3 disk RAID 5, with 2 disks being less than 6 months old and the third being 5 weeks old. Here is my set up:


    I have a Ryzen 5400G with 16 GB ram. I have a HP 250GB SSD for the OS and 3 Seagate Iron Wolf 6 TB NAS hard drives. My system has been running flawlessly for the last 5 weeks.


    I've done a scan to see if the third disk can be found in the system and I'm thinking the drive is done because OMV can't seem to find it (I'm not sure which one yet because I haven't opened up the case).


    I looked in the forum and didn't seen anything that jumped out at me as being my issue, except maybe one, but It was talk way over my understanding.


    Here is the info I was asked for when posting in here, I hope I got it correct.

    I am not sure what all that means, I've read through some of it, and if I understand it correctly it looks like I'm only showing the SSD and 2 of the 3 NAS drives and no Raid.


    So my question is what are my next steps? I know I will need to pull the crapped drive out and insert another, but will that reinstate my Raid 5? I'm curious why the Raid doesn't show "degraded" and allow me to access my information. Which is the way I understood Raid 5 to work with one dead disk. So is all my information lost?


    I did have some stuff which I hadn't backed up if it's lost oh well (but my daughter will be ticked off since it was her pix).


    Your help is greatly appreciated, so thank you in advance.

    If you can't be good, be good at it.


    System Information

    Version

    6.8.0-1 (Shaitan)

    Processor

    AMD Ryzen 5 3400G with Radeon Vega Graphics

    Kernel

    Linux 6.2.16-4-bpo11-pve

    Memory

    16 gig

    Video Card

    NVDIA Quadro P600 (for Plex HW transcoding)

    • Offizieller Beitrag

    I'm curious why the Raid doesn't show "degraded" and allow me to access my information. Which is the way I understood Raid 5 to work with one dead disk. So is all my information lost

    As far as mdadm is concerned a drive has been pulled (physically), mdadm is not hot swap you have to tell it what to do, the raid is currently inactive and the data should still be on the two remaining drives. Can you post the output of mdadm --detail /dev/md0

  • Sure: Here is what is says:


    What the heck, the drive is back? I shut it down last night and restarted it to ssh into it to get the info requested, and not all the drives are there... I'm confused.

    If you can't be good, be good at it.


    System Information

    Version

    6.8.0-1 (Shaitan)

    Processor

    AMD Ryzen 5 3400G with Radeon Vega Graphics

    Kernel

    Linux 6.2.16-4-bpo11-pve

    Memory

    16 gig

    Video Card

    NVDIA Quadro P600 (for Plex HW transcoding)

    Einmal editiert, zuletzt von billyp1970 ()

    • Offizieller Beitrag

    I'm curious why it thinks the drive has been pulled. Can the drive fail that catastrophically that it doesn't even register

    To answer the first part, your system can't 'see' it neither can mdadm, yes to part two.


    Now I'm scratching my head ?( because according to the above output the array is fine, have you rebooted since posting

  • Yes I've rebooted a third time. I rebooted it last night to see if that would take care of the issue (isn't the running joke in IT if it isn't working reboot it?) and after it was shutdown all night I started it up to ssh into it to get the requested info. Now the drive seems to be there, along with all the info...so far.

    If you can't be good, be good at it.


    System Information

    Version

    6.8.0-1 (Shaitan)

    Processor

    AMD Ryzen 5 3400G with Radeon Vega Graphics

    Kernel

    Linux 6.2.16-4-bpo11-pve

    Memory

    16 gig

    Video Card

    NVDIA Quadro P600 (for Plex HW transcoding)

  • Geaves, thank you for your help. But in your expert opinion should I be worried about a failing drive? Or do you think this is just a quirk and I'll be ok. Like I said originally, the two oldest drives are only about 6 months old and the newest one is 5 weeks old. All three are Seagate Iron Wolf 6TB NAS drives.

    If you can't be good, be good at it.


    System Information

    Version

    6.8.0-1 (Shaitan)

    Processor

    AMD Ryzen 5 3400G with Radeon Vega Graphics

    Kernel

    Linux 6.2.16-4-bpo11-pve

    Memory

    16 gig

    Video Card

    NVDIA Quadro P600 (for Plex HW transcoding)

    • Offizieller Beitrag

    isn't the running joke in IT if it isn't working reboot it

    ^^ before running round in ever decreasing circles.


    TBH I would login to the WebUI a few times a week for a while and see if it happens again, it could be a one of the drives is failing due to intermittent power loss, or it could be m'board or power supply related. It's not unheard of for a new drive to go 'tits up' (IT technical term) but it can happen. But you should trust your UPS that's what it's there for. :thumbup:

  • 'tits up' (IT technical term)

    What does that mean? Or is it better, not to ask this question?

    Or do you think this is just a quirk and I'll be ok.

    In any case I would check the correct seating of all connectors (data and power) on the drive and on the motherboard side.

    OMV 3.0.100 (Gray style)

    ASRock Rack C2550D4I C0-stepping - 16GB ECC - 6x WD RED 3TB (ZFS 2x3 Striped RaidZ1) - Fractal Design Node 304 -

    3x WD80EMAZ Snapraid / MergerFS-pool via eSATA - 4-Bay ICYCube MB561U3S-4S with fan-mod

  • In any case I would check the correct seating of all connectors (data and power) on the drive and on the motherboard side.

    Defiantly a good idea, I'll check that.

    It's like pear shaped but more terminal, it's a Brit saying meaning; dead, inoperable, bust.


    "My washing machine has gone tits up" :D

    I never realized that was a British saying, I've been saying it for years, I won't go into how I thought the saying came to be...LOL

    TBH I would login to the WebUI a few times a week for a while and see if it happens again, it could be a one of the drives is failing due to intermittent power loss, or it could be m'board or power supply related. It's not unheard of for a new drive to go 'tits up' (IT technical term) but it can happen. But you should trust your UPS that's what it's there for. :thumbup:

    Thank you for the suggestion..Wilco.

    If you can't be good, be good at it.


    System Information

    Version

    6.8.0-1 (Shaitan)

    Processor

    AMD Ryzen 5 3400G with Radeon Vega Graphics

    Kernel

    Linux 6.2.16-4-bpo11-pve

    Memory

    16 gig

    Video Card

    NVDIA Quadro P600 (for Plex HW transcoding)

  • I came back to the server just a little bit ago and that same drive had "disappeared" again, so for troubleshooting I disassembled my server, and swapped the cables on one "good" drive to the "bad" drive and vise-a-versa. Now I can't reach my server at all. I've checked both cables attached to the rear of the machine (ethernet and power cable) both are all the way in. The server LED shows it's on so I know it's getting power. So now my question is do I have an issue with my server, or is this a self-induced issue because OMV can't see the drives in the correct order?


    Any ideas?

    If you can't be good, be good at it.


    System Information

    Version

    6.8.0-1 (Shaitan)

    Processor

    AMD Ryzen 5 3400G with Radeon Vega Graphics

    Kernel

    Linux 6.2.16-4-bpo11-pve

    Memory

    16 gig

    Video Card

    NVDIA Quadro P600 (for Plex HW transcoding)

    • Offizieller Beitrag

    is this a self-induced issue because OMV can't see the drives in the correct order?

    That doesn't make any difference you can change the drives around on the board OMV will still locate them and so will mdadm they just get displayed in a different order.

    So now my question is do I have an issue with my server

    It's looking that way, but what, changing the cables was a good idea but it's left you in worst state, I would start by disconnecting your raid drives and don't reconnect them until you've done some testing. To me this points to either the m'board and/or power supply or, this I'm not sure of, a corrupt/degrading boot drive.

  • Thank you. This is really disheartening, I literally built this 5 weeks ago with all brand new parts. Except for the two drives which are only 6 months old.

    If you can't be good, be good at it.


    System Information

    Version

    6.8.0-1 (Shaitan)

    Processor

    AMD Ryzen 5 3400G with Radeon Vega Graphics

    Kernel

    Linux 6.2.16-4-bpo11-pve

    Memory

    16 gig

    Video Card

    NVDIA Quadro P600 (for Plex HW transcoding)

  • Yes, same result.


    I've installed a different hard dive and put a fresh install of OMV on it. It starts up fine. It isn't showing up on my router's client list, but I was able to access the server. I shut it down and started it up several times, then as a test, reconnected the drives with the data. Now the router says the port the server is connected to is dead. I'm calling it a bad MOBO, and have written ASRock for direction on getting a new board.


    I want to sincerely thank you for your help and patience, geaves. Have a great day.

    If you can't be good, be good at it.


    System Information

    Version

    6.8.0-1 (Shaitan)

    Processor

    AMD Ryzen 5 3400G with Radeon Vega Graphics

    Kernel

    Linux 6.2.16-4-bpo11-pve

    Memory

    16 gig

    Video Card

    NVDIA Quadro P600 (for Plex HW transcoding)

    Einmal editiert, zuletzt von billyp1970 ()

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!