I think I hosed my RAID 5

I would like to say what I did, and hopefully someone could tell me how I messed this up, because I'm sure I've horked this thing real good.

I started with an old HP All-In-One I had that was already running Debian, so I installed OMV 3.0.59 on it, along with 2 3TB Seagate USB drives, which I told OMV to link together as RAID 5. It took about 4 days, but it worked. This has been my NAS for the last few months. About 2 months ago, I decided to get serious about allowing my Roku and PS3 to start streaming from my NAS again (old solution was GoFlex Home; bad choice, because all movies had to either be in MP4, or 2 other formats to be played, since PS3 won't play MKV, etc), since I saw that PLEX might be a good solution to stream to both. So, installed PLEX Server as a plug-in on the OMV install on the HP. Worked great until last week.

All of a sudden, last week, even though my Roku and PS3 and both wired to my network, neither can stream very well. Both were having trouble with 320p. So, I moved the server to another box, but the server wasn't the problem; it was the HP having issues. I decided to get a Raspberry Pi and move everything there.

So, without asking questions, I just forged ahead (like the ID10T that I am), got the RPi3, installed OMV3 (which worked flawlessly), then followed this guide in the forum, not even thinking twice that it might not fit my particular situation.

When I restarted my RPi3, the dashboard would come up ok, but when I would click on File System, it would give me an unspecified "communication failure" over and over. Closing the window and logging in again would show the dashboard again, but this time, the CPU usage was at 100%, and stayed there for over 2 days, with the Load Average climbing and climbing until it hit 130, 130, 130. It finally backed down, and the "communication failure" pop-ups are at a minimal now, except for when I click on the File Systems link. Then, that's all I get, even if I successfully go somewhere else, like Physical Disks.

Now, I haven't turned the HP back on since I unplugged the USB drives. I might try to turn off the RPi3 and plug those drives back into the HP, turn it on, and see if it can still recognize the RAID 5. Otherwise, it looks like I might have to wipe both disks and start over.

Is there something I could've done to avoid this? What happens in the future when I want to expand? I just planned on buying a powered USB hub, plugging these drives into that, and two other drives next to those. Will this thing flip out if I do that?

I guess I should probably plan these things better. And sorry for the long post.

The GoFlex Home I had a few years ago died and took 2TB of my life away with it. Right now, I only stand to lose 1.5TB if I can't recover what I have. I can look into the unionfilesystems plugin after I see if I can recover my data )if I'm understanding you correctly). Do you think I should just wipe my drives and walk away with my hair not pulled out from my head? Is it possible I could unplug one of the USB drives (since they are supposed to be mirrored) and hope to recover at least one copy?

Well, then, I obviously need to go back several steps. I was looking to have a system where I could store data on my network and have that data duplicated so that, if one of the drives died, I could just unplug it, plug a new one in, have the system format it, re-duplicate it, and continue on until the other drive had to be replaced. I just want a NAS with back-up that's not complicated as f*ck (which every back-up system I've ever heard of is). I don't want to pick and choose which files to back up, I don't need it compressed, I don't want it to have to run at certain times, etc. I just want my data, and a mirror of my data. And apparently RAID isn't the way to do it. And now I'm more lost than before I found OMV on DuckDuckGo last year.

Zitat von Dropkick Murphy

But what happens, if 2 or 3 or whatever number of drives are failing the same time? What, if your NAS is burning? Right, all your data is lost and you´re gotten by the balls, despite of your Raidlevel...

Ok, I completely understand what you're saying, but please understand that while my experience with computers themselves is extensive, it's only connected with desktops. Anything larger with more than one hard drive is new to me, but in my experience, which goes back to before the movie WarGames and before I bought my first 300 baud modem, I have lost exactly 2 floppy disks, 1 CD (physical disk), 1 hard drive, and 3 optical drives. Honestly, I was surprised and pissed off when my GoFlex died on me. But, to give a frank response to your question, if that happened, it wouldn't put me in any different of a position than I'm in right now with 1.5TB trapped on 2 USB drives that I can't seem to access (that still has not been addressed).

Please, don't get me wrong; I'm very grateful and appreciative for all help, but would it be possible to see if I could recover my data first, and then bully me about "backup" after I have something to actually backup, because otherwise, the point is kind of moot, isn't it?

Zitat von Dropkick Murphy

In my case, I have an old Zyxel, which is every night automaticallay rsync-target for my OMV. What changes on my OMV goes to the backup. Not complicated...

So, you have to have a piece of software that monitors an entire drive in one state, compares it to another state, and takes a copy if there's a change? The entire drive, of all of the individual files? If the individual files, then where is it keeping the information about the former and current states? That has to reside somewhere. How many different states does it store? Do you have to choose how many different states it saves? How would you choose that? If I delete something on purpose, does the backup keep it, even if I want to get rid of it? If so, how do I truly get rid of it? If if doesn't, and I accidentally delete it, then will the backup get deleted? How can rsync-target tell the difference between the two circumstances? These kinds of questions (and others like it) and my personal low fail rate with hardware are the reasons I've stayed away from backup all of these years. If there is an easy way to do it, I'm more than willing to learn (the wife has been getting on me about it). But, I do have to either recover what I already have, or give up on it, wipe it clean, and start over from scratch. Otherwise, I have an RPi3 with a working copy of OMV, but no storage to do anything with.

Zitat von ryecoaaron

I use rsnapshot. Create a shared folder on one drive. Put data in it. Create a shared folder on another drive. Set rsnapshot to do an hourly,daily, weekly, monthly, yearly snapshot of the data folder to the backup folder. Very simple. This keeps multiple copies (but only uses more space when the file changes)

Huh? Hourly, daily, weekly, monthly, AND yearly backups? Sorry, snapshots? (what's the difference?) Multiple copies? If the first drive is totally full, and the second drive has multiple copies of the files from the first drive, won't the second drive fill up way before the first drive? Even if only one file has two copies on the second drive, I'd think there's still space on the first drive, but now the backup will fail, because the drive is full, which brings me to my next-to-last point...

Zitat von Dropkick Murphy

In my case, I have an old Zyxel, which is every night automaticallay rsync-target for my OMV. What changes on my OMV goes to the backup. Not complicated...
If my OMV dies in any way, I laugh and build a new one, get my data back from the Zyxel and thats it.

When my GoFlex died, it took me 8 months to save enough to buy the 2 USB drives. Another 4 months to save for the RPi3. I don't have the resources everybody else has. If My OMV dies, or any more of my equipment, my technological base moves back to last century, period. I was just trying to do the best I could in a bad situation, you know? And lastly...

Zitat von ryecoaaron

Do you really need a realtime copy? With a mirror, if you accidentally delete something, it is gone from both drives instantly.

No, I don't need a realtime copy. And, if I did accidentally delete something, I figured it would truly be gone, because of not having backups before, and the problems I have with backup as stated earlier.

Thanks

Zitat von ryecoaaron

Things don't last as long as they used to. And crypto viruses didn't exist back in the day (I have been working computers since before Wargames). So, you can lose files without hardware failures.

Ok, we're both oldschool, and I totally understand this. All of the optical drives I've lost have been in the last 2 years. When the first one went bad, it took me quite a while to diagnose it, because I just couldn't believe that it was the drive itself that failed. I've heard of these crypto viruses and would be devastated if I had one. I'm in complete agreement with you here.

You've both brought up great points about rsync and rsnapshot. They sound wonderful and fantastic, and they might even be what I'm looking for, which will be the NEXT step of my journey, AFTER I either recover my data or wipe the drives and start fresh. Please consider the following admittedly grossly exaggerated conversation.

Tech: So, you'd like to have a computer? Awesome. What kind of processor were you thinking about having in it?
Bum: I'd love to get a computer, but I'm homeless right now.
Tech: Ok, but you should also consider the motherboard and the memory. Have you thought about that?
Bum: Not really. I'd have no place to plug it in, you see?
Tech: Well, that's not important right now, we really have to talk about the processor. Intel or AMD? Sounds like you're on a budget, so AMD it is.

Do you see where I'm coming from? I have no concept right now of rsync apart from occasionally see it associated with a Linux install now and again, and this is the first I've heard of rsnapshot. I don't know what it is, how it works, not even the wrong idea of how either of them might work. But, I have a slight understanding of NAS, and of OMV (and have even had it working on 2 different platforms), and possibly a poor understanding of RAID, and that was the kind of advice I was hoping to get to maybe have a chance at recovering my data which had been missing for 2 days when I started this thread, and is now almost going on 4 days?

Look, I understand, because I've been "there".

Zitat von ryecoaaron

There are many, many people we have dealt with that have lost files because they have no backup. Just trying overly remind people to backup their files.

I understand this more than you know. But, first, I'm not asking you to recover my files. I'm just asking if anyone knows if there is even a possibility, or is there no chance, and should I just wipe the disks and start fresh? I'm not going to be upset if I lose the data; I know how volatile data is. My first computer was an Atari 800XL and I had to save my programs to cassette tape, and if my mother turned the dryer on without telling me, the house voltage would drop too low, and the whole thing would freeze up, and I'd lose everything. I stopped being upset at that stuff a very long time ago. Yeah it took a long time to gather 1.5TB of data, and it will take a long time to get it back, but it's not the end of the world for me. But, if I am going to replace it, I'd like to get started now, because if preping the RPi3, OMV and the USB disks with either rsync or rsnapshot is anything like the HP, OMV, the USB discs and RAID5, then I'm not going to be able to even start putting data on this thing until 4 days AFTER I get it all set up, which I can't do until I know if there is a chance to recover what I already have or not.

After I know if there is a chance if I can recover my data or not, I can look into rsync and rsnapshot and the multitude of other backup solutions that there are out there, because I already have the most important thing for backup; extra space (the second drive). Since I was going to use RAID anyways, I wasn't planning on using the entire 6TB for storage, so I'm very open to backup. And, if I do decide that rsync or rsnapshot are answers to what I'm looking for, I know 2 people who personally use it, and after this fiasco, I'm certainly not going to go using any more technology before fully understanding it first. Otherwise, I'm sure I'll install one of them, and then start loading data like normal, and the next day, realize I've messes the entire thing up, because I forgot one setting, or didn't fully understand what I was implementing (hmmm, like RAID, maybe? <- dig at myself).

Please. I'm not trying to be a dick. I know you are volunteering your time and expertise. I'll get to implementing a proper backup solution when I get to that point, and even though I messed this up, can't you see that I was at the very least trying to have some kind of a backup solution, even if it was a bad one? All I need to know is, does anyone know if there might be the slightest possibility of a chance that I might be able to recover the data that is on these drives, and what steps I might take towards that end? Again, after that, I will listen, read, and learn all this is to know about proper backup solutions (like I've admittedly been avoiding for years).

You've done your job. You've converted me. I will learn about backup. I just need to know if I can recover what I have, or if I have to start over, or if you don't know. If you have no idea, and I'm the first ID10T to mess this up in this particular way, that's fine, too. I've just got to be able to move on, one way or the other.

Thank you.

Zitat von ryecoaaron

Combine those two and you may not have good luck recovering any files on the RPi.

Ok. I'm hearing that chances aren't good. I can deal with this. It's both good and bad news. Bad that chances aren't good, but good that I might be able to know one way or the other soon.

Zitat von ryecoaaron

Being a mirror makes it easier. Maybe we could get it started in degraded mode with just one drive so it doesn't overwhelm the RPi trying to resync the array but it will have to be all command line.

Ok. Is there documentation on "degraded mode"? I just got an idea. What if I plugged one USB drive into the RPi and the second into the HP, then boot them up, and then I could SSH into both. Then try to access each drive on each 'puter. If I can get to the data on one, I can wipe the other, format to ext4, transfer the data, then start all over again. I will try that here in the next few hours.

Ok, I did like I said, only because I did it before I saw your post. Plus, I'll need one drive with data, and one drive empty to do what I'm planning.

The HP is reporting that (through the web interface, immediately), that the RAID volume is clean, but degraded, and it is accessible from my network. I was gonna wipe the disk on the RPi, format to ext4, copy everything from the HP to it, then wipe it, format it to ext4, plug it into the RPi, then learn to properly set up backup.