Replaced dead HDD- - unable to create new storage pool or edit existing

Dejh0n · 1. Januar 2024

Hi hive mind,

I need some assistance on this one please.

I was in the process of backing up my data to a newly acquired NAS for backup purposes when I had a HDD fail on me.

Great.
Tried connecting to it, OMV showed me there was IO errors - I was unable to read anything from the drive. Bugger. Dead drive.

Luckily the drive was still under warranty, so got it replaced with a brand new drive.

Plugged in the new drive and was able to mount it under 'File Systems' - formatted it and called it the exact name as the old drive - "DATA2"

Oddly enough, the failed drive still turns up and I can't seem to remove it either.

I did try replacing the UUID in FSTAB with the new drive UUID and still no luck!

When going to 'Union Filesystems' - I can see the main storage volume (Volume1) with all the drives in there and one showing N/A - obviously for the old DATA2 drive.

When I try to edit Volume1 to remove the old DATA2 drive, it does not show up in the list for me to remove.
When I try and make a new volume, the replacement DATA2 drive does not show up as an option.

Hopefully, someone can help me and let me know what I've done wrong in this process!

chente · 1. Januar 2024

You must remove all references to that drive in all services and shared folders. The openmediavault-resetperms plugin can help you identify them. Then you can remove the disk cleanly. https://wiki.omv-extras.org/do…6:omv6_plugins:resetperms

If you have manually changed configuration files you should revert it.

Dejh0n · 2. Januar 2024

Thanks for coming back to me.

I am on OMV 5.10 by the way - so the readme linked wasn't super clear to me.
I have reverted both FSTAB and snapraid.config file to include the original disk UUID now.

I am still having the same issue as described. When I go to remove the N/A drive I can't un-select DATA2 at all to remove it from the pool.

If I click 'Save' it appears to remove the N/A drive but when I hit apply then it throws a large error and will not let me apply the changes.

When looking through the 'File Systems' tab I can see the old UUID DATA2 drive says it is still referenced.

I did go to my shared folders where there was a folder which referenced the failed disk - I changed to DISK4 now. There was also a folder in there called Volume1 which I also deleted.

Unfortunately, I am still a bit lost at this point in how to find where this failed disk is still referenced.

Any further guidance would be appreciated

chente · 2. Januar 2024

Zitat von Dejh0n

I am on OMV 5.10 by the way - so the readme linked wasn't <i>super</I> clear to me.

The resetperms plugin worked the same in OMV5 as it did in OMV6.

Zitat von Dejh0n

Unfortunately, I am still a bit lost at this point in how to find where this failed disk is still referenced.
Any further guidance would be appreciated

Check the services, it could be some rsync job or something else. Also check the SMART section.

____________________________________________________________________

You should update the OMV version. OMV5 is obsolete. It could be a good time to do a fresh installation of OMV6 (even OMV7 if you dare) and forget about all these problems.

Dejh0n · 2. Januar 2024

Zitat von chente

The resetperms plugin worked the same in OMV5 as it did in OMV6.
Check the services, it could be some rsync job or something else. Also check the SMART section.

There was a couple of old SMART tests which had a N/A device, which I removed.
I tried deleting the shared folders from NFS which it would not let me and threw up big errors..

I feel like it is the MergerFS/UnionFS folder causing these issues as I cannot reset any of the permissions as that volume is not and seemingly cannot be mounted.

I need to try and restore as much data onto the replacement disk from my SnapRAID config.

Would I still be able to do this if I were to do a fresh install of OMV?

I have a few docker containers I use (I use portainer) which I really dont want to loose data from and have briefly read there is a compatibility issue with upgrading 5 > 6.

Dejh0n · 2. Januar 2024

I have also now wiped the replacement drive and called it DATA12.

OMV will NOT allow me to mount this wiped drive, attached is the error it throws up Error #0.txt.

This appears to be the same error code it throws up when I try and delete a share from NFS..

I'm way in over my head with these issues and errors.

chente · 2. Januar 2024

Any service that references that drive will prevent the GUI from allowing you to unmount it. It can be mergerfs or any other. You need to remove all references to be able to unmount that disk.

Your first problem would be to recover that data from the pool and Snapraid. Check the Snapraid documentation to do this. Once that was done I would do a new OMV6 installation as I already said.

Regarding the containers, you only need the compose files and the persistent data folders intact to reproduce them in another OMV installation.

Dejh0n · 2. Januar 2024

Zitat von chente

Any service that references that drive will prevent the GUI from allowing you to unmount it. It can be mergerfs or any other. You need to remove all references to be able to unmount that disk.
Your first problem would be to recover that data from the pool and Snapraid. Check the Snapraid documentation to do this.

This is the issue I seem to be having. I am unable to mount the replacement drive to the OS to then be able to add it into the SnapRAID setup.

According to the documentation here.

I need to add the replacement drive (now DATA12),

modify the config to change my SnapRAID 'DATA2' drive from my DATA2 disk to the DATA12 disk,

then I can fix the drive and begin restoring the data.

If I navigate to the SnapRAID service and attempt to make those changes I am hit with the following screen:

I don't even know what that text string in the drive box is referring to. None of my disks have that UUID.
If I click the drop down box I can see all the other drives, just not the recently re-added 'DATA12' drive

chente · 2. Januar 2024

I think the easiest way would be to do this recovery from the CLI, and once the data is recovered, do a new installation of OMV6.

I haven't used Snapraid in a while so I'm not sure I can help you properly with this. gderf Could you give a hand with this please?

gderf · 2. Januar 2024

This is how I am currently trying to recover a 12TB drive that failed two days ago. I am doing it via the command line because I don't use the SnapRAID plugin for anything other than adding new drives to the array.

Made a backup copy of /etc/snapraid.conf file.

Use the SnapRAID plugin to remove the settings (Data, Content) from the failed drive.

Do whatever you have to do in OMV to remove the failed drive from the configuration. (This is a PITA, be persistent. But you can continue without success)

Check /etc/fstab to see if OMV removed the failed drive there, if not comment out the entry for it.

Physically remove the failed drive, start the machine and see if it is behaving well without messages about the failed drive. But you can ignore them if they only refer to the drive being missing. This can be fixed later, hopefully.

Add a new drive to the machine and format it. It needs to be as large or larger than the data amount on the failed drive.

Here is my original snapraid.conf file section for the drive and how I changed it - per the SnapRAID manual. Please read the recovery section if you haven't yet.

Code

Original snapraid.conf file section for failed drive.

#####################################################################
# OMV-Name: d2  Drive Label: d2
content /srv/dev-disk-by-uuid-a6e6252d-5a8f-4e9b-88b3-46bef35b01a0/snapraid.content
disk d2 /srv/dev-disk-by-uuid-a6e6252d-5a8f-4e9b-88b3-46bef35b01a0
#####################################################################

Code

Changed snapraid.conf file section for failed drive

#####################################################################
# OMV-Name: d2  Drive Label: d2
#content /srv/dev-disk-by-uuid-a6e6252d-5a8f-4e9b-88b3-46bef35b01a0/snapraid.content
disk d2 /srv/dev-disk-by-uuid-2522391f-d1ca-40b4-b408-7d0bb5a4b0e3
#####################################################################

Changes made to snapraid.conf were to comment out the content line and replace the failed disk uuid with the new disk uuid. If OMV removed the entire section for this drive then put it back in by hand as changed above in an editor running as root.

Next, properly compose the snapraid command to do the fix. Here is mine, but read the docs.

Code

sudo snapraid -d d2 -l fix.log fix

The only variable in the above command is the OMV-Name (mine is d2) for the drive.

I suggest running the fix command in a screen session if you already know how to use screen or can learn it quickly.

To start the recovery open a shell and cd into a drive and folder that has plenty of free space. This is where the log file is written to. Then run the above command. Watch it for a while until you see the progress line at the bottom of the screen. It looks like this:

19%, 17250982 MB, 178 MB/s, 85 stripe/s, CPU 0%, 112:41 ETA

I think the ETA is in hours:minutes.

While this is running you can look at the log file. It's messy and hard to read. Also look in the root of the newly added drive and see if recovered files are showing up.

If you need to stop the recovery before it finishes, you can break out of it and continue it later. It will pick up where it left off but the percentage will start at zero. Restarting a fix overwrites the existing log file so you may want to make a copy of it first.

When I first started with SnapRAID many years ago I got it all set up, did the first sync and scrub to see it did its thing error free. I then immediately did a simulated drive failure by removing one of the data disks and proceeded as above. It was a 3TB disk that was full. I recovered to an identical but empty disk. As near as I could tell it worked perfectly and I erased the recovered disk for future use. I don't remember how long it took, but a couple of days seems about right. Slow is the keyword here. The ETA is an estimate only and will vary as it goes.

I can say that this time I am becoming disappointed with my current recovery attempt. It seems to be writing a lot of files that are truncated or otherwise corrupted. I am quite sure that they were in perfect condition on the disk before it failed. I have no explanation and will begin discussing this in the SnapRAID forum, but not in this forum.

Good luck. Feel free to ask any questions. But you may want to see if the question is already answered in the SnapRAID forum or google it.

Dejh0n · 8. Januar 2024

Hey guys,

Just a quick update.

After your advice gderf, I went to the FSTAB and commented the drive out in the top section.

I also found it referenced twice in the MargerFS pool section in the FSTAB. I deleted out BOTH of these references and after a reboot I was able to 'remove' the old faulty and already removed drive from the MergerFS pool without errors!

I then went on to mount the replacement of Drive2 with Drive12 and it was able to mount with no problems this time!
I also was able to go into the SnapRAID service and modify the data drive, Drive2 with the newly added and mounted Drive12.
I then ran the fix error via the GUI and left if for a few days and a lot of data has been put onto the replacement drive12.

I will investigate the SnapRAID commands via a Screen session as instructed shortly. I was just VERY HAPPY to be able to get the volume online even if only partially thanks to both of your help.

So thank you both!

I will post any potential updates after I see how the SnapRAID fixes go - I am less hopeful, as I don't believe I had set this up correctly initially or maintained it as I think I was supposed to..

Replaced dead HDD- - unable to create new storage pool or edit existing

Jetzt mitmachen!

Tags