MergerFS + SnapRAID - Am I missing something?

Zitat von johnvick

But are the merged 3x60 GB virtual disks the OP is using not considered one filesystem?

No. Each individual disk can have different file systems. The 3 disks are "pooled".

Zitat von johnvick

Sure, but one parity disk would cover a three-disk pool. So not sure what trapexit was meaning as OP said he copied his files "inside the shared merged disk."

What I think trapexit was getting at is:

The key to using SnapRAID is understanding that it was designed for data that is "relatively" static. That is mentioned -> here. SnapRAID was NOT designed for high I/O environments or relational databases.

Static data, which is distributed across more than one disk, is the basis for "on-demand" Parity calculations. As an example; if multiple files are deleted on a single disk, that is the same as a partial disk failure. With a single parity drive, the fix command can restore files on one drive, based on parity AND the data on the remaining "unaffected" disks.

If lots of deletes happen on more than one disk, this has the same effect of simulating multiple partial disk failures. A single parity disk, alone, can not restore missing data across multiple disks. In the rebuilding process (along with parity); if data from one disk is needed to rebuild a file on another disk and that data is not present (it was deleted) that will result in an unrecoverable error.

Note: Using tiny virtual drives AND large media files on those tiny drives will greatly increase the chances of an unrecoverable error in a fix operation, after deleting multiple files using Most Free Space. The chances of such an event happening with actual drives in the + terrabyte range is far less.

The problem with the MergerFS Most Free Space directive, when used in conjunction with SnapRAID, is that Most Free Space scatters files across pooled disks. Several deleted files (that appear to be in the same folder) may actually be deleted from multiple physical disks. In a fix situation, this can appear to be a partial disk failure on more than one drive that may result in an unrecoverable SnapRAID error.

Again, if data is mostly static, there's no problem. If normal sized drives are used, even with multiple deleted files across more than one disk in the pool, the chance of an unrecoverable error is very low.

How to deal with this:
If you want to delete a lot files and all else appears to be normal (no disk issues, no bad SMART stat's, etc.) run a sync command after a very large delete. Otherwise, you might think about using more than one parity drive.

For others who may read this thread:

I'm not sure if SnapRAID is calculating Parity based on hard drive sectors, filesystem inodes, or some other indexing method, but since the minimum parity drive size requirement is based on the largest protected hard drive size, from the file perspective, I believe Parity could line up with file sectors in a manner that may resemble the following graphic.

The following is obviously an exaggeration because files are never of identical size but the point made is still valid. The MergerFS Most Free Space Directive will scatter files, that appear to be in the same shared folder, over all MergerFS pooled drives based on percentage of fill. Since static data on all drives is part of the parity calculation, doing large file deletes (several files) then attempting to recover them could result in the following:
_________________________________________________________________________

This depiction assumes several files have been deleted AND that the file sectors of the deleted files (scattered over pooled drives) line up in a manner as shown, AND that a recovery (snapraid fix) operation is immediately done to recover the deleted files. In the noted instances, there may not be enough static data, +parity, for recovery. In that case, there would be an unrecovered file. This would also apply to a full drive recovery but, again, the only files that that would be unrecoverable would be those that line up in a manner similar to what's shown.

In the graphic immediately below file, F1 is recoverable. Files F3 and FB are unrecoverable.



In the 2 Parity disks scenario (immediately above):
Files F1,F3, and FB are recoverable. Files F4, F8 and FC would be lost.
In both instances, if there was an actual hard drive failure the remainder of the files on the failed disk would be fully recovered.

The reason why pa67 experienced the unrecoverable files issue is that (he/her/they) were using tiny virtual drives (60GB) AND doing large deletes of files which, I'm going to speculate, where probably large music files or enormous video files. This is not a realistic test scenario.

With realistic sized hard drives (+ 500GB) and realistic deletes (usually a few files here and there), the chances of the above occurring become vanishingly small to nearly non-existent.

_____________________________________________________________________________________________

Zitat von johnvick

I my use case (config below) say I rip a CD and add to the mergerfs pool, the tracks could possibly be spread over the six drives in the pool.

In your case, based on your signature, you're using 4TB drives. A set of compressed CD sound files (maybe 3MB total?), scattered over 4TB X 6 drives is extremely small. Adding to that you have two (2) Parity drives. This is what I meant by deleted file sectors aligning, in the above manner depicted above, being a vanishing small probability in a real world scenario. (Similar to a meteor taking out your house.)

With that said, (with the MergerFS Most Free Space directive in place) if you deleted 4TB of scattered video files AND immediately tried to do a snapraid fix to recover them, you might not be able to recover some of those files. The output of the log file would tell you which files failed and, in that scenario, that's what backup is for.

With the above noted, you're scrubbing frequently (every night). I hope you're using a good diff script. If not, the script that comes with the OMV6 SnapRAID plugin is pretty good and it's easy to configure.

Zitat von MarcS

I had exactly that problem and only realised when it was too late. I had 5x500GB data disks and 2x 1TB Partiy disks. When I deleted c. 1TB of media files (all jpgs and mp3/4's) which were scattered on 3 HDs,

Well, you'd have to admit that deleting 1TB of data from an available pool of 1.5TB of disk space, that is scattered across all disks in the pool, is not "normal". Multiple simultaneous disk failures, even partial failures, is an extremely unlikely event. What most people don't realize is, RAID5 couldn't deal with that either. Both forms of protection are geared toward whole disk failures, where the remaining healthy disks can be used to rebuild data.

Zitat von MarcS

It seems to be related to the point you are describing. What this means is that the two packages should never be used together and especially not in mergerfs MostFreeSpace mode.

You have a legitimate point but, as it seems, most users don't take the time to educate themselves on MergerFS storage policies. Existing path, most free space, is a good policy for "data type" consolidation but it creates storage imbalances for users with huge media stores (particularly where numerous video files are concerned.)

In your case, at least SNAPRAID allowed for some recovery (60%) of your data store and it gave you an indication that error conditions existed. With RAID5, that would not be the case. Also, where error conditions are concerned, RAID5 is absolutely silent. Further, RAID5 can silently corrupt data with no indication whatsoever and, in some cases, without the top level filesystem detecting it. Where SNAPRAID shines is it's ability to maintain data integrity and it makes it possible to selectively recover files. RAID5 can't do either.

In the bottom line, there is no substitute for 100% backup of your data store. While events like this are rare, as you know, they can happen. With full backup, recovery options are far less painful.

Zitat von MarcS

The MEDIA folder was mounted as persistent to a Docker container whereby the developers forgot to mention that any upgrade of the NC container will erase that mounted folder (incl all data)

What!!?? That doesn't seem like sound design to me. Docker containers can be torched and recreated in a matter of seconds. Data that containers collect or generate shouldn't be part of the create / destroy process. That would be a lot like OMV destroying all existing data, on data drives, as part of an OS rebuild. The OS and data drives are separated for a reason, to prevent data loss. For the exact same reasons, Dockers should be designed to do the same thing.

Zitat von MarcS

So its not so unusual that a large amount of data can get lost.

It should be unusual. There's no way to protect from this type of data disaster, on the local machine. Keep this in mind, without SNAPRAID, you would have recovered nothing. If you find a method that can 100% "regenerate" 1TB of lost data, from a 1.5TB pool of 3 drives, without using backup of some kind, I'd be really interested in what it is and how it's done.

Zitat von MarcS

I was actually in RAID6 using 2 parity disks and I still lost 40% of the data. I would say that is an unexpected result.

Mentioning RAID5 was for simplicity and feature comparison purposes. In your case, if you had only 1 parity drive (versus 2), the amount of recovered data would have been even less, with maybe around 30% recovered.
______________________________________________________________

I don't use a Docker for media purposes, but:
There must be some workaround method where the "Media folder" can be used as a transition location. As files are deposited in Media, maybe they could be rsync'ed out to a permanent location, then deleted from the Media folder. Otherwise, with the risk of something like this happening again, I'd be looking for another Docker.