Ok... I obviously don't understand RAID but..

You are doing two different things. On windows, it sounds like you are creating a JBOD device and then creating two partitions that are mirrored. If any drive failed, you would lose all data and the mirror wouldn't help anything.

On Linux with raid 5, mdadm uses the space of the smallest drive (256GB) on all three drives (256GB x 3) and subtracts one drive from array for parity. You could do what you are doing on Windows but I wouldn't and it would have to be done from the command line. You would be better off to just create a unionfilesystems pool (mergerfs) of all three disks for 2.2TB. The mirror is pointless as mentioned above.

Maybe it is more advanced. I have no idea since it is Windows. And you said three disks. You could never have 1.1TB mirrored with the three disks you told me about and survive a disk failure.

That said, if you have seven disks, you can create something with redundancy with different size disks. Lots of people are using unionfilesystem (mergerfs) and snapraid to accomplish what it sounds like you want. mergerfs pools the drives and snapraid provides redundancy.

Although do you really need redundancy? I hope you aren't using this setup as backup.

Zitat von ryecoaaron

mergerfs pools the drives and snapraid provides redundancy.

I always thought snapraid’s main feature was protection against bit rot of large unchanging files. I know “raid” is part of the name but I don’t hear anyone referring to it as redundancy. I’m not trying to cause a stir (looks like you’ve had plenty of that lately ) I’m just trying to understand.

Without redundancy there can be no bitrot protection, only bitrot detection at the most. However bitrot detection and backups can provide (manual) bitrot protection.

Snapraid provides both bitrot detection, checksums, and protection, parity drive redundancy.

To add to Adoby's comments, snapraid doesn't offer realtime redundancy. But it is still redundancy because it allow a drive to fail without losing data (it isn't backup though). And don't confuse redundancy with availability. Helpful link as well - https://www.snapraid.it/compare

@Adoby & @ryecoaaron Thank you both for your replies. I understand backup. I have implemented it a long time ago with a "Full Disk Mirroring / Backup with Rsync" from page 64 of crashtest's OMV Getting Started. It is an amazing thing because it is so simple, and it just works. I have swapped the data disk out for the rsync disk numerous times simply by repointing the shared folders to the rsync disk without even breaking a sweat. I have even figured out how to backup using Rsync and a second SBC using WastlJ's Rsync Two OMV Machines.

When I hear talk of silent data corruption the Snapraid solution sounds perfect: so easy to set up and necessary if you have a growing media server as I do. But I can't visualize a path forward for the way I backup and adding Snapraid (and maybe unionfilesystem). So where does Snapraid fit? On the main machine with its data and rsync disks, or on the second just-backup machine, or both? And how? Both machines have four sata ports each. I don't need 24/7 RAID availability or real time redundancy.

I agree, there is a lack of tools here.

You need checksums and redundancy to handle bitrot. But you also need backups. That can be wasteful.

Assuming you just do mirroring (using ZFS or BTRFS) for redundancy and bitrot protection, you would need four drives. Two each for data and backups.

If you used checksums manually to detect bitrot, you could use the backup copy to fix it. Or fix bitrot in the backups the other way. Just manually replace the rotten file as needed. This should be possible to automate. Then you would need just two drives. Original and backup. And checksums...

I am working on a simple backup utility. Spare time project. Works partially but is still very buggy. It works by creating backup snapshots similar to what rsync or rsnapshot can do. (Very similar to my rsync snapshot script.) But it is much simpler (and possibly somewhat faster) than rsync because it only works on whole files and local filesystems, including NFS. Written in C++.

During daily (or whatever) backups it checksums and backups new and changed files. And creates new snapshots using hardlinks. But it also does a bitrot check of 5% (or whatever) of the old (supposedly) unchanged files in both the backup copy and the original files. Since there are two copies of every file available, backup and original, there is a good chance that any bitrotten file can be replaced.

5% means that all files are checked for bitrot every 20 days.

This would work fine with drives in pairs. Either in the same NAS or in the LAN.

BTRFS, ZFS and Snapraid can already handle this. Faster. Directly. But, for my needs, also wastefully.

I would prefer to use my backups as redundancy for bitrot protection, without the need to introduce more redundancy. And I would be fine checking for bitrot only during backups.

Edit: Replace "drive" with "filesystem" or "partition" as appropriate above.

Perhaps I am inventing the wheel? Do you know of some similar way to semi-automatically provide bitrot protection using backup copies rather than a filesystem with builtin redundancy?

Well, any backup by definition is wasteful but it sure comes in handy when your data disk spins down (another vexing topic for me) and then refuses to spin back up. I can live with waste as long as it provides real insurance against loss. We do it all the time with home-auto-life-health insurance. I believe I have disk failure covered. I’m concerned with data corruption now. I am perfectly willing to add disks to incorporate bit rot protection into my system. I just don’t know how to pull the trigger.

@Adoby I have read your explanation of snapshots, checksums, etc. that you use and it is fascinating but I fear it is above my skill level. OMV’s GUI integration of Snapraid seems doable to me. It’s just a matter of making it work with my Rsync setup.

Zitat von Adoby

Perhaps I am inventing the wheel? Do you know of some similar way to semi-automatically provide bitrot protection using backup copies rather than a filesystem with builtin redundancy?

While it would probably be slow, rsync does have a --checksum flag that does this. This could be used with rsync, rsnapshot (pretty sure), or your script. Not as flexible as your self-written utility. Borgbackup does checksumming (and dedup and compression) but doesn't normally have the files always accessible. But you can mount a borgbackup archive as a filesystem.