Hi,
Background
I built a nas a while back on a budget - 20$ for rock64+sd+psu from ebay. With a 8TB HDD and a second 8TB HDD with the usb backup plugin to rsync the drives whenever i felt like plugging in the backup. Pretty solid setup - could reliably hit 100+MB/sec on transfers.
Recently (~2 years later from initial setup) i discovered that some of my movies wouldn't play or would have artifacts that i knew weren't present in the past. Some of the rsynced backup data is fine as i believe it only syncs off of timestamps and not checksums, but i have no idea when the primary drive started corrupting data or what exact data is corrupted.... thankfully i did generate an md5 of all files a long time back, so that's a start... i'll need to write some code to dig into this.
I started digging into SnapRaid as i could still use it with a pi4 or something i had onhand, and i could just buy another HDD and use as parity. After playing around I ran into some concerns
Questions/Concerns
SnapRaid works with USB drives, but guides suggest setting up a scheduled 'snapraid sync'. Seems like snapraid doesn't have knowledge of 'file actions' and will blindly sync and re-compute checksums on whatever the storage drives spit out. The only way to detect silent corruption or potentially bit rot is to run a diff, review the results and review all file differences, then either sync or rebuild data.
I'm looking for something that runs 'live' where parity is updated per file, per file action. So if i transfer a file to the NAS, then the nas generates parity for that file - not rebuild the entire parity for the whole SSD. Also i'd like something that runs transparently (obviously i'd like to know if a drive is corrupting data).
Is there anything that fits this description? I'm thinking the RAID 5 option would fit? Does OMV 'raid 5' use a software raid? This would require me to get a full computer with SATA and a couple more disks which I was hoping to avoid... At this point I'm thinking i need to get something more professional.
To help someone in a similar predicament:
Word of the wise - it's a very good idea to run a find and log an md5 of all files on your nas. Even with raid5 I've heard some horror stories long back (2005) of non protocol conforming hardware raid5 doing wonky stuff.
Finding problems with video files - Let's say you don't know what an MD5 should be, you can still run this and see if the video file has 'problems'. Problem with doing this is that even good video files can still have missing frames here and there . But if you get 100+ errors in a single video file - that's a red flag.
#search for all file types present in a folder
find . -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u
#manually make a list of video file types that you have present - feed into ffmpeg, log errors for each file.
find -name '*.avi' -exec echo "processing {}" > >(tee -a ~/avi.log) ';' -exec ffmpeg -v error -i {} -f null - 2> >(tee -a ~/avi.log) ';'
find -name '*.flv' -exec echo "processing {}" > >(tee -a ~/flv.log) ';' -exec ffmpeg -v error -i {} -f null - 2> >(tee -a ~/flv.log) ';'
find -name '*.mkv' -exec echo "processing {}" > >(tee -a ~/mkv.log) ';' -exec ffmpeg -v error -i {} -f null - 2> >(tee -a ~/mkv.log) ';'
find -name '*.mp4' -exec echo "processing {}" > >(tee -a ~/mp4.log) ';' -exec ffmpeg -v error -i {} -f null - 2> >(tee -a ~/mp4.log) ';'
find -name '*.mpeg' -exec echo "processing {}" > >(tee -a ~/mpeg.log) ';' -exec ffmpeg -v error -i {} -f null - 2> >(tee -a ~/mpeg.log) ';'
find -name '*.mpg' -exec echo "processing {}" > >(tee -a ~/mpg.log) ';' -exec ffmpeg -v error -i {} -f null - 2> >(tee -a ~/mpg.log) ';
...
Alles anzeigen