[Solved]SnapRaid + Unionfilesytem + Cron = snapraid sync (scheduled to run hourly?)

Hi all,

First of all, I am really a newbie when it comes to OMV's configuration on SnapRaid + Union file system.

I have set up 6 x 2TB hdd to be data disk.
I have set up 2 parity hdds = 2TB + 3TB.
I have set up Union filesystem = combined 6 x 2TB hdd (usable is only 10.74TB)
I have set up schedule cron job to run snapraid sync on every hourly basis.
(please see the pictures attached below)

Questions:

1) What other housekeeping schedule cron job that I need to set up? Scrub?
2) When I schedule the cron job to run snapraid sync on hourly basis, when does system begin to start counting? The moment when OMV is fully loaded after it was turned ON?
3) What extra steps I need to configure that is necessary for maintaining good redundancy (fault tolerant) within SnapRaid configuration?

Most appreciated if someone can give advice.
Thank you.

Quote from crashtest

For restoration purposes, note that you can only restore to the condition present in your array, as it was during the last SYNC command. While it's my opinion, I think running a SYNC every hour is way too often. The reason is, if you have a data disaster or some other unfortunate incident, your automated (1 hour) SYNC command is likely to run before you discover the problem, making the situation permanent by backing the problem up. With longer SYNC intervals, possibly triggered manually, you'd have time to discover the issue and to stop the SYNC command, giving you a restoration option. (You'd be able to restore that collection of disks to a clean state that existed before the problem occurred.)

@crashtest, I just want to confirm whether my understanding is correct based on what I read above.....
I am trying to understand your given case scenario here......
So.....what happens in the event that a disk has crashed in the array, and I have set the sync at hourly interval automatically?
What happen, if a disk crashed and within a few seconds.....an automatic SYNC command has started?
What happen, if I did not take any action to DISABLE the automatic / scheduled sync command?
The biggest Disaster will ever happen to me ......all data previously saved on the crashed disk WILL NOT be recoverable since a SYNC command was automated at hourly interval, am I right?

I am trying to learn as much about the little known SNAPRAID technology from its FAQ: https://www.snapraid.it/faq#whatisit
But I could not find an answer......to my questions below

1) What happen, if you accidentally write a file / update a file to the disk while SYNC operation started?

2) Should SYNC operation ONLY start when there is not disk write/read operation?

3) My NAS does not TURN ON 24/7, but only when I wants to backup data on demand basis, so what is the best time to do:
SYNC - just before I turn off the NAS?
SCRUB - just do it once a month should be sufficed, it depends how I would like to prevent bit-rot to occur?

Quote from gderf

@wepee

You need to read the SnapRaid manual. See section 4.4 https://www.snapraid.it/manual

@gderf, thanks for pointing to the manual.

Since I am not a linux pro and and hopeless in installing SNAPRAID using command line interface, I did not read the manual at all.... to be honest

Thanks again for your info given.....section 4.4 does help me to reconfirm my understanding:

STOP ALL SYNC & SCRUB operation to prevent changes take place in then event of hard disk failure

Quote from thedarkness

1 - I don't know, that's why you should run it after hours when there's no activity on the server. Snapraid first lists files and then syncs it, if becomes missing in this process I guess will throw an error2- This would be the ideal condition
3- My NAS is not running all the time either. I would recommend using the script I mentioned above and run the script when you turn on the system every day. You will know it's time to use the server because you will get an email report.

SCRUB frequency: it's totally up to you and how much you want to gamble. Bit rot is rare, but you don't need to scrub the whole system in one run every month. You can scrub a little part every day or so, like the script does.

@thedarkness Thank you for your info on the script, but I hate to ask this question, how automate script within Schedule Job?

Quote from thedarkness

1 - I don't know, that's why you should run it after hours when there's no activity on the server. Snapraid first lists files and then syncs it, if becomes missing in this process I guess will throw an error2- This would be the ideal condition
3- My NAS is not running all the time either. I would recommend using the script I mentioned above and run the script when you turn on the system every day. You will know it's time to use the server because you will get an email report.

SCRUB frequency: it's totally up to you and how much you want to gamble. Bit rot is rare, but you don't need to scrub the whole system in one run every month. You can scrub a little part every day or so, like the script does.

@thedarkness Thank you for your info on the script, but I hate to ask this question, how automate script within Schedule Job

I know that I need to rename the file to have extension: sh ---> new snapraid script.txt ====> change to: new snapraid script.sh

I have 2 parity disks in my array, what minor configuration is required to be done within the script?

Quote from gderf

Reading a disk will not interfere with SnapRAID. Writing new content to a disk while a sync is running will likely trigger a warning suggesting you rerun the sync when it completes.

@gderf ok thank you, the general thumb of rule is: DON'T RUN SYNC OPERATION WHEN YOU EXPECTING HEAVY WRITING/ UPDATING A FILE.

As @crashtest suggested, only perform SYNC operation during when NAS is sitting in "IDLE mode", perform it during wee hours at night.

Quote from crashtest

An hourly interval will not do you any good. Again, we're talking about largely static data. The SYNC operation takes a tally of new and deleted files, assigns a check number to new files, and updates the content file with new information. There's no need to do this every hour.

This is the reason why you need to think about your SYNC interval. Hard drives rarely die all at once. They tend to corrupt data as they're dying. Once hard drive troubles are noticed, if a SYNC has not taken place, you can restore to the previous state of the disk on a new drive. This is why the manual tells you to "STOP ALL SYNC & SCRUB operations" to avoid modifying the data in the last good SYNC. These reasons are why users should (1) run a SYNC manually or (2) use an extended interval between SYNC's. A very large interval might mean losing work done since the last SYNC.That's the tradeoff involved in SYNC intervals:
- A short interval = Increased risk that corruption will be backed up.
- A long interval = A greater chance that interim work will be lost when restoring to the previous SYNC.

My preference is for the longer interval, to preserve older data in a clean state. Newer data i,generally, easier to recreate. That may not be the case with older files.

I believe, with E-mail Notifications set up for "file systems" and "SMART", and with regularly scheduled SMART tests, a two week SYNC interval is pretty safe. (Again, others may have other opinions.)

BTW: When scheduling SMART tests and using the schedule mentioned above, it might be useful to test drives on the 7th and the 21rst, in between the SYNC and SCRUB cycle.

You may or may not get an error on the next scrub. Which may or may not get "fixed" during the next fix command. This is why it's best to avoid it.

This is preferred, and easy to do if these operations are scheduled for after-hours. (2 or 3AM)

Display More

@crashtest, ok got it. Now I have a better understanding of SNAPRAID..........thank you for your great post!

Should I bother about implementing script that @thedarkness suggested? Or just stick with manual way of doing SYNC and SCRUB?

I am not familiar with linux commands, I have no idea about how to run script within OMV after doing Google search for 1-2 hours.