[Solved]SnapRaid + Unionfilesytem + Cron = snapraid sync (scheduled to run hourly?)

  • Hi all,


    First of all, I am really a newbie when it comes to OMV's configuration on SnapRaid + Union file system.


    I have set up 6 x 2TB hdd to be data disk.
    I have set up 2 parity hdds = 2TB + 3TB.
    I have set up Union filesystem = combined 6 x 2TB hdd (usable is only 10.74TB)
    I have set up schedule cron job to run snapraid sync on every hourly basis.
    (please see the pictures attached below)


    Questions:


    1) What other housekeeping schedule cron job that I need to set up? Scrub?
    2) When I schedule the cron job to run snapraid sync on hourly basis, when does system begin to start counting? The moment when OMV is fully loaded after it was turned ON?
    3) What extra steps I need to configure that is necessary for maintaining good redundancy (fault tolerant) within SnapRaid configuration?


    Most appreciated if someone can give advice.
    Thank you. :|


    pic1.jpgpic2.jpgpic3.jpgpic4.jpg

  • Housekeeping for SNAPRAID:
    There's no one right way to do this. It's a matter of personal preference but discussing SNAPRAID considerations might help.


    First, SNAPRAID is intended for mostly static files. If there's a massive amount of turnover or active relational DB's, SNAPRAID's benefits are diminished. Assuming you're mostly warehousing data, as is usually the case for a home NAS, SNAPRAID works well.


    For restoration purposes, note that you can only restore to the condition present in your array, as it was during the last SYNC command. While it's my opinion, I think running a SYNC every hour is way too often. The reason is, if you have a data disaster or some other unfortunate incident, your automated (1 hour) SYNC command is likely to run before you discover the problem, making the situation permanent by backing the problem up. With longer SYNC intervals, possibly triggered manually, you'd have time to discover the issue and to stop the SYNC command, giving you a restoration option. (You'd be able to restore that collection of disks to a clean state that existed before the problem occurred.)


    The SYNC command sets the list of files and their checksums, in the content file. It stands to reason that running a SCRUB immediately after a SYNC would be of limited value. The probability is low, that something would change in a very short time frame. So, after the passage of time, running a SCRUB is useful to verify that data integrity still exists or to identify problems. A FIX command makes sense after a scrub, to fix issues found in the scrub.


    The scheduled tasks I have for my SNAPRAID+UnionFS box is in this -> post. I also set up E-mail reports/notifications to see if something is of concern.
    In a similar manner, when compared to a ZFS scrub, I believe roughly 2 weeks between a SYNC and follow up SCRUB and FIX makes sense for what I do.


    For me it's:
    Day1 SYNC
    Day12 SCRUB
    Day13 FIX


    Day14 SYNC
    etc., etc.


    To avoid modifying files during a SYNC operation or other, these actions are set to run after-hours.

  • Hourly sync is way too often. I sync manually after making file changes on the server (and after running snapraid diff) and periodically run manual scrubs based on the output of the snapraid status command.


    If I were going to use automatic scrubs, I would make sure to use a script that uses snapraid diff and aborts the sync if too many files have been deleted. You don't want to lose a drive and have that automatically synced.

  • There is no ideal configuration, but you could:


    - run one sync every day
    - run one scrub after every sync for N% data that is older than N days


    This is easily automated with the script attached to this post (rename it to sh)


    This script will:
    - Always sync every time you set it --- configure a cron job to run the script every 24 hours
    - If sync is successful, scrub 8% of the array older than 5 days --- this value can be customised see the script
    - Send you a nice email including smart disk stats ---- add your email address in the script


    You can increase scrub values, but I believe these are okay. Bear in mind that scrubbing is a slow process if you have lots of data.
    This script also supports custom conditions, where you can set a limit of deleted or updated files to run the script, but I don't use it. The script is well made and every feature is documented in the comments.


    I also believe that the script will stop if does not read any data from a disk in the Snapraid array.


    That's really it.


    EDIT: Script needs a minor change if you have multiple parity disks

    OMV BUILD - MY NAS KILLER - OMV 5.x + omvextrasorg


    Core i3-8300 - ASRock H370M-ITX/ac - 8GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker), 3x4TB HDD (Data) - Fractal Design Node 304 - Be quiet! Pure Power 11 350W

    Edited 3 times, last by auanasgheps ().

  • For restoration purposes, note that you can only restore to the condition present in your array, as it was during the last SYNC command. While it's my opinion, I think running a SYNC every hour is way too often. The reason is, if you have a data disaster or some other unfortunate incident, your automated (1 hour) SYNC command is likely to run before you discover the problem, making the situation permanent by backing the problem up. With longer SYNC intervals, possibly triggered manually, you'd have time to discover the issue and to stop the SYNC command, giving you a restoration option. (You'd be able to restore that collection of disks to a clean state that existed before the problem occurred.)

    @crashtest, I just want to confirm whether my understanding is correct based on what I read above.....
    I am trying to understand your given case scenario here......
    So.....what happens in the event that a disk has crashed in the array, and I have set the sync at hourly interval automatically?
    What happen, if a disk crashed and within a few seconds.....an automatic SYNC command has started?
    What happen, if I did not take any action to DISABLE the automatic / scheduled sync command?
    The biggest Disaster will ever happen to me ;( ......all data previously saved on the crashed disk WILL NOT be recoverable since a SYNC command was automated at hourly interval, am I right?

  • I am trying to learn as much about the little known SNAPRAID technology from its FAQ: https://www.snapraid.it/faq#whatisit
    But I could not find an answer......to my questions below


    1) What happen, if you accidentally write a file / update a file to the disk while SYNC operation started?


    2) Should SYNC operation ONLY start when there is not disk write/read operation?


    3) My NAS does not TURN ON 24/7, but only when I wants to backup data on demand basis, so what is the best time to do:
    SYNC - just before I turn off the NAS?
    SCRUB - just do it once a month should be sufficed, it depends how I would like to prevent bit-rot to occur?

  • 1 - I don't know, that's why you should run it after hours when there's no activity on the server. Snapraid first lists files and then syncs it, if becomes missing in this process I guess will throw an error
    2- This would be the ideal condition
    3- My NAS is not running all the time either. I would recommend using the script I mentioned above and run the script when you turn on the system every day. You will know it's time to use the server because you will get an email report.


    SCRUB frequency: it's totally up to you and how much you want to gamble. Bit rot is rare, but you don't need to scrub the whole system in one run every month. You can scrub a little part every day or so, like the script does.

    OMV BUILD - MY NAS KILLER - OMV 5.x + omvextrasorg


    Core i3-8300 - ASRock H370M-ITX/ac - 8GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker), 3x4TB HDD (Data) - Fractal Design Node 304 - Be quiet! Pure Power 11 350W

  • @wepee


    You need to read the SnapRaid manual. See section 4.4 https://www.snapraid.it/manual

    @gderf, thanks for pointing to the manual.


    Since I am not a linux pro and and hopeless in installing SNAPRAID using command line interface, I did not read the manual at all.... to be honest :)


    Thanks again for your info given.....section 4.4 does help me to reconfirm my understanding:


    STOP ALL SYNC & SCRUB operation to prevent changes take place in then event of hard disk failure

  • 1 - I don't know, that's why you should run it after hours when there's no activity on the server. Snapraid first lists files and then syncs it, if becomes missing in this process I guess will throw an error2- This would be the ideal condition
    3- My NAS is not running all the time either. I would recommend using the script I mentioned above and run the script when you turn on the system every day. You will know it's time to use the server because you will get an email report.


    SCRUB frequency: it's totally up to you and how much you want to gamble. Bit rot is rare, but you don't need to scrub the whole system in one run every month. You can scrub a little part every day or so, like the script does.

    @thedarkness Thank you for your info on the script, but I hate to ask this question, how automate script within Schedule Job?

  • 1 - I don't know, that's why you should run it after hours when there's no activity on the server. Snapraid first lists files and then syncs it, if becomes missing in this process I guess will throw an error2- This would be the ideal condition
    3- My NAS is not running all the time either. I would recommend using the script I mentioned above and run the script when you turn on the system every day. You will know it's time to use the server because you will get an email report.


    SCRUB frequency: it's totally up to you and how much you want to gamble. Bit rot is rare, but you don't need to scrub the whole system in one run every month. You can scrub a little part every day or so, like the script does.

    @thedarkness Thank you for your info on the script, but I hate to ask this question, how automate script within Schedule Job


    I know that I need to rename the file to have extension: sh ---> new snapraid script.txt ====> change to: new snapraid script.sh


    I have 2 parity disks in my array, what minor configuration is required to be done within the script?

  • The is biggest Disaster will ever happen to me ......all data previously saved on the crashed disk WILL NOT be recoverable since a SYNC command was automated at hourly interval, am I right?

    An hourly interval will not do you any good. Again, we're talking about largely static data. The SYNC operation takes a tally of new and deleted files, assigns a check number to new files, and updates the content file with new information. There's no need to do this every hour.


    STOP ALL SYNC & SCRUB operation to prevent changes take place in then event of hard disk failure

    This is the reason why you need to think about your SYNC interval. Hard drives rarely die all at once. They tend to corrupt data as they're dying. Once hard drive troubles are noticed, if a SYNC has not taken place, you can restore to the previous state of the disk on a new drive. This is why the manual tells you to "STOP ALL SYNC & SCRUB operations" to avoid modifying the data in the last good SYNC. These reasons are why users should (1) run a SYNC manually or (2) use an extended interval between SYNC's. A very large interval might mean losing work done since the last SYNC.
    That's the tradeoff involved in SYNC intervals:
    - A short interval = Increased risk that corruption will be backed up.
    - A long interval = A greater chance that interim work will be lost when restoring to the previous SYNC.


    My preference is for the longer interval, to preserve older data in a clean state. Newer data is, generally, easier to recreate. That may not be the case with older files.


    I believe, with E-mail Notifications set up for "file systems" and "SMART", and with regularly scheduled SMART tests, a two week SYNC interval is pretty safe. (Again, others may have other opinions.)


    BTW: When scheduling SMART tests and using the schedule mentioned above, it might be useful to test drives on the 7th and the 21rst, in between the SYNC and SCRUB cycle.


    1) What happen, if you accidentally write a file / update a file to the disk while SYNC operation started?

    You may or may not get an error on the next scrub. Which may or may not get "fixed" during the next fix command. This is why it's best to avoid it.


    2) Should SYNC operation ONLY start when there is not disk write/read operation?

    This is preferred, and easy to do if these operations are scheduled for after-hours. (2 or 3AM)

  • Reading a disk will not interfere with SnapRAID. Writing new content to a disk while a sync is running will likely trigger a warning suggesting you rerun the sync when it completes.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 5.x on ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.

  • Reading a disk will not interfere with SnapRAID. Writing new content to a disk while a sync is running will likely trigger a warning suggesting you rerun the sync when it completes.

    @gderf ok thank you, the general thumb of rule is: DON'T RUN SYNC OPERATION WHEN YOU EXPECTING HEAVY WRITING/ UPDATING A FILE.


    As @crashtest suggested, only perform SYNC operation during when NAS is sitting in "IDLE mode", perform it during wee hours at night.

  • @crashtest, ok got it. Now I have a better understanding of SNAPRAID..........thank you for your great post! :thumbsup:


    Should I bother about implementing script that @thedarkness suggested? Or just stick with manual way of doing SYNC and SCRUB?


    I am not familiar with linux commands, I have no idea about how to run script within OMV after doing Google search for 1-2 hours.

  • My diff, sync, scrub cycle starts at 3:00am and takes 2 1/2 hours to complete. It runs every fourth day.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 5.x on ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.

  • I am not familiar with linux commands, I have no idea about how to run script within OMV after doing Google search for 1-2 hours.

    I took this piece from your post, "run script within OMV" and googled it.


    Code
    https://www.google.com/search?q=run+script+within+OMV&oq=run+script+within+OMV&aqs=chrome..69i57&client=ubuntu&sourceid=chrome&ie=UTF-8

    It pointed me to the exact correct link, first in the list:




    Code
    https://forum.openmediavault.org/index.php/Thread/12562-run-script-on-OMV-from-gui/

    Maybe I'm just lucky?

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 5.x on ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.

  • Should I bother about implementing script that @thedarkness suggested? Or just stick with manual way of doing SYNC and SCRUB?

    In this area, others like @gderf and @thedarkness may have better advice.


    In my use case, SNAPRAID is on one of my backup servers which works well.
    - truly static content, that changes on a known schedule. (Rsync jobs)
    - uses a simple filesystem (EXT4 in my case)
    - works well with UnionFS plugin which allows me to pool a collection of older disks.
    _________________________________________________________________________________


    I really need to look at diff jobs, custom scripts, etc., but my needs for a backup server are simple.

  • There are a few good scripts out there and I strongly suggest using one after carefully configuring it. The part of these scripts that will keep you out of trouble is the one that will refuse to run a sync if the number of missing files exceeds a user defined threshold, and how many times the script should run before ignoring the threshold and performing a sync anyway.


    SnapRAID will ignore mergerfs mountpoints.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 5.x on ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.

  • @thedarkness Thank you for your info on the script, but I hate to ask this question, how automate script within Schedule Job
    I know that I need to rename the file to have extension: sh ---> new snapraid script.txt ====> change to: new snapraid script.sh


    I have 2 parity disks in my array, what minor configuration is required to be done within the script?


    Use the script attached here, but I have not tested it because I don't have split parity. Let me know if it runs.



    1. Rename the script to new-snapraid.sh
    2. Place it in a convenient folder, i suggest /usr/sbin/user
    3. Change permissions so the file can be executed, use WinSCP and enable read-write-execute to the owner at least
    4. Create a sheduled job in OMV, the path you'll have to type is will be something similar to /usr/sbin/user/new-snapraid.sh

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!