[Solved]SnapRaid + Unionfilesytem + Cron = snapraid sync (scheduled to run hourly?)

    • OMV 5.x (beta)
    • Resolved
    • Very, very few users run split parity. Is there some reason you suggested a script for it?

      Regular script here:

      Source Code

      1. https://gist.github.com/bfg100k/87a1bbccf4f15d963ff7
      --
      Google is your friend and Bob's your uncle!

      RAID - Its ability to disappoint is inversely proportional to the user's understanding of it.

      ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.
    • gderf wrote:

      wepee wrote:

      I am not familiar with linux commands, I have no idea about how to run script within OMV after doing Google search for 1-2 hours.
      I took this piece from your post, "run script within OMV" and googled it.

      Source Code

      1. https://www.google.com/search?q=run+script+within+OMV&oq=run+script+within+OMV&aqs=chrome..69i57&client=ubuntu&sourceid=chrome&ie=UTF-8
      It pointed me to the exact correct link, first in the list:



      Source Code

      1. https://forum.openmediavault.org/index.php/Thread/12562-run-script-on-OMV-from-gui/
      Maybe I'm just lucky?
      @gderf yes I did try this method but none of them mentioned about specifically installing script for SYNC.

      If I appear to be begging forum user, to spoon feed me information, then I apologize in advance.

      As I am novice in the area of Linux, I did not want to experiment. I need to get the exact step by step details before I attempt to install it.
      otherwise I may screw this whole project, and start it all over again.

      I did some reading on this forum.openmediavault.org/index…?postID=225307#post225307
      But I stop reading on page 8 or 10, I could not remember.

      While I was searching for answer in this area, I did stumble a FAQ about inserting script at: forum.openmediavault.org/index…/5594-OpenMediaVault-FAQ/

      This is what I found.........

      The post was edited 1 time, last by wepee ().

    • thedarkness wrote:

      wepee wrote:

      thedarkness wrote:

      wepee wrote:

      I am trying to learn as much about the little known SNAPRAID technology from its FAQ: snapraid.it/faq#whatisit
      But I could not find an answer......to my questions below

      1) What happen, if you accidentally write a file / update a file to the disk while SYNC operation started?

      2) Should SYNC operation ONLY start when there is not disk write/read operation?

      3) My NAS does not TURN ON 24/7, but only when I wants to backup data on demand basis, so what is the best time to do:
      SYNC - just before I turn off the NAS?
      SCRUB - just do it once a month should be sufficed, it depends how I would like to prevent bit-rot to occur?
      1 - I don't know, that's why you should run it after hours when there's no activity on the server. Snapraid first lists files and then syncs it, if becomes missing in this process I guess will throw an error2- This would be the ideal condition3- My NAS is not running all the time either. I would recommend using the script I mentioned above and run the script when you turn on the system every day. You will know it's time to use the server because you will get an email report.
      SCRUB frequency: it's totally up to you and how much you want to gamble. Bit rot is rare, but you don't need to scrub the whole system in one run every month. You can scrub a little part every day or so, like the script does.
      @thedarkness Thank you for your info on the script, but I hate to ask this question, how automate script within Schedule JobI know that I need to rename the file to have extension: sh ---> new snapraid script.txt ====> change to: new snapraid script.sh

      I have 2 parity disks in my array, what minor configuration is required to be done within the script?
      Use the script attached here, but I have not tested it because I don't have split parity. Let me know if it runs.


      1. Rename the script to new-snapraid.sh
      2. Place it in a convenient folder, i suggest /usr/sbin/user
      3. Change permissions so the file can be executed, use WinSCP and enable read-write-execute to the owner at least
      4. Create a sheduled job in OMV, the path you'll have to type is will be something similar to /usr/sbin/user/new-snapraid.sh
      @thedarkness OK thanks for your steps given, most appreciated, I will give it a try. :thumbsup:
    • wepee wrote:

      @thedarkness OK thanks for your steps given, most appreciated, I will give it a try. :thumbsup:
      You do not want to run a split parity SnapRaid helper script unless you are configured for split parity disks , and you aren't. You are running multiple parity disks. These are completely different things.
      --
      Google is your friend and Bob's your uncle!

      RAID - Its ability to disappoint is inversely proportional to the user's understanding of it.

      ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.
    • thedarkness wrote:

      Well... Because he has two parity drives, so I assumed he has this kind of config.
      Multiple parity drives and split parity drives are completely different things.
      --
      Google is your friend and Bob's your uncle!

      RAID - Its ability to disappoint is inversely proportional to the user's understanding of it.

      ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.
    • gderf wrote:

      thedarkness wrote:

      Well... Because he has two parity drives, so I assumed he has this kind of config.
      Multiple parity drives and split parity drives are completely different things.
      @gderf Ok thanks for the heads up! :thumbup:

      In that case, I have 2 options:
      A) I will try the very first script that @thedarkness shown me earlier: forum.openmediavault.org/index…-new-snapraid-script-txt/
      or
      B) I will stick with manual SYNC and SCRUB for simplicity sake.
      SYNC once a week
      SCRUB once every 2 weeks

      I will be most likely stick with option B) just to make my life simple.

      Thank you all who has provided valuable advice to my question/problem. :)
    • gderf wrote:

      thedarkness wrote:

      Well... Because he has two parity drives, so I assumed he has this kind of config.
      Multiple parity drives and split parity drives are completely different things.
      Thanks for the heads up, I completely misunderstood it. Right so is he running 2 parity disk to be fault tolerant with 2 drives?
      I'll get the script updated for this scenario too. I tried many things and for my use this script the best: set and forget, and if something goes wrong, a daily script mail will tell me.
      /// OMV BUILD IN PROGRESS - MY NAS KILLER /// omv 5.x + omvextrasorg

      i3 8300 - ASRock H370M-ITX/ac - 8GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker), 3x4TB HDD (Data) - Node 304 - Be quiet! Pure Power 11 350W
    • He is running two parity disks.

      How do I know this? Two ways. He posted his disk configuration. And the current OMV SnapRAID plugin does not support split parity, and he didn't mention configuring it by hand which would be required for split parity.

      I already posted one of the available proper scripts for standard implementations. Google for more.

      The larger problem with things like this is that many just don't understand what it does and more importantly its limitations. And of course very, very few have ever simulated a disk failure and actually tried to recover from it.
      --
      Google is your friend and Bob's your uncle!

      RAID - Its ability to disappoint is inversely proportional to the user's understanding of it.

      ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.
    • @wepee

      Try the script I have attached, should be working with your setup.
      Files
      /// OMV BUILD IN PROGRESS - MY NAS KILLER /// omv 5.x + omvextrasorg

      i3 8300 - ASRock H370M-ITX/ac - 8GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker), 3x4TB HDD (Data) - Node 304 - Be quiet! Pure Power 11 350W
    • thedarkness wrote:

      @wepee

      Try the script I have attached, should be working with your setup.
      You should also post the source of the script. Please provide the URL.
      --
      Google is your friend and Bob's your uncle!

      RAID - Its ability to disappoint is inversely proportional to the user's understanding of it.

      ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.
    • gderf wrote:

      thedarkness wrote:

      @wepee

      Try the script I have attached, should be working with your setup.
      You should also post the source of the script. Please provide the URL.
      I think the source of the script is within the txt file:
      # Inspired by Zack Reed (zackreed.me/articles/83-updated-snapraid-sync-script)
      # Modified version of mtompkins version of my script ()
      # Updated by sburke to fix behavior on Debian 10
    • thedarkness wrote:

      @wepee

      Try the script I have attached, should be working with your setup.
      @thedarkness thank you for following this thread, and updated the script. :thumbsup:

      As of now, the more I read about the comments/descriptions denoted by "#" the txt file of: snapraid script - dual parity.txt
      the more silly question that I have in mind.
      I decided to save it for another day.
      When I am more a pro in writing bash scripts then I may understanding better.
      Bottom line is, I simply not confident in running the script unless I get a FULL understanding it works.
      What is the pitfalls? what is the thing that I need to look for?
      It is better to get a full picture on how whole the script works, rather than install it and run it blindly without
      knowing the consequence. Otherwise, I will back to the forum asking for help if I do run into trouble. :/

      The post was edited 1 time, last by wepee ().

    • gderf wrote:

      thedarkness wrote:

      @wepee

      Try the script I have attached, should be working with your setup.
      You should also post the source of the script. Please provide the URL.
      Woops, I forgot.
      It's from here zackreed.me/snapraid-split-parity-sync-script, but does not work as-is, you need to change some things you can find in the comments in order to get it working on OVM5 and dual parity or single parity.

      wepee wrote:

      thedarkness wrote:

      @wepee

      Try the script I have attached, should be working with your setup.
      @thedarkness thank you for following this thread, and updated the script. :thumbsup:
      As of now, the more I read about the comments/descriptions denoted by "#" the txt file of: snapraid script - dual parity.txt
      the more silly question that I have in mind.
      I decided to save it for another day.
      When I am more a pro in writing bash scripts then I may understanding better.
      Bottom line is, I simply not confident in running the script unless I get a FULL understanding it works.
      What is the pitfalls? what is the thing that I need to look for?
      It is better to get a full picture on how whole the script works, rather than install it and run it blindly without
      knowing the consequence. Otherwise, I will back to the forum asking for help if I do run into trouble. :/
      I understand your point. The goal of the script is actually to make things easier for somebody who does not want to run many commands manually.
      As I've given the script to you, it will always run a sync whenever the script is triggered. This **could** be an issue if you deleted many files by mistake, but you should backup your files offsite anyway. If you want you can enable trigger conditions based on a number of deleted or updated files: if the threshold is reached, sync will not be triggered.
      The best for you is to read and ask questions, I will answer when I have time, but I believe the information in the ##comments is well documented.
      Also you could test this whole thing in a Virtual Machine with a bunch of virtual drives configured and with some files copied on them.
      /// OMV BUILD IN PROGRESS - MY NAS KILLER /// omv 5.x + omvextrasorg

      i3 8300 - ASRock H370M-ITX/ac - 8GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker), 3x4TB HDD (Data) - Node 304 - Be quiet! Pure Power 11 350W
    • wepee wrote:

      I think the source of the script is within the txt file:# Inspired by Zack Reed
      Nope.
      --
      Google is your friend and Bob's your uncle!

      RAID - Its ability to disappoint is inversely proportional to the user's understanding of it.

      ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.

      The post was edited 1 time, last by gderf ().

    • wepee wrote:

      @thedarkness thank you for following this thread, and updated the script. :thumbsup:
      As of now, the more I read about the comments/descriptions denoted by "#" the txt file of: snapraid script - dual parity.txt
      the more silly question that I have in mind.
      I decided to save it for another day.
      When I am more a pro in writing bash scripts then I may understanding better.
      Bottom line is, I simply not confident in running the script unless I get a FULL understanding it works.
      What is the pitfalls? what is the thing that I need to look for?
      It is better to get a full picture on how whole the script works, rather than install it and run it blindly without
      knowing the consequence. Otherwise, I will back to the forum asking for help if I do run into trouble. :/

      You are not going to find that kind of insight within the script. It is written by a chain of highly experienced users of the software and its audience is those who have at least a working understanding of the software.

      You need to read the SnapRAID documentation for this, not the script.

      Source Code

      1. https://www.snapraid.it/

      And I'll go even further. You will be making a major mistake by relying on software like SnapRAID without actually testing it to determine that you can recover from lost files or a lost disk. And the time to learn how to do this is before you really need to do this.
      --
      Google is your friend and Bob's your uncle!

      RAID - Its ability to disappoint is inversely proportional to the user's understanding of it.

      ASRock Rack C2550D4I C0 Stepping - 16GB ECC - Silverstone DS380 + Silverstone DS380 DAS Box.
    • yeah the first time you experience a failure can be pretty nerve wrecking. Being prepared and knowing what to do would be ideal, even though I didn't. I had a couple drives fail sooner than I had anticipated. I say 'fail' but they were only in a pre-fail state so I was able to rsync to a new drive and didn't really require a rebuild from parity. Just a few files got corrupted and snapraid fixed them up.

      Post by gderf ().

      This post was deleted by the author themselves ().