[HowTo] SnapRAID in OMV

Hello everyone,

I'm setting up my new OMV5 server and configuring the SnapRAID plugin is my current task. I'm in the same boat as aroundmyroom and am wondering if there any documentation on what the plug-in can and can't do? I haven't really found anything as in-depth as I would have liked. I found a Plugin Guide for an older version that was written by WastlJ in 2014, but nothing more recent.

Here is what I'm currently seeking info for:

1. Diff Script in settings - wanting to learn exactly what its doing so I can decide how to best set the settings for my uses or if I am better served with using a different third party script as gderf and thedarkness are doing.
2. Rules - Why does it need a volume when inputting a rule? My understanding is that SR applies exclusions across all drives? I want to exclude .DS_Store across all drives.
   
3. Information Tab - there are not entries for the tools Prehash+Sync, Fix Silent or Undelete. Prehash+Sync and Undelete seem pretty straight forward based on the SR manual, but I'm not sure what Fix Silent is?
   
   Thanks for the help!

Zitat von hotsauce

Hello everyone,

I'm setting up my new OMV5 server and configuring the SnapRAID plugin is my current task. I'm in the same boat as aroundmyroom and am wondering if there any documentation on what the plug-in can and can't do? I haven't really found anything as in-depth as I would have liked. I found a Plugin Guide for an older version that was written by WastlJ in 2014, but nothing more recent.

Here is what I'm currently seeking info for:

1. Diff Script in settings - wanting to learn exactly what its doing so I can decide how to best set the settings for my uses or if I am better served with using a different third party script as gderf and thedarkness are doing.
2. Rules - Why does it need a volume when inputting a rule? My understanding is that SR applies exclusions across all drives? I want to exclude .DS_Store across all drives.
3. Information Tab - there are not entries for the tools Prehash+Sync, Fix Silent or Undelete. Prehash+Sync and Undelete seem pretty straight forward based on the SR manual, but I'm not sure what Fix Silent is?
   
   Thanks for the help!

1. To learn those settings you'd better start with the snapraid manual, then search what people do with these options. After some research I came to the conclusion that SnapRAID is extremely powerful but MUST stay under control to be effective. That's why I am endorsing the use of scripts that do everything for you, with some logic built in. I also get daily notifications after the script has run. Annoying? Maybe. I know what happens every day? Absolutely yes.

2. I had the same question when I added a couple of exclusion rules. Just pick wathever drive, I noticed that if you open the SnapRAID config file, the rule will apply to all drives. Check and let me know. Maybe we need to get the GUI updated.

3. I think it's the SnapRAID feature called fix (manual section 5.8), which will fix any error found in parity data. I don't use those settings in the GUI (aside from Status a couple of times) because I don't have the need for it.

An advice: don't focus too much on scrubbing. Noways, data corruption during transfers is rare, very rare, and scrubbing can take a long time, depending on how much data you have. Obviously you want to scrub your data to ensure its integrity, but set it to a little value. SnapRAID's defaults are 8% of data older than 10 days.

My boat is sailing hotsauce

I have checked a lot. I am not using the diff script where you can add your own sauce to the supplied plugin script. Actually all is derived from each other.

I am now up with my new 30TB NAS for almost 1 week

Syncing started when I added the 14TB of data. After that I created an issue myself as in the diff for snapraid was stated that I wanted to scrub 100% of my data. Uhhh why? I dunno it stood configured. it took some time, but it was all ok. Now I have changed that. to 10% and 6 days. First days I checked the logs to see if there where some mount points or data folders I could exclude. There were and I added them.

Than I had to test a lot of things with SAMBA, this caused a lot of issues with UnionFS and Mergefs with/without issues for SNAPraid

I noticed that I had a high load on the server (higher than 1.0) and this is not common. Due to the speedtests I did during sync or scrub I made false assumptions.

High load was caused by the multiple re-configurations I did (probably). After a reboot no further issues found. Load is fine

auanasgheps
1. I'll take a look through the manual and the script (found the location of the script from another post) and report back with specific questions. Also, You posted the scripts you use earlier in this thread, right? I'll have to take a look at that after i look at the built in script.

2. It looks like it does that, so I'd agree that the GUI needs to be updated. Here's how it shows up in the config file, so I'm not sure why the GUI needs to ask for a volume. You can see that the volume is commented out.

exclude *.unrecoverable
exclude lost+found/
exclude aquota.user
exclude aquota.group
exclude /tmp/
exclude .content
exclude *.bak
exclude /snapraid.conf*

# Rule selected from /srv/dev-disk-by-label-Media1
exclude .DS_Store

3. I'm completely guessing here, but I'm gonna guess that 'Fix' in the plugin is just the regular fix in snapraid where according to the manual it "doesn't differentiate between errors and intentional modifications. It unconditionally reverts the file state at the last "sync"." And that 'Fix Silent' in the plugin is the section in 5.8 where it says "-e, --filter-error option. As difference from other filter options, with this one the fixes areapplied only to files that are not modified from the the latest "sync"."

No idea how to confirm my guess. Is there a way to see what commands the GUI runs when you press a button?

aroundmyroom - Since you're not using the built in script, did you end up using a script that someone posted here, or did you combine bits and pieces to make your own?

Zitat von hotsauce

auanasgheps
1. I'll take a look through the manual and the script (found the location of the script from another post) and report back with specific questions. Also, You posted the scripts you use earlier in this thread, right? I'll have to take a look at that after i look at the built in script.

2. It looks like it does that, so I'd agree that the GUI needs to be updated. Here's how it shows up in the config file, so I'm not sure why the GUI needs to ask for a volume. You can see that the volume is commented out.

Code

exclude *.unrecoverable
exclude lost+found/
exclude aquota.user
exclude aquota.group
exclude /tmp/
exclude .content
exclude *.bak
exclude /snapraid.conf*

# Rule selected from /srv/dev-disk-by-label-Media1
exclude .DS_Store

Alles anzeigen

3. I'm completely guessing here, but I'm gonna guess that 'Fix' in the plugin is just the regular fix in snapraid where according to the manual it "doesn't differentiate between errors and intentional modifications. It unconditionally reverts the file state at the last "sync"." And that 'Fix Silent' in the plugin is the section in 5.8 where it says "-e, --filter-error option. As difference from other filter options, with this one the fixes areapplied only to files that are not modified from the the latest "sync"."

No idea how to confirm my guess. Is there a way to see what commands the GUI runs when you press a button?

aroundmyroom - Since you're not using the built in script, did you end up using a script that someone posted here, or did you combine bits and pieces to make your own?

1. Use the script I posted attached to this post. I started from another script and made some adjustments to make it work with OMV. So far it's the most complete I have seen and takes care of all activities.

2. Fine, I will open an issue in github

Zitat von ryecoaaron

Because the directory tree view needs a volume. Not everyone is using unionfilesystems either. So, they have diffferent folders on each volume. If I did put a root directory browser, I would have to figure out how to get the relative path of the folder you are selecting (tough if I don't know the filesystem mountpoint) since the config is not using an absolute path.

Does that mean that there are multiple config files that the plugin is referencing to make an overall config file? Otherwise I'm not understanding where the relative path and absolute path is relevant to snapraid exclusion rule making because my understanding is that these exclusions patterns/filters are applied to all paths/files that SR is processing. In the snapraid manual, I'm looking at sections 7.7, 7.14, and 8 as the relevant sections for this.

I'd say the issue is the potential confusion. The way I understood that tab in the plugin initially while setting it up this week, was that if I wanted to exclude .DS_Store from being processed on all my volumes, I would have to add the exclusion rule several times so that it would be excluded from each volume.

So the volume selection is there for PATH/FILE and PATH/DIR as per the Patterns in section 8? If that is the case, maybe that could be clarified in the Rules tab? Or perhaps a modification where volume only becomes a selectable option if the user would like to do a specific file/folder?

auanasgheps read through the actual scripts and i'm feeling much better about it. I think if there was some more information in the information tab, that would probably be very beneficial for everyone. The top of the actual diff script file actually has a nice summary that would be great for the information tab. I could probably take a whack at writing out some more details for the info tab if the plugin managers would be willing to use it.

The biggest lines I was unsure about on the settings tab were how 'run scrub', scrub frequency, and scrub percentage tied in to the diff script. After reading through the script, and cross referencing the manual, this is my understanding. Please correct me if im wrong!

Run Scrub - After Syncing of parity is complete, run scrub -p $SCRUB_PERCENT -o $SCRUB_OLDER_THAN_DAYS

Scrub frequency is $SCRUB_OLDER_THAN_DAYS, and Scrub Percentage is $SCRUB_PERCENT, which is explained in section 4.1 of the snapraid manual. My understanding is that it will scrub up to $SCRUB_PERCENT of the array as long as the data hasn't been scrubbed in the past $SCRUB_OLDER_THAN_DAYS. So the default is to scrub the entirity of the data that hasn't been scrubbed in the last 100 days. The scrub isn't mentioned at the top of the script where it says what the script does, seems like scrub should be step 5 in that list with the email summary being step 6.

All that said, I read the script you provided and I like it even more. It looks like it can be configued to do everything I want, except for automating some scrubs (which i may or may not do, still thinking). But that is probably better left to another script or maybe a cronjob.

Your understanding about $SCRUB_PERCENT and $SCRUB_OLDER_THAN_DAYS are correct.

Honestly I've never read those steps at the top of the script, but I'll add the scrub description.

I will work with ryecoaaron to get this script integrated into the plugin. I think everyone would benefit, since is a huge leap forward the currently integrated script.

I will also propose default values which will work for many people, like scrub 8% of the array older than 10 days (default values from snapraid manual), so it will be even easier, along with better description of what each function does.

Zitat von ryecoaaron

No, it is there so that the plugin's directory tree can be populated.

Can you explain how the directory tree is used? I've read through the snapraid manual as well as your comments on the github issue that thedarkness opened, and I'm still not grasping it.

Zitat von ryecoaaron

Do you see the folder button on the right side of the Rule textbox? The volume populates that treeview. This really has nothing to do with snapraid other than the end result is the Rule path.

ok, i'm tracking now. So I think to remove the potential confusion, the rule tab would either need some more explanation or be rewritten in a way that hides the volume selection unless the user needs to select a file or folder to have the rule populated for them.

Zitat von hotsauce

aroundmyroom - Since you're not using the built in script, did you end up using a script that someone posted here, or did you combine bits and pieces to make your own?

I am using the GUI and the default plugin script. I have a few rules enabled and all is working. I want to be a 'generic' user of OMV without being in the shell all the time, but I do not see why the modified script would give me more benefit than what the default delivered script is giving me with GUI help as well. The question I am asking myself is why is that modified script not being used by the plugin builder .. and of course the existing script could and may be updated to get more features and functions. The only question is, do I need it? The only thing I want is that in case of emergency I must be able to restore a disk.

Zitat von aroundmyroom

I am using the GUI and the default plugin script. I have a few rules enabled and all is working. I want to be a 'generic' user of OMV without being in the shell all the time, but I do not see why the modified script would give me more benefit than what the default delivered script is giving me with GUI help as well. The question I am asking myself is why is that modified script not being used by the plugin builder .. and of course the existing script could and may be updated to get more features and functions. The only question is, do I need it? The only thing I want is that in case of emergency I must be able to restore a disk.

What I particularly like about the script from thedarkness is that it can force a sync if you've ignored the errors for a few rounds of the script running as a fail safe

Zitat von ryecoaaron

There are just some things that are very difficult to do in the web interface. How would the plugin "know" when want to select something from the browser? Clicking on the folder button and not doing anything other than exposing the volume dropdown is more confusing to me.

I would rather people just create documentation on how to use plugins. I really dislike putting tons of notes everywhere in the plugin itself. I will try to make the volume optional.

My initial thought for changing the plugin would be for it to be more of a flow chart that exposes relevant options as the user selects what they want to do.

Choose rule type: inclusion or exclusion?
What would you like to filter? Type of File, Type of Directory, Specific File, Specific Directory
If specific, then expose the volume dropdown so they can choose that specific file or directory. Otherwise, type in the name of the file type or directory type.

I'm going to do my best to find some time to learn github and how to add some more documentation to the information tab, and then after that maybe make a new plug in tutorial for the latest version.

Just a simple question: snapraid is generating a message which is large in size (5 to 10MB), no issue receiving, but outlook does not like this when having it some time in a folder stored. (it locks up Outlook)

Is there a way to get snapraid messages that it is finished or having a failure but without all detailed info (note: I do not use debug, I do use the plugin diff script)