Snapraid diff - wrong threshold condition?

  • Hi,


    This is my first post, so hello to everyone.


    I have some strange problem with latest omv-snapraid-diff. There was a thread with "else" error, but it was updated already. Now, tonight strange thing happened:


    First of all - scheduled diff completed and stopped - please note the start and stop times :


    Now, second mail - this is something new:


    So one second AFTER diff, there goes SYNC script, that says 1165 is BELOW 100 and start SYNCINC. I'm no script master, but trying to view current latest omv-snapraid-diff, it seems to me, there is no exit condition after DELETE threshold check. Can someone please confirm, if it's a bug, or there is something wrong with my installation, and I shall configure it from sractch?
    I have NO separate cron job for sync job - looks like its called from diff.
    All the threshold stuff was working till last two updates, but it's strange no one is complaining about it :/

  • I'm seeing this exact problem as well. Update threshold disabled (0), delete threshold enabled (50). Deleted files: 137, so it shouldn't run a SYNC, but it does anyways. I've disabled the script and am running manual diff/sync for now. Hopefully this gets fixed soon.

  • I've noticed the same issue. I haven't dug in too much, but my first guess is maybe it has something to do with the update threshold? I've been using snapraid-diff for years, and never had an issue until recently (within a month), and have noticed that an update threshold (which I have disabled) is now included in my daily sync emails. The behavior is the same as described above. It seems like the script is running twice. The first email I get gives a warning that deleted files is above the threshold, then a second email (just after) comes in saying that sync is complete.


    Example of first email where it works correctly to stop the sync:
    201615 equal
    87 added
    340 removed
    328 updated
    0 moved
    0 copied
    0 restored
    There are differences!
    SnapRAID DIFF finished - Sun Sep 8 23:30:33 EDT 2019
    ----------------------------------------
    Changes detected [A-87,D-340,M-0,C-0,U-328] -> there are updated files (328) but update threshold (0) is disabled. Running SYNC Command
    Number of deleted files (340) exceeded threshold (20). NOT proceeding with sync job.
    Please run sync manually if this is not an error condition


    Yours, SnapRAID-diff script



    Then right after the sync proceeds:
    Changes detected [A-87,D-340,M-0,C-0,U-328] -> deleted files (340) is below threshold (20). Running SYNC Command
    SnapRAID SYNC Job started - Sun Sep 8 23:30:34 EDT 2019

    • Offizieller Beitrag

    I wish someone who actually uses snapraid (not me) would update the script to be closer to this script - https://zackreed.me/snapraid-split-parity-sync-script/

    omv 7.0.4-2 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.10 | compose 7.1.2 | k8s 7.0-6 | cputemp 7.0 | mergerfs 7.0.3


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • ryecoaaron, I'd agree. I'm pretty novice at coding, and this type of script change and edits are way beyond my abilities. However, looking at the link you sent, the options do look cleaner, with a lot of added abilities to proceed with or stop the sync.


    I looked at the current snapraid-diff script some more, and it looks like the sync command (line 446) happens regardless of update or delete thresholds (there's no if block for the sync command except if there are changes). Note, I didn't copy that code here due to the message length. So from what I can tell from current script, if the update threshold is exceeded, an email will be sent, but it will continue with the other checks and continue to sync. If the delete threshold is exceeded, similarly an email will send, but the sync will continue. This also explains why we're getting multiple emails from the same sync call. There's no "exit" if the thresholds are exceeded.


    I've made a new file snapraid-diff.fix and I manually commented out the update threshold (since I don't use it) and added an else for the sync and scrub commands - so they won't execute if the deleted files is above the threshold. Note this isn't very clean, but it's a quick fix that works for me. Here's line 385 and down as an example if this helps others. Again, probably not a good long term fix, but more of a makeshift that seems to work for me in a few quick tests:



    • Offizieller Beitrag

    I use the script that can be found here and have no problems with it.

    I could remove the diff script and diff script parameters from the plugin if people used that. So, I like that idea :)

    omv 7.0.4-2 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.10 | compose 7.1.2 | k8s 7.0-6 | cputemp 7.0 | mergerfs 7.0.3


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • I use the script that can be found here and have no problems with it.


    https://gist.github.com/mtompk…be36064c237da3f39ff5cc49d


    There are others that can be found by searching.

    Can you use it as a drop in - i.e. without installing anything else and just modifying a few parameters? Looking at the script, it says it requires mailx and python markdown. I guess my concern is having to monkey with something else, install a few more programs, etc. The nice thing about the snapraid plugin with OMV is you just push a button to install, configure with the webui, and done. Very simple, and a few buttons to push.


    Personally, I realize it means maintenance, but I like the parameters in the plugin. I'd imagine most beginners would too, vs trying to play with a different script and make sure it works for their needs.


    I'll dig in more into the mtompkins script and the existing v 1.1 omv-diff script and see if I can combine some new functions into it. May take a little while as I'm still learning / pretty newb at bash coding.

  • I don't recall having to install anything extra to use that script, whatever it requires must already be installed in my use case.


    As I said there are other scripts out there. I used to use this one:



    Code
    https://gist.github.com/bfg100k/87a1bbccf4f15d963ff7

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • All - I know there's other scripts for SnapRaid, but I didn't spend too much time looking in to those as it seemed like more of starting over, vs just having the plugin that worked as it always had. Anyway, I created a fork, where I corrected the omv-snapraid-diff script to correctly stop the sync if the deleted or update threshold is met or exceeded. I've been using this updated script for almost a month with zero issues. This update / correction can be merged with the main branch, but we need some more testers per discussion with ryecoaaron.


    To those that are still using the plugin script and having this issue of sync continuing despite thresholds being met, you can download the omv-snapraid-diff here: https://github.com/danmcg13/op…raid/tree/master/usr/sbin. If you do use it please report back with any issues you have or if it's working great reply back with that too so we can get more testing in. Hope this helps!


    Note, this doesn't add any new features or improve the script, but just corrects the issue of the script telling snapraid to sync regardless of updated or deleted file counts, as it does today. Thanks.

  • I use the script that can be found here and have no problems with it.


    https://gist.github.com/mtompk…be36064c237da3f39ff5cc49d


    There are others that can be found by searching.

    I too have the issue with the built in "omv-snapraid-diff" not adhering to the thresholds set. I have set the script you recommend to run under scheduled jobs in the gui, but am getting the following error.

    Code
    Error #0:
    OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export SHELL=/bin/sh; sudo --shell --non-interactive --user='root' -- /var/lib/openmediavault/cron.d/userdefined-83a50b50-0695-4aed-a4b0-6389adf7b955 2>&1' with exit code '127': /var/lib/openmediavault/cron.d/userdefined-83a50b50-0695-4aed-a4b0-6389adf7b955: 4: /var/lib/openmediavault/cron.d/userdefined-83a50b50-0695-4aed-a4b0-6389adf7b955: /home/scripts/snapScript.sh: not found in /usr/share/openmediavault/engined/rpc/cron.inc:185
    Stack trace:
    #0 /usr/share/php/openmediavault/rpc/serviceabstract.inc(588): Engined\Rpc\Cron->Engined\Rpc\{closure}('/tmp/bgstatusXn...', '/tmp/bgoutputXu...')
    #1 /usr/share/openmediavault/engined/rpc/cron.inc(189): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure))
    #2 [internal function]: Engined\Rpc\Cron->execute(Array, Array)
    #3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
    #4 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('execute', Array, Array)
    #5 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Cron', 'execute', Array, Array, 1)
    #6 {main}
  • Verify that the location you gave for the script in the scheduled job is correct, that the script is actually there, and it has its executable bit set.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • Verify that the location you gave for the script in the scheduled job is correct, that the script is actually there, and it has its executable bit set.

    Sorry, I should have stated that I did verify that before posting.

    Code
    root@OpenMediaVault:/home/scripts# ls -l
    total 16
    -rwxrwxrwx 1 root root 16054 Feb 14 18:18 snapScript.sh
  • Can you execute the script from the command line and if so what does it spit out?


    You didn't mention this so I'll ask - did you properly edit the script's USER CONFIGURATION section?

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • Can you execute the script from the command line and if so what does it spit out?


    You didn't mention this so I'll ask - did you properly edit the script's USER CONFIGURATION section?

    I did. Speaking of that, what is correct for having the script email the output? In the gui, there's the option to email the output, which works for the builtin "omv-snapraid-diff" script. Not sure what to set in the user config section of this script, if anything all, or just comment out, as I've not gotten it to run yet.

  • Can you execute the script from the command line and if so what does it spit out?


    You didn't mention this so I'll ask - did you properly edit the script's USER CONFIGURATION section?

    Sorry I missed this question before my last post. When I am ready to run a sync later, I will try and post back the results.

  • Put a valid email address in for EMAIL_ADDRESS=


    You can use a fully qualified email address or a bare username on the system that gets mail and that you read that user's mail.


    Make sure you have installed the specified requirements package python-markdown


    I suggest also installing the discount package and making the below change in the script:


    Change this line from:

    $MAIL_BIN -a 'Content-Type: text/html' -s "$SUBJECT" "$EMAIL_ADDRESS" < <(python -m markdown $TMP_OUTPUT)


    Change to:

    $MAIL_BIN -a 'Content-Type: text/html' -s "$SUBJECT" "$EMAIL_ADDRESS" < <(markdown $TMP_OUTPUT)

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • Can you execute the script from the command line and if so what does it spit out?


    You didn't mention this so I'll ask - did you properly edit the script's USER CONFIGURATION section?


    Sorry I missed this question before my last post. When I am ready to run a sync later, I will try and post back the results.


    I ran the script from the command line and it failed with the following:

    Code
    root@OpenMediaVault:/home/scripts# ./snapScript.sh
    -bash: ./snapScript.sh: /bin/bash^M: bad interpreter: No such file or directory

    It seems it may not have liked some characters or text formatting in the script. I originally pasted it in notepad++, in windows 10, edited, then copied it over to OMV.

    I rebuilt the script by pasting directly into nano on OMV, then making the user section edits there. Now it will run from the command line. It will probably run fine from the schedule now as well.


    Thank you for the quick replies and tips for changing the mail parameters in the script.

  • OK, good luck.

    Still having an issue. I ended up running a manual sync last night because the script wouldn't run all the way. The script starts to run and then will sit, as shown below, indefinitely. If I run top when I start the script, I can see the snapraid process running but then the snapraid process is no longer in the list when the script freezes.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!