Snapraid SCRUB-Cycle count not taking days with nothing to do in consideration

Unipo · Sep 23rd 2020

Hi,

It is in my understanding that in the snapraid configuration page, when you click on "Schedule Diff" then save it create a script "/usr/sbin/omv-snapraid-diff" populated with the info you put under "Diff Script settings" in the said configuration page.

My needs (might evolve) were to sync everyday without updated/deleted thresholds and scrub 25 percent of the array every week.

It appeared to be working great the first few days as I had always new of updated files. And each morning I had the pleasure to be greeted by a nice email keeping me informed of what happened: (Seriously, I love it)

Code

SnapRAID Script - Version: 1.1.0

running SnapRAID 11.3 by Andrea Mazzoleni, http://www.snapraid.it

---------------------------------------------

SnapRAID Job started - Sun Sep 20 00:00:01 CEST 2020

Warnings about 'Ignoring mount point' are possibly due to AUFS or mhddfs-Pooling in OMV


SnapRAID DIFF started - Sun Sep 20 00:00:01 CEST 2020

Loading state from /srv/dev-disk-by-label-s1/snapraid.content...

Comparing...

add [Edited]


12183 equal

1 added

0 removed

0 updated

0 moved

0 copied

0 restored

There are differences!

SnapRAID DIFF finished - Sun Sep 20 00:00:04 CEST 2020

----------------------------------------

Changes detected [A-1,D-0,M-0,C-0,U-0] -> there are updated files (0) but update threshold (0) is disabled. Running SYNC Command

Changes detected [A-1,D-0,M-0,C-0,U-0] -> there are deleted files (0) but delete threshold (0) is disabled. Running SYNC Command

SnapRAID SYNC Job started - Sun Sep 20 00:00:04 CEST 2020

----------------------------------------

Self test...

Loading state from /srv/dev-disk-by-label-s1/snapraid.content...

Scanning disk s1...

Scanning disk s2...

Scanning disk s3...

Using 135 MiB of memory for the file-system.

Initializing...

Hashing...

0%, 0 MB 1%, 86 MB 6%, 316 MB 11%, 549 MB 16%, 789 MB 22%, 1026 MB, 235 MB/s, CPU 7%, 0:00 ETA 27%, 1261 MB, 235 MB/s, CPU 7%, 0:00 ETA 32%, 1497 MB, 235 MB/s, CPU 7%, 0:00 ETA 37%, 1736 MB, 235 MB/s, CPU 7%, 0:00 ETA 42%, 1974 MB, 235 MB/s, CPU 7%, 0:00 ETA 47%, 2201 MB, 235 MB/s, CPU 7%, 0:00 ETA 52%, 2432 MB, 235 MB/s, CPU 7%, 0:00 ETA 57%, 2672 MB, 235 MB/s, CPU 7%, 0:00 ETA 62%, 2903 MB, 235 MB/s, CPU 7%, 0:00 ETA 67%, 3136 MB, 235 MB/s, CPU 7%, 0:00 ETA 72%, 3376 MB, 235 MB/s, CPU 7%, 0:00 ETA 77%, 3612 MB, 235 MB/s, CPU 7%, 0:00 ETA 82%, 3855 MB, 236 MB/s, CPU 7%, 0:00 ETA 88%, 4093 MB, 236 MB/s, CPU 7%, 0:00 ETA 93%, 4336 MB, 236 MB/s, CPU 7%, 0:00 ETA 98%, 4573 MB, 236 MB/s, CPU 7%, 0:00 ETA 100% completed, 4650 MB accessed in 0:00

Everything OK

Resizing...

Saving state to /srv/dev-disk-by-label-s1/snapraid.content...

Saving state to /srv/dev-disk-by-label-s2/snapraid.content...

Saving state to /srv/dev-disk-by-label-s3/snapraid.content...

Verifying /srv/dev-disk-by-label-s1/snapraid.content...

Verifying /srv/dev-disk-by-label-s2/snapraid.content...

Verifying /srv/dev-disk-by-label-s3/snapraid.content...

Syncing...

Using 32 MiB of memory for 32 cached blocks.

0%, 0 MB 1%, 91 MB 3%, 167 MB 5%, 247 MB 6%, 319 MB 8%, 397 MB, 76 MB/s, CPU 18%, 0:00 ETA 9%, 464 MB, 74 MB/s, CPU 18%, 0:00 ETA 11%, 547 MB, 76 MB/s, CPU 18%, 0:00 ETA 13%, 628 MB, 76 MB/s, CPU 18%, 0:00 ETA 15%, 715 MB, 78 MB/s, CPU 19%, 0:00 ETA 17%, 798 MB, 78 MB/s, CPU 19%, 0:00 ETA 18%, 877 MB, 78 MB/s, CPU 20%, 0:00 ETA 20%, 960 MB, 79 MB/s, CPU 21%, 0:00 ETA 22%, 1042 MB, 79 MB/s, CPU 21%, 0:00 ETA 24%, 1123 MB, 79 MB/s, CPU 20%, 0:00 ETA 25%, 1204 MB, 79 MB/s, CPU 21%, 0:00 ETA 27%, 1275 MB, 79 MB/s, CPU 21%, 0:00 ETA 29%, 1361 MB, 79 MB/s, CPU 21%, 0:00 ETA 31%, 1447 MB, 80 MB/s, CPU 21%, 0:00 ETA 32%, 1523 MB, 79 MB/s, CPU 22%, 0:00 ETA 34%, 1602 MB, 80 MB/s, CPU 22%, 0:00 ETA 35%, 1670 MB, 79 MB/s, CPU 22%, 0:00 ETA 37%, 1761 MB, 80 MB/s, CPU 22%, 0:00 ET

A 39%, 1852 MB, 80 MB/s, CPU 22%, 0:00 ETA 41%, 1925 MB, 80 MB/s, CPU 22%, 0:00 ETA 43%, 2001 MB, 80 MB/s, CPU 23%, 0:00 ETA 44%, 2075 MB, 79 MB/s, CPU 23%, 0:00 ETA 46%, 2155 MB, 79 MB/s, CPU 23%, 0:00 ETA 48%, 2235 MB, 79 MB/s, CPU 23%, 0:00 ETA 49%, 2313 MB, 79 MB/s, CPU 23%, 0:00 ETA 51%, 2391 MB, 80 MB/s, CPU 23%, 0:00 ETA 53%, 2468 MB, 80 MB/s, CPU 23%, 0:00 ETA 54%, 2553 MB, 80 MB/s, CPU 23%, 0:00 ETA 56%, 2630 MB, 80 MB/s, CPU 23%, 0:00 ETA 58%, 2703 MB, 79 MB/s, CPU 22%, 0:00 ETA 59%, 2781 MB, 79 MB/s, CPU 22%, 0:00 ETA 61%, 2858 MB, 79 MB/s, CPU 22%, 0:00 ETA 63%, 2933 MB, 79 MB/s, CPU 22%, 0:00 ETA 64%, 3010 MB, 79 MB/s, CPU 22%, 0:00 ETA 66%, 3087 MB, 79 MB/s, CPU 22%, 0:00 ETA 68%, 3170 MB, 79 MB/s, CPU 22%, 0:00 ETA 69%, 3246 MB, 79 MB/s, CPU 22%, 0:00 ETA 71%, 3327 MB, 79

MB/s, CPU 22%, 0:00 ETA 73%, 3409 MB, 79 MB/s, CPU 22%, 0:00 ETA 75%, 3489 MB, 79 MB/s, CPU 22%, 0:00 ETA 76%, 3566 MB, 79 MB/s, CPU 22%, 0:00 ETA 78%, 3638 MB, 79 MB/s, CPU 22%, 0:00 ETA 79%, 3715 MB, 78 MB/s, CPU 22%, 0:00 ETA 81%, 3802 MB, 79 MB/s, CPU 22%, 0:00 ETA 83%, 3875 MB, 79 MB/s, CPU 22%, 0:00 ETA 85%, 3954 MB, 78 MB/s, CPU 22%, 0:00 ETA 86%, 4026 MB, 78 MB/s, CPU 21%, 0:00 ETA 88%, 4105 MB, 78 MB/s, CPU 22%, 0:00 ETA 90%, 4187 MB, 78 MB/s, CPU 21%, 0:00 ETA 91%, 4262 MB, 78 MB/s, CPU 21%, 0:00 ETA 93%, 4341 MB, 78 MB/s, CPU 21%, 0:00 ETA 95%, 4417 MB, 78 MB/s, CPU 21%, 0:00 ETA 96%, 4495 MB, 78 MB/s, CPU 21%, 0:00 ETA 98%, 4574 MB, 78 MB/s, CPU 21%, 0:00 ETA 100% completed, 4650 MB accessed in 0:00


s1 0% |

s2 22% | *************

s3 0% |

parity 57% | ***********************************

raid 9% | *****

hash 5% | ***

sched 3% | **

misc 1% |

|______________________________________________________________

wait time (total, less is better)


Everything OK

Saving state to /srv/dev-disk-by-label-s1/snapraid.content...

Saving state to /srv/dev-disk-by-label-s2/snapraid.content...

Saving state to /srv/dev-disk-by-label-s3/snapraid.content...

Verifying /srv/dev-disk-by-label-s1/snapraid.content...

Verifying /srv/dev-disk-by-label-s2/snapraid.content...

Verifying /srv/dev-disk-by-label-s3/snapraid.content...

SnapRAID SYNC Job finished - Sun Sep 20 00:01:46 CEST 2020

----------------------------------------

SnapRAID SCRUB-Cycle count (7) not met (6). No scrub was run. - Sun Sep 20 00:01:46 CEST 2020


Yours, SnapRAID-diff script

Display More

BUT one morning, as i was drinking my coffee and checking my emails... None from my NAS...

A tad worried I started looking for the reason, which seems to be an error in the script. (But please tell me if I'm wrong, I am slowly learning...) The logs:

Code

omv-snapraid-diff: INFO: 'SCRUB_OLDER_THAN_DAYS not set in '/etc/snapraid-diff.conf''

omv-snapraid-diff: INFO: 'set it to 0'

omv-snapraid-diff: INFO: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

omv-snapraid-diff: INFO: 'SnapRAID Job started - Script-version: 1.1.0'

omv-snapraid-diff: INFO: 'SnapRAID-Version: 11.3 by Andrea Mazzoleni, http://www.snapraid.it'

omv-snapraid-diff: INFO: 'Checking 'disks' in '/etc/snapraid.conf''

omv-snapraid-diff: INFO: 'SnapRAID DIFF started'

omv-snapraid-diff: INFO: 'SnapRAID DIFF finished'

omv-snapraid-diff: INFO: '----------------------------------------'

omv-snapraid-diff: INFO: 'SUMMARY of changes since last sync:'

omv-snapraid-diff: INFO: 'Added: [0] - Deleted: [0] - Moved: [0] - Copied: [0] - Updated: [0]'

omv-snapraid-diff: INFO: 'No change detected. Nothing to do'

omv-snapraid-diff: INFO: 'SnapRAID Job ended.'

Display More

I think the script stops after a DIFF if the output is "nothing has changed".

Why is it an issue I hear you ask. Well the day after, the script ran and found some changes, sync the array and send me an email. At the end of the email I notice that the SCRUB-Cycle count missed one day, the day the script stopped at " Nothing has changed, nothing to do".

What should I change in the script so it would still take in consideration the times it runs even if nothing changed in the array in the SCRUB-Cycle count?

Thank you

Unipo · Oct 2nd 2020

Up?

crashtest · Oct 3rd 2020

You're SYNC'ing everyday?

There's a sequence of operations that's needed to keep a SNAPRAID array clean. Without house cleaning operations, the SYNC command is simply reassigning new checksums to files regardless of their condition, corrupted or not. House keeping must be done before a SYNC operation. Here's a -> link to a sequence of SNAPRAID operations I used on a backup server.

The purpose of a diff script is described. It's a barrier set up to prevent over writing the last SYNC operation so, if a large number of files are deleted, changed, etc., (exceeding the threshold set) a new SYNC operation is halted. This preserves the last SYNC state as a potential recovery point; allowing for files, folders or even a full drive to be restored to their condition as of the last SYNC operation, before a potential data disaster event occurred.

Snapraid SCRUB-Cycle count not taking days with nothing to do in consideration

Unipo Sep 23rd 2020

Participate now!