[HowTo] SnapRAID in OMV

  • Great. Thanks for your help.

    Did that process work for you? I'm getting the same error, and I follwed the process listed here. Deleted drive from OMV gui, went to terminal and ran command snapraid -E sync, and got this error:

    Code
    Disk 'd2' with uuid 'abee6d94-71ad-42ec-ab28-bc921527b6a8' not present in the configuration file!
    If you have removed it from the configuration file, please restore it

    How did you resolve your error? Thanks.

  • Did that process work for you? I'm getting the same error, and I follwed the process listed here. Deleted drive from OMV gui, went to terminal and ran command snapraid -E sync, and got this error:


    Disk 'd2' with uuid 'abee6d94-71ad-42ec-ab28-bc921527b6a8' not present in the configuration file!
    If you have removed it from the configuration file, please restore it How did you resolve your error? Thanks.

    Hello kmal. No, that didn't do the job for me, I got the same error as you quote here (different disk ID, obvs) and as a result I have not yet been able to remove the SSD from my Snapraid array. I haven't followed up or tried anything else yet, as my priorities were elsewhere, but I'm glad you reminded me: if @jollyrogr or anyone else can point me in the right direction here I would be grateful.


    Thanks!

  • Not sure that it matters, but looking at the snapraid manual, I see that they list the command as
    snapraid sync -E


    Try running with the -E at the end instead of the way I had it before?


    They also list the steps to remove the disk as follows:

    Not sure how that's supposed to work with OMV since I don't think you can use the GUI to edit the configuration file to point to an empty directory. May need to try logging into the shell, and (as root) do the steps listed.

  • I think @bmhj2 has already manually edited the config file and had the same results/error. I read the instructions from the SnapRAID guide that you referenced, but wasn’t sure about editing the config file manually, so I searched the forum and came across this thread. My real problem is that I can’t get the first sync to finish. I’ve tried just about everything I can find or think of and still get the “Warning! The array is not fully synced. You have a sync in progress at 99%” when checking Snapraid status. I’ve read that docker files and Plex data don’t mix well with Snapraid, so that’s why I’m trying to remove a drive(the drive that contains these files).

  • So you can't even get the initial sync to complete? How long did you let it go? It can take many hours to complete. I don't use docker or plex and have had no issues. Snapraid is designed to be used in situations where files change very infrequently. Perhaps docker and plex are constantly changing files and causing issues that way. If there are certain affected directories, those can be easily excluded from the snapraid array. If plex or docker is mucking with all your media, then it probably just wont work.

    • The initial sync ran and the log shows it as finishing, although while syncing it did throw errors for modifying files during sync, and those files were from docker. When I check Snapraid status is where I get the error about “sync in progress”. That combined with the errors during the sync leads me to believe docker is modifying files during the sync, causing the inability to completely sync. To remedy this I tried to remove the disk that the docker files live on from the array, but am not able to remove the disk, which lead me here. I think if I can remove this disk from the array, the sync should run w/o error.
  • Got it to sync error free by excluding /plex & /docker. Semi-related question though, how does mergerfs play with plex metadata and docker files? Any issues using it with the drive that contains those files? That’s my next mission.

  • I do not have my Plex metadata directory or docker files in mergerfs managed mount points. I would not want that data spread across multiple drives. I am not saying it wouldn't work; I just don't want to do it.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • I thought that simply excluding those directories should work. We should be able to remove drives though.


    Not sure about your mergerfs question since don't use plex or docker. Mergerfs does play well with snapraid though.

  • @jollyrogr yeah thanks for that tip. I would rather remove the drive and then keep that drive out of both Snapraid & mergerfs, but this will work for now. Wish I would have known what I know now prior to installing the plugin. Ha. On that note, would uninstalling and reinstalling/reconfiguring the Snapraid plugin create any problems? What happens to the data on the parity drive and content files when the plugin is uninstalled? May have to look into that as well if I can’t get removing a drive to work. @gderf thanks for the info, I’ll keep that drive out of the pool.

  • You shouldn't have to uninstall and reinstall the SnapRaid plugin to get the configuration you want.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • Originally I tried to remove the drive that contains the plex & docker data from the array but got errors every way I tried it, so I used the exclusions as a band aid until I can I remove that drive from the array.

  • Thanks to others for further discussion here.

    I think @bmhj2 has already manually edited the config file and had the same results/error. I read the instructions from the SnapRAID guide that you referenced, but wasn’t sure about editing the config file manually, so I searched the forum and came across this thread. My real problem is that I can’t get the first sync to finish. I’ve tried just about everything I can find or think of and still get the “Warning! The array is not fully synced. You have a sync in progress at 99%” when checking Snapraid status.


    Yes, that's right: I have tried editing the config file via CLI and via the plugin, with no success using either method.


    It seems as if @kmal808 have been encountering the same error, caused by files which changed somehow without that being picked up by the parity. In my case these errors have only related to two drives:

    • The SSD shown in my earlier post, which I wanted to remove because it contains Plex databases and Docker config data, and I realised (too late) that Snapraid is not the best way to back those up.
    • An HDD not mentioned before, which has developed problems since I last posted. This is almost certainly because I used Beets to re-tag the metadata on a large music collection, meaning that many of the files now show errors on Snapraid, presumably because of some snafus created by that Beets import taking place while I was also messing around with the "snapraid sync -F" command trying to remove the SSD from the array (I have also, as indicated in my original post, since added a second HDD parity drive).

    Stating the problem very simply, having first misunderstood how Snapraid works and created a setup which was likely to have problems, I have then compounded my error by making lots of file changes in quick succession so that I am left with parity files which are incorrect. As a consequence I can't run a successful sync, because Snapraid is returning false positives - suggesting there are data errors where I am confident there aren't.


    For (1) I have managed to stop the error messages by adding exclusions for the Docker and Plex folders on the SSD as described by @kmal808 above. If I understand things correctly, this has removed files on the SSD from the snapraid.parity and the snapraid.content files on a subsequent sync, and this seems to have acted as the bandaid that @kmal808 describes above. The outstanding question is still this: how to remove the drive from the array completely?


    For (2) I have not yet done anything, but I am wondering whether the following will work:

    • add temporary exclusions for the HDD to the config
    • run a sync (on the assumption that this will then remove the faulty parity data)
    • remove the exclusions
    • run another sync (on the assumption that this will force a recalculation of the parity)

    Or...


    it does strike me that another less fiddly solution would be (as @kmal808 seems to suggest, and @gderf advises against) to just nuke the array and start from scratch. @kmal808 suggests uninstalling the plugin, but doesn't know how this would affect the snapraid.parity and snapraid.content files. I assume it would be possible to exercise this option by simply deleting all the parity and content files and running a new sync, but I have already proven myself to be dumber than I thought here so would prefer to . I'd be willing to take the risk with the data, since if necessary I can restore from a backup - but I suppose I would ask the following questions:

    • Can I remove the SSD from the array and force a complete rebuild of the HDD parity data without exercising my nuclear option?
    • How complicated will this be?
    • If I exercise the nuclear option, what is the simplest means of doing so?

    I should finally add: forgive me, fellow forum users, for I have sinned! Hope you don't mind helping me dig myself out of this hole.

  • Hi everyone. I can't seem to find the ability to search this thread, so I hope this isn't already answered. I have snapraid setup to send an e-mail to me from its nightly sync/scrub per the diff script.


    Everything is working fine except the e-mail is a bit spammy. It gives me every change to the % and time remaining. Is it possible to exclude these redundant lines from the e-mail?


    example below -
    Using 4527 MiB of memory for the FileSystem.
    Initializing...
    Scrubbing...
    Using 168 MiB of memory for 32 blocks of IO cache.
    0%, 2 MB
    0%, 348 MB
    0%, 1487 MB
    0%, 2710 MB
    0%, 3909 MB
    0%, 5143 MB, 1198 MB/s, CPU 72%, 1:53 ETA
    0%, 6331 MB, 1196 MB/s, CPU 72%, 1:53 ETA
    0%, 7568 MB, 1203 MB/s, CPU 72%, 1:53 ETA
    0%, 8767 MB, 1202 MB/s, CPU 72%, 1:53 ETA
    0%, 9975 MB, 1203 MB/s, CPU 72%, 1:53 ETA
    0%, 11201 MB, 1214 MB/s, CPU 71%, 1:52 ETA
    0%, 12415 MB, 1214 MB/s, CPU 71%, 1:52 ETA
    0%, 13614 MB, 1212 MB/s, CPU 71%, 1:52 ETA
    0%, 14846 MB, 1214 MB/s, CPU 71%, 1:52 ETA
    0%, 16045 MB, 1213 MB/s, CPU 71%, 1:52 ETA
    0%, 17262 MB, 1212 MB/s, CPU 71%, 1:52 ETA
    0%, 18476 MB, 1212 MB/s, CPU 70%, 1:52 ETA
    0%, 19704 MB, 1213 MB/s, CPU 70%, 1:52 ETA
    0%, 20880 MB, 1211 MB/s, CPU 70%, 1:52 ETA
    .... all the way to 100%


    is there a way to have this removed from the nightly e-mail?

  • You would need to edit the diff script. Line 422 sends all output of the sync command to a variable and elsewhere in the script that output is included in the email.

    Code
    $SNAPRAID_BIN sync${prehash} >> $TMP_OUTPUT


    Line 444 includes all the scrub output.


    Code
    $SNAPRAID_BIN scrub -p $SCRUB_PERCENT -o $SCRUB_OLDER_THAN_DAYS >> $TMP_OUTPUT
  • Hi,


    I'm starting with snapraid and trying to set up an automatic schedule for sync and scrub tasks.


    With the snapraid-diff schedule job will I be covered? I've read in the script that it does a diff, and then a sync. I have also make the option to make a scrub after a sync (but the panel option "Scrub frequency" set to 7 days I have no idea what does) . And it does it once a day. At what time exactly?


    Is making a sync once a day really necessary if you do not change your files so often?

  • I've read in the script that it does a diff, and then a sync. I have also make the option to make a scrub after a sync (but the panel option "Scrub frequency" set to 7 days I have no idea what does)

    The diff checks whether there have been changes, and the the sync updates the parity based on those. By default, the sync does not go ahead if you have more than X number of files (by default, X = 50) deleted, which is a safeguard against the parity for files being removed if they have been deleted in error. In that situation, you have to sync the file manually from the CLI or the GUI.


    Scrubbing checks for errors by comparing the parity against the hashes computed when the 'sync' command was run, thus ensuring you against (for example) file system rot. By default it checks a small percentage each time and doesn't check the ones that have been done recently, to avoid duplicating effort. You can change the parameters for this; according to the snapraid manual (sections 4.1 and 5.7) 7 days means that it runs a scrub once a week, checking ~8% of files each time and not those checked in the last ten days. With these default settings all files will be checked about every three months. I can't advise on whether these are good settings but I've been happy to assume that they are.

    With the snapraid-diff schedule job will I be covered?

    Yes.

    And it does it once a day. At what time exactly?

    Mine runs at midnight.

    Is making a sync once a day really necessary if you do not change your files so often?

    No, but the purpose of the diff is to detect and sync only those files that have changed, are new, or have been deleted. If the diff detects no changes then no further sync is required, so it does no harm to run the script daily.

  • hello all snapraid users !


    I see in the diff script 100% of the array is scrub after 7 synch.


    I run diff script one a day.


    if I add files every day. My system will run scrub once a week and so 100% of the array will be processed each week.
    That seems like a lot to me?
    Do you have any advice ?

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!