SnapRAID Questions

  • Hello all!


    I have recently installed OMV with the SnapRAID plugin. I finished the OMV setup process and transferred my data to the respective shared folders. SnapRAID was set up and I did an initial run with Prehash+Sync. It started running, then I left to run some errands. When I came back my PC had gone to sleep and the web client had obviously disconnected from OMV. I logged back in but cannot find the status of the Prehash+Sync process anywhere. I tried to run another Prehash+Sync but I only get:

    Code
    Self test...
    The lock file '/srv/dev-disk-by-label-sda/snapraid.content.lock' is already locked!
    SnapRAID is already in use!

    From this I suppose the Prehash+Sync process that I started a few hours ago is still running. Is there any way to see its progress and estimated time to completion? Also, is there a tutorial anywhere about automating things like Sync, Diff, and Scrub in SnapRAID with OMV? What I have gathered from reading is that you run an initial Sync and then you just need to run a Diff, so that it only syncs files added after the last Sync. Is this correct? Any suggestions on when (and how frequently) to run any of these three processes? Thanks!

  • Running a sync from the shell should show you progress, but the initial sync on a large data set can take a long time. You could look within OMV Diagnostics | System Information | Processes and see if it listed as running.


    You will want to use a helper script to run a diff, sync, scrub cycle. Running a diff does not in and of itself run a sync afterwards. OMV ships with one located at /usr/sbin/omv-snapraid-diff and you can configure it within the SnapRAID Settings | Diff Script Settings.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

    Einmal editiert, zuletzt von gderf ()

  • Thanks for the response gderf. I did check before my initial post and there was a SnapRAID process running. It still is. Unfortunately, I cannot find a way of knowing how much longer it has left. I tried running the sync command from shell and I get the same message, as the previous Prehash+Sync is still running. I know this will take a long time (I have 12TB of data in the drives, with three 8TB drives total capacity for data and two 8TB drives for parity).


    If the diff is not automatically followed by a sync, then what is the main purpose of it? I see the Diff Script Setting, but how do you schedule this? Also, how do you schedule the rest (sync and scrub) and when? Does each sync or scrub operation take a long time, even after the initial sync? I am concerned that the system will be running on a regular basis just to complete all of these tasks, which would defeat the purpose of having my drives in standby.

  • I would expect it to take quite a few hours to initially sync 12TB. You could kill the sync process and then run it again in the shell so you can get some progress feedback. The sync will pick up where it left off so you won't lose any work in progress. I don't recall how long it took to fully resync my 16TB of data but I know it was overnight and then some.


    SnapRAID has many commands available. You can run them one at a time but you need a helper script to chain commands together.


    The supplied omv-snapraid-diff script is a bit of a misnomer. Depending on how it is configured, it will run a diff, followed by a sync, followed by a scrub.


    The scripted sync jobs after the first one will go a lot faster as only new data added to the array since the last sync will need to be processed.


    You can setup a scheduled command to run the script. But I find the OMV built in scheduler won't allow a lot of flexibility in the schedule. I run mine every other day and I don't see any way to use the scheduler from the OMV web GUI to do that. So I put this in the root user's crontab. This runs the script at 3:00am on odd numbered days.


    Code
    0 3 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31 * * /usr/sbin/omv-snapraid-diff >/dev/null 2>&1

    I don't have any of my drives go into standby. I feel there is more wear and tear on them if they are restarted frequently.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • Thanks again for the info. So basically once I set my Diff Script Settings under SnapRAID in OMV, should I just need to click on Schedule diff above it? Will it give me options of run times then? I also do not see an option to sync after a diff under the Diff Script Settings. I just see the scrub option after a successful sync.


    When you say to run sync in shell, could it be through Putty or Shell in a Box? In either case, I guess if my PC goes to sleep I will again lose track of the progress and would need to kill the process and run it again. Is this correct?

    • Offizieller Beitrag

    When you say to run sync in shell, could it be through Putty or Shell in a Box? In either case, I guess if my PC goes to sleep I will again lose track of the progress and would need to kill the process and run it again. Is this correct?

    To avoid this you can use "screen"
    https://linuxize.com/post/how-to-use-linux-screen/

  • If you press the Schedule Diff button and enable it within it will schedule a Daily run of the script. There is no other scheduling option available that way. But the item will appear in OMV's list of scheduled jobs and you can edit the frequency of run there: Specific date, Hourly, Daily, Weekly, Monthly, Yearly, or At reboot.


    The script will run a sync automatically only if a diff is detected.


    Having the PC go into Sleep or Standby might interfere with running processes. I don't do that here - my server is up, running and available 24/7/365.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • @macom Thanks for the suggestion. This looks really useful. Will get it installed to monitor these processes.


    @gderf Thanks for clarifying the script. I will try killing the process and then restarting through shell to check current progress. Once this clears, I will test the script with my current standby settings (thanks again to @macom for the hdidle help on my previous thread). I would certainly prefer to have my system running 24/7, but in my case this is not justified, as it will be accessed once or twice daily under normal conditions. But I will certainly test it to make sure it works, even with the HDDs gone to standby. BTW, is there a place or log where I can check when the last diff/sync/scrub was run?


    Thanks again!

  • BTW, is there a place or log where I can check when the last diff/sync/scrub was run?



    Thanks again

    Look in /var/log/snapraid.log

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • So SnapRAID finished the initial sync after about a day or so. I then ran the diff and sync commands manually and everything works fine. The disks wake up from standby and the process completes normally. I have not tested automating these using the script, but I am sure it will work. I will post after testing.


    However, I am facing a more serious issue that I found out during the last testing phase. My transfer speeds to OMV on SMB are terrible. They are rarely going above 30MB/s. When I transferred the bulk of my data a couple of weeks ago, I was getting 100MB/s+, maybe 90MB/s+ on smaller files. Is this due to using SnapRAID? I am trying to use iperf to check network speeds but can't get it to work. I am very likely doing something wrong when running it.


    Edit: I managed to run iperf with the server on OMV and I am getting 940Mbits/s, which is expected with a 1Gbps link. I really hope this is not caused by SnapRAID.

  • SnapRAID has nothing to do with your transfer speed problem.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • Sorry, I mistakenly referred to the problem potentially caused by SnapRAID, where I should have said mergerfs. My reasoning is that since the file is being written to a single disk, the speeds will depend on the HDD speeds. Still, it would not make sense to me since these disks (WD 8TB Red) are rated much faster than that. I will keep testing. Any suggestions?


    Edit: Tested from another PC (also wired) and speeds were nomal at 110MB/s+. It seems this may be a SMB issue specifically to my main PC, which was not there a couple of weeks ago.

  • I would expect some reduction in thruput with mergerfs, but I don't know how much. I am fairly sure writes to it will be more impacted than reads from it. Have a look at the mergerfs homepage for suggestions.


    https://github.com/trapexit/mergerfs

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • Seems that I edited my post at the same time you posted yours. I had no issues achieving the expected speeds from another system. It could be a potential problem with SMB on my main PC. Still, it is odd that this happened without any changes to the system, but I know SMB can sometimes be a pain.


    Edit: After checking with further file transfers, I sometimes even get speeds in the 1-3MB/s range. Pretty slow. I am looking for a guide to troubleshoot SMB connections from my PC to OMV.

  • Speed issues have been resolved on their own. I still believe the problem was related to SMB or some other issue with my main PC. Nothing to do with OMV. I am in the process of setting up some automated tasks and will post results when complete. Thanks!

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!