[HowTo] SnapRAID in OMV

    • Official Post
    Quote from ryecoaaron

    Seems like the plugin (I assume that is what you mean by automation) could be enhanced if people provided suggestions.

    I don't know. The plugin is pretty damned good as it is right now.

    Setting drive recovery considerations aside:
    Since a sync operation removes any chance of recovery of previously deleted or changed files, the real question is to when (or when not) to sync. That question, and where to set thresholds in the Diff section of the plugin, is a matter of the user's use case and personal preference.

    - If a user wants to automate with Diff AND they're downloading everything under the sun, new files would have to be set high. (To allow a range that's typical for what they do.)
    - If they're worried about losing files or, say, changed documents, that setting should be low.

    What it really boils down to is, the user must familiarize themselves with how SnapRAID works, they must understand their own use case (their own wants and needs) and tweak Diff settings accordingly.

    In this case, the plugin allows new users to focus on Diff settings versus worrying about the in's and out's of an actual Diff script. That's exactly what the plugin should do.

    Quote from ryecoaaron

    Seems like the plugin (I assume that is what you mean by automation) could be enhanced if people provided suggestions.

    The plugin is fine. By automation I was referring to the diff script. I'm not sure there's any enhancement that would make me trust it. If there are changes to the array, I want to know what they are before syncing.

    Quote from ryecoaaron

    Just that no one wants to help integrate that with plugin.

    I am the maintainer of the snapraid-aio-script and I am humbled by the amount of people who use and recommend it.


    I would love my script to be bundled in the plugin, because everyone would benefit from it, but I don't know how to accomplish this. I am not really a coder and learned shell for this specific project and for fun.


    The project has also evolved a lot from a "simple diff script". If we would integrate it in OMV, we would have to expose every single option via the GUI, for example Docker containers to be paused/stopped, custom commands before/end, notification services and so on.


    Also, don't forget new features. They don't come often, but I try.


    This is beyond my abilities.


    I already do my best to make this script as plug and play as possible for OMV users, since I'm a passionate OMV user as well: auto install dependencies, reuse mailx config from OMV, documentation for OMV users.


    Quote from jollyrogr

    I'm not sure there's any enhancement that would make me trust it. If there are changes to the array, I want to know what they are before syncing.

    Have you tried my script? Has many logics to prevent unwanted syncs.

    OMV BUILD - MY NAS KILLER - OMV 6.x + omvextrasorg (updated automatically every week)

    NAS Specs: Core i3-8300 - ASRock H370M-ITX/ac - 16GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker Apps), 2x16TB HDDs w/ SnapRAID - Fractal Design Node 304 - Be quiet! Pure Power 11 350W


    My all-in-one SnapRAID script!

    • Official Post
    Quote from auanasgheps

    I am the maintainer of the snapraid-aio-script and I am humbled by the amount of people who use and recommend it.


    I would love my script to be bundled in the plugin, because everyone would benefit from it, but I don't know how to accomplish this. I am not really a coder and learned shell for this specific project and for fun.


    The project has also evolved a lot from a "simple diff script". If we would integrate it in OMV, we would have to expose every single option via the GUI, for example Docker containers to be paused/stopped, custom commands before/end, notification services and so on.


    Also, don't forget new features. They don't come often, but I try.


    This is beyond my abilities.

    I'm not asking you to make the changes to the plugin. I just need to know how the plugin should interact with the script. I don't use snapraid or the script. So, it would take a lot of effort on my part to figure it out. Just not something I have time to do. But if you could tell me what the form would look like and what buttons you would want to perform actions, I could code that in short order. This is how the diff script stuff was added.


    And if you make changes to the script that the plugin needs to change for, I would need you to let me know. Or maybe once the script was implemented in the plugin, it would be easier for you to tell what needed to change.

    omv 7.4.7-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.3 | k8s 7.2.0-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.8


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Quote from auanasgheps

    Have you tried my script? Has many logics to prevent unwanted syncs.

    Nice update to the original script, and correct me if I'm wrong, but it still just uses a threshold value to determine if a sync will be performed. If a single file is corrupted due to a bad sector on a disk and your threshold is >1, then it will sync and wipe out any possibility of recovering that file. I'm not interested in that. It isn't that difficult to manually sync my array when adding/removing files.

    • Official Post
    Quote from jollyrogr

    If a single file is corrupted due to a bad sector on a disk and your threshold is >1, then it will sync and wipe out any possibility of recovering that file. I'm not interested in that. It isn't that difficult to manually sync my array when adding/removing files.

    Sounds like you are using snapraid as backup and snapraid isn't backup (just like raid).

    omv 7.4.7-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.3 | k8s 7.2.0-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.8


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Quote from ryecoaaron

    Sounds like you are using snapraid as backup and snapraid isn't backup (just like raid).

    Yeah that's pretty much what I am doing and it has worked well for the last 6 years or however long I've been running this (since omv2). So there is a level of risk I have accepted and I'm not going to increase it by automating the sync.

    • Official Post
    Quote from jollyrogr

    Yeah that's pretty much what I am doing and it has worked well for the last 6 years or however long I've been running this (since omv2). So there is a level of risk I have accepted and I'm not going to increase it by automating the sync.

    While I was reminding you of that, that message was really for other people reading this. You are free to do whatever you want. I think it is strange that you are willing to risk backup on something that provides availability and redundancy but are worried about losing one file. But if it works for you, so be it.

    omv 7.4.7-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.3 | k8s 7.2.0-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.8


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Quote from ryecoaaron

    While I was reminding you of that, that message was really for other people reading this. You are free to do whatever you want. I think it is strange that you are willing to risk backup on something that provides availability and redundancy but are worried about losing one file. But if it works for you, so be it.

    Not sure I get what you mean there. I like snapraid because it provides redundancy (snapraid is really not for availability IMO) and data protection without having to invest in more drives for a 1 to 1 backup. As my drives age and as larger drives become cheaper, I could start to look into adding a second array for backing up the first.


    My first server only had 8 bays and my current one has 24, so I do have room to expand a bit.

    Quote from ryecoaaron

    I'm not asking you to make the changes to the plugin. I just need to know how the plugin should interact with the script. I don't use snapraid or the script. So, it would take a lot of effort on my part to figure it out. Just not something I have time to do. But if you could tell me what the form would look like and what buttons you would want to perform actions, I could code that in short order. This is how the diff script stuff was added.


    And if you make changes to the script that the plugin needs to change for, I would need you to let me know. Or maybe once the script was implemented in the plugin, it would be easier for you to tell what needed to change.

    Allright let's do this. I'll open a discussion/issue on GitHub so we can discuss it there.


    Quote from jollyrogr

    The plugin is fine. By automation I was referring to the diff script. I'm not sure there's any enhancement that would make me trust it. If there are changes to the array, I want to know what they are before syncing.

    I would recommend taking a look at borg as a proper backup solution. It is super, super reliable. It has versioning, encryption, compression, deduplication.

    It takes care of my off-site backups to a NAS at my parent's house. Whatever happens on my local SnapRAID array I have a weekly backup, along with a history/versioning of the last 3 months.


    It virtually runs on anything: for this I'm using a 10 years old Synology NAS and a shucked drive I paid very little for.


    The cost is small, you can automate it using borgmatic and then you'll sleep well!

    OMV BUILD - MY NAS KILLER - OMV 6.x + omvextrasorg (updated automatically every week)

    NAS Specs: Core i3-8300 - ASRock H370M-ITX/ac - 16GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker Apps), 2x16TB HDDs w/ SnapRAID - Fractal Design Node 304 - Be quiet! Pure Power 11 350W


    My all-in-one SnapRAID script!

    Thanks 1

    That does look neat. Offsite backup is ideal but I would neat a bit more upload speed. And I would need to build another 30tb NAS.

    • Official Post
    Quote from auanasgheps

    Allright let's do this. I'll open a discussion/issue on GitHub so we can discuss it there.

    Before this goes to github (where it will disappear for most forum users), what are you proposing? As noted earlier, the plugin already allows the customization of variables where, afterward (and saved), a Diff script is generated. With your version of a generated Diff, where the plugin is concerned, what would be different?

    Quote from crashtest

    Before this goes to github (where it will disappear for most forum users), what are you proposing? As noted earlier, the plugin already allows the customization of variables where, afterward (and saved), a Diff script is generated. With your version of a generated Diff, where the plugin is concerned, what would be different?

    Ideally we would expose all of my script features in the Plugin GUI. It has many more features compared to the existing script.

    OMV BUILD - MY NAS KILLER - OMV 6.x + omvextrasorg (updated automatically every week)

    NAS Specs: Core i3-8300 - ASRock H370M-ITX/ac - 16GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker Apps), 2x16TB HDDs w/ SnapRAID - Fractal Design Node 304 - Be quiet! Pure Power 11 350W


    My all-in-one SnapRAID script!

    • Official Post
    Quote from auanasgheps

    Ideally we would expose all of my script features in the Plugin GUI. It has many more features compared to the existing script.

    If the scripts are that different, it should be possible for the plugin to config things for either script. We could have a checkbox to select which script you want.

    omv 7.4.7-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.3 | k8s 7.2.0-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.8


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Quote from ryecoaaron

    If the scripts are that different, it should be possible for the plugin to config things for either script. We could have a checkbox to select which script you want.

    Honestly, with all due respect to the creator(s) of the original script, I believe there's no reason in keeping that one. Mine has all the features and beyond.
    If the user does not care about advanced features, they can be ignored.

    Please somebody correct me if I'm wrong.

    OMV BUILD - MY NAS KILLER - OMV 6.x + omvextrasorg (updated automatically every week)

    NAS Specs: Core i3-8300 - ASRock H370M-ITX/ac - 16GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker Apps), 2x16TB HDDs w/ SnapRAID - Fractal Design Node 304 - Be quiet! Pure Power 11 350W


    My all-in-one SnapRAID script!

    • Official Post
    Quote from auanasgheps

    Honestly, with all due respect to the creator(s) of the original script, I believe there's no reason in keeping that one. Mine has all the features and beyond.
    If the user does not care about advanced features, they can be ignored.

    I'm not worried about the original script if the new one does everything it did and more. I just don't like the idea of changing the basic behaviors substantially. I would much rather use a maintained script (by someone other than me) that does a better job and have the plugin only support that script.

    omv 7.4.7-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.3 | k8s 7.2.0-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.8


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    • Official Post
    Quote from ryecoaaron

    If the scripts are that different, it should be possible for the plugin to config things for either script. We could have a checkbox to select which script you want.

    And, possibly, one of those check boxes would enable @auanasgheps default script settings.
    (A sort of, middle of the road, script for SnapRAID newbies.)

    Quote from crashtest

    And, possibly, one of those check boxes would enable @auanasgheps default script settings.
    (A sort of, middle of the road, script for SnapRAID newbies.)

    Yeah, I would populate all fields with the script defaults, which work for most users.

    OMV BUILD - MY NAS KILLER - OMV 6.x + omvextrasorg (updated automatically every week)

    NAS Specs: Core i3-8300 - ASRock H370M-ITX/ac - 16GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker Apps), 2x16TB HDDs w/ SnapRAID - Fractal Design Node 304 - Be quiet! Pure Power 11 350W


    My all-in-one SnapRAID script!

    EDIT: Never mind. Sync finished at some point much before the ~80h or so were lapsed. No idea what happened (that would still interest me though).


    Original post:


    I think I made a mistake. I had the AiO script through an error last night due to too many changed files. That was due to a larger than usual backup run, so all okay, but to figure that out I initially triggered a check instead of a diff. When I saw that that was not what I had wanted, I stopped it.


    I think that messed with the setup, because a manually triggered sync now want to start from scratch. That is not ideal, but far worse is that it is impossibly slow.


    I suspect this is due to the low RAM amount it uses for the sync: "Using 64 MiB of memory for 64 cached blocks." The hash was much faster (with much more memory). Can I influence this somehow?


    The system is an Asrock Deskmini X600 w/ Ryzen 5 5600G, 32GB RAM and 2x SATA + 2x NVMe SSD. Syncs used to be much faster (until the SSDs ran hot).

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login