SnapRaid - cheat sheet / capacity calculator / best practice

1. offizieller Beitrag

    Hello OMV folks,


    I am testing OMV on my old N40L HP Microserver, and plan to move from XigmaNAS after using that for over 8 years (!)


    I was hoping to find a calculator to put all the numbers into it that would spit out the best way to set the drives up with SnapRaid and UnionFS/mergerfs.


    I am moving from 3 datadisks:

    • 2x2 Tb WD Red EFRX in ZFS Mirror (1.5 Tb used)
    • 1x3 Tb WD Red EFRX UFS (no redundancy) (2.6 used)


    Old disks lying around, just burn testing them with this script to see if they are still fit for storage.

    • 3x1.5 Tb Samsung HD154UI - in progress, but one of them failed already


    I also have a bunch of old notebook drives and external USB disks

    • WD My Passport Essential 320 Gb (USB2) - PASSED
    • WD Elements SE Portable 320 Gb (USB3) - PASSED
    • WD Blue 500 Gb (SATA) - in progress
    • two other waiting to be tested, although one of them SMARTed out already


    Plan to use one of the WD externals as system drive to free up SATA ports. And also I hear that USB pendrive is not best practice to have the system on. I wonder though if a USB connected spinning drive is any better?


    • Plan to have the 3TB as parity disk and have the two WD 2Tb drives and one Samsung 1.5 Tb as data drives. Is there a better setup out of this mix?
    • Can I still throw in the small drives in the mix with just that one big (the 3Tb drive) parity?
    • Is it the number of disks that is key, or does the overall data amount play a role too?


    Thanks a lot

    • Offizieller Beitrag
    Zitat von danzirulez

    And also I hear that USB pendrive is not best practice to have the system on.

    There is nothing wrong with using USB thumbdrives as a boot drive. I'm been using them on two different boxes, for years, and haven't had a failure yet. (I've been using 16 and 32GB drives - preferably USB3). The only requirement is installing the flashmemory plugin, for longer drive life.
    On the other hand, if you want to use external disks or an SSD, there's nothing wrong with that either.


    Zitat von danzirulez

    Plan to have the 3TB as parity disk and have the two WD 2Tb drives and one Samsung 1.5 Tb as data drives. Is there a better setup out of this mix?

    For SNAPRAID, that will work fine.

    Zitat von danzirulez

    Can I still throw in the small drives in the mix with just that one big (the 3Tb drive) parity?

    You can. And if you want to get an even larger drive than 3TB, you could use it as well. Simply turn off parity on the 3TB drive, reformat it and add it to SNAPRAID as a data disk. Then add a 4TB drive (as an example), set it as the new parity drive, and run a SYNC. Done.

    Zitat von danzirulez

    Is it the number of disks that is key, or does the overall data amount play a role too?

    The number (and size) of disks is driven by the amount of data to be stored. As a matter of personal preference, I like to start with a storage fill rate between 25 to 50%. This allows room for growth. At 75%, I'm looking to add more storage.
    When considering the number of disks, setting 1 parity disk means you can recover 1 drive. This may be enough for your needs. If you're running older drives, stay on top of them with automated SMART tests. But when the number of disks to be protected reaches 4, 5, or more , it might be time to consider 2 parity drives, for protection from up to 2 drive failures.
    _________________________________________________________________


    Setting up UnionFS (MergerFS) is another matter. When it comes to data and I'm thinking about file size, primarily, what will you be storing?

    Zitat von crashtest

    The number (and size) of disks is driven by the amount of data to be stored. As a matter of personal preference, I like to start with a storage fill rate between 25 to 50%. This allows room for growth. At 75%, I'm looking to add more storage.
    When considering the number of disks, setting 1 parity disk means you can recover 1 drive. This may be enough for your needs. If you're running older drives, stay on top of them with automated SMART tests. But when the number of disks to be protected reaches 4, 5, or more , it might be time to consider 2 parity drives, for protection from up to 2 drive failures.
    _________________________________________________________________


    Setting up UnionFS (MergerFS) is another matter. When it comes to data and I'm thinking about file size, primarily, what will you be storing?


    The 2x2 Tb ZFS mirror holds all the important data, which consists of family photos, documents, videos etc. Plenty of small files here, but very static. This is also backed up to cloud. Not to have too many files, I usually zip the photos of a particular event and use an image viewer that can handle the zip for slideshow.


    The 3Tb drive holds movies, series, iTunes library. I'd say 70+ % of these are huge files, but there are also plenty small here. On the other hand, this is also static mostly, and newly added movies, series I am not to worried about.


    Perhaps I could still use one mirror in the mix? So 3Tb for parity, 2x2 Tb in Raid1 and the two healthy Samsung drives as data disks?

    • Offizieller Beitrag

    You could keep the zmirror, but I wouldn't put it in a UnionFS pool. I'm not saying it wouldn't work, I don't know, but I wouldn't do it. ZFS is fine on it's own but when it comes to adding volume management over top of a zpool, which already has integrated volume management, it just doesn't seem like a good idea. Also, SNAPRAID is best used with a simple file system like EXT4 or XFS.
    ___________________________________________________________


    The reason why I asked about files and sizes:
    A lot of users want to store Video files in a UnionFS pool. Video files can easily eclipse the sum total of everything else you have and then some. The default UnionFS (storage) policy, "Existing Path, Most free space", works fine for a mix of files but if you're leaning toward storing a lot of Video files that will continue to grow, there's a issue you should be aware of. This -> thread covers the topic. If you're still interested in the "Existing Path, Most free space" policy, there is a work around. Otherwise, with a lot of video content, "Most free space" would be the policy to use.

    Thank you, will read the thread.


    Actually I don't think UnionFS is that important. Mostly convenience... Perfectly happy with multiple shares as long as I can keep my data safe - be it the important files or just movies. And then safety would be taken care of by SnapRaid.


    Also, OMV has CrashPlan in docker, which I have been using for years now. And one of some other reasons to move away from XigmaNAS. The other being emby, cos Plex is a constant bother. Docker in itself is a huge plus comparing OMV to XigmaNAS.

    • Offizieller Beitrag
    Zitat von danzirulez

    1. Actually I don't think UnionFS is that important. Mostly convenience... Perfectly happy with multiple shares as long as I can keep my data safe - be it the important files or just movies. And then safety would be taken care of by SnapRaid.


    2. Docker in itself is a huge plus comparing OMV to XigmaNAS.

    1. I agree with you here. It's just not that much trouble to spread network shares over a few disks. But note that data safety doesn't get much better than it is with a zmirror. The problem is the cost - 1/2 of your disk real-estate (50%). If you decide to leave the zmirror, the cost of SNAPRAID protection depends on the number of drives protected. For a single parity disk, it can range from 33% (2 protected disks) to as little as 20% (4 protected disks).


    2. Docker is a huge bonus. Almost every conceivable server add-on, other than an actual hypervisor, is available as a Docker. There's nothing to compare to it.

    • Offizieller Beitrag
    Zitat von crashtest

    Almost every conceivable server add-on, other than an actual hypervisor, is available as a Docker.

    Actually, you can run qemu/kvm inside docker :)

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    • Offizieller Beitrag
    Zitat von ryecoaaron

    Actually, you can run qemu/kvm inside docker :)

    Something seems "eerily" familiar about a comment like that. :D
    I was thinking of a type 1 hypervisor like Proxmox or ESXi, but I suppose KVM crosses the type 1 line as well.

    • Offizieller Beitrag

    I know. I was trying not to be that way lol

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!

Benutzerkonto erstellen Anmelden