UnionFS + SnapRaid concept

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • UnionFS + SnapRaid concept

      Dear Gurus,

      I have searched google quite a bit but still couldn't find any conceptual overview of how unionFS works with SnapRaid.

      Coming from NAS4Free / XigmaNAS and ZFS backgorund I think I miss the essence of how snapraid works. I was wondering if there was a good source of the basics you could point me to.

      I simply cannot grasp how you would unionFS something with individual filesystems on individual disks and THEN 'raid' them together... ?(

      Many thanks in advance.
      :)
    • danzirulez wrote:

      Dear Gurus,

      I have searched google quite a bit but still couldn't find any conceptual overview of how unionFS works with SnapRaid.

      Coming from NAS4Free / XigmaNAS and ZFS backgorund I think I miss the essence of how snapraid works. I was wondering if there was a good source of the basics you could point me to.

      I simply cannot grasp how you would unionFS something with individual filesystems on individual disks and THEN 'raid' them together... ?(

      Many thanks in advance.
      :)
      SnapRAID explicitly ignores UnionFS (mergerfs) mount points. It will throw warnings like this when it syncs:

      WARNING! Ignoring mount point '/srv/dev-disk-by-label-d1/sftp/outgoing/music' because it appears to be in a different device

      In the above warning, the specified mountpoint is a mergerfs mountpoint (a manually created one, not the result of using the UnionFS plugin).
      --
      Google is your friend and Bob's your uncle!

      OMV 4.x - ASRock Rack C2550D4I - 16GB ECC - Silverstone DS380
    • oh, ok, so UnionFS is really a userspace convenience to see multiple disks mounted in one big 'folder' that can then be shared in just a single provider.

      e.g.
      • DiskA has all the music
      • DiskB has all the movies
      Share01 is a unionFS of of the two disks shared via Samba and so the client can see a both the music and the movies in one single share..

      Now, with SnapRaid I would have a third drive... DiskC for parity?

      Assuming DiskA and DiskB changes quite rarely... say I am adding only 1-2 albums and movies a month... and sync them with snapraid every week...

      Is that a somewhat real-life scenario?

      Thanks a lot
    • UnionFS merges drives together under a common mount point. Basically, it logically adds all directories together. If two directories exist that are identical, on two or more drives, UnionFS shows one directory and merges the file and sub-dir contents of these directories together, as if they were one.
      UnionFS also provides for a policy where new files are distributed among the drives of the pool. UnionFS is independent of SNAPRAID.

      SNAPRAID is independent of UnionFS. It lines up all drives, start to finish, (size doesn't matter as long at the parity drive is largest) and does an odd-even parity calculation.

      Drive1
      Drive2
      Drive3
      Parity Drive
      1.5TB
      2TB
      2.5TB
      3TB
      0
      1
      0
      1
      1
      0
      1
      0
      0
      1
      0
      1

      0
      0
      0


      1
      1



      0

      (This is "even" parity. There's more to it than this, but this is the concept.)

      The difference is RAID5 interleaves parity info among all array drives and does it on the fly. SNAPRAID stores all parity info on one disk and recalculates it on demand. Just as it is with RAID5 array, SNAPRAID can reconstruct a drive based on this data.
      Since there's a "content file", SNAPRAID can also use parity info to restore files and folders as of the last SYNC (when parity is recalculated and the content file is updated). One SNAPRAID feature, that's truly outstanding, is the ability to detect and correct silent corruption (bit-rot).

      __________________________________

      The advantages? There are several. One notable advantage is that a disk of any size can be added at any time. If the disk is larger than the rest, add it, set the new disk as the SNAPRAID Parity drive, add the old parity disk to the UnionFS pool, run a SYNC and it's done. Users can back out of either one (SNAPRAID or UnionFS) at any time and no data is lost.

      When used together, the two provide an array that's similar to RAID5, but without traditional RAID risks and there are many more features for data preservation and restoration.

      The post was edited 2 times, last by crashtest: edit ().

    • Your understanding of UnionFS, as currently implemented in OMV via mergerfs, is basically correct.

      Yes, you would need a parity drive to use Snapraid which would protect you from the total loss of one disk. You can have more than one parity drive though, and each additional one will protect one more disk from total loss.

      SnapRAID is more or less designed to protect large files that change infrequently. So your use case of only adding one or two albums or movies a month will allow the sync to run quite quickly. However, I would sync more frequently. Perhaps every other day or so because SnapRaid doesn't protect any newly added content since the last sync. The smaller the amount of newly added content, the faster a sync will run. Even if you don't add any new content you should still run a scrub frequently to protect you from bit rot. A scrub is highly tunable so you balance its run time vs it thoroughness.

      I run a scheduled sync and scrub job on cron at 3:00am every other day on an array of 6 data disks totaling 29TB of which 23TB are used. It takes about 3.5 hours to run with SCRUB_PERCENT=10
      SCRUB_AGE=6.

      I add 1TB or more to my array every month with most days adding ~30GB.
      --
      Google is your friend and Bob's your uncle!

      OMV 4.x - ASRock Rack C2550D4I - 16GB ECC - Silverstone DS380

      The post was edited 1 time, last by gderf ().

    • New

      Many thanks, I am testing OMV and extremely excited to see:
      • Emby in a docker
      • Cockpit running
      • Pi-Hole in a docker
      • snapraid and mergerFS


      Couple of ToDo's:
      • gderf has a USB saving script.. actually about to use the internal USB port with a USB-connected HDD, have a few of those lying around.. this frees up some space. Although imaging this would be a bit of a pain
      • burntest some old disks just for the heck of it - based on this article blog.linuxserver.io/2018/10/29/new-hard-drive-rituals/
      • remote mount my current samba shares somehow inside of mergerFS for Emby to see them
      • Also pondering on how to move off my 2x2 Tb ZFS mirror and a 3Tb UFS (not redundant) disks... planning ahead


      Last time I played with debian it was like 6-8 years ago... Man it has changed! All for the better! :)

      Thanks guys for all your input
    • New

      danzirulez wrote:



      • gderf has a USB saving script.. actually about to use the internal USB port with a USB-connected HDD, have a few of those lying around.. this frees up some space. Although imaging this would be a bit of a pain

      Actually, shouldn't be a pain. My OMV runs from a 16GB SSD mounted in an external USB case plugged into a USB port. This is the drive that my script images. You just need to properly determine the device name to use as the source device for the dd command in the script.
      --
      Google is your friend and Bob's your uncle!

      OMV 4.x - ASRock Rack C2550D4I - 16GB ECC - Silverstone DS380
    • New

      gderf wrote:

      danzirulez wrote:


      • gderf has a USB saving script.. actually about to use the internal USB port with a USB-connected HDD, have a few of those lying around.. this frees up some space. Although imaging this would be a bit of a pain

      Actually, shouldn't be a pain. My OMV runs from a 16GB SSD mounted in an external USB case plugged into a USB port. This is the drive that my script images. You just need to properly determine the device name to use as the source device for the dd command in the script.
      ye, it's just the disk I will be using is either 500 Gb or 320 Gb... this is how I meant it.... I suppose I could compress the images to save space?
    • New

      danzirulez wrote:

      ye, it's just the disk I will be using is either 500 Gb or 320 Gb... this is how I meant it.... I suppose I could compress the images to save space?
      Compression might save some space but with higher run times for both creating the image and restoring it. It also might not be reliable and cause the process to fail. I suggest not doing this.
      --
      Google is your friend and Bob's your uncle!

      OMV 4.x - ASRock Rack C2550D4I - 16GB ECC - Silverstone DS380