mergfs balance on SnapRaid Question

    • OMV 4.x
    • mergfs balance on SnapRaid Question

      Hello,
      i would like to create a Snapraid with 4 8TB HDD`S
      3 Data 1 Parity.
      It woul be nice, if there is a possibility to balance the Data on my pool.
      What i understand is, that the tool mergerfs-tools/ is a balancer that can handle this.
      Does this Tool run automaticly in the "Background" or is it triggerd via a cron-job ?

      Would be very nice if someone could explain this.
    • mergerfs.balance can be run from the commandline to balance drives. It does not run in the background. It could be run on cron, but a better choice would be to use a mergerfs create policy that keeps the drives balanced once you have them initially in balance.
      --
      Google is your friend and Bob's your uncle!

      OMV 4.x - ASRock Rack C2550D4I - 16GB ECC - Silverstone DS380
    • gderf wrote:

      mergerfs.balance can be run from the commandline to balance drives. It does not run in the background. It could be run on cron, but a better choice would be to use a mergerfs create policy that keeps the drives balanced once you have them initially in balance.
      Can you pls explain which policy i should take ?
      So do i understand this right:
      first i have to choose a policy which will take always the most free space hard-drive
      i move all my date to my snapraid => the drive will not be balanced cause i will move a lot of data at once
      then i have to run the balance-plugin => to get the drives in balance

      since then i will only move smaller amounts of data, so the policy will trigger and keep my drives in balance - is this correct ?
    • First off, leave SnapRAID out of the conversation. It isn't relevant. This all about mergerfs or the Union Filesystem plugin if you are using it.

      For a mergerfs policy you could choose Most Free Space or Existing Path Most Free Space depending on your needs. If you set this up properly first, moving data into the pool would try to balance it automatically.

      You can run the mergerfs.balance tool at any time. If the pool is already balanced, nothing should happen.
      --
      Google is your friend and Bob's your uncle!

      OMV 4.x - ASRock Rack C2550D4I - 16GB ECC - Silverstone DS380
    • I am new to mergerfs and have made a pool with the LFS (least-free-space) policy.
      My reasoning behind it was that it made sense to keep as much of a specific kind of data on the same disk; for instance all the episodes of a tv series.
      It also makes it easier to find files when a directory/file tree is not equally distributed among all the drives of the pool.
      Also, using most-free-space (or an afterwards balanced mergerfs pool) would create a situation where it is unlikely for a disk to spin down, because it would more frequently be asked for data.

      So my question: why would you balance the data among all the drives? What is the advantage?
    • T.Underhill wrote:

      So my question: why would you balance the data among all the drives? What is the advantage?
      There is no requirement to write data into the pool via the mergerfs mountpoint. If you want certain data to be stored on certain disks, then put it there directly yourself.

      The advantage of letting mergerfs do the balancing is that you don't have to manually place data onto individual drives. With careful planning and initial implementation it's possible to configure mergerfs one time only. The way I have it set up here, I have added new disks multiple times over the last two years and haven't touched the mergerfs configuration. However, I do not use the OMV UnionFS plugin, I set it up by hand and use globbing to identify disks in the fstab configuration.
      --
      Google is your friend and Bob's your uncle!

      OMV 4.x - ASRock Rack C2550D4I - 16GB ECC - Silverstone DS380
    • Paradido wrote:

      I haven't found a reason why LFS or for more placement control EPLFS (which requires manual intervention when a drive runs full) shouldn't always be used for media.
      I configure mergerfs manually here rather than using the OMV UnionFS plugin. I use globbing to identify disks and paths and LFS for the policy. I haven't touched the configuration in years and have added numerous new disks over that period.
      --
      Google is your friend and Bob's your uncle!

      OMV 4.x - ASRock Rack C2550D4I - 16GB ECC - Silverstone DS380
    • gderf wrote:

      There is no requirement to write data into the pool via the mergerfs mountpoint. If you want certain data to be stored on certain disks, then put it there directly yourself.
      The advantage of letting mergerfs do the balancing is that you don't have to manually place data onto individual drives.
      I think you completely misunderstood the question from both T.Underhill and me.

      The question is what benefits the MergerFS policy LUS/MFS has for media in comparison to LFS? Why not always use LFS? And the question came up because: why is romibaer even trying to evenly spread the data across all drives thus scattering around everything?
    • Using LFS will result in filling disks one by one before moving on to the next disk. Using MFS will result in a roughly even amount of free space across all disks as they fill.

      Not sure what the objection to scattering data around is. If you operate on the data only via a mergerfs mountpoint you are not even aware of the scattering.
      --
      Google is your friend and Bob's your uncle!

      OMV 4.x - ASRock Rack C2550D4I - 16GB ECC - Silverstone DS380
    • gderf wrote:

      Using LFS will result in filling disks one by one before moving on to the next disk. Using MFS will result in a roughly even amount of free space across all disks as they fill.
      Doesn't answer the question, you're just explaining what it does. What is the benefit of LUS/MFS for media?

      gderf wrote:

      Not sure what the objection to scattering data around is. If you operate on the data only via a mergerfs mountpoint you are not even aware of the scattering.
      T.Underhill already mentioned drawbacks of scattering. E.g. when you copy over a TV show with multiple seasons/folders. With LFS the whole series ends up on the same drive. With LUS/MFS it ends up scattered around on multiple drives.
      Thus if you remove one drive you don't have the whole series on it. Or when you lose one drive and somehow can't restore it with SnapRAID then you have an incomplete series in your pool. Good luck finding out manually what episodes are now missing. With LFS the whole series would be gone so one knows what's missing way more easily.
      T.Underhill also mentioned a second drawback: possible drive spindown prevention. If you are e.g. listening to an album and every file is on a different drive, all drives would have to stay awake while I'm listening to that album. With LFS only one drive has to spin up.

      I'm sure there are more drawbacks. So the question once again: what benefits does the MergerFS policy LUS/MFS have for media in comparison to LFS?
    • Paradido wrote:

      I'm sure there are more drawbacks. So the question once again: what benefits does the MergerFS policy LUS/MFS have for media in comparison to LFS?
      Perhaps this might help with some examples, for my own choice (once some drives arrive) I shall go with MFS, the real problem with media is when Existing Path is used. As to a drive failing and SnapRaid not being able to recover, that's what a backup is for :)

      If you start looking at advantages and disadvantages and whether drives spin up/down you might as well deploy single drives for each specific media type, then one running rsync to all drives, then a backup policy.
      Raid is not a backup! Would you go skydiving without a parachute?
    • Users Online 1

      1 Guest