mergfs balance on SnapRaid Question

  • Hello,
    i would like to create a Snapraid with 4 8TB HDD`S
    3 Data 1 Parity.
    It woul be nice, if there is a possibility to balance the Data on my pool.
    What i understand is, that the tool mergerfs-tools/ is a balancer that can handle this.
    Does this Tool run automaticly in the "Background" or is it triggerd via a cron-job ?


    Would be very nice if someone could explain this.

  • mergerfs.balance can be run from the commandline to balance drives. It does not run in the background. It could be run on cron, but a better choice would be to use a mergerfs create policy that keeps the drives balanced once you have them initially in balance.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • mergerfs.balance can be run from the commandline to balance drives. It does not run in the background. It could be run on cron, but a better choice would be to use a mergerfs create policy that keeps the drives balanced once you have them initially in balance.

    Can you pls explain which policy i should take ?
    So do i understand this right:
    first i have to choose a policy which will take always the most free space hard-drive
    i move all my date to my snapraid => the drive will not be balanced cause i will move a lot of data at once
    then i have to run the balance-plugin => to get the drives in balance


    since then i will only move smaller amounts of data, so the policy will trigger and keep my drives in balance - is this correct ?

  • First off, leave SnapRAID out of the conversation. It isn't relevant. This all about mergerfs or the Union Filesystem plugin if you are using it.


    For a mergerfs policy you could choose Most Free Space or Existing Path Most Free Space depending on your needs. If you set this up properly first, moving data into the pool would try to balance it automatically.


    You can run the mergerfs.balance tool at any time. If the pool is already balanced, nothing should happen.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • I am new to mergerfs and have made a pool with the LFS (least-free-space) policy.
    My reasoning behind it was that it made sense to keep as much of a specific kind of data on the same disk; for instance all the episodes of a tv series.
    It also makes it easier to find files when a directory/file tree is not equally distributed among all the drives of the pool.
    Also, using most-free-space (or an afterwards balanced mergerfs pool) would create a situation where it is unlikely for a disk to spin down, because it would more frequently be asked for data.


    So my question: why would you balance the data among all the drives? What is the advantage?

  • So my question: why would you balance the data among all the drives? What is the advantage?

    I'd also like to know this. I haven't found a reason why LFS or for more placement control EPLFS (which requires manual intervention when a drive runs full) shouldn't always be used for media.

  • So my question: why would you balance the data among all the drives? What is the advantage?

    There is no requirement to write data into the pool via the mergerfs mountpoint. If you want certain data to be stored on certain disks, then put it there directly yourself.


    The advantage of letting mergerfs do the balancing is that you don't have to manually place data onto individual drives. With careful planning and initial implementation it's possible to configure mergerfs one time only. The way I have it set up here, I have added new disks multiple times over the last two years and haven't touched the mergerfs configuration. However, I do not use the OMV UnionFS plugin, I set it up by hand and use globbing to identify disks in the fstab configuration.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • I haven't found a reason why LFS or for more placement control EPLFS (which requires manual intervention when a drive runs full) shouldn't always be used for media.

    I configure mergerfs manually here rather than using the OMV UnionFS plugin. I use globbing to identify disks and paths and LFS for the policy. I haven't touched the configuration in years and have added numerous new disks over that period.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • There is no requirement to write data into the pool via the mergerfs mountpoint. If you want certain data to be stored on certain disks, then put it there directly yourself.
    The advantage of letting mergerfs do the balancing is that you don't have to manually place data onto individual drives.

    I think you completely misunderstood the question from both T.Underhill and me.


    The question is what benefits the MergerFS policy LUS/MFS has for media in comparison to LFS? Why not always use LFS? And the question came up because: why is romibaer even trying to evenly spread the data across all drives thus scattering around everything?

  • Using LFS will result in filling disks one by one before moving on to the next disk. Using MFS will result in a roughly even amount of free space across all disks as they fill.


    Not sure what the objection to scattering data around is. If you operate on the data only via a mergerfs mountpoint you are not even aware of the scattering.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • Using LFS will result in filling disks one by one before moving on to the next disk. Using MFS will result in a roughly even amount of free space across all disks as they fill.

    Doesn't answer the question, you're just explaining what it does. What is the benefit of LUS/MFS for media?


    Not sure what the objection to scattering data around is. If you operate on the data only via a mergerfs mountpoint you are not even aware of the scattering.

    T.Underhill already mentioned drawbacks of scattering. E.g. when you copy over a TV show with multiple seasons/folders. With LFS the whole series ends up on the same drive. With LUS/MFS it ends up scattered around on multiple drives.
    Thus if you remove one drive you don't have the whole series on it. Or when you lose one drive and somehow can't restore it with SnapRAID then you have an incomplete series in your pool. Good luck finding out manually what episodes are now missing. With LFS the whole series would be gone so one knows what's missing way more easily.
    T.Underhill also mentioned a second drawback: possible drive spindown prevention. If you are e.g. listening to an album and every file is on a different drive, all drives would have to stay awake while I'm listening to that album. With LFS only one drive has to spin up.


    I'm sure there are more drawbacks. So the question once again: what benefits does the MergerFS policy LUS/MFS have for media in comparison to LFS?

    • Offizieller Beitrag

    I'm sure there are more drawbacks. So the question once again: what benefits does the MergerFS policy LUS/MFS have for media in comparison to LFS?

    Perhaps this might help with some examples, for my own choice (once some drives arrive) I shall go with MFS, the real problem with media is when Existing Path is used. As to a drive failing and SnapRaid not being able to recover, that's what a backup is for :)


    If you start looking at advantages and disadvantages and whether drives spin up/down you might as well deploy single drives for each specific media type, then one running rsync to all drives, then a backup policy.

    Raid is not a backup! Would you go skydiving without a parachute?


    OMV 6x amd64 running on an HP N54L Microserver

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!