mergerfs - distribution over several disks

  • Hello friends of OMV!

    Can someone explain to me in simple terms what setting I need to make in mergerfs for creating new files?

    I really tried to figure it out using THIS site but I'm just more confused.


    I have 5 hard drives of different sizes. One is an SSD for the OS.

    Now I want to use ALL of the 4 remaining disks. I have no need for redundancy and am aware of the risks. The OMV only serves as a backup NAS.


    I would like the data to be saved in its entirety and NOT split across multiple disks. In the event of a failure, I could swap the disk and rsync would re-save the missing data.

    I wouldn't have to save EVERYTHING again.


    Now I wish that the data would be stored evenly distributed on all disks. So the individual hard drives should all be filled to the same percentage.


    I chose "Existing Path - most free space", but my rsync job is currently writing everything completely on the largest one of the discs without using the others :(

    Even if the disk is filled so much, that the others have more space left...


    Is it possible to change the data writing policy for the mergerfs plugin after creating the file system? And what would be the right policy for my needings?


    Thanks very much !!



    Settings:



    Dashboard:

    OMV 6.0.27-1 (Shaitan) // ASRock J5040-ITX // be quiet! ATX 500W (BN46) // 2x4GB Samsung M471A5244BB0-CRC

    Fractal Design Node 304 // 4-Port PCIE to SATA 3.0 ControllerCard // 1x 128GB SSD, 3x 4TB and 1x 5TB HDD

    Edited 2 times, last by Kamikaze01 ().

  • - Just wipe all my disks

    - creat new BTRFS file systems for each disk

    - mount each one new

    - create a pool with mergerfs over all 4 Disks


    tried it with nearly all settings (existing paths,.... shared paths... least free space, least used space, etc etc...).

    But mergerfs always create new files only on my largest disk !!

    LoL - even if i choos "RANDOM" only one of my disks getting filled...


    What is the right settings, so ALL my 4 Disks gettings same amount of % filled with data...

    do i have to put some settings in fstab at the bottom?

    Thank you in advance...

    OMV 6.0.27-1 (Shaitan) // ASRock J5040-ITX // be quiet! ATX 500W (BN46) // 2x4GB Samsung M471A5244BB0-CRC

    Fractal Design Node 304 // 4-Port PCIE to SATA 3.0 ControllerCard // 1x 128GB SSD, 3x 4TB and 1x 5TB HDD

  • try to do 3 manual copy of several gigas of data , if this time works and data is distrubted, the answer is that policy only works on separate copies, not in the same copy of several files ( police do not work on each file copy, works on the "general copy job")


    PD: Really I do not know the answer to your question, but the test I suggest is really fast and clear.

  • do i have to put some settings in fstab at the bottom?

    No. And don't check the fstab checkbox unless you know why you are doing it (almost no reason to).


    It looks like your system is still using an existing path policy. When you are changing the policy, did you reboot? The policy won't change unless the filesystem is remounted or the system is rebooted.

    omv 6.0.30-1 Shaitan | 64 bit | 5.15 proxmox kernel | plugins :: omvextrasorg 6.0.9 | kvm 6.1.19 | mergerfs 6.1.1 | zfs 6.0.9
    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.
    Please don't PM for support... Too many PMs!

  • thank u for your answer...


    yeah - i already tried this.


    after fresh settings with empty disks and only one shared folder .... i tried to copy few GB of Data.

    Doesnt matter what settings i set - all data will only be createt on my largest disk :(

    i dont know what to do next...


    i thought with this settings it should distributed over all disks...



    but rsync (and manually copying different files) always create on the same disk :(



    in case someone could help - this is what my fstab looks like



    Thank u !!

    OMV 6.0.27-1 (Shaitan) // ASRock J5040-ITX // be quiet! ATX 500W (BN46) // 2x4GB Samsung M471A5244BB0-CRC

    Fractal Design Node 304 // 4-Port PCIE to SATA 3.0 ControllerCard // 1x 128GB SSD, 3x 4TB and 1x 5TB HDD

  • your disk show diferent free size space.


    3 disk have 3.64 Tib free and one 4.27 iB


    So police works as expected, copy files on the disk that have most free space.


    until your 4,27 disk grow and only have 3.64 TiB free space , you do not notice that files where distributed by pool.

  • thought with this settings it should distributed over all disks...

    existing path will keep everything in the existing path on the same disk. If you want them balanced and you are not a control freak, just use most free space. But as raulfg said, you will have an imbalance at the beginning because you have one larger drive.

    omv 6.0.30-1 Shaitan | 64 bit | 5.15 proxmox kernel | plugins :: omvextrasorg 6.0.9 | kvm 6.1.19 | mergerfs 6.1.1 | zfs 6.0.9
    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.
    Please don't PM for support... Too many PMs!

  • Thank u so much for your answers...


    But in my first post I already filled up the largest disk with more than 40% and nothing got created on the other disks...


    Then I wiped all data and did everything from beginning. But again the data are always written only to the 4,27TB disk..


    I tried different settings in mergerfs, but nothing changed...


    I will wait my rsync job run over night an totally fill the 4,27TB disk. Then I will see what happend next...

    OMV 6.0.27-1 (Shaitan) // ASRock J5040-ITX // be quiet! ATX 500W (BN46) // 2x4GB Samsung M471A5244BB0-CRC

    Fractal Design Node 304 // 4-Port PCIE to SATA 3.0 ControllerCard // 1x 128GB SSD, 3x 4TB and 1x 5TB HDD

  • existing path will keep everything in the existing path on the same disk. If you want them balanced and you are not a control freak, just use most free space. But as raulfg said, you will have an imbalance at the beginning because you have one larger drive.

    yes - I want it definitely balanced.

    But I don't want every single file split over more than once disk (striped).


    I want it that way, so I can simply change a disk (if it's broken) and my rsync job will only copy these missing files to that new disk.

    All the files stored on the other disks should still be accessible normal.

    OMV 6.0.27-1 (Shaitan) // ASRock J5040-ITX // be quiet! ATX 500W (BN46) // 2x4GB Samsung M471A5244BB0-CRC

    Fractal Design Node 304 // 4-Port PCIE to SATA 3.0 ControllerCard // 1x 128GB SSD, 3x 4TB and 1x 5TB HDD

  • yes - I want it definitely balanced.

    But I don't want every single file split over more than once disk (striped).


    mergerfs will never split a file across multiple disks.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 6.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 16GB ECC RAM.

  • mergerfs will never split a file across multiple disks.

    Thank u... This was clearly a misunderstanding from me.


    I just change setting to "most free space" and restart my omv (last time I tried different settings I did not restart).


    This time the files are stored/created balanced over all disks :P:thumbup:


    But what do u mean "if I am not an Control freak"??! ryecoaaron:(

    OMV 6.0.27-1 (Shaitan) // ASRock J5040-ITX // be quiet! ATX 500W (BN46) // 2x4GB Samsung M471A5244BB0-CRC

    Fractal Design Node 304 // 4-Port PCIE to SATA 3.0 ControllerCard // 1x 128GB SSD, 3x 4TB and 1x 5TB HDD

  • But what do u mean "if I am not an Control freak"??

    I think the existing path policy was created for control freaks. If you put your files in specific folders so that they are in the right folders on one disk, they will never be distributed to other disks until you run out of space on that disk. Just years of dealing with mergerfs/unionfs plugins and hearing how people want the files distributed across disks. control freaks really want mergerfs to do artificial intelligence and hook into their mind to ask exactly where to put each file lol

    omv 6.0.30-1 Shaitan | 64 bit | 5.15 proxmox kernel | plugins :: omvextrasorg 6.0.9 | kvm 6.1.19 | mergerfs 6.1.1 | zfs 6.0.9
    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.
    Please don't PM for support... Too many PMs!

  • LoL... Very good explained ^^


    But I must confess (shame on me): I also have a specific folder structure.... Each folder contains a group of files which belongs together.

    It would be perfect to keep all files in one folder as well.


    So if one of my disks in the omv machine crashes, the other disks should contain folders with complete content...


    Now with setting "most free space" the whole content of each folder is distributed over different disks ;)


    Soo... Yeah... Best would be if mergerfs could read my mind ^^

    Now I feel like having a 1.st world problem ;)

    OMV 6.0.27-1 (Shaitan) // ASRock J5040-ITX // be quiet! ATX 500W (BN46) // 2x4GB Samsung M471A5244BB0-CRC

    Fractal Design Node 304 // 4-Port PCIE to SATA 3.0 ControllerCard // 1x 128GB SSD, 3x 4TB and 1x 5TB HDD

  • The capability you are wanting is not within the scope of mergerfs.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 6.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 16GB ECC RAM.

  • Can you explain this statement?

    I was really hoping not to...


    The plugin by default uses systemd mount files to mount the mergerfs filesystems because systemd does better dependencies resolving. If you check the box, it goes back to using fstab mounts. This exists for legacy reasons.

    omv 6.0.30-1 Shaitan | 64 bit | 5.15 proxmox kernel | plugins :: omvextrasorg 6.0.9 | kvm 6.1.19 | mergerfs 6.1.1 | zfs 6.0.9
    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.
    Please don't PM for support... Too many PMs!

  • The capability you are wanting is not within the scope of mergerfs.

    i thought "existing path" would do exactely what i want....? keep everything together in it's folders (in the last folder level at the bottom.)

    In other words... mergerfs should look at folders as if they where files....


    And "existing path - most free space" should do the trick: keep everything in its folder and distripute the folders balances over my disks...


    as i remember - i got this working in omv 5. But unfortunally i dont remember my settings there...


    existing path will keep everything in the existing path on the same disk. [...]

    yeah - thats what i want... but if i choos "existing path" my files/folders are stored only on the largest disk... even if the other disks have more free space left...

    i do not clearly understand the disciption for "most shared path" --> walk back the path if it fails to find a branch at that level

    i am sure my problem exists between keyboard and chair ( ;) ) and i am very sorry for that - but i try very hard to clearly understand everything...

    OMV 6.0.27-1 (Shaitan) // ASRock J5040-ITX // be quiet! ATX 500W (BN46) // 2x4GB Samsung M471A5244BB0-CRC

    Fractal Design Node 304 // 4-Port PCIE to SATA 3.0 ControllerCard // 1x 128GB SSD, 3x 4TB and 1x 5TB HDD

    Edited 2 times, last by Kamikaze01 ().

  • yeah - thats what i want... but if i choos "existing path" my files/folders are stored only on the largest disk... even if the other disks have more free space left...

    If that is what you want, then don't be surprised when your disks aren't balanced.

    omv 6.0.30-1 Shaitan | 64 bit | 5.15 proxmox kernel | plugins :: omvextrasorg 6.0.9 | kvm 6.1.19 | mergerfs 6.1.1 | zfs 6.0.9
    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.
    Please don't PM for support... Too many PMs!

  • If that is what you want, then don't be surprised when your disks aren't balanced.

    you're right... but why doesn't it create Folders on the other disks?


    My settings:

    NAS --> Shared Folder --> SubFolder --> SubFolders 0001 - 9999 --> manyFiles belong together


    my disks now have ALL exact the same free space:




    now if i store everything with setting "existing path - most free space" following happend:

    Code
    Disk1 --> nothing
    Disk2 --> nothing
    Disk3 --> nothing
    Disk4 --> Shared Folder --> SomeFolder --> SubFolder 0001 -->manyFiles belong together
            > Shared Folder --> SomeFolder --> SubFolder 0002 -->manyFiles belong together
            > Shared Folder --> SomeFolder --> SubFolder 0003 -->manyFiles belong together
            > Shared Folder --> SomeFolder --> SubFolder 0004 -->manyFiles belong togethe




    With "most free Space" it happend this way (balanced over all disks what is great)

    but if eg. Disk3 fails, i can not use Disk1, because in SubFolder 0003 are some files missing.




    What i expect mergerfs to do is to store my SubFolders with its whole content...

    in this case if Disk3 fails, the other disks contain full and complete SubFolders.

    I know that this way it can not be balanced 100%... but it should completely fill a SubFolder befor change the disk.



    I am awfully sorry for annoying but i just want to understand it right...

    is this clearly not possible?!?! i am sure i got this working on my old omv 5

    OMV 6.0.27-1 (Shaitan) // ASRock J5040-ITX // be quiet! ATX 500W (BN46) // 2x4GB Samsung M471A5244BB0-CRC

    Fractal Design Node 304 // 4-Port PCIE to SATA 3.0 ControllerCard // 1x 128GB SSD, 3x 4TB and 1x 5TB HDD

    Edited 2 times, last by Kamikaze01 ().

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!