MergeFS Balancing

Sandro_Rocha · Jan 23rd 2023

Good Morning. I'm doing some tests with the OMV in a virtual machine to, in the future, migrate to a real system. So far I've had a few issues that I've managed to resolve. My last test consisted of adding a disk in MergerFS, as I intend to do this on a real machine and didn't want to have any problems. The disk addition went well, I didn't lose any data or anything. But when trying to balance the data, which I will need to do, I received the following message:

Code

file: aquota.user
from: /srv/dev-disk-by-uuid-92addc17-a125-4d05-ac30-bf176d76ba73
to: /srv/dev-disk-by-uuid-33c98b26-12fd-4bc7-a163-3031fdf6c463
rsync -avlHAXWE --relative --progress --remove-source-files /srv/dev-disk-by-uuid-92addc17-a125-4d05-ac30-bf176d76ba73/./aquota.user /srv/dev-disk-by -uuid-33c98b26-12fd-4bc7-a163-3031fdf6c463/
sending incremental file list

sent 65 bytes received 20 bytes 170.00 bytes/sec
total size is 7,168 speedup is 84.33
ERROR - exited with exit code: 23
Branches within 2.0% range: * /srv/dev-disk-by-uuid-fb1c7ef8-78b2-4a91-bd2a-16ddfa5b7e04: 95.62% free * /srv/dev-disk-by-uuid-33c98b26-12fd-4bc7-a163-3031fdf6c463: 99.92% free

END OF LINE

Display More

In the virtual machine I currently have 4 disks. 1 for the system, 3 merged with MergerFS, under the name "volume". Within that "volume" path, I have 3 folders created and shared: Downloads, Movies and TV Shows. For testing purposes I copied two movies totaling 3.59 GB to the "Movies" folder, a TV Show episode with a size of 440 MB to the "TV Shows" folder and two files with a total size of 442 MB to the "Downloads" folder.

sdb is 5% of 20 GB full, sdc is 19% of 20 GB full, and sdd is empty. I wanted to try to do the balancing but it is giving the error above and nothing happens with the files. I expected at least one of the movies on sdc to be moved to sdd but it didn't. What is the problem and how to solve it?

ryecoaaron · Jan 23rd 2023

The balance doesn't work very well on these small tests especially if your min free space is a very large percent of the drives size. I don't think you have anything wrong here.

gderf · Jan 23rd 2023

It looks like the balancer is trying to move an aquota.user file from one drive to another. This is not allowed no matter how it is attempted because the aquota.* files have their immutable bits set (which is proper).

crashtest · Jan 23rd 2023

Quote from gderf

It looks like the balancer is trying to move an aquota.user file from one drive to another.

So when setting up mergerfs, if the balancer is to used, do you think a solution would be to turn the quota service off and remove aquota.* files?

gderf · Jan 23rd 2023

Quote from crashtest

So when setting up mergerfs, if the balancer is to used, do you think a solution would be to turn the quota service off and remove aquota.* files?

Well, if you don't use quotas, the should work. However, the mergerfs.balance tool has an EXCLUDE filter, so I would try that first. No idea if the OMV webGUI can make use of it though.

ryecoaaron · Jan 23rd 2023

Quote from gderf

No idea if the OMV webGUI can make use of it though.

Nope. It just runs the simple command. Guess I could always use an aquota exclude though.

crashtest · Jan 23rd 2023

Quote from ryecoaaron

Guess I could always use an aquota exclude though.

I believe that's a good idea. A good number of MergerFS support requests are due to unbalanced pools. The Balance Pool tool fixes that issue.

ryecoaaron · Jan 23rd 2023

Quote from crashtest

A good number of MergerFS support requests are due to unbalanced pools. The Balance Pool tool fixes that issue.

While that is true, many of the people that have an unbalanced pool are caused by not understanding (or not reading) the policy they are using. So, the balance tool will fix it once and then the pool will potential get out of balance again.

crashtest · Jan 23rd 2023

Quote from ryecoaaron

So, the balance tool will fix it once.

I'm almost finished with the mergerfs doc. I explained, in an overview and with an illustration, what can happen with Existing Path - Most Free Space (the default). I also demo'ed the use of the Balance Tool.

Quote from ryecoaaron

then the pool will potential get out of balance again.

That may not necessarily be true, when using the default policy (EP-MFS). The problem is almost always Video files (enormous file sizes) going to a single folder - "Videos". The Balance Tool automatically applies the old manual fix of creating "Videos" folders on additional drives and spreading files between them. Thereafter, the second directive kicks in "most free space" which might keep things balanced. (Of course, there will be a few users out there who can find a way to unbalance their pool. )

I'm going to try to nudge users toward "Most Free Space" & SNAPRAID / Backup.
_____________________________________________________________________

BTW: It would be good if you can apply an aquota.* exclude to the dedup tool as well. I have a scenario were aquota.* files are detected as duplicates.

ryecoaaron · Jan 24th 2023

mergerfs plugin 6.3.5 in the repo. balance and dedupe should skip aquota.user and aquota.group

crashtest · Jan 24th 2023

Thanks!

Sandro_Rocha · Jan 25th 2023

Quote from ryecoaaron

mergerfs plugin 6.3.5 in the repo. balance and dedupe should skip aquota.user and aquota.group

I updated the plugin and ran the tool to do the balancing. No mistakes. So-so. The file has been copied from the sdc disk to the sdd disk. But in the "Balance Pool" window the task kept appearing indefinitely as if it was copying the files over and over again. I had to stop the process, go to the file manager and file system and verify that the files had indeed been moved. Is that so? Every time I do the balancing will I have to interrupt the process manually? When there are few files it won't be a problem, I think. But if there are many, it already becomes one more element to make a mess.

ps: I ran the tool again and it appeared again copying the same files as before. Seems like a problem.

I'm so interested in MergerFS balancing because I'm really going to need this feature. I have an Asustor NAS with 2 disks. I set up a server with, initially, 3 disks. I intend to install the OMV on it, copy the data from the NAS to it and use the NAS disks on the server, getting 5 disks.

Sandro_Rocha · Jan 25th 2023

I ran the command mergerfs.balance in the terminal to see what was going on while monitoring the file system and apparently the command kept moving the same file between disks all the time. Before running the command disk 1 had 2.65 GB occupied, disk 2, 1.7 GB and disk 3, zero. When I run the command, disk 3 is 1.85 GB occupied and disk 1 is 800 MB. However the command does not stop. It identifies that disk 1 now has more free space and moves the file back. When disk 3 runs out, it has more space and the file is moved again. This repeats endlessly. I thought the command ran only once until the disks were balanced, as far as possible. Is this an error in my configuration or a problem with the tool?

ps: my mergerfs is configured with "cache.files= full", because Deluge was not downloading the files, and "most free space". It was like "Existing Path - Most Free Space" but I changed it so I didn't have to create folders beforehand.

crashtest · Jan 25th 2023

Quote from Sandro_Rocha

apparently the command kept moving the same file between disks all the time.

Is this a really large single file? (And) Do you have a collection of a couple hundred meg of small files in the Merged array?

Quote from Sandro_Rocha

It identifies that disk 1 now has more free space and moves the file back. When disk 3 runs out, it has more space and the file is moved again. This repeats endlessly. I thought the command ran only once until the disks were balanced, as far as possible.

Balancing is on a "per file" basis. (When the command is complete, member disks are "roughly" the same.)

For example:
If you have 1TB sized disks or a bit less and a file that is several GB in size (say 20GB or more), moving that one file will reduce the percentage fill of the drive that's losing the file and increase the percentage fill of the drive that's gaining the file. I can see where something like that might trigger a ping-pong effect.

Take a look at Trapexit's -> doc on the Balance Tool. (Trapexit is the Developer of MergerFS.) Try running the Balance Tool from the command line with a 10% range. mergerfs.balance -p 10 You could exclude the troublesome file as well.

Quote from Sandro_Rocha

"most free space".

Most Free Space will, eventually, balance MergerFS member disks on it's own. If you maintain this policy, you won't need to use the Balance Tool. (Assumes you'll have ongoing additions and deletions.)

Quote from Sandro_Rocha

Before running the command disk 1 had 2.65 GB occupied, disk 2, 1.7 GB and disk 3, zero. When I run the command, disk 3 is 1.85 GB occupied and disk 1 is 800 MB.

Are all of these disks the same size? Remember, the Balance Tool shuffles files in accordance with drive fill "percentage". 10% of a 1TB drive is 100GB. 10% of a 800GB drive is 80GB. From the Balance Tool's perspective, percentage wise, 100GB and 80GB are equal.

Sandro_Rocha · Jan 25th 2023

Quote from crashtest

Is this a really large single file? (And) Do you have a collection of a couple hundred meg of small files in the Merged array?

It's a file of just under 2 GB. The three disks are 20GB each.

Quote from crashtest

Most Free Space will, eventually, balance MergerFS member disks on it's own. If you maintain this policy, you won't need to use the Balance Tool. (Assumes you'll have ongoing additions and deletions.)

I plan to use balancing once after completing my server. I have 3 disks on the server, 2 disks on the NAS and 1 disk on the desktop. My idea is to copy all the data from the NAS and the desktop to the server, erase the data from the NAS and desktop disks, add them to the server and the pool, leaving me with 6 disks in total, and do the balancing. Then let MergerFS itself manage the files.

ryecoaaron · Jan 25th 2023

Quote from Sandro_Rocha

It's a file of just under 2 GB. The three disks are 20GB each.

And this is why the balancing has problems. As I said before, It is known that balancing gets in a loop on very small filesystems with relatively large files. I know you are testing but it is pointless to test the balancing on these tiny filesystems.

Sandro_Rocha · Jan 25th 2023

Quote from ryecoaaron

And this is why the balancing has problems. As I said before, It is known that balancing gets in a loop on very small filesystems with relatively large files. I know you are testing but it is pointless to test the balancing on these tiny filesystems.

What do you suggest I do? I'm testing to avoid issues that could cause me to lose data or make a mess.

If the balancing doesn't work the way I expect, I'll have to do it the hard way, which I wanted to avoid: create the same folders that I intend to use on all the disks, copy the data doing a manual balancing and only after that the data all 6 disks are copied and all 6 disks installed, make the pool, choosing a policy, if any, that does nothing, and start handling the data on the disks manually. In the end, the pool would only help to concentrate the paths in one place so that 6 folders with the same name do not appear in the share. In this scenario, using Sonarr especially will be tricky.

crashtest · Jan 25th 2023

Quote from Sandro_Rocha

It's a file of just under 2 GB. The three disks are 20GB each.

2GB is 10% of a 20GB disk where the Balance Tool is attempt to balance to 2%. As ryecoaaron has already stated, herein lays the problem. The only way 20GB disks could be used in a test, with the Balance Tool, would be with music or document files that are w-a-y smaller - in the lower MB range.

When using realistic drive sizes, 1TB drives and larger, a 2GB file would be 0.2% or less.

crashtest · Jan 25th 2023

Quote from Sandro_Rocha

What do you suggest I do? I'm testing to avoid issues that could cause me to lose data or make a mess.

See the post above. If you want to use tiny VM drives, use smaller files. And be aware of the 2% balancing goal, that is realistic when using real world hard drives.

Participate now!