Minio - constant I/O

I have OMV 6.9.15-2 (Shaitan) on Linux 6.1.21-v8+ (rpi4/ARM) with minio plugin installed and configured with just one s3 bucket used as backup store.

It's working as expected, I can see when backups are transferred there, I can use minio built in feature: monitoring->watch->start to see connections with their speed.

The thing that confused me was that I noticed HDD light indicator blinking all the time, this suggests some i/o

iotop points me to minio service:

it's 2-4 processes that constantly make about 100-400kB/s read.

Minio admin panel monitoring shows nothing at that time.

Power indicator shows me that drives are spinning all the time without any ability to go to any suspend. This draws constantly about 12W of energy instead of about 5W. If minio is disabled all i/o is gone.

Any ideas how can I make this service to not work that hard? Replication, encryption, quota, object locking - all off. It's just simple bucket for automated backups, about 85G now.

ok, quick update,

I got confirmation from minio team that this is expected behavior and cannot be turned off

I found in docs about usage scanner: https://github.com/minio/minio…docs/config#usage-scanner

It may be reconfigured to sleep more but cannot be off. Minio will spin hard drives preventing them from sleep and therefore I needed to remove it, it was well designed plugin with elegant and promising functionality but completely useless power drain, even with empty buckets and not much activity. I'll look for alternatives for local s3, if there is none - I will need to use nfs for my app.

just be warned that minio may be terrible idea,

others also learned that: https://www.reddit.com/r/selfh…_selfhosted_s3compatible/

In contrast to any enterprise use I'm paying electricity bill here (as well as all hardware) so I need to take that into account. The fact is that it caused about 20-30% constant cpu usage, low priority - so did not affect any other operations, but certainly constant I/O keeps warm all disks in RAID and drains energy (even with no clients for few days). With this service it just destroys power saving modes for hard drives and system causing about 45% increase of power consumption of this small, absolutely non professional home NAS.

Scalability of this solution is great, but bottom limit is clearly not the home users. I found some reports on github about that and there is no will to give users an option to decide about background tasks and activity. It wont be fixed as it's expected behavior. If You need it for something really simple it really can be terrible idea on home NAS.

For those who are interested in details from monitoring - here is graph showing how much cpu usage is reduced:

and here are reduced io:

For power consumption graph I need to wait some time.

Yet again it's not any enterprise usage

Zitat von ryecoaaron

You don't need raid for a home NAS either.

Also it's not forbidden to use it at home if You want

Zitat von ryecoaaron

And raid is the reason your energy use and cpu spike is so high.

Every RAID for sure uses more energy for write than with single, simple drive. This is perfectly clear.
But the reason here for much higher energy consumption is i/o activity. Minio is designed to perform constant background operations and You can't change this setting. It's like a car that has always engine turned on and You need it only once a day to go to work.

You will get same effect with some loop that reads, writes some file and pauses for fraction of second.

Zitat von ryecoaaron

Every time minio writes to the storage location, it has to write to all of your drives in the array. And if it is mdadm, btrfs, or zfs raid, those are software raid and use cpu especially if you chose a parity raid type.

I'm using hardware RAID here, that of course need some extra power for controller, cpu is not affected much. But the problem here is not that controller but minio i/o activity that keeps disks at high state all the time. When minio was disabled disks were able to suspend with system default options.

Zitat von ryecoaaron

If you had minio storage on a single ssd drive, you would use a lot less energy and cpu.

Obviously using less faster drives and simple setup reduces energy On the other hand probably You will still get better results with setup that can use right c-states, is able to sleep and turn on power save modes.
I'm not sure if minio is able to finish it's tasks with faster drive or it's never-ending background job. Are You using it on single ssd or nvme and can confirm it may not do anything in background because on very fast storage it's different?

So basically You just confirmed that minio will perform background activity soon after launch,

maybe less than 1%, maybe more on smaller cpus. Also that may depend on data size, I had there about 82GB.

It will perform i/o in background preventing most power savings and without control over this process.

At work I also have several clusters to manage and I never saw electricity bill for them. Different requirements. At home I pay extra attention that all services need to idle properly

Thanks for Your time for tests,

On limited pi4 resources and IO it may be relatively more than on fast, modern server. Also I was not using ssd drives here, because ethernet is limiting transfers here and I wanted to use spare drives. Minio was deployed as plugin, which launched podman with two or three images. As I said earlier hardware RAID should not cause extra CPU usage (I checked that earlier) but using more than one drive for sure increased overall power consumption.

After some time I can clearly see that power consumption dropped from 12W to about 8W, CPU about those 15% and some memory, not much. Most of power consumption probably was caused by hdd constantly working. Minio has some option to change background tasks sleep time, but it will not stop those, maybe process will be 1% but still doing constant i/o when cpu is free. Usually it's ok, on servers with massive data.

Thanks for Your time,

According to docs background tasks are automatically adjusted depending on system load and with low priority to utilize "unused resources". How many files were in Your bucket? Was anything else using this disk? Was it deployed via podman/omv plugin? Have You checked iotop?

On my side it was used for backups (once a day) and several small services like plex/jellyfin (mostly idle, used maybe once per week). I think that there is not much difference in power consumption with 0.13% and 5%, most of that comes from hdd (4 in my case) that could not go into suspend mode.

Zitat von ryecoaaron

141. All .iso files. I can add small files though.

This should not be much different than my content (which was snapshots backup). I don't remember how much there were files.

Zitat von ryecoaaron

I have a nightly rsync job that backs up to this system. Nothing else other than occasionally me uses it.

That should not change much. I think that more difference comes from podman and it's release. I'm not sure when minio started to do it's io on my side, it was constant 150Kb/s, not much but enough to keep drives working all the time.

Soon I'll deploy new OMV instance on RK3588 board, I'll give a try Your compose version, should not be that much different but it's not podman.

It was not high CPU at any point

The problem was constant i/o and just few percent of CPU needed for those.

This is not much but enough to keep my HDD in constant operation and of course destroy any power saving mode.

For second device that should be used only from time to time I needed services that can sleep on idle. Minio was the only one constantly doing something in background and I could not find any option to disable this. When I turned off minio io gone as well as cpu from about 4-5% to 1-2%. With device that is constantly doing something those are really small numbers, but for device that should idle this is issue.

Please take a look at iotop monitor and let me know if Yor instances also reindex files all the time. Do any of You have cpu/mem/network/io graph?