Minio - constant I/O

  • I have OMV 6.9.15-2 (Shaitan) on Linux 6.1.21-v8+ (rpi4/ARM) with minio plugin installed and configured with just one s3 bucket used as backup store.

    It's working as expected, I can see when backups are transferred there, I can use minio built in feature: monitoring->watch->start to see connections with their speed.


    The thing that confused me was that I noticed HDD light indicator blinking all the time, this suggests some i/o

    iotop points me to minio service:

    it's 2-4 processes that constantly make about 100-400kB/s read.


    Minio admin panel monitoring shows nothing at that time.

    Power indicator shows me that drives are spinning all the time without any ability to go to any suspend. This draws constantly about 12W of energy instead of about 5W. If minio is disabled all i/o is gone.

    Any ideas how can I make this service to not work that hard? Replication, encryption, quota, object locking - all off. It's just simple bucket for automated backups, about 85G now.

  • ok, quick update,

    I got confirmation from minio team that this is expected behavior and cannot be turned off


    I found in docs about usage scanner: https://github.com/minio/minio…docs/config#usage-scanner

    It may be reconfigured to sleep more but cannot be off. Minio will spin hard drives preventing them from sleep and therefore I needed to remove it, it was well designed plugin with elegant and promising functionality but completely useless power drain, even with empty buckets and not much activity. I'll look for alternatives for local s3, if there is none - I will need to use nfs for my app.


    just be warned that minio may be terrible idea,

    others also learned that: https://www.reddit.com/r/selfh…_selfhosted_s3compatible/

    • New
    • Official Post

    just be warned that minio may be terrible idea,

    It is only "terrible" if you want to spindown your drives. It is a great product and used in the enterprise. Very reliable. So, I wouldn't call it terrible for any other use case.

    omv 7.1.0-2 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.2 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.5 | scripts 7.0.7


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • In contrast to any enterprise use I'm paying electricity bill here (as well as all hardware) so I need to take that into account. The fact is that it caused about 20-30% constant cpu usage, low priority - so did not affect any other operations, but certainly constant I/O keeps warm all disks in RAID and drains energy (even with no clients for few days). With this service it just destroys power saving modes for hard drives and system causing about 45% increase of power consumption of this small, absolutely non professional home NAS.


    Scalability of this solution is great, but bottom limit is clearly not the home users. I found some reports on github about that and there is no will to give users an option to decide about background tasks and activity. It wont be fixed as it's expected behavior. If You need it for something really simple it really can be terrible idea on home NAS.


    For those who are interested in details from monitoring - here is graph showing how much cpu usage is reduced:



    and here are reduced io:



    For power consumption graph I need to wait some time.


    Yet again it's not any enterprise usage :)

    • New
    • Official Post

    n contrast to any enterprise use I'm paying electricity bill here (as well as all hardware) so I need to take that into account. The fact is that it caused about 20-30% constant cpu usage, low priority - so did not affect any other operations, but certainly constant I/O keeps warm all disks in RAID and drains energy (even with no clients for few days). With this service it just destroys power saving modes for hard drives and system causing about 45% increase of power consumption of this small, absolutely non professional home NAS.

    You don't need raid for a home NAS either. And raid is the reason your energy use and cpu spike is so high. Every time minio writes to the storage location, it has to write to all of your drives in the array. And if it is mdadm, btrfs, or zfs raid, those are software raid and use cpu especially if you chose a parity raid type. If you had minio storage on a single ssd drive, you would use a lot less energy and cpu.

    omv 7.1.0-2 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.2 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.5 | scripts 7.0.7


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • You don't need raid for a home NAS either.

    Also it's not forbidden to use it at home if You want :)


    And raid is the reason your energy use and cpu spike is so high.

    Every RAID for sure uses more energy for write than with single, simple drive. This is perfectly clear.
    But the reason here for much higher energy consumption is i/o activity. Minio is designed to perform constant background operations and You can't change this setting. It's like a car that has always engine turned on and You need it only once a day to go to work.

    You will get same effect with some loop that reads, writes some file and pauses for fraction of second.


    Every time minio writes to the storage location, it has to write to all of your drives in the array. And if it is mdadm, btrfs, or zfs raid, those are software raid and use cpu especially if you chose a parity raid type.

    I'm using hardware RAID here, that of course need some extra power for controller, cpu is not affected much. But the problem here is not that controller but minio i/o activity that keeps disks at high state all the time. When minio was disabled disks were able to suspend with system default options.


    If you had minio storage on a single ssd drive, you would use a lot less energy and cpu.

    Obviously using less faster drives and simple setup reduces energy :) On the other hand probably You will still get better results with setup that can use right c-states, is able to sleep and turn on power save modes.
    I'm not sure if minio is able to finish it's tasks with faster drive or it's never-ending background job. Are You using it on single ssd or nvme and can confirm it may not do anything in background because on very fast storage it's different?

    • New
    • Official Post

    Also it's not forbidden to use it at home if You want

    Key word here was "need". I didn't say can't.

    Are You using it on single ssd or nvme and can confirm it may not do anything in background because on very fast storage it's different?

    I'm not using it at home at the moment. At work, it is four node cluster of large servers with a lot of flash storage with i/o around 1-5 gbps all day long.


    I just spun up container with minio in the compose plugin. I uploaded a few hundred megs of files to it. It has been running for 10 mins and I haven't see the cpu go above 0.29% for this container. Granted this is on an amd64 VM but I don't see why that would change background tasks.


    omv 7.1.0-2 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.2 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.5 | scripts 7.0.7


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • So basically You just confirmed that minio will perform background activity soon after launch,


    maybe less than 1%, maybe more on smaller cpus. Also that may depend on data size, I had there about 82GB.

    It will perform i/o in background preventing most power savings and without control over this process.


    At work I also have several clusters to manage and I never saw electricity bill for them. Different requirements. At home I pay extra attention that all services need to idle properly :)

    • New
    • Official Post

    So basically You just confirmed that minio will perform background activity soon after launch,

    I confirmed Docker will perform background activity. I do not know if that is minio or something else in the container. None of my docker containers are using 0.00% cpu. And your cpu activity is also a couple orders of magnitude more activity than my container. That doesn't seem right. I just looked at one of the minio nodes at work with 6TB of data on just that node and the load is 0.18 on a 16 cpu system with people/systems accessing it. I guess I will put minio in a container on an RPi4 and put 80+ GB in it to see if the load is as high as your system.

    omv 7.1.0-2 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.2 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.5 | scripts 7.0.7


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Thanks for Your time for tests,


    On limited pi4 resources and IO it may be relatively more than on fast, modern server. Also I was not using ssd drives here, because ethernet is limiting transfers here and I wanted to use spare drives. Minio was deployed as plugin, which launched podman with two or three images. As I said earlier hardware RAID should not cause extra CPU usage (I checked that earlier) but using more than one drive for sure increased overall power consumption.


    After some time I can clearly see that power consumption dropped from 12W to about 8W, CPU about those 15% and some memory, not much. Most of power consumption probably was caused by hdd constantly working. Minio has some option to change background tasks sleep time, but it will not stop those, maybe process will be 1% but still doing constant i/o when cpu is free. Usually it's ok, on servers with massive data.

    • New
    • Official Post

    So, I put 140GB on a minio container last night on an RPi4. Storage is a single spinning hard drive. Highest cpu % for the container that I have seen is 0.13% Load on the RPi is under 0.15 but this rpi is doing other things.

    omv 7.1.0-2 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.2 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.5 | scripts 7.0.7


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Thanks for Your time,

    According to docs background tasks are automatically adjusted depending on system load and with low priority to utilize "unused resources". How many files were in Your bucket? Was anything else using this disk? Was it deployed via podman/omv plugin? Have You checked iotop?

    On my side it was used for backups (once a day) and several small services like plex/jellyfin (mostly idle, used maybe once per week). I think that there is not much difference in power consumption with 0.13% and 5%, most of that comes from hdd (4 in my case) that could not go into suspend mode.

    • New
    • Official Post

    How many files were in Your bucket?

    141. All .iso files. I can add small files though.


    Was anything else using this disk?

    I have a nightly rsync job that backs up to this system. Nothing else other than occasionally me uses it.

    Was it deployed via podman/omv plugin?

    compose plugin.



    Have You checked iotop?

    Nope. Just docker stats and top.

    omv 7.1.0-2 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.2 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.5 | scripts 7.0.7


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • 141. All .iso files. I can add small files though.

    This should not be much different than my content (which was snapshots backup). I don't remember how much there were files.



    I have a nightly rsync job that backs up to this system. Nothing else other than occasionally me uses it.

    That should not change much. I think that more difference comes from podman and it's release. I'm not sure when minio started to do it's io on my side, it was constant 150Kb/s, not much but enough to keep drives working all the time.


    Soon I'll deploy new OMV instance on RK3588 board, I'll give a try Your compose version, should not be that much different but it's not podman.

  • I;m using regular compose file:

    version: '3'

    services:

    minio:

    image: quay.io/minio/minio

    volumes:

    - /zdata/s3:/data

    ports:

    - 9000:9000

    - 9001:9001

    environment:

    MINIO_ROOT_USER: 'admin'

    MINIO_ROOT_PASSWORD: 'Tumper'

    MINIO_ADDRESS: ':9000'

    MINIO_CONSOLE_ADDRESS: ':9001'

    command: minio server /data

    restart: always

    • New
    • Official Post

    I have almost 19k files using 140+GB and still don't see the high cpu use on the RPi.

    omv 7.1.0-2 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.2 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.5 | scripts 7.0.7


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!