Docker Space issues on USB flash drive. /var/lib/containerd taking up space on drive

  • Interesting. Try running the following in terminal and post the results.


    Code
    docker info | grep "Storage Driver"
    Code
    fred@omv:~$ docker info | grep "Storage Driver"
    
    Storage Driver: overlay2

    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.

    OMV AMD64 8.x on headless Tyan Thunder SX GT86C-B5630 1U Server with Intel Xeon Silver 4110 CPU @ 2.10GHz & 32GB DDR4 ECC RAM.

    • Official Post

    overlay2 is the default. shouldn't have to put it in daemon,json.

    omv 8.0.10-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.1 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • I have this issue too. Started yesterday after everything running fine for months on my prod system.


    I am getting my logs filled with these messages - literally 10 plus logs per second


    Code
    2/1/2025, 4:47:21 PM
    containerd[19150]: time="2025-12-01T16:47:21.849619415+11:00" level=error msg="unable to parse \"max 0\" as a uint from Cgroup file \"/sys/fs/cgroup/system.slice/docker-fc3b3f825c60e54c614cd7e14e02458d18833d3a2a90cb5612832a4248afa658.scope/hugetlb.1GB.events\""


    I have googled this error to death but I can't find anything that fixes this issue.


    Looking at my omv-os disk it is half full but seems to have stabilised but this is not good - 13GB in io.containerd.snapshotter.v1.overlayfs





    OP can you step me through the issue and solution so I can see if I can fix this?


    I have replicated this issue on a clean installation - I thought it might be hardware issue but looks like I can fix it

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

  • Downed/stopped all containers

    Stopped docker service

    Deleted containerd directory

    Deleted docker storage location on ssd

    Reinstalled docker using location on system disk (SSD - not omv install disk)


    started one container - dozzle and same issue


    noted the following when reinstalling docker (in omv reinstall window) but not sure if this is a clue...


    Have replicated this issue with omv7 and a clean omv8 install.


    Code
    Package 'docker.io' is not installed, so not removed
    Package 'containerd' is not installed, so not removed
    Package 'docker-compose' is not installed, so not removed


    docker info

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

  • OP can you step me through the issue and solution so I can see if I can fix this?


    I don't how the issue happened, but I know what solved it. Its all an overlayfs issue. I have the steps in an earlier post. Docker is using overlayfs and this is causing containerd to be used instead of your docker folder for the docker overlays.

    1. I shutdown all my containers.

    2. systemctl docker containerd stop

    3. deleted the symlink and recreated the containerd dir.


    4. ran

    docker info | grep "Storage Driver"

    results

    Storage Driver: overlayfs


    5. sudo nano /etc/docker/daemon.json

    {

    "storage-driver": "overlay2",

    "data-root": "/srv/yourdrive/dockcontain/docker"

    }


    You need to add "storage-driver":"overlay2", to the daemon.json,

    6. systemctl docker containerd start

    7.reboot


    This should solve your issue.





    • Official Post

    . Its all an overlayfs issue. I have the steps in an earlier post. Docker is using overlayfs and this is causing containerd to be used instead of your docker folder for the docker overlays.

    Ah, I missed the overlay2 vs overlayfs difference. That would explain it. Maybe the plugin should specify overlay2 to prevent this.

    omv 8.0.10-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.1 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Edited once, last by ryecoaaron ().

  • Thank you redneckgamer88. I will try again making the change to the storage driver.


    My system minipc does not seem to have containerd installed on omv8


    I have omv8 with docker on a rpi5 but don’t have this issue. [edit - actually I do have the issue but I only run 2 containers on it so hadn't noticed disk usage]


    Not sure what’s going on here but very strange and thanks for your help

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

    Edited once, last by jata1 ().

  • So just checked all of my omv systems running docker. Results below.


    Looks like a recent change to docker defaults but maybe only applied to new installs?


    rpi4 - omv7 (have not reinstalled or changed docker for ages)

    • Architecture: aarch64
    • Storage Driver: overlay2
    • driver-type: n/a


    rpi5 - omv8 (clean install of omv8 so docker install very recent)

    • Architecture: aarch64
    • Storage Driver: overlayfs
    • driver-type: io.containerd.snapshotter.v1



    minipc - omv8 (clean install of omv8 so docker install very recent)

    • Architecture: x86_64
    • Storage Driver: overlayfs
    • driver-type: io.containerd.snapshotter.v1

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

  • I can confirm the issue and fix is to change storage driver in daemon.json


    I can see that containerd snapshotter is false when docker starts from my logs after changing the storage driver


    So I think this might cause some issues for new installs of docker/compose plugin


    Code
    dockerd[227333]: time="2025-12-02T07:32:21.347650513+11:00" level=info msg="Docker daemon" commit=9a84135 containerd-snapshotter=false storage-driver=overlay2 version=29.1.1

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

    • Official Post

    So I think this might cause some issues for new installs of docker/compose plugin

    I will be pushing this change shortly - https://github.com/OpenMediaVa…8eada54b9419bdad00780e79f - but I hope omv7 doesn't have this issue. I don't want to keep changing omv7 plugins.

    omv 8.0.10-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.1 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • I’m also seeing this on a fresh Debian 13.2 install with Docker 29.1.1. Journalctl is getting flooded with errors like:

    Code
    Dec 02 08:14:24 firebat containerd[738]: time="2025-12-02T08:14:24.846853387+01:00" level=error msg="unable to parse \"max 0\" as a uint from Cgroup file \"/sys/fs/cgroup/system.slice/docker-140e0740e7d1bce6a88e3288e95fe7e6f3a358423097999195249672af19b6ba.scope/hugetlb.2MB.events\""
    Dec 02 08:14:24 firebat containerd[738]: time="2025-12-02T08:14:24.847152514+01:00" level=error msg="unable to parse \"max 0\" as a uint from Cgroup file \"/sys/fs/cgroup/system.slice/docker-140e0740e7d1bce6a88e3288e95fe7e6f3a358423097999195249672af19b6ba.scope/hugetlb.1GB.events\"

    They repeat constantly for every running container.

    I already switched the Docker storage driver to overlay2 (confirmed in docker info), but the errors continue. My system is using:


    • Official Post

    If you are on OMV 8, I already pushed a change to enforce the overlay2 storage driver. Did you install it? Not seeing any problems on my omv8 docker systems.

    omv 8.0.10-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.1 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • I’m also seeing this on a fresh Debian 13.2 install with Docker 29.1.1. Journalctl is getting flooded with errors like:

    Code
    Dec 02 08:14:24 firebat containerd[738]: time="2025-12-02T08:14:24.846853387+01:00" level=error msg="unable to parse \"max 0\" as a uint from Cgroup file \"/sys/fs/cgroup/system.slice/docker-140e0740e7d1bce6a88e3288e95fe7e6f3a358423097999195249672af19b6ba.scope/hugetlb.2MB.events\""
    Dec 02 08:14:24 firebat containerd[738]: time="2025-12-02T08:14:24.847152514+01:00" level=error msg="unable to parse \"max 0\" as a uint from Cgroup file \"/sys/fs/cgroup/system.slice/docker-140e0740e7d1bce6a88e3288e95fe7e6f3a358423097999195249672af19b6ba.scope/hugetlb.1GB.events\"

    They repeat constantly for every running container.

    I already switched the Docker storage driver to overlay2 (confirmed in docker info), but the errors continue. My system is using:

    This issue is unrelated to the overlayfs issue. My mistake for thinking they were.


    This issue is a bug with containerd and is being looked at by containerd devs. I posted a new thread on the forum on the containerd bug

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

    • Official Post

    containerd.io 2.2.0 in the omv7 and 8 repos. This is upgrading the package from 2.1.5. Are people still seeing the issue?


    Unpacking containerd.io (2.2.0-2~debian.12~bookworm) over (2.1.5-1~debian.12~bookworm) ...


    Unpacking containerd.io (2.2.0-2~debian.13~trixie) over (2.1.5-1~debian.13~trixie) ...

    omv 8.0.10-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.1 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • ryecoaaron - I think this issue is with containerd 2.2


    I am already on 2.2 and have the issue :)


    One temporary solution I have seen on github is to install 2.1.5 but I haven't tried this

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

  • Does anyone have a shell command or other action that can be used to reliably trigger this journal flooding on demand?

    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.

    OMV AMD64 8.x on headless Tyan Thunder SX GT86C-B5630 1U Server with Intel Xeon Silver 4110 CPU @ 2.10GHz & 32GB DDR4 ECC RAM.

  • Does anyone have a shell command or other action that can be used to reliably trigger this journal flooding on demand?

    I know that dozzle container with a standard setup triggers it.


    more info here... bug in containerd impacting dozzle and potentially other containers


    The issue on github dozzle linked above has some useful links to the containerd issue

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

    • Official Post

    I think this issue is with containerd 2.2


    I am already on 2.2 and have the issue

    My systems just got the update. So, I thought it was just released. I don't have the issue making it harder to know.

    omv 8.0.10-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.4 | compose 8.1.2 | cterm 8.0 | borgbackup 8.1 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!