Despite setting up shares and setting up docker to use my SSD drive and not my 64gig flash drive for the docker. /var/lib/containerd was taking up 35gigs of free space on my flash drive. deleting containerd breaks docker in OMV. I ended up bind mounting /var/lib/containerd to my ssd. I then had to delete all my containers, networks and volumes to get it working again. was not the end of the world but I thought that having my docker install the the SSD would solve this issue? Now that I am doing what I am doing, OMV only takes up 3.14 gigs on my flash drive instead of the near 40 gigs it was taking before. Maybe the location of containerd can be included in the docker settings plugin? IDK. Ether way. my stuff seems to be working allot better now.
Docker Space issues on USB flash drive. /var/lib/containerd taking up space on drive
-
votdev
Approved the thread. -
What path do you have in:
In OMV | Services | Compose | Settings | Docker | Docker storage
Using a path on your system drive there will be a problem.
Leaving that setting empty will cause the default to be used which is under /var/library on the system drive.
-
-
-
I used ncdu found that 35gigs was being used up on my containerd folder.
What is using all this space? In my case the directory /var/lib/containerd is less than 3MB.
I cannot remember any post here regarding issues with size of that directory. Only /var/lib/docker
-
What is using all this space? In my case the directory /var/lib/containerd is less than 3MB.
I cannot remember any post here regarding issues with size of that directory. Only /var/lib/docker
Yep, With 50 to 60 containers running, my /var/lib/containerd is just over 8MB.
What I want to know is. Where is cointainerd supposed to go after setting the docker location? Is this because I installed the kernal plugin.? Could that have something to do with this? Do I need containerd? I have read other threads saying that it is not needed. Maybe I just needed to delete containers, networks, volumes then spin all the containers up.
Containerd is installed as a docker dependency, so yes you need it.
You need to figure out what is using all the space. Just deleting and recreating everything with the same configuration will likely create the same result.
This process will most likely require taking all containers down, perhaps a removal and reinstall of docker/containerd removing the contents of the offending directory in between, and then bringing up the containers one at a time, checking the usage of that offending directory each time.
Hopefully that will let you find the container that is causing this.
-
-
This is what my containerd looks like right now. It is syslinked to my NVME drive at the moment. I stopped using a bind mount. I don't know what is causing this or why. Keep in mind I did do a full reinstall and I am getting the same issue. Could the Kernal plugin be the cause? I don't really have anything crazy. Every single one of my containers seem to be getting snapshots taking up this space. -
Could the Kernal plugin be the cause?
Definitely not.
It is syslinked to my NVME drive at the moment.
The bind mount might have caused that if docker couldn't create its overlayfs mounts. Never tried. Seems like it would cause the snapshot issue. None of my docker machines have more than 1mb in /var/lib/containerd. Why are you symlinking to a different drive?
-
Definitely not.
The bind mount might have caused that if docker couldn't create its overlayfs mounts. Never tried. Seems like it would cause the snapshot issue. None of my docker machines have more than 1mb in /var/lib/containerd. Why are you symlinking to a different drive?
Because those snapshots would fill my 64gb flash drive and cause major performance issues. The first time I had this issue I ended up reinstalling OMV and doing a start from scratch. I am doing my backups using the docker plugin. not sure if that is causing the snapshot issue. I didn't do anything weird accept setup syslinks in my /srv/ to all my shares before I made the shares.
-
-
Because those snapshots would fill my 64gb flash drive and cause major performance issues. The first time I had this issue I ended up reinstalling OMV and doing a start from scratch. I am doing my backups using the docker plugin. not sure if that is causing the snapshot issue. I didn't do anything weird accept setup syslinks in my /srv/ to all my shares before I made the shares.
Performance issues? That's a big understatement.
What is the rationale for doing this?: "I didn't do anything weird accept setup syslinks in my /srv/ to all my shares before I made the shares."
-
Performance issues? That's a big understatement.
What is the rationale for doing this?: "I didn't do anything weird accept setup syslinks in my /srv/ to all my shares before I made the shares."
I am pretty sure I have having usb space issues before doing this. But doing a syslink to all my shares in my /srv/ drive was the only way I could get more then one SMB share to work on my mergerFS. I needed a low perm share and a high perm share. No matter what I did I could not get the 2nd share working. Admittedly Grok recommended using syslinks to bypass some sort of mergerFS permission issue nonsense. I couldn't get sources from the stupid thing, but it worked. It actually worked. So I went ahead and setup all my shares that way.
/srv$ tree -I "dev-disk-by-uuid-*" -t -L 1
.
├── mergerfs
├── GAMES -> /srv/mergerfs/merge/sharestorage/GAMES/
├── PLEX -> /srv/mergerfs/merge/sharestorage/PLEX/
├── OMVbackup -> /srv/mergerfs/merge/sharestorage/#backup/OMVbackup/
├── PS2SMB -> /srv/mergerfs/merge/sharestorage/PS2SMB/
├── #pictures -> /srv/mergerfs/merge/sharestorage/#pictures/
├── SYNCTHING -> /srv/mergerfs/merge/sharestorage/SYNCTHING/
├── compose -> /srv/dev-disk-by-uuid-065797a6-b491-41b2-94c4-1b3ee3e4c260/compose/
├── dockerbackups -> /srv/mergerfs/merge/sharestorage/#backup/dockerbackups/
├── port -> /srv/dev-disk-by-uuid-065797a6-b491-41b2-94c4-1b3ee3e4c260/portainer/
├── dockercontain -> /srv/dev-disk-by-uuid-065797a6-b491-41b2-94c4-1b3ee3e4c260/dockcontain/
├── OMVbackups -> /srv/mergerfs/merge/sharestorage/#backup/OMVbackup/
Not sure how any of this would even matter considering that my docker install targets the UUID and not a syslink.
/srv/dev-disk-by-uuid-065797a6-b491-41b2-94c4-1b3ee3e4c260/dockcontain/docker
and I have reinstalled it several times using the GUI. -
I can't even begin to follow what you are doing or why. And I am willing to say nobody else is doing these things in the way you are doing them either. I suggest starting over from scratch and not doing things like this.
And it's symlinks not syslinks.
-
-
Because those snapshots would fill my 64gb flash drive and cause major performance issues.
They shouldn't. All of my docker machines don't even have a 64gb os drive. I have no idea why you are filling the drive with "snapshots".
I am doing my backups using the docker plugin. not sure if that is causing the snapshot issue.
The docker (openmediavault-compose) plugin does not use snapshots. It either backs up the container live or stops it. Are you using snapshots somewhere? I honestly don't know why someone would.
-
so this folder is nearly empty on everyone else's setup?
-
-
Yep
Code
Display More/var/lib/containerd# du -hd1 . 544K ./io.containerd.content.v1.content 1.7M ./io.containerd.metadata.v1.bolt 0 ./io.containerd.runtime.v1.linux 0 ./io.containerd.runtime.v2.task 0 ./io.containerd.snapshotter.v1.btrfs 0 ./io.containerd.snapshotter.v1.native 0 ./io.containerd.snapshotter.v1.overlayfs 0 ./tmpmounts 0 ./io.containerd.snapshotter.v1.blockfile 0 ./io.containerd.snapshotter.v1.erofs 0 ./io.containerd.sandbox.controller.v1.shim 2.2M . -
so this folder is nearly empty on everyone else's setup?
Yes.
Code
Display Morefred@omv:~$ sudo du -hd1 /var/lib/containerd 4.0K /var/lib/containerd/io.containerd.snapshotter.v1.blockfile 432K /var/lib/containerd/io.containerd.content.v1.content 8.0K /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs 4.0K /var/lib/containerd/io.containerd.runtime.v1.linux 1.4M /var/lib/containerd/io.containerd.metadata.v1.bolt 4.0K /var/lib/containerd/tmpmounts 8.0K /var/lib/containerd/io.containerd.snapshotter.v1.native 72K /var/lib/containerd/io.containerd.runtime.v2.task 4.0K /var/lib/containerd/io.containerd.snapshotter.v1.btrfs 4.0K /var/lib/containerd/io.containerd.sandbox.controller.v1.shim 4.0K /var/lib/containerd/io.containerd.snapshotter.v1.erofs 2.0M /var/lib/containerd -
Ok, I think I fixed it and now have it behaving like all your setups. I got rid of the syslinks, /var/lib/containerd is all on the USB.
1. I shutdown all my containers.
2. systemctl docker containerd stop
3. deleted the symlink and recreated the containerd dir.4.
docker info | grep "Storage Driver"results
Storage Driver: overlayfs
5.sudo nano /etc/docker/daemon.jsonMy results did not have a storage driver listed. just my dataroot. I edited it to the following.
5. turned everything back on, turned on all my containers.6. /var/lib/containerd is all on the USB. The following is the size after turning on all the containers. Keep in mind I would see the USB fill up in the past when I turned on each container.
Everything seems to be working now. Ill be back if it starts messing up. I am wondering what the contents of /etc/docker/daemon.json is on everyone else's install? I kinda want to know what caused this in the first place? But if it is never replicated again, I don't care that much. I appropriate all your help.
-
-
/etc/docker/daemon.json
Code{ "data-root": "/srv/d0/docker/docker-data" } -
/etc/docker/daemon.json
Code{ "data-root": "/srv/d0/docker/docker-data" }Interesting. Try running the following in terminal and post the results.
Codedocker info | grep "Storage Driver"
Participate now!
Don’t have an account yet? Register yourself now and be a part of our community!