LXC containers vs. docker

Krisbee · Aug 24th 2024

IIRC, on OMV installing the compose plugin alone does not in install docker-ce. etc. It's only after you go through the compose settings page and save a config that the docker packages are installed. As you can specify the location of "Docker storage" at this stage, the content of the default location "/var/lib/docker" folder is written to the location specified in the settings. Hence, docker can be installed on ZFS in one step as I indicated in #17 above, to which you can add the output:

Code

root@omv7vm:~# cat /etc/docker/daemon.json
{
  "data-root": "/pool2/dockersys"
}
root@omv7vm:~#

So the "/var/lib/docker" folder may never be installed on an EXT4 filesystem in the first case and it will be using the overlayfs storage driver.

Installing docker to its default location on the system OS EXT4 filesystem and later moving the "/var/lib/docker" folder to a non-EXT4 system is a scenario I've not tested. But while you'd expect the "data-root" dir to change correctly in etc/docker/daemon.json, are all the files/dirs on the old "data-root" moved?

BernH · Aug 24th 2024

Quote from Krisbee

Installing docker to its default location on the system OS EXT4 filesystem and later moving the "/var/lib/docker" folder to a non-EXT4 system is a scenario I've not tested.

I don't use ZFS, but just to expand on your statement about not testing moving docker storage to a non-EXT4 filesystem, I moved my docker storage from the OS EXT4 filesystem to an XFS filesystem years ago when I first started playing with docker on OMV (back in about OMV4) without issue.

Of course, this is no shock as XFS is just a filesystem with a long history of native linux kernel support, while ZFS is a combination filesystem/raid amongst other things, and is newer to the linux world.

chente · Aug 24th 2024

Quote from Krisbee

IIRC, on OMV installing the compose plugin alone does not in install docker-ce. etc. It's only after you go through the compose settings page and save a config that the docker packages are installed. As you can specify the location of "Docker storage" at this stage, the content of the default location "/var/lib/docker" folder is written to the location specified in the settings.

That depends on the order of setting the fields in the plugin GUI. The vast majority of users will start by configuring the first field, the compose folder, press the Save button and, without knowing it, they will have installed docker in the /var/lib/docker folder and it will be up and running. After that they will configure the docker folder field but at that point docker is already installed. This behavior may change in the future if ryecoaaron modifies it but it remains an unknown.

Someone could even be running docker on EXT4 and after that decide to switch to ZFSor BTRFS. If that is the case, should the user manually modify the docker driver?

I would really like to know if that is going to work or not, just to explain it on the wiki. Since it seems like you don't know, I'll try if I find the time, but it won't be soon. In any case, I will maintain in the wiki the recommendation to use EXT4 as recommended in the Docker documentation, I think it is the most prudent.

crashtest · Aug 24th 2024

It seems that a full fledged debate is in now progress.
As a gentle reminder, for the purposes and intent behind this thread, the user's original questions have been answered.

Krisbee · Aug 24th 2024

Quote from chente

Someone could even be running docker on EXT4 and after that decide to switch to ZFSor BTRFS. If that is the case, should the user manually modify the docker driver?

They don't have to if using ZFS version >= 2.2.0. I believe docker on BTRFS has defaulted to overlayfs since about docker 23, so no need to change the storage driver. I'm sure you can find relevant discussion of r/zfs and r/btrfs.

chente · Aug 24th 2024

Quote from Krisbee

They don't have to if using ZFS version >= 2.2.0. I believe docker on BTRFS has defaulted to overlayfs since about docker 23, so no need to change the storage driver. I'm sure you can find relevant discussion of r/zfs and r/btrfs.

If I am understanding correctly what you are saying is that after a change from EXT4 to ZFS in the /var/lib/docker folder docker would work without any additional action if the ZFS version is equal to or greater than 2.2.0. You also say that BTRFS can work with the overlay2 driver (overlayfs?). However the docker documentation states that overlay2 is only compatible with XFS and EXT4. See here -> https://docs.docker.com/engine…orted-backing-filesystems

jata1 · Aug 25th 2024

In the interest of science, I will setup my system disk as zfs and then install docker clean/new on this drive. 😂😂

I will let you know how I get on.

My minipc has arrived from china and should be delivered tomorrow

Krisbee · Aug 25th 2024

jata1 I hope the new minpc lives up to expectations. To clarify, OMV OS device defaults to EXT4. You need at least one additional device to get docker on a non-EXT4 filesystem.

Quote from chente

If I am understanding correctly what you are saying is that after a change from EXT4 to ZFS in the /var/lib/docker folder docker would work without any additional action if the ZFS version is equal to or greater than 2.2.0. You also say that BTRFS can work with the overlay2 driver (overlayfs?). However the docker documentation states that overlay2 is only compatible with ZFS and EXT4. See here -> https://docs.docker.com/engine/storage/drivers/select-storage-driver/#supported-backing-filesystem

You perhaps meant XFS and EXT4 in your last sentence. This is clearly a case where docker docs are out of date and at odds with other docs (even their own at https://docs.docker.com/engine/release-notes/23.0/) and practical experience.

An example of docker on BTRFS:

Code

root@omvt:/# docker info | grep Storage
 Storage Driver: overlay2
root@omvt:/# cat /etc/docker/daemon.json
{
  "data-root": "/srv/dev-disk-by-uuid-ea7a408d-775d-470a-8ad5-74ee3e2fd779/dockersys"
}
root@omvt:/# btrfs sub li -t /srv/dev-disk-by-uuid-ea7a408d-775d-470a-8ad5-74ee3e2fd779
ID      gen     top level       path
--      ---     ---------       ----
256     33      5               apps
257     9       5               apps_backup
258     31      5               compose
259     33      5               dockersys
root@omvt:/# btrfs fi sho
Label: none  uuid: ea7a408d-775d-470a-8ad5-74ee3e2fd779
        Total devices 1 FS bytes used 226.11MiB
        devid    1 size 20.00GiB used 1.52GiB path /dev/sda

root@omvt:/# docker ps 
CONTAINER ID   IMAGE                  COMMAND     CREATED              STATUS              PORTS                                                                      NAMES
bc7814223563   linuxserver/heimdall   "/init"     About a minute ago   Up About a minute   0.0.0.0:99->80/tcp, :::99->80/tcp, 0.0.0.0:444->443/tcp, :::444->443/tcp   heimdall
066a88e0670c   amir20/dozzle:latest   "/dozzle"   4 minutes ago        Up 4 minutes        0.0.0.0:8888->8080/tcp, :::8888->8080/tcp                                  dozzle
root@omvt:/# ls -l /srv/dev-disk-by-uuid-ea7a408d-775d-470a-8ad5-74ee3e2fd779/dockersys
total 4
drwx--x--x 1 root root  152 Aug 25 10:24 buildkit
drwx--x--- 1 root root  256 Aug 25 10:48 containers
-rw------- 1 root root   36 Aug 25 10:24 engine-id
drwx------ 1 root root   16 Aug 25 10:24 image
drwxr-x--- 1 root root   10 Aug 25 10:24 network
drwx--x--- 1 root root 2198 Aug 25 10:48 overlay2
drwx------ 1 root root   20 Aug 25 10:24 plugins
drwx------ 1 root root    0 Aug 25 10:24 runtimes
drwx------ 1 root root    0 Aug 25 10:24 swarm
drwx------ 1 root root    0 Aug 25 10:45 tmp
drwx-----x 1 root root   56 Aug 25 10:24 volumes
root@omvt:/#

Display More

jata1 · Aug 26th 2024

Krisbee - understood regarding os disk and not using zfs. No problem!

chente -

installed OMV os on internal m.2 sata slot (formatted ext4)

then installed proxmox kernel

then installed zfs plugin

then created zfs pool (basic) for system disk using the other internal slot m.2 nvme

then docker installed on zfs pool

then copied across all of my docker persistent data to the zfs

checked and updated all the docker mappings/configs

done & beer.

Everything running great. No errors. Performance is great. Overall this is a big upgrade over my rpi5 8gb.

jata1 · Aug 26th 2024

Will monitor for a while. Keeping my rpi5 online but dedicated to all my home automation (home assistant etc) stuff.

Have found one issue but not with any of the above config (maybe it is my lack of understanding of zfs). I added a new/formatted usb ssd disk to the minipc but I can't add another new pool - the new device is not shown in the dropdown. Any suggestions?

crashtest · Aug 26th 2024

Quote from jata1

Will monitor for a while. Keeping my rpi5 online but dedicated to all my home automation (home assistant etc) stuff.

Give data backup some thought. (Because, it's just a matter of time...)

jata1 · Aug 26th 2024

crashtest. Yes I do backup. Basically everything using rsync to offsite server.

Both prod and backup data storage using multi disk btrfs so I’m good.

jata1 · Aug 27th 2024

zfs experiment is over in any case. Decided to delete the disk and setup as ext4 as I don't have a need for zfs on this single disk. And this is my prod server so stability is important to me.

All worked fine using zfs by the way - so i guess it is an option - but only really makes sense if you have more than one disk (and they are not USB).

Krisbee · Aug 27th 2024

@jatal Mentioning USB and ZFS in the same sentence makes me nervous, you've come to a sensible conclusion.

jata1 · Aug 27th 2024

Agreed. The disk i used for testing was not usb. It was an internal (pcie i think) m.2 nvme. But all the same, ext4 is the way to go for this type of use - together with a good backup strategy.

And I am very happy with my $200 ($120 US) minipc. It's a GMKtec G3. It totally kills my rpi5 and I paid less for it too.

chente · Aug 31st 2024

Quote from Krisbee

You perhaps meant XFS and EXT4 in your last sentence. This is clearly a case where docker docs are out of date and at odds with other docs (even their own at https://docs.docker.com/engine/release-notes/23.0/) and practical experience.

Yes, it was a typographical error, it has been corrected, thank you.

I guess you're right, there may be contradictions in the docker documentation. I think you are referring to this specifically. https://docs.docker.com/engine…-fixes-and-enhancements-6

I hope they stop discouraging ZFS and BTRFS on production systems soon. In the meantime I will keep the docker wiki on omv-extras as is to avoid problems.

Quote from jata1

installed OMV os on internal m.2 sata slot (formatted ext4)
then installed proxmox kernel
then installed zfs plugin
then created zfs pool (basic) for system disk using the other internal slot m.2 nvme
then docker installed on zfs pool

then copied across all of my docker persistent data to the zfs
checked and updated all the docker mappings/configs
done & beer.

Everything running great. No errors. Performance is great. Overall this is a big upgrade over my rpi5 8gb.

Display More

I'm glad this works apparently without problems, that's good news. Thanks for the effort to try it.

LXC containers vs. docker

jata1 Aug 27th 2024

Participate now!