LXC containers vs. docker

1st Official Post

    I have been using OMV on a rpi for a few years and have it working well using docker for a number of services such as plex, transmission, unifi, influx, grafana, mysql and others.


    Now I have a minipc (Intel N100 based, 16g ram) and my plan is to migrate most services to the minipc. So this opens up the KVM plugin and LXC containers in addition to docker.


    I have setup plex in a LXC as a test and am happy with the progress I have made once I got my head around the basics of LXC containers and the KVM plugin!


    Questions:

    • Given omv is based on debian bookworm (I use proxmox kernel) - should I aim to use debian bookworm lxc containers where possible?
    • I am thinking that LXC containers will work faster / more efficiently than docker containers. Is this a reasonable assumption?
    • Any input and thoughts appreciated. Especially in relation to stability and ease of maintenance of LXC containers?


    ryecoaaron - just wanted to say that the KVM plugin is very nice. Thank you!


    Thanks all!

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

    • Official Post
    Quote from jata1

    should I aim to use debian bookworm lxc containers where possible?

    No reason to if you don't want to. Just like docker, you can run whatever container you want since it won't affect the host.


    Quote from jata1

    I am thinking that LXC containers will work faster / more efficiently than docker containers. Is this a reasonable assumption?

    Nope. They are both containers. docker is actually probably more efficient since it is only running the primary service instead of basically the full OS.

    Quote from jata1

    Especially in relation to stability and ease of maintenance of LXC containers?

    The maintenance and stability of an LXC container is pretty much the same as a VM. You will have to patch the system unlike docker where you just update to a new image.

    omv 8.0.10-2 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.6 | compose 8.1.3 | cterm 8.0 | borgbackup 8.1.5 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Thanks 1

    This is of course simple way with docker,but usually you cannot just change image name and update(lets say gitlab), you need to go version by version. And of course use default container images as much as you can, because the security of those containers is in maintaners hands.

    I use a combination of Docker LXC and full VM.


    as was suggested, LXC does "borrow" some things from the host but not as much as docker, so my general way if working is this:


    I will use docker is I can find a suitable image and like the way it works.


    I will use an LXC if I can't find that image and/or I need the service to operate as an independent computer with it's own IP, better host isolation, a stable OS that is "update controlled" independent from the host, etc. LXC is almost as fast as docker and not much more of a load on the host than docker is.


    I will use a full VM if I need all those features of the LXC, but need a GUI or need to run a non-linux os.

    Asrock B450M, AMD 5600G, 64GB RAM, 6 x 4TB RAID 5 array, 2 x 10TB RAID 1 array, 100GB SSD for OS, 1TB SSD for docker and VMs, 1TB external SSD for fsarchiver OS and docker data daily backups

    Edited 2 times, last by BernH ().

    Like 1 Thanks 1

    Thanks all. Very helpful input.


    I will try a few lxc containers but it sounds like docker is mostly a good choice for the services I run.


    Much less work for me now switching to the new minipc. Although it has been interesting getting lxc containers running.


    I guess it will be interesting to see how intel N100 performs v rpi5

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

    I have got Plex running in a lxc today but it sounds like there are no advantages over a docker container on the same hardware.


    Is that correct?


    I’m asking as I haven’t worked out (yet) how to map/access my media from the lxc so I might not bother with progressing that now.

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

    Quote from jata1

    I have got Plex running in a lxc today but it sounds like there are no advantages over a docker container on the same hardware.


    Is that correct?


    I’m asking as I haven’t worked out (yet) how to map/access my media from the lxc so I might not bother with progressing that now.

    You can map media in an lxc by using the filesystem passthrough configuration. I do this for my nextcloud lxc.


    The only real advantage of an lxc over docker for most things, as I see it, would be the extra isolation from the host from a user standpoint, as the user accounts used in an lxc don't exist on the host system, while the docker user does. This can be viewed as a little bit of extra security.

    Asrock B450M, AMD 5600G, 64GB RAM, 6 x 4TB RAID 5 array, 2 x 10TB RAID 1 array, 100GB SSD for OS, 1TB SSD for docker and VMs, 1TB external SSD for fsarchiver OS and docker data daily backups

    Edited once, last by BernH ().

    Thanks 1

    I agree with many good points made above -- I frequently favor Docker containers over LXC & VM except when a Proxmox Cluster is available, which can provide more high availability & uptime than a single host. Clustering Docker containers can also be accomplished with Kubernetes or other services (Docker Swarm was supposed to do this but has been heavily criticized for inadequacies and networking issues).


    Overall there are extremely few services for which I have not found adequate solutions in Docker images!

    One more question…


    Should I use zfs or ext4 on the internal system storage disk (not os) for docker and kvm containers?

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

    • Official Post

    I wouldn't use ZFS for Dockers. There was an issue, some years ago, where Docker's use of a type of overlayFS made Dockers look like a "legacy filesystem" to ZFS. At that time ZFS required a "driver" to be compatible with Docker. That issue may have been solved in later versions of ZFS but I don't know that for sure. Perhaps someone else can chime in with the answer to that.

    While I use ZFS for NAS storage / shares, for utilities purposes (Dockers, storage for a UrBackup server, etc.), I use a standalone disk and EXT4. To me that seemed like a clean solution.

    That's great. Thanks. I am using btrfs for my media/data storage so i will keep things as they are and use ext4 for docker etc.

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

    The past issue with using docker on zfs filesystems was that there was no overlayfs support. Using docker zfs storage driver led to poor performance and generated very large numbers of snapshots. But the good news is that since Oct 2023 zfs has overlayfs support:


    Release zfs-2.2.0 · openzfs/zfs
    Supported Platforms Linux: compatible with 3.10 - 6.5 kernels FreeBSD: compatible with releases starting from 12.2-RELEASE New Features Block cloning…
    github.com

    Quote
    Linux container support (#12209, #14070, #14097, #12263) - Added support for Linux-specific container interfaces such as renameat(2), support for overlayfs, idmapped mounts in a user namespace, and namespace delegation support for containers.

    Thanks Krisbee - So would you recommend using zfs on a system/services drive instead of ext4.


    I might give zfs a try and see how it goes

    OMV 8 (latest) on N100 minipc (16GB) and rpi5 (8GB). OS on SSD/SD. System ext4 on SSD. Data BTRFS on HDDs

    • Official Post

    For both ZFS and BTRFS you must configure the storage controller and take into account the uniqueness of each of them. You can read here:

    BTRFS -> https://docs.docker.com/engine/storage/drivers/btrfs-driver/

    ZFS -> https://docs.docker.com/engine/storage/drivers/zfs-driver/

    If you have the possibility to do so, my recommendation is to use EXT4 for docker.

    chente I thought the link in #12 above makes it clear that you can use the recommended default docker "overlayfs" storage driver in ZFS since OpenZFS version 2.2.0. You don't need to use the docker zfs storage driver any more. E.g:


    • Official Post

    Well, I haven't studied this in depth so I'm not sure what you mean. I only limit myself to reading the current docker documentation and if I understand correctly it is necessary to configure docker storage. He also talks about using block devices.

    If I am understanding it wrong, please correct me.

    Furthermore, the docker documentation recommends not using ZFS with docker in production. So as long as that doesn't change, I prefer to follow that docker recommendation and recommend the use of EXT4 for docker in the forum.

    To the quote docker docs at https://docs.docker.com/engine…rs/select-storage-driver/


    "overlay2 is the preferred storage driver for all currently supported Linux distributions, and requires no extra configuration.

    The Docker Engine has a prioritized list of which storage driver to use if no storage driver is explicitly configured, assuming that the storage driver meets the prerequisites, and automatically selects a compatible storage driver. You can see the order in the source code for Docker Engine 27.1.2"


    As ZFS & BRTFS both support overlayfs you do not have to configure the storage controller unless you have some specific requirement to do so.

    • Official Post

    I have doubts about that. I understand that if you install docker on a system with ZFS or BTRFS, the appropriate driver will be automatically configured. But in the case of OMV docker is always installed on an EXT4 file system by default. You can later modify the /var/lib/docker folder to another location which may be a different file system such as ZFS or BTRFS. But reading the ZFS document it seems that it is necessary to manually modify the docker configuration file to use the zfs driver. https://docs.docker.com/engine…th-the-zfs-storage-driver. It would be good to know if this is so. Have you tried it?


    Anyway I keep reading everywhere that it is recommended to use overlay2 for maximum stability, that is, EXT4.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login