Docker in LXC with kvm plugin AppArmor issue

Jip · Aug 21st 2023

Is anyone successfully running Docker inside an LXC container made with the KVM plugin? My test lxc container (Debian 12) runs into this Read-only file system error when doing a docker run:

Code

docker: Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running `/usr/sbin/apparmor_parser apparmor_parser -Kr /var/lib/docker/tmp/docker-default2194068202` failed with output: apparmor_parser: Unable to replace "docker-default".  Unknown error (30): Read-only file system

The only way I can get it working is uninstalling AppArmor inside the container.

But when I use lxc-create to make a similar Debian 12 container (based on the same image), then I don't need to remove AppArmor in order to get docker working. I've noticed the fs mounts look very different compared to creating a similar container using lxc-create.

The LXC container XML looks like this:

Code

<domain type='lxc'>
  <name>test</name>
  <uuid>95ef5e04-74cc-4349-8fa0-9e83a3f9e391</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/sbin/init</init>
  </os>
  <features>
    <privnet/>
  </features>
  <cpu>
    <topology sockets='1' dies='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
    <filesystem type='mount' accessmode='mapped'>
      <source dir='/home/'/>
      <target dir='/'/>
    </filesystem>
    <interface type='network'>
      <mac address='52:54:00:8f:54:4f'/>
      <source network='default'/>
    </interface>
    <console type='pty'>
      <target type='lxc' port='0'/>
    </console>
  </devices>
</domain>

Display More

These are my installation steps:

Code

root@LXCNAME:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
719385e32844: Pull complete 
Digest: sha256:dcba6daec718f547568c562956fa47e1b03673dd010fe6ee58ca806767031d1c
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running `/usr/sbin/apparmor_parser apparmor_parser -Kr /var/lib/docker/tmp/docker-default2194068202` failed with output: apparmor_parser: Unable to replace "docker-default".  Unknown error (30): Read-only file system

error: exit status 226.
ERRO[0003] error waiting for container:                 


root@LXCNAME:~# systemctl status apparmor | cat
○ apparmor.service - Load AppArmor profiles
     Loaded: loaded (/lib/systemd/system/apparmor.service; enabled; preset: enabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: inactive (dead)
       Docs: man:apparmor(7)
             https://gitlab.com/apparmor/apparmor/wikis/home/

root@LXCNAME:~# mount
devpts on /dev/pts type devpts (rw,nosuid,relatime,gid=5,mode=620,ptmxmode=666)
devfs on /dev type tmpfs (rw,nosuid,relatime,size=64k,mode=755,inode64)
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net/ipv4 type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net/ipv6 type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (ro,nosuid,nodev,noexec,relatime)
libvirt on /proc/meminfo type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
devpts on /dev/ptmx type devpts (rw,nosuid,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/tty1 type devpts (rw,nosuid,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/console type devpts (rw,nosuid,relatime,gid=5,mode=620,ptmxmode=666)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=801972k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/sda1 on /var/lib/docker type ext4 (rw,relatime,errors=remount-ro)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=104856k,nr_inodes=26214,mode=700,inode64)


Docker starts working after `apt remove apparmor` and adding under <features>:

    <capabilities policy='allow'>
      <mknod state='on'/>
    </capabilities>

Display More

Compared to lxc-create:

Code

lxc-create -t download -n test

nano /var/lib/lxc/test/config
#lxc.apparmor.profile = generated
#lxc.apparmor.allow_nesting = 1

lxc-start test
lxc-attach test
docker run hello-world
# docker run works


root@test:~# systemctl status apparmor
○ apparmor.service - Load AppArmor profiles
     Loaded: loaded (/lib/systemd/system/apparmor.service; enabled; preset: enabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: inactive (dead)
       Docs: man:apparmor(7)
             https://gitlab.com/apparmor/apparmor/wikis/home/


mount
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro)
none on /dev type tmpfs (rw,relatime,size=492k,mode=755,inode64)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/devices/virtual/net type sysfs (rw,relatime)
sysfs on /sys/devices/virtual/net type sysfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
devpts on /dev/lxc/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
none on /proc/sys/kernel/random/boot_id type tmpfs (ro,nosuid,nodev,noexec,relatime,size=492k,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666,max=1024)
devpts on /dev/ptmx type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666,max=1024)
devpts on /dev/lxc/tty1 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666,max=1024)
devpts on /dev/lxc/tty2 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666,max=1024)
devpts on /dev/lxc/tty3 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666,max=1024)
devpts on /dev/lxc/tty4 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666,max=1024)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=801972k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)

Display More

I had to comment the AppArmor settings in /var/lib/lxc/test/config since AppArmor is not installed on the OMV host. But I didn't have to uninstall it inside the container. As you can see the mounts made by lxc and libvirt are very different... E.g. the read-only securityfs on /sys/kernel/security which seems to be the problem in libvirt lxc does not exists in the container made with lxc-start.

Is it possible to get docker working in the LXC container made with the KVM plugin without uninstalling AppArmor inside the container?

ryecoaaron · Aug 21st 2023

Quote from Jip

Is anyone successfully running Docker inside an LXC container made with the KVM plugin?

Why is the only question I have?

Quote from Jip

But when I use lxc-create to make a similar Debian 12 container (based on the same image), then I don't need to remove AppArmor in order to get docker working.

lxc-create and libvirt are evidently doing different things.

Quote from Jip

Is it possible to get docker working in the LXC container made with the KVM plugin without uninstalling AppArmor inside the container?

Not with the way libvirt is creating the lxc container. I honestly don't know why you are doing this though. Running a container in a container is something I don't want to make changes for.

Jip · Aug 21st 2023

Thanks for replying! I'm actually asking for a different setup (not OMV) where there's only libvirt available (and no docker). Since OMV has the KVM plugin and a community of libvirt users I was wondering if there was already some 'magic' going on in the plugin to make this work. But I was able to reproduce my issue also on OMV so now I'm wondering if anyone here ran into this (and found a solution). Not asking to make changes, but looking for advice/experience.

ryecoaaron · Aug 21st 2023

Ah. I would run containers in a VM instead of LXC in that situation. Since libvirt is doing so much of the LXC setup, I'm not even sure where to start to look for how to change it. On the other hand, I don't see an issue with removing apparmor if the containers are not on the internet.

Jip · Aug 21st 2023

Thanks! A VM is what I'm trying to avoid actually

I noticed that inside the lxc-create container the output of docker info shows:

Security Options:

seccomp

Profile: builtin

cgroupns

So AppArmor is not enabled there by default.

On the container made with kvm plugin (libvirt) docker info shows:

Security Options:

apparmor

seccomp

Profile: builtin

cgroupns

There it tries to use AppArmor, which it can't due to /sys/kernel/security being mounted read-only. So in order to make the libvirt container behave like the lxc-create container I just found out I can run:

umount /sys/kernel/security && systemctl restart docker

Then docker info will no longer list apparmor and docker run hello world works.

So I think I have to look for a way to avoid /sys/kernel/security being mounted in the first place...

ryecoaaron · Aug 21st 2023

Quote from Jip

A VM is what I'm trying to avoid actually

Any reason why?

Quote from Jip

So I think I have to look for a way to avoid /sys/kernel/security being mounted in the first place...

It is probably baked into the images the plugin is downloading.

Jip · Aug 21st 2023

I'm trying to avoid a VM so I can have direct access to the files on the host system using bind mounts (although maybe 9pfs could work, haven't tried yet). And I prefer not to have to pre-allocate RAM.

I don't think it is baked in the image though. Even when using the same rootfs (created from the Debian LXC image from linuncontainers.org), the behavior is different when starting with lxc-start or virsh. Only in the second instance the /sys/kernel/security is mounted.

ryecoaaron · Aug 21st 2023

Quote from Jip

I don't think it is baked in the image though. Even when using the same rootfs (created from the Debian LXC image from linuncontainers.org), the behavior is different when starting with lxc-start or virsh. Only in the second instance the /sys/kernel/security is mounted.

It was just a guess. I haven't looked into it.

Jip · Aug 21st 2023

Admittedly a very ugly hack, but I created an empty directory to be mounted at /sys/kernel/security and added this to the XML config:

Code

    <filesystem type='mount' accessmode='mapped'>
      <source dir='/empty/'/>
      <target dir='/sys/kernel/security'/>
    </filesystem>

Now docker doesn't complain, and runs without trying to use AppArmor and I didn't need to modify the image or manually uninstall AppArmor.

The full XML looks like this:

Code

<domain type='lxc' id='7617'>
  <name>test</name>
  <uuid>95ef5e04-74cc-4349-8fa0-9e83a3f9e391</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/sbin/init</init>
  </os>
  <features>
    <privnet/>
    <capabilities policy='allow'>
      <mknod state='on'/>
    </capabilities>
  </features>
  <cpu>
    <topology sockets='1' dies='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
    <filesystem type='mount' accessmode='mapped'>
      <source dir='/home/'/>
      <target dir='/'/>
    </filesystem>
    <filesystem type='mount' accessmode='mapped'>
      <source dir='/empty/'/>
      <target dir='/sys/kernel/security'/>
    </filesystem>
    <interface type='network'>
      <mac address='52:54:00:8f:54:4f'/>
      <source network='default' portid='b98239f5-46ab-47a0-817a-a1f161768a72' bridge='virbr0'/>
      <target dev='vnet8'/>
      <guest dev='eth0'/>
    </interface>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='lxc' port='0'/>
      <alias name='console0'/>
    </console>
  </devices>
</domain>

Display More

trythat · Aug 22nd 2023

Quote from ryecoaaron

Not with the way libvirt is creating the lxc container. I honestly don't know why you are doing this though. Running a container in a container is something I don't want to make changes for.

I follow a youtube channel called @scottibyte and he explains as others have that a lxc container uses up less resources than a vm. He use docker within a lxc container a lot, I do understand to some extent why he does it.

I've not tried it yet, as I've not seen many posts on here regarding lxc, which to me says either it's a little harder than using bare metal ubuntu and lxc, or people really just want to use docker an vms, which is pretty easy to do.

The scottibyte channel is a good jumping in place to consider using lxc, very well put together tutorials. Plus I believe lxc is extremely popular with proxmox.

I might actually have a go with my test vm for OMV and see what I can accomplish, probably more in the autumn/winter though, as I expect it might become a bit of a rabbit warren once I start playing with it

ryecoaaron · Aug 22nd 2023

Quote from trythat

he explains as others have that a lxc container uses up less resources than a vm.

It does.

Quote from trythat

He use docker within a lxc container a lot, I do understand to some extent why he does it.

Quote from trythat

I believe lxc is extremely popular with proxmox.

lxc is popular because there is no other container option in proxmox. People use docker in lxc for this reason. If proxmox had a native docker option, I doubt people would use lxc for docker. Since OMV has an option for VMs, LXC, and docker, there is very little reason to combine any of those.

Docker in LXC with kvm plugin AppArmor issue

Participate now!

Tags