Hey everyone,
I plan on changing my network card from AQC100 to RTL8126. In preparation for this, I decided to change the kernel with the built in tool to the latest ProxMox kernel 6.8, as it should have native support for the new NIC.
After rebooting, I noticed a very high system load that did not go down after several minutes. I investigated and found that some of the docker containers I am running behave unexpectedly, thus causing the high system load. I also can't reach my containers from the web.
Here are the relevant parts of the logs of some of the containers, and the log from dmesg:
immich-postgres:
2024-10-01 19:53:55.210 UTC [1] FATAL: called `Result::unwrap()` on an `Err` value: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }
2024-10-01 19:53:55.210 UTC [1] LOG: database system is shut down
watchtower:
time="2024-10-01T21:56:22+02:00" level=error msg="permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get \"http://%2Fvar%2Frun%2Fdocker.sock/v1.25/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D\": dial unix /var/run/docker.sock: socket: permission denied"
time="2024-10-01T21:56:22+02:00" level=info msg="Waiting for the notification goroutine to finish" notify=no
vaultwarden:
thread 'main' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.39.2/src/signal/unix.rs:58:53:
failed to create UnixStream: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
dmesg:
kauditd_printk_skb: 1390139 callbacks suppressed
[ 769.341735] audit: type=1400 audit(1727812722.236:209006964): apparmor="DENIED" operation="create" class="net" info="failed type and protocol match" error=-13 profile="docker-default" pid=5391 comm="su-exec" family="unix" sock_type="stream" protocol=0 requested="create" denied="create" addr=none
[ 769.341739] audit: type=1400 audit(1727812722.237:209006965): apparmor="DENIED" operation="create" class="net" info="failed type and protocol match" error=-13 profile="docker-default" pid=5391 comm="su-exec" family="unix" sock_type="stream" protocol=0 requested="create" denied="create" addr=none
[ 769.341741] audit: type=1400 audit(1727812722.237:209006966): apparmor="DENIED" operation="create" class="net" info="failed type and protocol match" error=-13 profile="docker-default" pid=5391 comm="su-exec" family="unix" sock_type="stream" protocol=0 requested="create" denied="create" addr=none
[ 769.341743] audit: type=1400 audit(1727812722.237:209006967): apparmor="DENIED" operation="create" class="net" info="failed type and protocol match" error=-13 profile="docker-default" pid=5391 comm="su-exec" family="unix" sock_type="stream" protocol=0 requested="create" denied="create" addr=none
Display More
The dmesg-logs were spammed with the audit message.
Judging from the logs, it seems like the kernel update caused some unwanted behaviour of apparmor.
Debian/OMV is running natively on the system, and all packages in my system are up to date. The versions of docker are:
docker-ce 5:27.3.1-1~debian.12~bookworm and docker-compose-plugin 2.29.7-1~debian.12~bookworm, I am using the OMV compose plugin to manage my containers.
Rebooting with the old kernel fixes the issue, so that's what I'm doing for now. But still, dmesg is spammed with this message:
[ 984.577643] audit: type=1400 audit(1727813824.884:40140): apparmor="ALLOWED" operation="open" profile="php-fpm" name="/usr/share/openmediavault/datamodels/rpc.notification.json" pid=26810 comm="php-fpm8.2" requested_mask="r" denied_mask="r" fsuid=999 ouid=0
[ 984.577647] audit: type=1400 audit(1727813824.884:40141): apparmor="ALLOWED" operation="file_lock" profile="php-fpm" name="/usr/share/openmediavault/datamodels/rpc.notification.json" pid=26810 comm="php-fpm8.2" requested_mask="wk" denied_mask="wk" fsuid=999 ouid=0
What can I do to resolve this issue, so I can use the latest ProxMox kernel?
Best regards,
Bastian