Docker Cloudflare Tunnel container logs error "failed to sufficiently increase send buffer size" despite setting this on OMV 6

Hello everyone, hope you can possibly help me configure my OMV6 server settings to resolve this error...

I have installed a Docker image for creating a Cloudflare Tunnel via the following YML on the new Services > Compose > Files OMV plugin:

version: "3.9"
services:
  tunnel:
    container_name: cloudflared
    image: cloudflare/cloudflared:latest
    restart: always
    command: tunnel run
    environment:
      - TUNNEL_TOKEN=<REMOVED>

My account for such Cloudflare tunnel is active and working, and both tunnels on two different OMV 6 servers using this image/container work well and are reported as "Healthy" on the Cloudflare Tunnel dashboard.

However, I recently had encountered this error in the container logs:

2024/02/08 23:28:00 failed to sufficiently increase send buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB).
See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.

So I studied what is proposed and found this interesting blog post, and applied the parameters that survive an OMV 6 reboot.

I checked the current UDP/IP receive buffer default and limit by typing the following commands:

$ sysctl net.core.rmem_default
net.core.rmem_default = 2097152

$ sysctl net.core.rmem_max
net.core.rmem_max = 2097152

These default values are less than 26,214,400 bytes (25MB) so I added the following lines at the bottom of the /etc/sysctl.conf file:

$ nano /etc/sysctl.conf

net.core.rmem_max=26214400
net.core.rmem_default=26214400

Changes to /etc/sysctl.conf took effect upon reboot, and were confirmed. But the Cloudflare container error persisted...

I even tried a little above the 25MB, still nothing, despite the settings surviving a reboot and seemingly being applied across the OMV server settings:

$ sysctl net.core.rmem_default
net.core.rmem_default = 27262976

$sysctl net.core.rmem_max
net.core.rmem_max = 27262976

However the above log in the container logs still persists... any ideas what may be wrong or not applicable in this case?

P.S. Currently running OMV 6.9.13-1 with 2x4GB RAM but this error has been there since some time ago.

Thank you!

Hi macom  ryecoaaron can I possibly pick your brain for ideas, when you can, as you are much more experienced than me? Would Docker inherit for its containers such system-wide setting that seems to be applied OK via /etc/sysctl.conf ? Perhaps the setting must be in the Docker level, or container level, for this Cloudflare tunnel service to not "complain"? Not sure, I admit, your feedback would be very appreciated. Thanks!

Thank you ryecoaaron ! What a great tip, I was not aware you could add sysctls to compose ! I will do that and report back tonight.

Question: If such settings affect the performance of the hosting router, like in the case of OMV with Docker, will these compose-level settings be respected? In this case cloudflared is asking for more "buffer size" but will it be given, via Docker, from the system overall ? Thanks.

Zitat von ryecoaaron

It is not commonly needed. This is actually the first real world use case I have seen for them.

These settings will not be applied to the system since it is in a different namespace.

Happy to have contributed to your first ever real-world case!

But then is there a point applying these on the compose-level...? If they aren't going to be respected in terms of "buffer size" do you see any real benefit except silencing this log error?

Anyway it seems these specific sysctls are not accepted by docker:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /proc/sys/net/core/rmem_default: no such file or directory: unknown

According to this Stack Overflow article:

Zitat

Since Docker containers share the host system's kernel and its settings, a Docker container usually can't run sysctl at all. (You especially can't disable security-critical settings like this one.) You can set a limited number of sysctls on a container-local basis with docker run --sysctl, but the one you mention isn't one of these.

Furthermore, you also can't force changes like this in a Dockerfile. A Docker image only contains a filesystem and some associated metadata, and not any running processes or host-system settings. Even if this RUN sysctl worked, if you rebooted your system and then launched a container from the image, that setting would be lost.

Given what you've shown in this Dockerfile – customized Linux kernel settings, no specific application running, an open-ended ssh daemon as the container process – you might consider whether a virtual machine fits your needs better. You can use a tool like Packer to reproducibly build a VM image in much the same way a Dockerfile builds a Docker image. Since a VM doeshave an isolated kernel, you can run that sysctl command there and it will work, maybe via normal full-Linux-installation methods like an /etc/sysctl.conf file.

When running these on OMV's CLI:

% docker run --sysctl net.core.rmem_max
invalid argument "net.core.rmem_max" for "--sysctl" flag: sysctl 'net.core.rmem_max' is not allowed
See 'docker run --help'

% docker run --sysctl net.core.rmem_default
invalid argument "net.core.rmem_default" for "--sysctl" flag: sysctl 'net.core.rmem_default' is not allowed
See 'docker run --help'

So I guess this is a dead end ryecoaaron. Anyways. Thanks for your help. I can live with the error, not sure the real impact in buffer/speed etc.

Yes, my connection is fast (at home). But no worries. Not much knowledge with UDP either. In any case, thank you for your kind help. I opened a bug request on their Github with the hope someone can shed more light. Thanks again for your time ryecoaaron !

Hello everyone, after posting the issue on GitHub too, it appears there is a solution for this type of error in containers. Originally I got it on Cloudflared but now also on Syncthing recent versions (image in docker).

According to member Dezrin there, he found something interesting when perusing Red Hat documentation:

Zitat

I attempted to follow the /etc/sysctl.conf fix, but upon checking the RedHat Documentation, their example for increasing the socket recieve buffer size includes 2 parameters that you hadn't stated:

Code

net.core.rmem_default=26214400
net.core.wmem_default=26214400
net.core.rmem_max=26214400
net.core.wmem_max=26214400

Confirmed that this resolves the problem, and I am no longer experiencing the failure within the logs.

To make the changes without editing the /etc/sysctl.conf file, use the following commands:

Code

sysctl -w net.core.rmem_default=26214400
sysctl -w net.core.wmem_default=26214400
sysctl -w net.core.rmem_max=26214400
sysctl -w net.core.wmem_max=26214400

Restart the container and this error should no longer be present within the logs.

This bug can be resolved.

For making the changes permanent, obviously include these at the bottom of /etc/sysctl.conf as I did, I rebooted, and the error doesn't appear now on either container logs.

Many thanks for the find, Dezrin, not sure what's your alias here (as you mentioned you read this thread).

And not sure why the two extra parameters you found were not mentioned in the original Wiki page that the error prompts us to visit.

Hi macom and ryecoaaron any risk on OMV 6.x (or later?) stability if we include these parameters permanently in /etc/sysctl.conf ?