Docker custom network stopped resolving ip of container name

godfuture · 12. November 2021

Hi guys, I have a very urgent issue.

Some of my docker container stopped starting because of "no route to host" error. In the past, I have setup a container for fluentd as logging host and configured its container name as host of the logging driver flags in other containers. Basically it is this feature:
https://stackoverflow.com/ques…ame-instead-of-ip-address

This setup worked without any flaws since quite a time. Now restarting my server since long runtime, docker does not resolve the internal fluentd container ip anymore, but instead resolves my public ip.

Do you know where and how I could start tackling this issue? Did something with docker internal dns change recently? I am not using docker-compose.

Glad for any help!

Update: now I additionally have different other issues. When opening office documents in Nextcloud with collabora, I get "unauthorized wopi host". Don't know yet how this all comes together...

raulfg3 · 13. November 2021

do you use a custom ntwork like my-net>?.

use this to all of your stacks:

networks:

default:

external:

name: my-net

an example:

Code

version: "2.1"
networks:
  default:
    external:
      name: my-net
services:
  ouroboros:
    image: pyouroboros/ouroboros
    container_name: ouroboros
    environment:
      - LATEST=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped

Alles anzeigen

if swag is in my-net you can resolve dockers name whitout problem

chente · 13. November 2021

Zitat

Intel Pentium G3460T @ 3GHz

Debian GNU / Linux 9.2 (stretch)

Versión: 4.0.10-1 Arrakis

OMV6 is close to stable. If your system matches your signature, you are still in OMV4. In your case I would consider updating the system. Perhaps this is the root of the problem.

Zoki · 13. November 2021

Maybe you can post the fluentd stack and one other which can not find it.

Do not forget to zero out sensible information.

godfuture · 14. November 2021

Zitat von chente

OMV6 is close to stable. If your system matches your signature, you are still in OMV4. In your case I would consider updating the system. Perhaps this is the root of the problem.

No. did not update the signature. I am on OMV5, luckily.

Zitat von raulfg3

do you use a custom ntwork like my-net>?.

Yes. And it seems I can resolve those names inside the container. But still when starting, using those names for log driver does not work any longer.

Zitat von raulfg3

use this to all of your stacks:

I think stacks are really useful. But as I am using a lot of encrypted drives, and I had issues with dependencies of crypt partitions used by docker, I have written my own systemd service files.

I do not really know what logs to provide, as this is happening during the startup phase and the error message is quite clear. So I try to explain the setup a bit:
My containers are configured with restartPolicy "never". On start, there are systemd services that correctly work together with crypttab dependencies. So systemd is starting the container in correct order. Those docker services with least dependencies are started first like fluentd or portainer. When fluentd is up, other containers are started like mariadb. When mariadb is up, nextcloud and firefox sync goes up. In the past, the logging driver host flag on every container was able to resolve the fluentd container with its name and returned its local ip in network.

Well, today resolving only works in container. Maybe not trying to fix it, but to use a different approach. I do not want to start using fixed ips. Too much effort and if something changes, nothing works anymore. Does someone know how to achieve the same with help of docker dns?

Zoki · 14. November 2021

Just to check my understanding:

You have set up fluentd container as logging target.

You are trying to instruct the docker daemon (not the software running inside the containers) to sende the logs (stdout of the containers) to the fluentd container.

You are trying touse the fluentd containers hostname in the config on the docker host.

So the host has to resolve the container hostname.

godfuture · 14. November 2021

Zitat von Zoki

So the host has to resolve the container hostname.

As this was possible till few weeks / months ago...do you think this has changed? Are there possibilities to return the IP of the app host instead of asking the dns server for the container name which returns the public ip?

Zoki · 14. November 2021

Maybe you can map the port of your fluentd container to the port on 127.0.0.1 and use localhost instead of the container hostname?

So the port is not open to the public, but can be reached from the host.

Zoki · 14. November 2021

and if you add the container hostname onto your hosts file as 127.0.0.1 you don't have to change your scripts

godfuture · 14. November 2021

Zitat von Zoki

Maybe you can map the port of your fluentd container to the port on 127.0.0.1 and use localhost instead of the container hostname?
So the port is not open to the public, but can be reached from the host.

It works to open the ports on host itself and to refer localhost in fluentd address. This is also okay or even better, because I do not need to attach fluentd container in all networks.

Now I just need to find out why collabora and nextcloud are not working anymore. I guess this is also related to changed dns behavior.

Many thanks!

Docker custom network stopped resolving ip of container name

chente 14. November 2021

Jetzt mitmachen!

Tags