Docker communication between containers ... again

    • OMV 4.x
    • Docker communication between containers ... again

      I know that this has been posted before, but there appears to be no satisfactory list of steps to solve it. I had this working on my old OMV4 setup - but it crashed and I am unable to remember how to do it.

      Environment:
      • OMV 4.1.8 installed on Dell PE R720
      • Docker-UI plugin and Docker installed via OMV UI of the same version
      • Docker Client v18.03.1-ce
      • Docker Server v18.03.1-ce
      • OMV says I have all the latest patches
      • No Docker network changes to default setting


      Scenario #1:
      • Pull linuxserver/tt-rss:latest image
      • Pull postgres:9 image
      • Run Postgres image as "postgres9" on Bridge network with either no ports exposed, or 5432->5432
      • Run TTRSS image as "ttrss" on Bridge network with all the appropriate ports exposed and the extra arguments "--link postgres9:db"
      • TTRSS web application is unable to connect to the Postgres9 database. I receive the following error: "Unable to connect to database (as ttrss to postgres9, database ttrss):Unable to connect to database (as ttrss to postgres9, database ttrss):"
      • I run an interactive terminal on ttrss (docker exec -it ttrss bash) and not only can I ping the IP Address which postgres9 is using, if I ping via the name "postgres9" it resolves properly.


      Scenario #2:.
      • Pull linuxserver/tautulli (formerly PlexPy)
      • Run Tautulli as "plexpy" on Bridge network with proper ports exposed
      • Access the exported port on my OMV server (172.17.0.2:8181 exposed as 8181)
      • The first step is for Tautulli to connect to Plex.TV and collect a token
      • Web application attempts to fetch token ... but fails with the following error:

      Source Code

      1. 2018-06-12 22:50:32 - INFO :: CP Server Thread-7 : Tautulli PlexTV :: Fetching a new Plex.tv token for Tautulli.
      2. 2018-06-12 22:50:52 - WARNING :: CP Server Thread-7 : Failed to access uri endpoint /users/sign_in.xml. Is your server maybe accepting SSL connections only? HTTPSConnectionPool(host='plex.tv', port=443): Max retries exceeded with url: /users/sign_in.xml (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4b0635d450>: Failed to establish a new connection: [Errno -3] Try again',))


      The reason I posted two different scenarios is that I believe the problems are linked. I seem to remember that on an earlier version of Docker (before they went to a CE release) on OMV3 (yes, I inline upgraded to OMV4), there was a command or configuration I needed to change to allow containers on the default docker0 bridge to a) be able to talk to each other even with the "--link" argument and b) to NAT through the host OMV server to the outside world.

      I've tried to look at the easy thing:

      Source Code

      1. root@server:~# cat /proc/sys/net/ipv4/ip_forward
      2. 1
      3. root@server:~# iptables -L
      4. Chain INPUT (policy ACCEPT)
      5. target prot opt source destination
      6. ACCEPT all -- 192.168.2.0/24 anywhere
      7. ACCEPT all -- anywhere anywhere
      8. Chain FORWARD (policy DROP)
      9. target prot opt source destination
      10. DOCKER-USER all -- anywhere anywhere
      11. DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
      12. ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
      13. DOCKER all -- anywhere anywhere
      14. ACCEPT all -- anywhere anywhere
      15. ACCEPT all -- anywhere anywhere
      16. Chain OUTPUT (policy ACCEPT)
      17. target prot opt source destination
      18. Chain DOCKER (1 references)
      19. target prot opt source destination
      20. ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:8181
      21. ACCEPT tcp -- anywhere 172.17.0.7 tcp dpt:postgresql
      22. ACCEPT tcp -- anywhere 172.17.0.10 tcp dpt:https
      23. ACCEPT tcp -- anywhere 172.17.0.10 tcp dpt:http
      24. Chain DOCKER-ISOLATION-STAGE-1 (1 references)
      25. target prot opt source destination
      26. DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
      27. RETURN all -- anywhere anywhere
      28. Chain DOCKER-ISOLATION-STAGE-2 (1 references)
      29. target prot opt source destination
      30. DROP all -- anywhere anywhere
      31. RETURN all -- anywhere anywhere
      32. Chain DOCKER-USER (1 references)
      33. target prot opt source destination
      34. RETURN all -- anywhere anywhere
      Display All

      Is there anything I have overlooked?
      Would any additional information help in finding the root of the issue?

      Many thanks in advance! Logically Yours,

      --rogue
      Files

      The post was edited 1 time, last by LogicallyRogue ().

    • No clue if anyone is following this thread - but I am nearing an answer.

      Based on my initial report, we can distill the issue down to this:
      • Spin up a Docker container on the bridge network
      • Connect to it with an interactive shell (docker exec -it CONTAINERNAME bash)
      • ping google.com - this will work
      • cat /etc/passwd - look for a non-root user
      • su NONROOTUSER
      • ping google.com - this will fail both on the name resolution and pinging



      It appears on my old server, I lived through upgrading from "docker" to "docker-engine". This may have made some configuration changes during the migration which allowed my existing containers to continue to work. In retrospect, new Docker containers I brought up failed to work - and I chocked it up to a feature being broken in the application the container was serving.

      But on a fresh install of OMV4 with the Docker Plugin, all of my containers are affected. Some don't appear to be on first glance, but every single one of them is isolating the non-root users so that they cannot access any outbound network services. Even using the docker container linking between a web server and a db server doesn't work.

      [Edit: removed ranting about AppArmor as it was incorrect]

      The post was edited 1 time, last by LogicallyRogue: Bad data - don't want to confuse others ().

    • flvinny521 wrote:

      I am not really knowledgeable about your issue, but I was just wondering if you have moved your default Docker location to somewhere other than the default location.
      I have not changed any Docker defaults.

      I've added a bunch of AppArmor profiles (which disappear on reboots - but that's on me), and assigned them to containers - but no configuration changes or location changes.
    • The root cause is this: Docker generates the /etc/resolv.conf and /etc/hosts files.

      For some reason, on my OMV, the permissions of those two files in each container is set to 600. Only root users can do DNS Name resolution in that manor.

      The last several months have shown that Docker image authors are switching their images to run everything as a non-root user.

      For my containers only using resolv.conf - I can mount the Docker servers /etc/resolv.conf in the container as read only. That solves the permissions issue for me - but only because I want all of my Docker instances to point to my internal DNS rather than dnsmasq or other tricks.

      For my containers using hosts (anything I use --link for in the extra options), I need to interactively enter the Docker container (docker exec -it CONTAINER bash) and manually chmod the file to 644. This won't survive a modification - and I'm unsure if this will survive a reboot... But for the moment it's the best I can do.

      Any thoughts on configuration or modification of how Docker generates the two.filea would be helpful.