docker composer - networks

Jubri · Jul 1st 2023

Hi all,

I am trying to migrate from Portrainer to Composer, and I have a question about the network configuration.

Prior, in Portainer, this was my config:

Code

- [Portainer] Network Configuration
    Endpoints: local
        Networks: Add networks
            Name: 192.168.1.x.[GW1].config
            Driver: macvlan
            Configuration
            Parent network lan: eno1
            Subnet: 192.168.1.0/24
            Gateway: 192.168.1.1
            >> Create the network
        Networks: Add networks
            Name: 192.168.1.x
            Driver: macvlan
            Configuration: 192.168.1.x.[GW1].config
            >> Create the network

Display More

... and I was using 192.168.1.x as the network in all my containers. Each with a different IP.

Now I'm trying to use this config, creating a file to each container:

Code

services:
  pihole:
    container_name: "pihole"
    ...
    networks:
      macvlan_network:
        ipv4_address: 192.168.1.102
networks:
  macvlan_network:
    driver: macvlan
    driver_opts:
      parent: eno1
    ipam:
      driver: default
      config:
       - subnet: 192.168.1.0/24
         ip_range: 192.168.1.102/32
         gateway: 192.168.1.1

Display More

Only with this container, all was fine. But when I try to add another container with same config (only changing ipv4_address and ip_range) it says: Error response from daemon: failed to allocate gateway (192.168.1.1). Address already in use

Any suggestions?

Thank you.

Regards,

Jubri.

BernH · Jul 1st 2023

Do you really need macvlan networks for all of your docker containers? This will give each container a real ip address on your lan.

Pihole needs that because it has to expose port 53 to the lan and not interfere with OMV. As a general rule docker containers do not need to run like that as they are just supposed to be applications that are isolated from the host os to prevent causing host os problems.

If you need to have multiple containers on the same network, all you need to do is declare the same bridge network in each one.

Here is an example of how to declare that bridge network.

Code

version: "3.7"

services:
  service_name:
    container_name: container_name
    image: docker/image_name
    ports:
      - "3030:3030"
    environment:
      - PUID=1000
      - PGID=1000
      - UMASK=002
      - TZ=Etc/UTC
      - PLEX_LOGIN
      - PLEX_PASSWORD
    volumes:
      - /<host_folder_config>:/config

networks:
  default:
    name: network_name
    driver: bridge

Display More

chente · Jul 1st 2023

Quote from Jubri

Any suggestions?

BernH is right in what he said, you probably don't need an IP in your network for every container. But if you still need to have two or more containers using the same vlan, I think the solution goes through the following:

1. You don't need to create the same network twice. Create it with the first container and attach the second container to the same network, but don't create it again, it already exists.

2. Use smaller network ranges, instead of /24 as the network mask you can set a smaller range, between /27 and /29 might be fine.

References:

- https://www.reddit.com/r/selfh…nd_macvlan_from_the_same/

- https://www.reddit.com/r/docke…iled_to_allocate_gateway/

- https://tonylawrence.com/posts…free-your-synology-ports/

Jubri · Jul 1st 2023

Thank you BernH,

In principle, I had several containers and I liked the idea of isolating them in the network with their own mac/ip address, so I didn't have any port collisions between them. It's not critical, I could go for the bridging option in case I can't move forward.

Thank you chente,

I tried creating the network first, and then adding it to the container definition as external. But the limitation, maybe it's the GUI, is that I can't nest the configurations as I did in Portrainer.

Even if I use smaller ranges, the default gateway is always set to my network router (192.168.1.1). I understand that the default gateway should be for inside the container, but I don't understand why it collides in different containers. I need to study more

I'm going to investigate the command line, and see if I can nest (define parents) the configurations as I had in Portainer.

Thank you!

Jubri · Jul 1st 2023

Hi,

I have been able to move forward,

I created the network from the command line:

Code

docker network create -d macvlan --subnet=192.168.1.0/24 --gateway=192.168.1.1 -o parent=eno1 192.168.1.x.config

And in composer file:

Code

services:
  pihole:
    container_name: "pihole"
    ...
    networks:
      192.168.1.x.config:
        ipv4_address: 192.168.1.102
networks:
  192.168.1.x.config:
    external: true

With this config, I can run now two differents containers with differents mac/ip.

The containers have connectivity to each other and to the other computers on the network, but the problem is that they don't have connectivity with omv/docker host...

Thank you.

chente · Jul 1st 2023

Quote from Jubri

The containers have connectivity to each other and to the other computers on the network, but the problem is that they don't have connectivity to the host where I run docker...

Interesting.

This problem is the same with networks created with the KVM plugin. There it is solved with a bridge on the host, you can see it here. https://wiki.omv-extras.org/do…0_bridge_interface_on_omv

But I wouldn't know how to reproduce that in docker networks. Maybe the same way? using a br0 bridge on the host?

Jubri · Jul 1st 2023

Yes, that is!

Code

ip link add macvlan_local link eno1 type macvlan mode bridge 
ip address add 192.168.1.99/32 dev macvlan_local
ip link set macvlan_local up
ip route add 192.168.1.102/32 dev macvlan_local
ip route add 192.168.1.103/32 dev macvlan_local

192.168.1.101, IP from my omv/docker host. It remains unchanged.

192.168.1.99 a new IP for my docker host
192.168.1.102, container1 IP (pi-hole)
192.168.1.103, container2 IP (plex)

And now it works!

Every time I add a new container, I'll have to remember to add a route if I need it to talk to the host where docker is. (Or define a range of IPs to use and add a single route)

It may not be very elegant. I'll keep reading...

Thank you!

chente · Jul 1st 2023

Quote from Jubri

It may not be very elegant. We'll keep reading...
Thank you!

No. It's not very elegant. I had to read it 4 times to understand it and I still don't fully understand it, I'll keep studying it carefully

Thank you for the solution

Jubri · Jul 1st 2023

Perdona chente, por el ingles de la montaña.

chente · Jul 1st 2023

Quote from Jubri

Perdona chente, por el ingles de la montaña.

Don't worry, I'm hardly going to notice, I don't speak English, I use a translator

I managed to get one container working with its own IP, but the second container on another IP is unreachable. I can't locate the problem.

chente · Jul 1st 2023

Quote from Jubri

With this config, I can run now two differents containers with differents mac/ip.

I can't get out of here, I have no way to get it to work in the second container. Do you have any idea what could be the reason?

It's not important, I don't really need it at all... for now, until I come up with a utility, I'm just playing to follow your thread.

Jubri · Jul 1st 2023

Quote from chente

I can't get out of here, I have no way to get it to work in the second container. Do you have any idea what could be the reason?

It's not important, I don't really need it at all... for now, until I come up with a utility, I'm just playing to follow your thread.

The whole process:

1- Create macvlan network manually:

Code

docker network create -d macvlan --subnet=192.168.1.0/24 --gateway=192.168.1.1 -o parent=eno1 192.168.1.x.config

eno1, is my ethernet device name.
192.168.1.1 is my default gateway, main router IP.
192.168.1.x.config is the name I will use (you can choose any string).

2-Create both containers, with different IPs:

Code

services:
  container1:
    container_name: "container1"
    ...
    networks:
      192.168.1.x.config:
        ipv4_address: 192.168.1.102
networks:
  192.168.1.x.config:
    external: true

Code

services:
  container2:
    container_name: "container2"
    ...
    networks:
      192.168.1.x.config:
        ipv4_address: 192.168.1.103
networks:
  192.168.1.x.config:
    external: true

3- Add communications with omv/docker host

Code

ip link add macvlan_local link eno1 type macvlan mode bridge 
ip address add 192.168.1.99/32 dev macvlan_local
ip link set macvlan_local up
ip route add 192.168.1.102/32 dev macvlan_local
ip route add 192.168.1.103/32 dev macvlan_local

192.168.1.99, It's a new IP for omv/docker host

macvlan_local is a ip link name (you can choose any string)
Added two routes, to each container IP.

Check the IPs you are going to use in each container, maybe that's it.

Edit: Very important, when editing the composer file, save the configuration before continuing.

Thank you.

chente · Jul 1st 2023

I have limited myself to points 1 and 2. For the moment I have discarded point 3.

My network is different but the IPs are correctly configured in both containers. I've tried a different container and have tried different IPs. I checked on the router that they were not assigned to another device.

The strange thing is that the first container I can start on any IP, but I can't start any other on any IP even if the first one is stopped.

So I created a second container with the same content as the first. It is a handbrake container, the second one I have called handbrake2.

It turns out that handbrake2 is not able to boot with this network even though the first handbrake is stopped. It is as if the first container has hijacked this network and no other container can use it. I'm confused.

I guess you don't have any other configuration besides what you have written right?

chente · Jul 1st 2023

Sorry, I explained it wrong. The second container can boot. But it is not reachable on that IP.

Without using that network it works correctly of course.

chente · Jul 1st 2023

Incredible. That container has taken over that network.

I removed those lines from container1's compose file. I added the same lines to container2. Container2 is not accessible. A reboot does nothing either.

It is as if docker has assigned that network only to the first container that used it. No other container can use it.

Jubri · Jul 1st 2023

It is strange.

For my part, I haven't done anything else. I came from a previous configuration using Portainer, but I removed all network settings and started from scratch.

The third step, the configuration was lost on reboot.

I am going to be away for a few days. When I come back I will install OMV from scratch, and I will try these steps again.

chente · Jul 1st 2023

Quote from Jubri

It is strange.

Yeah. It's strange. If I have a while I'll try to research this, but I won't waste a lot of time, I don't really need it. I just wanted to know if your procedure works for anyone, in order to add it to the wiki somewhere, but there seems to be something wrong.

BernH · Jul 1st 2023

Quote from Jubri

Thank you BernH,

In principle, I had several containers and I liked the idea of isolating them in the network with their own mac/ip address, so I didn't have any port collisions between them. It's not critical, I could go for the bridging option in case I can't move forward.

You will not get port collisions on containers id you just map the ports. The is hand in the Ports section of a compose file. The number to the left of the : is the host port, the number to the right is the container port. You can move around the host side to whatever you need. Pihole needed an ip because of the port 53 requirement. Port 53 is reserved for DNS activity. Port 53 is already in use on the OMV host, so pihole needs an IP.

I run pihole, but I opted to run it as an lxc container instead of docker. This way it gets an IP but doesn't require docker macvlan shennanigans.

It's much simpler to just map ports than to do a bunch of macvlan's for the other containers, and for me, it seemed much simpler to do an lxc for pihole.,

chente · Jul 2nd 2023

Quote from chente

Yeah. It's strange. If I have a while I'll try to research this, but I won't waste a lot of time, I don't really need it. I just wanted to know if your procedure works for anyone, in order to add it to the wiki somewhere, but there seems to be something wrong.

okay. I figured out what the problem was, although I still don't understand why it's a problem. One of the containers I set up had the always mapped port accessible on the host changed. Custom made me change it to 5850:5800 so they don't interfere when it really isn't necessary because they are different IPs (I forgot that in the rush). So setting that port to 5800:5800 on the two containers is now both accessible.

But the question now is why can't I change that port if I use network vlan? If I don't use the network vlan I can change the port and it works normally.

Does network vlan affect port mapping?

chente · Jul 2nd 2023

Interesting. Only one of the containers with exposed ports appears in the plugin GUI. The other appears blank.

docker composer - networks

chente Jul 1st 2023

Participate now!