Access to VLAN on openwrt from container on OMV

matkoh · Aug 23rd 2023

Hello,

I run a container "Solaranzeige" which is created with the compose-plugin. This container have to access a "Solaredge SE7k", which is placed in the VLAN IOT on an openwrt-router. This worked until end of july on which I switched the formerly on portainer created container to the compose-file. But even put the with compose created container down and start the old container on portainer don't solve the problem.

Here some information on my environment:

OMV is running on a NAS and have ip 192.168.178.57.

The container Solaranzeige has a brigded network 172.27.0.0/16.

The openwrt-router has internal ip 192.168.1.1 and gets wan-ip 192.168.178.73 from my main-router (192.168.178.1).

The VLAN has ip 192.168.103.0/24.

The Solaredge SE7k has ip 192.168.103.111.

In my main-router a static route is set to:

network = 192.168.103.0
subnetmask = 255.255.255.0
gateway = 192.168.178.73

In the container-log I see "connection timed out".

Here a tcpdump from inside the container:

Code

tcpdump -v -n -i eth0 | grep 192.168.103.111
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
    172.27.0.2.43834 > 192.168.103.111.1502: Flags [S], cksum 0xd463 (incorrect -> 0x4dd2), seq 489377900, win 64240, options [mss 1460,sackOK,TS val 2357886925 ecr 0,nop,wscale 7], length 0

A tcpdump on the wan-interface of openwrt gives no result:

Code

tcpdump -n -v -i eth1 | grep 192.168.103.111
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes

A tcpdump on OMV for the container-interface:

Code

tcpdump -n -v -i br-b7332f814f7a | grep 192.168.103.111
tcpdump: listening on br-b7332f814f7a, link-type EN10MB (Ethernet), snapshot length 262144 bytes
    172.27.0.2.33918 > 192.168.103.111.1502: Flags [S], cksum 0xd463 (incorrect -> 0xf6e1), seq 821241838, win 64240, options [mss 1460,sackOK,TS val 2358186539 ecr 0,nop,wscale 7], length 0
    172.27.0.2.33918 > 192.168.103.111.1502: Flags [S], cksum 0xd463 (incorrect -> 0xf2eb), seq 821241838, win 64240, options [mss 1460,sackOK,TS val 2358187553 ecr 0,nop,wscale 7], length 0
    172.27.0.2.33918 > 192.168.103.111.1502: Flags [S], cksum 0xd463 (incorrect -> 0xeb0b), seq 821241838, win 64240, options [mss 1460,sackOK,TS val 2358189569 ecr 0,nop,wscale 7], length 0

And a tcpdump on OMV for the main-interface gives no result:

Code

tcpdump -n -v -i enp3s0 | grep 192.168.103.111
tcpdump: listening on enp3s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes

Has anybody an idea, what the cause is? And what I can do to solve it?

Thanks in advance

Matthias

chente · Aug 23rd 2023

You could use this procedure so that the container has its own IP in the range of your IOT VLAN 192.168.103.0/24 generated by openwrt. I guess that should solve the problem. https://wiki.omv-extras.org/do…_the_same_lan_as_the_host

chente · Aug 23rd 2023

You will have to adapt the size of the network range in the container so that it does not collide with other IPs used. And probably restrict the VLAN range in DHCP on the openwrt so it doesn't match either.

matkoh · Aug 23rd 2023

chente, Thank you.

I tried, but it don't work yet. I create a new network:

Code

docker inspect network mynet-iot
[
    {
        "Name": "mynet-iot",
        "Id": "e6a5164b78d130c75a635e284c3f5d6450b27660335c7ee036e1955c4ede1075",
        "Created": "2023-08-23T13:19:30.351515653+02:00",
        "Scope": "local",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.103.0/24",
                    "IPRange": "192.168.103.50/29",
                    "Gateway": "192.168.103.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "parent": "enp3s0"
        },
        "Labels": {}
    }
]

Display More

Then I change my docker-compose file:

Code

File:
---
version: "2"
services:
  solaranzeige:
    image: takealug/solaranzeige:latest
    container_name: Solaranzeige
    volumes:
      - <path to appdata>/solaranzeige/data/:/solaranzeige
      - <path to appdata>/solaranzeige/influxdb/:/var/lib/influxdb
      - <path to appdata>/solaranzeige/grafana/:/var/lib/grafana
      - <path to appdata>/solaranzeige/pvforecast/:/pvforecast
      - <path to appdata>/solaranzeige/www/:/var/www
    networks:
      mynet-iot
        ipv4_address: 192.168.103.51 
    ports:
      - 3000:3000
      - 1883:1883
      - 8084:80
    tmpfs:
      - /tmp
      - /var/log
    restart: unless-stopped
networks:
  mynet-iot:
    external: true

Environment:
USER_ID=99
GROUP_ID=100
TZ=Europe/Berlin
UPDATE="yes"
MOSQUITTO="yes"
INFLUXDB="yes"

Display More

When I try to create the container with up, I get:

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; docker-compose --file '/srv/dev-disk-by-uuid-b1c1814b-2fa8-4238-9485-baac66654178/compose-files/solaranzeige/solaranzeige.yml' --env-file '/srv/dev-disk-by-uuid-b1c1814b-2fa8-4238-9485-baac66654178/compose-files/solaranzeige/solaranzeige.env' --env-file '/srv/dev-disk-by-uuid-b1c1814b-2fa8-4238-9485-baac66654178/compose-files/global.env' up -d 2>&1': parsing /srv/dev-disk-by-uuid-b1c1814b-2fa8-4238-9485-baac66654178/compose-files/solaranzeige/solaranzeige.yml: yaml: line 20: mapping values are not allowed in this context

OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; docker-compose --file '/srv/dev-disk-by-uuid-b1c1814b-2fa8-4238-9485-baac66654178/compose-files/solaranzeige/solaranzeige.yml' --env-file '/srv/dev-disk-by-uuid-b1c1814b-2fa8-4238-9485-baac66654178/compose-files/solaranzeige/solaranzeige.env' --env-file '/srv/dev-disk-by-uuid-b1c1814b-2fa8-4238-9485-baac66654178/compose-files/global.env' up -d 2>&1': parsing /srv/dev-disk-by-uuid-b1c1814b-2fa8-4238-9485-baac66654178/compose-files/solaranzeige/solaranzeige.yml: yaml: line 20: mapping values are not allowed in this context in /usr/share/openmediavault/engined/rpc/compose.inc:609
Stack trace:
#0 /usr/share/php/openmediavault/rpc/serviceabstract.inc(620): OMVRpcServiceCompose->{closure}('/tmp/bgstatuswc...', '/tmp/bgoutputrD...')
#1 /usr/share/openmediavault/engined/rpc/compose.inc(612): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure))
#2 [internal function]: OMVRpcServiceCompose->doCommand(Array, Array)
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#4 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('doCommand', Array, Array)
#5 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Compose', 'doCommand', Array, Array, 1)
#6 {main}

Display More

Where is my mistake?

Matthias

chente · Aug 23rd 2023

Quote from matkoh

line 20: mapping values are not allowed in this context

The error points to line 20 of the yaml. Since the plugin adds several commented lines to the beginning of each yaml, line 20 corresponds to this:

Quote from matkoh

networks:

So the error could be that the version of the yaml file is too old, I'm not sure about this, you would have to consult the docker documentation. Maybe change:

Quote from matkoh

version: "2"

For this?

Code

version: "3"

chente · Aug 23rd 2023

On the other hand when you provide an IP to your container in this way it is useless to publish ports, all ports are accessible on that IP, so you can remove these lines:

Quote from matkoh

Code

    ports:
      - 3000:3000
      - 1883:1883
      - 8084:80

Port 80 will also be port 80 in the IP you have given to the container, it will not be port 8084.

chente · Aug 23rd 2023

Introducing the container published here https://hub.docker.com/r/takealug/solaranzeige :

Code

docker run \
  -d \
  -e USER_ID="99" \
  -e GROUP_ID="100" \
  -e TIMEZONE="Europe/Berlin" \
  -e UPDATE="yes" \
  -e MOSQUITTO="yes" \
  -e INFLUXDB="yes" \
  -p 3000:3000 \
  -p 1883:1883 \
  -p 8080:80 \
  -v {SOLARANZEIGE_STORAGE}:/solaranzeige \
  -v {INFLUXDB_STORAGE}:/var/lib/influxdb \
  -v {GRAFANA_STORAGE}:/var/lib/grafana \
  -v {PVFORECAST_STORAGE}:/pvforecast \
  -v {WWW_STORAGE}:/var/www \
  --name=Solaranzeige \
  --restart unless-stopped \
  --tmpfs /tmp \
  --tmpfs /var/log \
  takealug/solaranzeige:tag

Display More

On the composerize website https://www.composerize.com/ He gave me this:

Code

version: '3.3'
services:
    solaranzeige:
        environment:
            - USER_ID=99
            - GROUP_ID=100
            - TIMEZONE=Europe/Berlin
            - UPDATE=yes
            - MOSQUITTO=yes
            - INFLUXDB=yes
        ports:
            - '3000:3000'
            - '1883:1883'
            - '8080:80'
        volumes:
            - '{SOLARANZEIGE_STORAGE}:/solaranzeige'
            - '{INFLUXDB_STORAGE}:/var/lib/influxdb'
            - '{GRAFANA_STORAGE}:/var/lib/grafana'
            - '{PVFORECAST_STORAGE}:/pvforecast'
            - '{WWW_STORAGE}:/var/www'
        container_name: Solaranzeige
        restart: unless-stopped
        tmpfs: '/tmp,/var/log'
        image: 'takealug/solaranzeige:tag'

Display More

You can use a guide to adapt your yaml to the version of docker-compose 3.3

matkoh · Aug 23rd 2023

OK, I take the compose-file from the last post and change it to my volumes and the new macvlan-network. I was able to create and start the container.

But it has no network connection (no route to host) when first try to update. route inside the container gives:

Code

route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.103.1   0.0.0.0         UG    0      0        0 eth0
192.168.103.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0

OMV has ip 192.168.178.57 and this is in another network segment.

If I could define 192.168.178.73 as gateway in the macvlan network, it maybe works. This is the openwrt-router wan-port. But I can only define a gateway in the range of the subnet and 192.168.103.1 is only reachable through 192.168.178.73.

Matthias

chente · Aug 23rd 2023

Quote from matkoh

If I could define 192.168.178.73 as gateway in the macvlan network, it maybe works

~~hmm I think that shouldn't be necessary. The container should receive the gateway from openwrt.~~ Edit: This is not correct.

In any case, why does the container need to communicate with the host? Sorry, I don't know how that container works, but I thought it was enough to access the Solaredge SE7k device on its IP.

You could define a second network interface inside the container in another IP range, but I don't know if that would be too complicated?

matkoh · Aug 23rd 2023

I just have the same idea. I now use this compose-file:

Code

version: '3.3'
services:
    solaranzeige:
        environment:
            - USER_ID=99
            - GROUP_ID=100
            - TIMEZONE=Europe/Berlin
            - UPDATE=yes
            - MOSQUITTO=yes
            - INFLUXDB=yes
        ports:
            - '3000:3000'
            - '1883:1883'
            - '8080:80'
        volumes:
            - '<path to appdata>/solaranzeige/data:/solaranzeige'
            - '<path to appdata>/solaranzeige/influxdb:/var/lib/influxdb'
            - '<path to appdata>/solaranzeige/grafana:/var/lib/grafana'
            - '<path to appdata>/solaranzeige/pvforecast:/pvforecast'
            - '<path to appdata>/solaranzeige/www:/var/www'
        container_name: Solaranzeige
        restart: unless-stopped
        tmpfs: '/tmp,/var/log'
        image: 'takealug/solaranzeige:latest'
        networks:
            default:
            mynet-iot:
                ipv4_address: 192.168.103.51
networks:
  default:
  mynet-iot:
    external: true

Display More

The container mainly do this:

1. Update from https://github.com/StefaE/PVForecast.git/ at each start of container.

2. Scheduled get data from the configured devices. In my case this is a "Solaredge SE7k".

3. Show the data in a grafana-frontend.

4. Send the data to a mqtt-broker. In my case this is another container which runs mosquitto.

route inside container now gives:

Code

route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.103.1   0.0.0.0         UG    0      0        0 eth0
172.26.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth1
192.168.103.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0

I still have the error "no route to host" when the container try to update. I guess, the problem is the gateway 192.168.103.1 for all routing. It should be only the gateway for the macvlan-network and the default-network should have gateway 172.26.0.1 as in the other container.

How can I define the correct network-settings in the compose-file?

Matthias

chente · Aug 23rd 2023

Have you generated the mynet-iot network in the plugin GUI? Allows you to set the gateway when you generate it.

In the document that I linked are the steps to follow to do it.

matkoh · Aug 23rd 2023

Yes, I create it in the plugin GUI, as described. But I can only define a gateway in the range of the subnet. I try to define 192.168.178.73 as gateway and get this error:

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; docker network create --driver macvlan --opt parent=enp3s0 --subnet=192.168.103.0/24 --gateway=192.168.178.73 --ip-range=192.168.103.50/29 mynet-iot 2>&1' with exit code '1': no matching subnet for gateway 192.168.178.73

OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; docker network create --driver macvlan --opt parent=enp3s0 --subnet=192.168.103.0/24 --gateway=192.168.178.73 --ip-range=192.168.103.50/29 mynet-iot 2>&1' with exit code '1': no matching subnet for gateway 192.168.178.73 in /usr/share/php/openmediavault/system/process.inc:242
Stack trace:
#0 /usr/share/openmediavault/engined/rpc/compose.inc(1317): OMV\System\Process->execute(Array, 1)
#1 [internal function]: OMVRpcServiceCompose->setNetwork(Array, Array)
#2 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#3 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('setNetwork', Array, Array)
#4 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Compose', 'setNetwork', Array, Array, 1)
#5 {main}

chente · Aug 23rd 2023

And what happens if you define the gateway 192.168.103.1 for that interface? Shouldn't openwrt take care of connecting that IP?

matkoh · Aug 23rd 2023

openwrt runs on another router, not inside OMV.

In the main router I have a static route, which forward all traffic for 192.168.103.0/24 to 192.168.178.73 which is the DHCP address of the openwrt router.

It seems, that OMV don't use this route for traffic from a container. In OMV network itself it is no problem to reach the 192.168.103.x addresses.

chente · Aug 23rd 2023

Yeah. And that's why it worked for you initially. The container lived in the same network range as OMV. And the static route of the first router provided the path into the network range of the VLAN created by openwrt.

What I don't understand is why that doesn't work now, that's why I proposed to give the container an IP in the network range of the VLAN to which it has to have access, that way you avoid all that routing.

What I didn't know when I proposed that is that the container must also have access to the host.

I have the scheme of your network in my head but I don't see the solution. I'm still thinking...

chente · Aug 23rd 2023

Quote from chente

And what happens if you define the gateway 192.168.103.1 for that interface?

hmm haven't you tried this?

That added to the container's default network should provide access to both networks.

chente · Aug 23rd 2023

Reviewing the previous posts I see that you already did it like this in post #4. But at that time the container did not work for you. Have you changed it afterwards?

matkoh · Aug 24th 2023

The actual configuration is:

- network mynet-iot with ip-address 192.168.103.51 and gateway 192.168.103.1

- network solaranzeige2_default with ip-address 172.26.0.2 and gateway 172.26.0.1

As i wrote on post #10 this leads to a default gateway 192.168.103.1 and the OMV-host is not reachable.

A good solution were

- network default with ip-address 172.26.0.2 and gateway 172.26.0.1, which is default and connect to the OMV-host

- additional gateway 192.168.178.73 for all ip 192.168.103.0/24

But I have no idea, how to configure this.

Quote from chente

What I don't understand is why that doesn't work now

Me too... I only can guess, that an update changed something related to docker networks.

On 07/25 I installed a update for the packages

docker-buildx-plugin docker-ce docker-ce-cli docker-compose-plugin
and that was the day, when the Solaredge SE7k was last reachable in VLAN IOT. On the same day, I switched the container "Solaranzeige" to compose-file, but I don't think, this caused the issue.

Matthias

chente · Aug 24th 2023

If the plugin doesn't let you create the network with two gateways you can do it manually:

Code

docker network create \
  --driver=bridge \
  --subnet=192.168.103.0/24 \
  --ip-range=192.168.103.0/24 \
  --gateway=192.168.103.1 \
  --opt "com.docker.network.gateway=192.168.178.73" \
  mynet-iot

matkoh · Aug 24th 2023

That command don't work:

Code

docker network create \
  --driver=bridge \
  --subnet=192.168.103.0/24 \
  --ip-range=192.168.103.0/24 \
  --gateway=192.168.103.1 \
  --opt "com.docker.network.gateway=192.168.178.73" \
  mynet-iot
Error response from daemon: cannot create network cbcdb8f941a5ab90a5938adf4f89eac46535e8546ec0a17da9ee3d1aad9b878c (br-cbcdb8f941a5): conflicts with network f82a8db84db90ab82c4f4590da034c25b6ac5c2d724d14607a29f5604cddd53a (br-f82a8db84db9): networks have overlapping IPv4

When I use driver macvlan instead of bridge, it works:

Code

docker network create   --driver=macvlan   --subnet=192.168.103.0/24   --ip-range=192.168.103.0/24   --gateway=192.168.103.1   --opt "com.docker.network.gateway=192.168.178.73"   mynet-iot
32b20d60aef930ba89c89901016020485cb3b3a4fd0c4c9ebd38eae3cb9db2ef

But the result is the same as before, default gateway for both networks is 192.168.103.1:

Code

route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.103.1   0.0.0.0         UG    0      0        0 eth0
172.26.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth1
192.168.103.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0

and the OMV-host is not reachable. The

Code

--opt "com.docker.network.gateway=192.168.178.73"

is not visible in the routing table. But it is in the created network:

Code

docker inspect network mynet-iot
[
    {
        "Name": "mynet-iot",
        "Id": "32b20d60aef930ba89c89901016020485cb3b3a4fd0c4c9ebd38eae3cb9db2ef",
        "Created": "2023-08-24T09:59:00.510917331+02:00",
        "Scope": "local",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.103.0/24",
                    "IPRange": "192.168.103.0/24",
                    "Gateway": "192.168.103.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.gateway": "192.168.178.73"
        },
        "Labels": {}
    }
]

Display More

Participate now!