I have a couple of Docker containers running on my OMV host that worked better having their own, unique IP on my LAN. I thus created a docker macvlan subnet (within my LAN's regular subnet) to allocate IPs to those docker containers.
However, as is documented on the omv wiki, there is a limitation with docker macvlan's such that they cannot communicate with the host, so whilst other devices on the LAN can connect just fine to the containers with their unique IPs, the OMV host itself cannot.
This is probably fine for a lot of use cases, but it can cause issues. For me, it meant that when I was working remotely and using the Wireguard interface running on the OMV host to VPN back home, I couldn't connect to those docker containers, because my IP traffic is coming out of the OMV host.
Thus the requirement to create a bridge interface. A guide for this has already been added to the documentation, but, because I ran into some issues, I believe there is room for a slightly more comprehensive solution, as (as far as I can tell) there's currently no masqerade rule included in the docs, thus any traffic destined for the macvlan docker containers will still have the host IP as its source.
This post is thus intended as a guide for people who may also have issues when trying to achieve this setup (like origin2000 on this post here: docker macvlan <-> omv host communication), and also might be able to be implemented into the wiki to improve the guide for those who may want this setup (though I'm not sure whether that is possible, as my current solution relies on a systemd service file and I don't know how to best integrate this into OMV's config from the gui).
TL;DR: For those who want/require use of macvlan for some of their Docker containers, and also want host <-> container communication (without creating routing conflicts for regular traffic), a more reliable method of achieving this is at the bottom of this post. HOWEVER: THIS REQUIRES CLI COMMANDS TO IMPLEMENT. I HAVE USED A SYSTEMD SERVICE FILE TO MAKE THESE COMMANDS AUTOMATIC. THOSE MORE FAMILIAR WITH OMV'S GUI MAY KNOW HOW TO BETTER IMPLEMENT THIS FROM WITHIN OMV'S WEB-UI. SKIP TO POST#2 FOR THE SERVICE FILE.
Through trial-and-error, I have found that the key to making a macvlan setup work reliably is to use a combination of IP based Policy Rules and Routing Tables to capture and route traffic destined for the Docker Macvlan Subnet, and for matching traffic that is routed that direction to be masqueraded as the IP of the bridge interface. This ensures that all the docker containers in the macvlan subnet can communicate with each other, all devices on the LAN can communicate with the containers, and, importantly, the OMV host itself can too (this is useful for remote wireguard clients, or also perhaps if you're using Docker to run an AdGuard / PiHole DNS server that you'd like the OMV host to also use [side note, not sure how advisable that is... Not without a backup dns server at least]).
Below is first an overall summary / "visualisation" of the end objective, and the visualised example assumes you would have already used the wiki instructions to set-up a macvlan with a specific "IPRange": subnet for docker containers from within OMV's Compose plugin (in the below example, that macvlan had already been created with an "IPRange": "192.168.42.248/29" subnet, and named docknet).
The commands in the below "visualisation" can be used to give you an overview of your own setup and display the end results, not to actually "do" anything (for those, skip to the end for the service file). You can run the commands in the visualisation pre- and post-setup to get an idea of what we are doing and what has changed:
OMV HOST (with its regular IP address of say, 192.168.1.5, which is within the main LAN subnet of 192.168.1.0/24 served by the router sitting at 192.168.1.1 [these will likely all be different in your own setup])
┌──────────────────────────────────────────────┐
INTERFACES & IP ADDRESSES
eth0: 192.168.1.5/24 ←Main physical interface (your interface name may be different, like en0 etc, you would have to check in Network Interfaces to confirm what this interface is called and make sure that is what you use in any commands)
|
docknet-bridge (parent: eth0): 192.168.1.254/32 ← Macvlan bridge interface. Can be named anything you like, mine's docknet-bridge. This exists in the docker macvlan subnet (conveniently called docknet!) previously created in OMV's Compose plugin. In this example, the docker macvlan "IPRange" subnet is 192.168.1.248/29, which is a pool encompassing 7 usable IPs ranging from 248 – 254. I then have it set in my router's DHCP server to only dynamically allocate IP addresses up to 192.168.1.247. This 'docknet-bridge' interface is set to the same IP as the Auxiliary Host Address set in the Docker macvlan's settings by the Compose plugin, so that Docker doesn't give this IP out to a container.
wgnet1: 10.192.1.254/24 ← WireGuard VPN server #1 (automatically created by the omv wireguard plugin)
wgnet2: 10.192.2.254/24 ← WireGuard VPN server #2 (also created by the omv wireguard plugin)
└──────────────────────────────────────────────┘
┌──────────────────────────────────────────────┐
MAIN ROUTING TABLE (ip route show)
default via 192.168.1.1 dev eth0 (this will be the route to your gateway address, so likely your router's IP)
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.5 (your standard route to your LAN subnet via the eth0's IP address [src])
10.192.1.0/24 dev wgnet1 proto kernel scope link src 10.192.1.254 (Route to the Wireguard subnet added by the Wiregaurd plugin. 10.192.1.254 is the wg "server" address of your OMV host communicating with wg devices (most likely clients) in the wgnet1 subnet)
10.192.2.0/24 dev wgnet2 proto kernel scope link src 10.192.2.254 (As above, for wgnet2)
NOTE: There's no route to Docker's macvlan subnet 192.168.1.248/29 ('docknet') in the main table. I've found that it's important not to have the docker macvlan route in the main routing table as it sometimes caused my setup conflicts, with OMV host traffic meant for my LAN going out via the bridge interface instead of the main eth0 interface.
└──────────────────────────────────────────────┘
┌──────────────────────────────────────────────┐
ROUTING POLICY RULES (ip rule show)
Priority 0: from all lookup local (standard Linux policy rule)
Priority 100: from all to 192.168.1.248/29 lookup 99 ◄─ KEY Anything destined for IPs in the docker macvlan subnet is caught by this policy rule, and is told to lookup Table 99 (numbers don't matter too much, as long as the Priority Number is higher than 0 and less than 32766)
Priority 32766: from all lookup main (standard Linux policy rule)
Priority 32767: from all lookup default (standard Linux policy rule)
└──────────────────────────────────────────────┘
┌──────────────────────────────────────────────┐
ROUTING TABLE: 99 (ip route show table 99)
192.168.1.248/29 dev docknet-bridge scope link src 192.168.1.254 (IP traffic destined for the docker mavlan subnet will be caught by the policy rule, told to lookup Table 99, and then given this route via the the bridge interface that we will have created, which has a src address of 1.254
└──────────────────────────────────────────────┘
┌──────────────────────────────────────────────┐
IPTABLES NAT (iptables -t nat -L POSTROUTING -v) (there will likely be a few others printed if you are using docker and/or wireguard, but this one below is the important one)
target prot opt in out source destination
MASQUERADE all -- any docknet-bridge anywhere 192.168.1.248/29 (This means: any traffic from any source that is destined for the .248/29 subnet (i.e. the docker macvlan subnet), gets its address masqueraded to the src IP address of the docknet-bridge interface (i.e. .254))
└──────────────────────────────────────────────┘
END RESULT:
┌──────────────────────────────────────────────┐
DOCKER CONTAINERS (Inside Docknet Macvlan)
Parent Interface: eth0
│
Network: 192.168.1.248/29 (192.168.1.248 - 192.168.1.255)
Auxiliary Address (Host): 192.168.1.254 (same IP as docknet-bridge, not mandatory to be the same, but logical to be, as you can be sure Docker won't allocate this IP)
│
Container IPs:
• 192.168.1.249 ┐
• 192.168.1.250 │
• 192.168.1.251 ├<-Available for containers
• 192.168.1.252 │
• 192.168.1.253 ┘
• 192.168.1.254 ←Reserved host auxiliary address, and src address for the bridge interface
└──────────────────────────────────────────────┘
MACVLAN ISOLATION WORKAROUND
- The Problem: Parent (eth0) cannot talk to child (Docker containers)
- Solution: Macvlan bridge interface (docknet-bridge) as sibling macvlan to docknet
POLICY ROUTING (DESTINATION-BASED)
- Rule priority 100: "to 192.168.1.248/29 lookup 99"
- Table 99 only has route applicable to Docker's macvlan subnet
- All other traffic uses main table (normal routing)
MASQUERADE (SOURCE NAT)
- All traffic TO Docker containers gets NAT-ted to 192.168.1.254
- Containers see the OMV host as 192.168.1.254 (they never see real source)
- Return traffic automatically un-NAT-ted
/32 ASSIGNMENT ON DOCKNET-BRIDGE
- Prevents the kernel auto-creating a route for the bridge interface in the main table
- Forces explicit control via policy routing
- No metric conflict with eth0's /24 route (this was something that caused me issues before I went with this setup)
TARGETED ROUTING
- Destination-based policy routing (priority 100) means only Docker traffic (192.168.1.248/29) uses special routing
- All traffic types (WireGuard, Host) to Docker follow the same path through table 99
- MASQUERADE makes the host appear as 192.168.1.254 to containers
- Normal OMV host traffic (NFS, LAN, Internet, Remote WG Clients, etc.) bypasses the special routing, and use normal routing via eth0
- Correct source addresses preserved for non-Docker traffic (again, this was something I had an issue with before this setup)
