Remote mounts, reboots, and Docker...

I'm using OpenMediaVault to host my Plex Docker container, and all my media files are on SMB shares on another host. I've had success using the Remote Mount feature and manually mounting those CIFS shares, then mapping the mount points into the Plex Docker compose file, and then bringing up the Plex container. If I reboot OpenMediaVault, Remote Mount does not automatically mount those shares, so I need to bring down Plex, mount the shares, then bring Plex up manually, which is not ideal.

Is the lack of automounting of Remount Mount storage a limitation or a bug? If I could get that storage to auto mount, then maybe I could look into leveraging Docker swarm features to implement a container healthcheck that could use "mountpoint -q" to test whether that storage is mounted and then auto-restart the Plex instance based on it's state.

Alternatively, I have found a lot of mentions about using Docker to create CIFS volumes as part of the container so they would mount/unmount inside of and part of the container (which makes a lot of sense):

However no matter what I did when I tried to start my container in compose, I would always get, "Container plex Starting Error response from daemon: error while mounting volume [...] : no route to host". There are references (including in the docs above) about Docker needing to have the IP address at least passed in via addr or just have the CIFS share referenced by IP address, and I tried that however I would still get that "no route to host" error message. I think that leaves two other possibilities. One possibility is that there is some issue with the way Docker is interacting with the Debian firewall that is causing this issue (however there is no issue for the Plex server if I instead do the CIFS mount outside of Docker, map that into the container, and then bring up the Plex container, Plex operates normally on my network). The other possibility is that this is a security issue, and that Docker is blocking the ability of mounting a remote filesystem in the container. In years past there were mentions of working around those limitations by loosening Docker security with flags like "--cap-add SYS_ADMIN --cap-add DAC_READ_SEARCH", and later people mentioning they also needed "--security-opt apparmor:unconfined", and then later still people saying the container just needs to be run "--privileged". I'm not keen on that if there is a better way, but maybe that's necessary now?

Quote from ryecoaaron

It is an issue with your system. The remotemount plugin creates a systemd mount file which should mount the remote share once networking is online. If your filesystem is not being mounted, I would look at the systemctl or journalctl logs for that mount file.

The plugin used to create an automount file but then the filesystem shows up as autofs which is explicitly not supported by OMV (hard to tell what kind of filesystem it really is).

I'm glad it doesn't use autofs, that would not work well with Docker I would think.

Looking through journalctl, I see messages like this:

omv mount[777]: mount error: could not resolve address for asgard.home.local: Unknown error

and this:

omv systemd[1]: srv-remotemount-music.mount: Failed with result 'exit-code'.

Looking in systemctl, the RemoteMount status before I manually mount the CIFS shares post-boot is loaded, ACTIVE=failed, SUB=failed. If I manually mount them, they change to active, active.

Is there some other place I can check with more information than this?

Quote from ryecoaaron

I think you have everything you need. The fact that your OMV system can't resolve the hostname at mount time is why it can't mount the remote mount. You could switch the remote mount to use the remote system's IP address to see if that fixes the issue.

Like I mentioned, I tried using just IP addresses in the Docker Compose file, but that had no impact, the error message was still "no route to host", which is why I was asking if this was a Debian firewall issue or a Docker container privilege issue (and note that I do not have any issues getting to the SMB host by name or IP addresses outside of Docker Compose).

Quote from ryecoaaron

I think you have everything you need. The fact that your OMV system can't resolve the hostname at mount time is why it can't mount the remote mount. You could switch the remote mount to use the remote system's IP address to see if that fixes the issue.

I just changed Remote Mount to use IP addresses instead of hostnames, and I see a new error on boot:

Oct 19 10:09:11 omv kernel: CIFS: Attempting to mount \\10.0.0.20\video

Oct 19 10:09:11 omv kernel: CIFS: VFS: Error connecting to socket. Aborting operation.

Oct 19 10:09:11 omv kernel: CIFS: VFS: cifs_mount failed w/return code = -101

Oct 19 10:09:11 omv kernel: CIFS: Attempting to mount \\10.0.0.20\music

Oct 19 10:09:11 omv kernel: CIFS: VFS: Error connecting to socket. Aborting operation.

Oct 19 10:09:11 omv kernel: CIFS: VFS: cifs_mount failed w/return code = -101

Looking further down the logs, that's happening a few seconds before dhclient performs DHCPDISCOVER on my ethernet interface:

Oct 19 10:09:14 omv dhclient[582]: DHCPDISCOVER on enp3s0 to 255.255.255.255 port 67 interval 12

Oct 19 10:09:14 omv sh[582]: DHCPDISCOVER on enp3s0 to 255.255.255.255 port 67 interval 12

Oct 19 10:09:14 omv sh[582]: DHCPOFFER of 10.0.0.10 from 10.0.0.1

Oct 19 10:09:14 omv dhclient[582]: DHCPOFFER of 10.0.0.10 from 10.0.0.1

Oct 19 10:09:14 omv sh[582]: DHCPREQUEST for 10.0.0.10 on enp3s0 to 255.255.255.255 port 67

Oct 19 10:09:14 omv dhclient[582]: DHCPREQUEST for 10.0.0.10 on enp3s0 to 255.255.255.255 port 67

Oct 19 10:09:14 omv dhclient[582]: DHCPACK of 10.0.0.10 from 10.0.0.1

Oct 19 10:09:14 omv sh[582]: DHCPACK of 10.0.0.10 from 10.0.0.1

Oct 19 10:09:14 omv avahi-daemon[632]: Joining mDNS multicast group on interface enp3s0.IPv4 with address 10.0.0.10.

Oct 19 10:09:14 omv avahi-daemon[632]: New relevant interface enp3s0.IPv4 for mDNS.

Oct 19 10:09:14 omv avahi-daemon[632]: Registering new address record for 10.0.0.10 on enp3s0.IPv4.

Oct 19 10:09:14 omv dhclient[582]: bound to 10.0.0.10 -- renewal in 1406 seconds.

Oct 19 10:09:14 omv sh[582]: bound to 10.0.0.10 -- renewal in 1406 seconds.

Oct 19 10:09:14 omv sh[1280]: enp3s0=enp3s0

So OMV is trying to mount the Remote Mount CIFS shares before networking is up, which is why it is failing at boot.

Quote from ryecoaaron

systemd is being told to wait for networking to be online. Not sure what else I can tell systemd to do. I would try changing your OMV system to a static ip if dhcp is that slow.

I could do that, but that sounds like we'd be working around a bug in systemd. I'd rather look into what's going on with systemd and file a bug report with them if that's where the issue lies.

And regarding the issues with doing the CIFS mount from the Docker container, I guess I'll follow up with the Docker folks on that as well?

Quote from ryecoaaron

I don't think it is a bug. It is figuring out the ordering of dependencies on your system. I would consider a router taking that long to assign a dhcp address more of a bug.

Personally, I would not mount a cifs share from inside the container. I would mount it on the host and then pass it through as a volume.

My router is pfSense running on an Intel(R) Xeon(R) E-2134 CPU @ 3.50GHz (8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads), running off of an SSD with 16GB of RAM. I've never seen any performance issues with it, DHCP server or otherwise (CPU usage averages 3%, RAM 3%).

If there's a problem with the dependencies, then it's part of the OpenMediaVault ISO, since that's what I've installed from, and I haven't tweaked anything other than adding OMV-Extras, then configuring a couple of remote mounts and a container.

How do you recommend I isolate whether this is a systemd, pfSense, or OMV ISO out-of-the-box dependency issue, so I can file the bug report with the right entity?

I fail to see what the advantage of mounting a CIFS share in the host has - it adds an additional layer that complicates the setup. If the share is mounted as part of the container, then when the share is inaccessible, the container doesn't start, which is what I want since the Plex container is useless without the share. Having Plex run when it can't get to the share results in Plex presenting content that it has no access to, and in a worst-case scenario could result in Plex pruning the indexed media from its database, which would require re-indexing once it is available. This is not wanted. Mounting the CIFS share at the host level means more work figuring out ways I can make the share an operational dependency to the container. If the container doesn't start when the share isn't available, then I'm 50% to that solution already.

Quote from ryecoaaron

If you are using pfsense, then that is a dhcp server I have run for years. I have no idea why your system takes a long time to get an address. There are lot of pieces here. I don't know the right place to "fix" this. Many people would probably say I need to write a better mount file for remotemounts. Since I don't have this issue on my network, I don't have a good way to determine what that is. I am open to suggestions but I don't need a "bug" report for it. I also don't have time (or patience) to try and get enough info from a system I don't own by forum posts or github issues. This plugin was originally written with the goal of just doing backups to a remote mount that was always available on an OMV system that wasn't shutdown often. People use it for just about anything now and it is almost impossible for me to come up with dependencies that make it work generically.

Understood. My OMV system boots in about 3 seconds (it runs off fast NVMe storage in a modern computer with 16GB of RAM), and I don't think there is any issue with pfSense and it's DHCP server, the problem is probably with Debian/systemd since a static IP doesn't demonstrate any issues with CIFS remote mounts. What I'll do is find another distro and replicate my setup there and when I reproduce the bug, I'll hopefully find someone else in another open source project who's receptive to getting to the bottom of it.