NPM proxied hosts not accessible until I access the NPM management interface

Does this happen from outside your LAN or inside?

If it only happens from the inside, I'd suspect you are not running a local DNS server like pihole or dnsmasq that points access to those domain names to NPM and therefore you desktop machine is looking to the internet for DNS name resolution, but "hairpinning" is not supported by your ISP's equipment, which is typical. Hairpinning is basically reflecting the traffic back to the origin, so a DNS resolution pointing to your public ip allows traffic to "bounce" off of their equipment and back to you.

Depending on the equipment you have, you can sometimes set up local DNS in your router, but if not pihole or dnsmasq can do it easily by setting the DNS server in your router to point to them instead of your ISP's dns server, and then you configure them to resolve your local addresses but pass anything they don't have an entry for on to your ISP's servers and/or google, and/or clouflare, and/or ....

The alternatives to pihole or dnsmasq are a full fledged dns server like bind9 or unbound, or editing your hosts file. Host file edits will work on a computer but not a phone, and the full fledged dns servers are a lot more complicated to set up, so I would recommend pihole or dnsmasq. Pihole can also block some malicious ad sites and if required act as a dhcp server, enabling the dhcp server does require disabling the one in your router as there can only be one dhcp server on a network.

The idea still stands.

When you enter a URL in a browser, your computer or devices queries a dns server to find out what the actual IP address is. With no dns servicing your LAN, the browser relies on the dns cache in your computer or router, but a cache is not permanent. Once a URL to ip resolution calls out of the cache, there is nothing left to point to the ip address so another lookup is performed. If the only dns server found is out on the internet then that is where the query is sent and since the internet dns servers can only resolve as far as you public ip but know nothing about your lan, there is nothing available to fill that request. You should always be able to access them on your lan using the ip and port, but some services will not work right, such as nextcloud or vaultwarden since nextcloud uses the domain for authorized acces and share link creation and vault warden required ssl certificates.

If you don’t want to run Pihole or dnsmasq and direct your devices to use it as the dns server, the only other option is a hosts file edit on things that allow it.

Quote from VaryingLegwarmer

In duckdns I am configuring the url to resolve to my local IP, thought that was meant to account for that DNS routing to my local ip no? In the guides I followed they never mention this misbehavior.. besides that, why then after like 30 seconds of hitting the url it eventually brings back the service I meant to access originally? That's why I fear a container or service is "going to sleep". Routing works eventually but service is inaccessible for a while.

Duckdns is out on the internet and can't resolve LAN ip addresses. It knows nothing about your LAN, doesen't know where it is and can't see it because it is a LAN address and not an internet address, with your LAN segregated by your router. There are likely many people around the world that have the same LAN ip ranges used that you have so even if duckdns could see into a LAN, how would it know to send the traffic to you instead of the guy down the road? and even if it did, as I mentioned, hairpinning is not allowed by a lot of the ISP's equipment.

I can pretty well guarantee you that you have a DNS issue, but if you want to confirm for yourself, the next time you can't access a container using the domain, try using the ip address of your server and the port number of the container. You will be able to access it.

Quote from VaryingLegwarmer

I can access the resources using the local IP while this happens, like I mentioned I keep connected via ssh even when the urls were not resolving in the browser. Of course duckdns doesn't know anything about my LAN but when I hit the duckdns url (that is pointing to my local ip in duckdns) it doesn't need to know it, it just needs to resolve it to my local IP. If I'm inside my network of course it resolves to a local resource (my omv machine with NPM). If this wasn't the case my setup would not work at all but it does work for most of the time. This is a pretty standard setup for home servers in order to have a domain name and https for local resources so I thought others might know what was causing my issue since I don't think most are experiencing this. Are you familiar with this setup? Your explanation doesn't account for the fact that the resource is inaccessible only for a little while, after hitting the url several times eventually it does resolve back to the resource again.

My explaination does account for that. If you look above I mentioned dns caching. Once an address is resolved, the entry is stored in a cache, but the cache is not permanent. over time cached entries will get dropped and until a new attempt is successful in resolving the ip it will not work.

This setup is not standard for a home setup as you mention because you are asking one network (the internet) to resolve addresses on another (your LAN), which it has no direct access to. Will it it work sometimes? Perhaps. Is it reliable, no. There are many write-ups on the internet about runing local DNS for this exact reason. If your setup was the standard way of doing it, there would be no need for those write-ups or things like dnsmasq or any other dns server to be set up as a local dns server.

The reliable way on a LAN is to use some kind of local DNS server (or intercetper like pihole and dnsmasq) or host file edits. Host file edits are fine for devices that allow it and if they will always stay on the LAN side of your router, but mobile devices (ie a smart phone) often don't allow host file edits (there is a dns changer app for android) and if they did, the edit would stop working once they are not on your LAN. Likewise your semi-working setup will not allow a smart phone access from the internet because the duckdns resolution is pointing to a LAN address.

There is a guide in the guides section of the forum to setting up pihole in docker on OMV. It is not overly complicated and it is not overkill, but it will allow for local dns definitions that can be reliably resolved on your LAN, and unknown ones forwarded to the WAN. If you don't want full pihole, you can use the guide as a base to change out to a dnsmaq container but it may require a bit trial and error to get it going since it isn't a documented guide.

Quote from VaryingLegwarmer

So in my setup, if I use dnsmasq then I wouldn't need NPM? With NPM I also get automatic letsencrypt renewal btw but besides this I only use NPM for descriptive url for local resources. All external access I do is through wireguard right now because I haven't set aside the time to harden my local network enough to feel comfortable exposing the services over the internet (and want to avoid CF).

No, you are not understanding the role everything plays.

NPM is a reverse proxy (I run it myself and I am the one that wrote the guide that is posted for it). Basically a reverse proxy is a specialized web server It allows for one point of access to all the services you run, without having to open multiple ports in a firewall. It does this routing to them based on the domain used for the service to the ip address and port, and allows for encryption via ssl certificates.

A dns server like duckdns, google, your isp. etc. does domain to internet ip (public) address lookup, but has nothing to do with encryption or hosting. It is simply a reference table to equate a domain with an ip address, but they do automatically pass configurations around the internet to all the other dns servers and try to verify the entries. Because of this sharing of information, all it takes is for one person to have done the same thing that you have tried using the same LAN ip listed on any DNS service to cause problems. When the information is traded around and verified, 2 entries that use the same LAN IP address but with different domains associated with conflict with each other and become invalid because an ip address on a network has to be unique. This is the point of the 'duckdns knows nothing about your LAN' comment. it can't see it or verify anything on it and as such dns server exchanges will cause breakage of the configuration. When a resolution is established and cached on your somputer it will work for a while, but the next time it needs to resolve again it may not be right because the last time the dns servers exchanged data something became invalid.

DNSmask or pihole act like a DNS server for your LAN, for domain to LAN ip (private/LAN side of your router) address lookup, but unlike a full-blown dns server, they are better termed as dns re-directors or interceptors since they are configured with your domain to LAN ip entries, but a DNS request for anything that they don't have defined is passed on to a full-blown dns server (or several) out on the internet. They look after your LAN and google/cloudflare/your ISP/etc., looks after the internet (WAN)

If you want to use domain names on your lan and have encryption, you still need the reverse proxy (NPM).

The basic concept is this (and this is outlined in the guides and the documentation for pihole) You set your router's dhcp server to issue the ip of pihole or dnsmasq as the dns server to use. You make the entries in there to point to your local ip address of your server for your domain names and set an upstream dns server (or severs). devices that join your network should then be told to use dnsmasq or pihole as the dns server. When a device asks for domain resolution they either answer with an entry they have or pass the request to the upstream server if they don't have the answer.

If you can't set your router to give the address of pihole or dnsmask as the dns server you can make a manual dns server entry on your devices network configuration.

I would recommend pihole over dnsmasq though. Pihole actually uses dnsmask as the redirector, but it brings ease of setup and configuration to the talble as well as the ability to block malicious website access for every device on your LAN that is using it for dns resolution. DNSmasq by itself will not do that.

The duplicate ip address entries by someone else somewhere in the world and the subsequent sharing of the DNS info between servers is the root of you loosing the resolutions. local DNS will stop that from happening. It may even stop your ISP problems because private LAN ip addresses are not supposed to exist on the internet.

Quote from VaryingLegwarmer

Where is the guide, is it this one? [HOWTO] install PIHOLE on OMV using portainer WEBGUI on a MacVlan

That is the one. If you use the compose plugin instead of portainer, it is still valid, you just do the settings in the compose plugin instead.

I don't run pihole in a docker container, mine is running in an lxc which I have placed a guide on the forum for, so I can't give you the specifics of the docker setup. The author of that docker guide perhaps can help you in better detail than I can if you run into problems.

Doing that will not survive a container update unless you copy that file out to a directory to be used for persistent data and then map it back in.

Persistent data files and folders that are mapped back into a container are “overlayed” on top of the containers files and used as the active files.

Quote from VaryingLegwarmer

Not sure I understand this. The file is being written into the container's tmp folder. So far there hasn't been an issue after I restart the container (which wipes the tmp folder) so why would I need this file to survive container updates?

If the tmp directory is outside the container and is mapped back into it ie. if your compose files has a Volume mapping that encompasses the tmp directory), you are good. Those files essentially replace the ones that the container build creates. This is the whole idea of the persistent data (ie. appdata) directories for docker.

Editing files that are inside a container but are not mapped back in from an external directory, they get deleted whenever a container is recreated.

Once again, I do not run pihole as a docker, so I am not familiar with what files get put in the appdata directory.

If that edit is for NPM, then I believe the directory is already in the persistent data directory if I recall correctly.

For local dns resolutions, you are still better off using Pihole or dnsmasq as your local dns interceptor. All requests will go to them automatically, and you will not have to deal with the configurations of NPM, which can get complicated for some services, requiring additional entries in the advanced configuration and/or custom locations sections in order to let the service’s web pages pass through correctly.

Pihole or dnsmasq will also let you address any machine on the LAN by name for accessing things like a samba share by simply equating the machines lan ip with <machine_name>.local

I personally use both setups (I run a Nextcloud server which requires access by its web accessible address for share link generation). Pihole maps my internet accessible domains to my server where NPM is running, in order too keep the request from reaching the internet where it would get lost because a “hairpin” connection is almost never allowed to occur. NPM catches the activity and gives access to the web page services with the same functionality as from the internet, but Pihole also gives me the friendly name access for things that are not web page accessed like samba.

However if you are using the let’s encrypt ssl certificates from NPM, you have to open ports 80 and 443 to it from your router and have an internet verifiable domain so now your web page services are internet accessible even if you don’t want them to be as you mentioned (unless you are only opening those ports to trigger a certificate refresh when they are about to expire), but there is no need for ssl on a lan unless you are afraid your house mates might intercept and spy on your access to your server.

I don't think you are quite understanding the way docker works.

You said doing an edit in the container fixed your problem and it persisted after a container down/removal/up. The removal will delete the container and the up will re-create the container in it's default state. A container upgrade will also re-create the container in it's default state. The default state will not have that edit you made. Persistent files can also survive an OS re-install that will save you a lot of configuration time if you have to start fresh. That is the point I am making.

If you want an edit inside a container to persist after those events, your yaml file will need to map a persistent copy of that file or it's directory back into the container.

Now you are saying you don't care if the edited file survives a container re-creation, so you don't care if your fix is permanent? Kind of confusing there, but whatever floats your boat, I'm not gonna fight about it.

For the purposes of clarification for anyone else reading, mapping the permanent edited file back in is as simple as copying that file out of the container into a directory with a docker cp container_name:/path/to/file/in/container/filename /path/to/permanent/file/filename and then putting a volume mapping back in the compose file similar to this so the permanent file gets layerd on top of the container files and is used as to active version:

volumes:
- /path/to/permanent/file/nginx.conf:/etc/nginx/nginx.conf