Sudden Web GUI issue with port 80 and 443?

1. offizieller Beitrag

    Hello,


    I ran into a real weird behaviour of my OMV Server, and would like to get your opinion and experience before I dig deeper and maybe bother the community with a github issue.


    Code
    uname -a
Linux MyNAS 5.10.0-0.bpo.8-amd64 #1 SMP Debian 5.10.46-4~bpo10+1 (2021-08-07) x86_64 GNU/Linux


    Hardware - Homebuilt NAS Tower with 4 HDD Bays (currently 3 in use), one 120GB SSD as root disk, and one extra eSATA PCI Adapter for an external eSATA Docking Station (for external backup disks).



    I startet with a Debian Stretch installation and installed OMV manually following the guide, and finally updated to Debian Buster in August 2021. (I took my time, as I did not want to be early adopter this time :) ). I did not install any additional software, besides some linux tooling like the lshw package etc. No pihole or other stuff. This server has the only purpose to run my OMV.

    I also updated OMV on the CLI and verified it on the web gui shortly after. Yes, I did a reboot.

    I was running with the default configuration for the web gui, which means port 80 and 443.


    Everything was fine, until I did a small stupid mistake. I created a folder on my root disk instead my backup disk, and synced backup data there, resulting in the root disk being 100% full. I did not realize it immediately as all the shared were still working fine, even after a reboot. I recognized something is wrong when I wanted to access the web gui and it was not accessible (Timeout). So I plugged in my screen to the NAS and saw a bunch of error messages when booting. The journalctl entries clearly pointed to the full disk. I found the full root disk and deleted the misplaced backup data, rebooted and did an apt-get update / upgrade and autoremove and cleanup, as this was my original intend.

    All error messages that I saw disappeared and everything looked fine.


    But weirdly enough, the web gui was still not accessible. I was assuming some side effect of the full disk, but now I think this is not the case.

    I used omv-firstaid to reset the web gui configuration (IPv4, Port 80 and 443, http and https enabled, IPv6 disabled), but nothing changed.

    The nginx service was up and running, and looked fine.


    When having a look at netstat -a I saw that there are no ports 80 and 443 listening for IPv4. But there were entries for IPv6 listening, like :::443 and :::80.

    A google search showed some results in which IPv6 interferred with the IPv4 configuration when using the same ports (weird). I checked with ifconfig -a and saw that my ethernet adapter had no IPv6 configured, but the localhost had still an IPv6 entry. To be sure I disabled IPv6 manually by creating /etc/sysctl.d/70-disable-ipv6.conf with the content: net.ipv6.conf.all.disable_ipv6 = 1 and activated and rebooted.

    The IPv6 entry finally disappeared for localhost, but the web gui was still not there.

    As another post mentioned a possible port conflict, I switched my https port to 9443 and http to 8080 just to give it a test.

    The web gui worked!!!

    Now I checked on this with multiple combinations of 80, 8080, 443 and 9443.


    I can verify that nginx starts correctly, but it seems to get a conflict on port 80 and 443 and does not provide the web gui as a result. I was not able to find the conflict, but obviously there ist some. As I have written I do not have any other software running, and especially no other software that provides a web interface. My setup is stable for some time now, and everything worked fine, until I did the update and filled the disk right after it.


    I did not activate the firewall on the OMV server and I am trying to access it from my regular windows Desktop which is located in the same network.




    My questions are:

    - Did anyone have the same experience?

    - Do you have any idea about the root cause? What could possibly cause such a weird conflict?

    - Am I blind? I do not see anything else using 443 or 80? So I assumed a port conflict. Any ideas what this can be?


    I came to a point where I wanted to ask for your oppinion and help in investigating on this. I do not think this is an issue related to my full disk issue anymore, but rather something that got introduced with my last package update.


    Edit: Removed typos and corrected formatting.

    I had a closer look but could not find any hint about the root cause.


    The /etc/nginx/sites-available/openmediavault-webgui contains the correct values, if I change them.


    working:

    Code
    listen 8080 default_server;
listen 9443 default_server ssl deferred;


    not working:

    Code
    listen 80 default_server;
listen 443 default_server ssl deferred;


    when checking netstat everything looks ok as well. nginx is listening on port 80 / 443 only on IPv4 (as I disabled IPv6). But if I want to connect, my connection just gets a timeout.


    I checked the logs in /var/log but could not find anything suspicious.

    Code
    /var/log/nginx/error.log does not show anything useful as well:
2021/10/25 21:03:19 [alert] 533#533: *78856 open socket #11 left in connection 4
2021/10/25 21:03:19 [alert] 533#533: *78689 open socket #7 left in connection 10
2021/10/25 21:03:19 [alert] 533#533: aborting


    Do you have any further idea where I could look for identifying the root cause? This seems to be reproducible behaviour, as there is already a second post in this forum describing the exact same issue.

    By the way ... I am currently not thinking it is a port conflict issue anymore, because:

    - I do not see any duplicate port usage in IPv4

    - IPv6 is deactivated


    I suspect there is something broken in the webserver. The nginx configuration looks great, but there must be something that prevents it from delivering anything on port 80 and 443.

    Any ideas where to look? I am not a nginx specialist.

    While writing my last entry, I thought about iptables, even though I did not activate or configure it.

    I see there is an REJECT entry for the IP of my desktop computer for the f2b-omv-webgui chain. But I do not see any reference to 80 or 443.

    Can anyone please have a look and help me interpret the iptables output? I had a horrible night and cannot even focus on walking straight :D

    Ok. I grabbed a coffee and started to feel human again. I dug deeper and here is what I found:

    This is stupid but reasonable.

    Obviously there is a block entry for my desktop IP in iptables, which includes http and https.

    So where does it come from?


    I remembered that I activated fail2ban for preventing brute force attacks.


    I found an entry in /var/log/fail2ban.log

    Code
    2021-10-19 00:07:12,635 fail2ban.actions        [516]: NOTICE  [omv-webgui] Restore Ban 192.168.1.20


    Ok. So I checked fail2ban


    Code
    fail2ban-client status omv-webgui
Status for the jail: omv-webgui
|- Filter
|  |- Currently failed: 1
|  |- Total failed:     1
|  `- File list:        /var/log/auth.log
`- Actions
|- Currently banned: 1
|- Total banned:     1
`- Banned IP list:   192.168.1.20


    There it is. Obviously my Desktop simply gets blocked on http and https.

    This is the reason why there is no error message anywhere and everything looks fine. It is because everything IS fine and behaving as intended.


    So I removed my IP from the fail2ban jail with:

    Code
    fail2ban-client set omv-webgui unbanip 192.168.1.20


    and switched the omv webgui ports back to 80 and 443 again with omv-firstaid.


    I tried accessing the webgui on 443 and.... Tadaaaa it works.

    IT is amazing ... just do it the correct way, and it works. 8o


    So how did this happen?

    Following scenario:

    - My root disk was 100% full. The shares started, but nginx and the php webgui had an issue with that and did not start properly.

    - I tried to connect to the webgui during this, but it failed. Anyhow there were log entries about my failed connection attempts in /var/log/auth.log

    - These failed connection attempts have been recognized by fail2ban, as intended, and my IP has been jailed.

    - After I emptied the disk, updated omv and everything was fine again, the fail2ban entry persisted.


    When starting my investigation I got mislead by some posts having a similar issue, but having unintended port conflicts. That was why I first started looking there.



    My issue is solved now, and I hope my description will help others in the future :D

    Nice. Would never remeber about the "fail2ban".

    Zitat von Shakesbeer

    So I removed my IP from the fail2ban jail with:

    You can make an exception on the "fail2ban" service to ignore LAN IPs to prevent this, ;)


    In my case, I have in the "IgnoreIP" field:

    Code
    127.0.0.1 192.168.1.0/24

    That way, I'm never locked out by mistake.

    Had this since the beginning and never remembered about it, :)


    This also gives me an idea about another thread, thank you

    Zitat von Soma

    Nice. Would never remeber about the "fail2ban".

    You can make an exception on the "fail2ban" service to ignore LAN IPs to prevent this, ;)

    Yeah. I did not add my computer to the exception list, as I was a bit paranoid and wanted to protect the omv server against brute force even from local systems in my LAN, in case one of these systems gets infected. But maybe this was too paranoid, and finally it caused more harm than use :)


    Thank you for pointing out the ignore option. I think I will add my regular desktop to this list now :)


    Zitat von Soma

    This also gives me an idea about another thread, thank you

    Which one do you mean?

  • Shakesbeer

    Hat das Label gelöst hinzugefügt.
    Zitat von Shakesbeer

    Which one do you mean?

    This one: rsync over ssh - RSync - openmediavault


    The OP isn't able to wireguard other services on the LAN so I'm wondering if it's the fail2ban/iptables that's blocking something (or maybe I'm way off)

    Zitat von macom

    Might be worth to enable notifications so that you get an email if an IP gets banned on the server.

    See "action" menu in fail2ban GUI in OMV.

    Thank you very much. This is a good hint.

    I will add this as well. In parallel I was thinking about some notification by a messenger. Maybe signal. I wrote an asynchronous interface for one of my servers (not reachable through the internet, only outgoing traffic allowed) by using signalcli. Maybe I will revive this.


    In parallel I am looking for a good epaper display that I can use to display basic information about my setups on a dashboard, which I can place somewhere nice, like a picture frame. Because I am a little fed up by all the notifications.


    But you are right. E-Mailing is the most basic notification and works when more sophisticated stuff fails. Will definitely add this.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!

Benutzerkonto erstellen Anmelden