Compose Plugin Settings is hitting Fail2ban on a SWAG reverse-proxy

1st Official Post

    ryecoaaron && votdev

    This is making me nuts.

    I use a SWAG reverse-proxy conf to access OMV via WAN.

    All works great but on the last days, I've been getting banned on the WAN due to my SWAG internal IP beeing banned on the fail2ban from OMV.


    After some test and trial, I figured out what is causing it but not how to sort it.


    Whenever I go to the Services-> Compose-> Settings page, the fail2ban increases a count on the nginx-404 JAIL coming from the 172.21.0.5 (docker IP from SWAG).

    Since I have a really low count value (3 on a 10 minute time), after a while, my Page goes 502.

    Haven't yet, updated openmediavault-compose to 6.9.4and don't know if the same will keep happening.

    Versions:


        



    If I go to any other page from the compose plugin, this doesn't happen.


    Does this say something to you both on what is happening when going to the Settings page?


    Thank for any input.

    • Official Post
    Quote from Soma

    Does this say something to you both on what is happening when going to the Settings page?

    Have you tried ctrl-shift-R? Do you see anything in the browser's F12 debugger? Do you see anything in /var/log/nginx/error.log or /var/log/nginx/openmediavault-webgui_error.log when accessing the page?

    omv 8.2.2-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.2.1 | compose 8.1.6 | cterm 8.0 | borgbackup 8.1.7 | tempmon 8.0.3 | mergerfs 8.0.1 | scripts 8.0.1 | writecache 8.1.7


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Quote from ryecoaaron

    Have you tried ctrl-shift-R?

    Yes, all the time, ;)

    Quote from ryecoaaron

    Do you see anything in /var/log/nginx/error.log or /var/log/nginx/openmediavault-webgui_error.log when accessing the page?

    Only this from access.log:


    Quote from ryecoaaron

    Do you see anything in the browser's F12 debugger?

    Well, can you explain me better what to look for?

    • Official Post
    Quote from Soma

    can you explain me better what to look for?

    Any error in the console part. It is usually pretty obvious if there is an error. If not, I really don't know what would being giving fail2ban issues. The Settings tab isn't doing anything out of the ordinary. I guess I will have to try setting up a system with fail2ban on it.

    omv 8.2.2-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.2.1 | compose 8.1.6 | cterm 8.0 | borgbackup 8.1.7 | tempmon 8.0.3 | mergerfs 8.0.1 | scripts 8.0.1 | writecache 8.1.7


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    • Official Post

    If you are using the default nginx-404 jail, it is only looking at /var/log/nginx*/*access*.log. What is the output of sudo grep 404 /var/log/nginx*/*access*.log

    omv 8.2.2-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.2.1 | compose 8.1.6 | cterm 8.0 | borgbackup 8.1.7 | tempmon 8.0.3 | mergerfs 8.0.1 | scripts 8.0.1 | writecache 8.1.7


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Quote from ryecoaaron

    If you are using the default nginx-404 jail, it is only looking at /var/log/nginx*/*access*.log.

    Yes, the default JAIL that is done from installation. Just activated it and changed some settings.

    Quote from ryecoaaron

    I would try deleting the older logs in that folder.

    The logs just rotated yesterday.


    I can (and will) bypass this by setting an exclusion on that IP.

    Was just trying to figure out why this is happening in the last days/weeks.


    I thought maybe something changed on the workbench that made it hitting several access in a short period of time.


    Quote from ryecoaaron

    Any error in the console part. It is usually pretty obvious if there is an error. If not, I really don't know what would being giving fail2ban issues. The Settings tab isn't doing anything out of the ordinary. I guess I will have to try setting up a system with fail2ban on it.

    Don't really see anything out of the ordinary. Although I'm not that experienced on it.


    Let me try to update the plugin and see if it still does the same.

    If all comes to it, I'll just make the exclusion, to prevent beeing locked out.


    Thank you for the inputs, ;)

    • Official Post
    Quote from Soma

    Let me try to update the plugin and see if it still does the same.

    I don't think an update is going to change this. There would have to be a bug causing a 404 in the plugin. I don't think I have every seen that in this plugin. Did you see anything from sudo grep 404 /var/log/nginx*/*access*.log?

    omv 8.2.2-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.2.1 | compose 8.1.6 | cterm 8.0 | borgbackup 8.1.7 | tempmon 8.0.3 | mergerfs 8.0.1 | scripts 8.0.1 | writecache 8.1.7


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Quote from ryecoaaron

    There would have to be a bug causing a 404 in the plugin.

    The error that shows is a "502 bad Gateway" on the reverse-proxy. It's understandable since the internal IP is then, banned on the JAIL due to too many hits (as seen on the *.access.log every ~2seconds)


    Is it the same as a 404?


    What I'm been trying to change is some settings on the proxy.conf I created to see if it prevents that by adding some proxy headers:



    Quote from ryecoaaron

    Did you see anything from sudo grep 404 /var/log/nginx*/*access*.log?


    Yes:

    Code
    pi@panela:~ $ sudo grep 404 /var/log/nginx*/*access*.log
/var/log/nginx/openmediavault-webgui_access.log:172.21.0.5 - - [10/Jul/2023:09:55:39 +0100] "POST /rpc.php HTTP/1.1" 200 404 "https://<MYDOMAIN>.duckdns.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.67"
/var/log/nginx/openmediavault-webgui_access.log:172.21.0.5 - - [10/Jul/2023:10:53:46 +0100] "POST /rpc.php HTTP/1.1" 200 404 "https://<MYDOMAIN>.duckdns.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.67"
/var/log/nginx/openmediavault-webgui_access.log:172.21.0.5 - - [10/Jul/2023:11:51:22 +0100] "POST /rpc.php HTTP/1.1" 200 404 "https://<MYDOMAIN>.duckdns.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.67"
/var/log/nginx/openmediavault-webgui_access.log:172.21.0.5 - - [10/Jul/2023:11:57:30 +0100] "POST /rpc.php HTTP/1.1" 200 404 "https://<MYDOMAIN>.duckdns.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.67"
/var/log/nginx/openmediavault-webgui_access.log:172.21.0.5 - - [10/Jul/2023:12:04:39 +0100] "POST /rpc.php HTTP/1.1" 200 404 "https://<MYDOMAIN>.duckdns.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.67"
/var/log/nginx/openmediavault-webgui_access.log:172.21.0.5 - - [10/Jul/2023:12:04:47 +0100] "POST /rpc.php HTTP/1.1" 200 404 "https://<MYDOMAIN>.duckdns.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.67"


    For now, I added the IP to the "Ignore IP" setting of fail2ban.

    Next week I'll be home and will make some more "wacky" tests to see if I sort this although it's just a minor issue.

    When I log in via LAN IP (while WIREGUARDED) it doesn't happen but I have my LAN on the "Ignore IP list so, maybe if I remove it (the ignore list) it might also happen.


    Will keep this updated.

    • Official Post

    This definitely seems like something caused by swag. Not sure why it only happens with the compose plugin though.

    omv 8.2.2-1 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.2.1 | compose 8.1.6 | cterm 8.0 | borgbackup 8.1.7 | tempmon 8.0.3 | mergerfs 8.0.1 | scripts 8.0.1 | writecache 8.1.7


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Quote from ryecoaaron

    Not sure why it only happens with the compose plugin though.

    That's what got me going nuts.

    And the funny thing is only on the "settings" sub-page.

    All other pages don't make the counter increase, go figure

    • Official Post

    Soma

    Off topic. Memory of other threads accessing your system remotely with Wireguard. Why did you switch to swag to access the OMV interface? Isn't this a bit risky? Among other things, if someone finds your domain, the admin user could block you directly in OMV after several attempts, regardless of their IP.

    Quote from chente

    Memory of other threads accessing your system remotely with Wireguard.

    Why did you switch to swag to access the OMV interface?

    I've always used SWAG to access via https.

    Wireguard is a last resort, in case I get blocked or SWAG stops working.

    Quote from chente

    if someone finds your domain, the admin user could block you directly in OMV after several attempts, regardless of their IP.

    You make a good point and I never thought about it.

    But also that is covered, I can WIREGUARD the server and run omv-firstaid to reset the failed counter.


    And even if it WIREGUARD fails, there's SSH portforward from WAN to LAN, in really last resort.


    If that happens, all I need to do is change the proxy.conf to a complete new name for the subdomain, and the old https link stops working, ;)

    • Official Post

    Yeah. I already imagine that you are covered for eventualities of access.

    Either way it would not expose the OMV interface to the internet directly. I guess that will be a sweet tooth for any hacker ^^

    Quote from chente

    Either way it would not expose the OMV interface to the internet directly.

    That's what the fail2ban is there for and I have it really restricted (3 hits on a 10 minute period and it's IP banned)


    Up until now, I only had 1 attempt (with several hits from a huge amount of different IPs from Russia, Singapore, China, Marrocco and more that I didn't inspect) via SSH (got the info from fail2ban a while ago) because I was using an AP on an Italian house that was probably already hacked.

    It wasn't even on https JAIL, it was ssh JAIL.

    As soon I saw that, I re-did my portforward and SSH port (never used defaults nor will) on the system and hits were gone.


    How the port for SSH that I was using at the time was discovered, was a mistery to me.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login