Report Page Fails with 504 Gateway Timeout

David B · May 2nd 2022

Hello OMV Community,

I am having a recurring issue with the "Diagnostics → Report" page on my OMV installation. Every time I try to load it, it tries to load for several minutes before ultimately following with this error: "Http failure response for http://192.168.1.2/rpc.php: 504 Gateway Time-out"

Has anyone else seen this and does anyone know if there is a way to fix it?

My OMV version is 6.0.25-1 (Shaitan).

David B · May 3rd 2022

votdev Is there any information I could provide for getting this addressed?

votdev · May 3rd 2022

Please check /var/log/syslog for any error messages. If you do not find any, please run the following commands and then try to open the page in the UI again. Are there any errors shown in the CLI?

Code

# monit stop omv-engined
# omv-engined -d -f

Afterwards stop with CTRL-C and run monit start omv-engined.

David B · May 3rd 2022

I didn't see any errors in /var/log/syslog, so I tried doing what you described, and I still don't see any errors. I have attached the output from omv-engined -d -f. Perhaps you will be able to find something that I can't? It looks pretty unhelpful though.

votdev · May 3rd 2022

The attached file does not contain the time range when you select the diagnostic report page in the UI; the logs do not contain information about the RPC that is called when visiting the page. Have you logged the whole time, maybe you missed the important part.

You can also have a look into /var/log/nginx/openmediavault-webgui_error.log.

David B · May 3rd 2022

Hmm alright. I will try again and post back here again.

In the meantime, I do have these entries in /var/log/nginx/openmediavault-webgui_error.log, but that doesn't really help much with solving the problem, I would think.

Code

2022/05/03 09:24:13 [error] 2483#2483: *32247 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.4, server: openmediavault-webgui, request: "POST /rpc.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm-openmediavault-webgui.sock", host: "192.168.1.2", referrer: "http://192.168.1.2/"
2022/05/03 11:24:35 [error] 2483#2483: *32547 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.4, server: openmediavault-webgui, request: "POST /rpc.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm-openmediavault-webgui.sock", host: "192.168.1.2", referrer: "http://192.168.1.2/"
2022/05/03 11:26:14 [error] 2483#2483: *32547 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.4, server: openmediavault-webgui, request: "POST /rpc.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm-openmediavault-webgui.sock", host: "192.168.1.2", referrer: "http://192.168.1.2/"
2022/05/03 11:31:35 [error] 2483#2483: *32705 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.4, server: openmediavault-webgui, request: "POST /rpc.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm-openmediavault-webgui.sock", host: "192.168.1.2", referrer: "http://192.168.1.2/"
2022/05/03 14:17:34 [error] 2483#2483: *33100 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.4, server: openmediavault-webgui, request: "POST /rpc.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm-openmediavault-webgui.sock", host: "192.168.1.2", referrer: "http://192.168.1.2/"
2022/05/03 14:21:30 [error] 2483#2483: *33089 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.4, server: openmediavault-webgui, request: "POST /rpc.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm-openmediavault-webgui.sock", host: "192.168.1.2", referrer: "http://192.168.1.2/"
2022/05/03 14:24:16 [error] 2483#2483: *33100 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.4, server: openmediavault-webgui, request: "POST /rpc.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm-openmediavault-webgui.sock", host: "192.168.1.2", referrer: "http://192.168.1.2/"

David B · May 3rd 2022

Alright I just tried again, but I still don't see anything.

votdev · May 4th 2022

Please run the command time omv-sysinfo and post the last lines of the output. I guess it will take a very long time. Please tell us also which hardware you are using. What plugins do you have installed?

David B · May 4th 2022

The time omv-sysinfo command has been running for about 10 minutes and it's not done yet. I will post an update when it's finished, but for now, OMV lists my CPU as Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz. The machine has 8GB of RAM.

As for the plugins I use, here it the list of what I have installed:

openmediavault-cputemp 6.1.1
openmediavault-fail2ban 6.0.2
openmediavault-flashmemory 6.1
openmediavault-omvextrasorg 6.0.8
openmediavault-remotemount 6.0
openmediavault-sharerootfs 6.0-4
openmediavault-symlinks 6.0.4
openmediavault-wol 6.0.1

I also use several Docker containers, but I would that that they would be irrelevant.

David B · May 4th 2022

I have attached a copy of the output from time omv-sysinfo. I think I have a guess as to what is causing the problem: the fail2ban output is very lengthy, and it seems to take some time for that information to print out when running the command in the terminal.

votdev · May 4th 2022

Try the same command after you have unistalled the fail2ban command.

ryecoaaron Any ideas what's going on here?

ryecoaaron · May 4th 2022

fail2ban is working overtime blocking on the scans on your ssh port since it is exposed to the internet. And while it seems to be doing its job, I would highly recommend moving ssh off port 22. I don't know why that is slowing down omv-sysinfo other than the raw amount of entries.

David B · May 4th 2022

votdev After clearing all of my fail2ban entries and trying again, the command runs instantly and the timeout no longer happens when using the web UI.

ryecoaaron I can certainly move it, but I thought that the way those scanners typically worked meant that it would still get hammered. I would actually like to close it off, but unfortunately, I need SSH access to be open.

ryecoaaron · May 4th 2022

Quote from David B

I can certainly move it, but I thought that the way those scanners typically worked meant that it would still get hammered. I would actually like to close it off, but unfortunately, I need SSH access to be open.

Most of the scanners check regular ports for speed reasons. You could set it to 20123 and I doubt you would see any traffic. And I'm not suggesting closing the ssh port just changing it to a different port (I do).

ssh hostname

becomes

ssh -p 20123 hostname

And you wouldn't have to change the port on OMV just on your port forwarding 20123 (wan) -> 22 (lan)

David B · May 4th 2022

Alright I will experiment with this. Thank you!

bw1235 · Jun 10th 2022

FWIW - came here for this issue. I noticed when I went to reinstall Docker that it returned a "DNS Error". I'd been using Adguard Home with DNS-over-TLS. Reverting to default ISP's DNS on my router fixed this. It also resolved this 504 timeout issue. It ALSO resolved the "Please Wait, Configuration Changes..." taking FOREVER. Now back to normal.

I suspect that all the various networks I have were all having to time out on similar DNS error, causing everything to take too long and for this 504 timeout to trigger.

Hope this helps. I'm going to investigate more. I may revert resolv.conf to 8.8.8.8 and see if I can operate that way while maintaining my router's DNS settings to use AGH.

Participate now!