Any ideas what's going on here? Possible Nginx issue

conorlap · May 7th 2020

Hi all,

My OMV seems to randomly hang every so often for no apparent reason, when I access the Web GUI quite often when I select Docker it gives me a Bad Gateway error. Today at 09.41 approx I see on the logs stuff about nginx failing - this was around the time I was trying to open Docker on webgui to check stuff.

Any idea what's happening and how this can be fixed? I've suffered this sort of problem for quite a while now but it's getting quite frustrating!

Thank you!

Full code dump at Pastebin: https://pastebin.com/nh6EgBwS

Code

May  7 09:39:07 myservername systemd[1]: Starting Clean php session files...
May  7 09:39:07 myservername CRON[7082]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
May  7 09:39:34 myservername systemd[1]: Started Clean php session files.
May  7 09:41:28 myservername monit[1386]: 'nginx' failed protocol test [HTTP] at [127.0.0.1]:450 [TCP/IP SSL] -- HTTP: Error receiving data -- Resource temporarily unavailable
May  7 09:42:59 myservername monit[1386]: 'nginx' trying to restart
May  7 09:43:00 myservername monit[1386]: 'nginx' stop: '/bin/systemctl stop nginx'
May  7 09:43:00 myservername monit[1386]: 'nginx' failed to stop (exit status -1) -- Program '/bin/systemctl stop nginx' timed out after 30 s
May  7 09:43:09 myservername dockerd[1278]: time="2020-05-07T09:43:01.044730798+01:00" level=error msg="stream copy error: reading from a closed fifo"
May  7 09:43:10 myservername dockerd[1278]: time="2020-05-07T09:43:01.173256438+01:00" level=error msg="stream copy error: reading from a closed fifo"
May  7 09:43:19 myservername monit[1386]: 'nginx' process is running after previous exec error (slow starting or manually recovered?)
May  7 09:43:20 myservername monit[1386]: 'nginx' connection succeeded to [127.0.0.1]:450 [TCP/IP SSL]
May  7 09:43:28 myservername systemd[1]: Stopping A high performance web server and a reverse proxy server...
May  7 09:43:28 myservername systemd[1]: nginx.service: Stopping timed out. Terminating.
May  7 09:43:30 myservername dockerd[1278]: time="2020-05-07T09:43:30.293324522+01:00" level=error msg="Handler for GET /containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
May  7 09:43:30 myservername dockerd[1278]: time="2020-05-07T09:43:30.331203466+01:00" level=error msg="Handler for GET /containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
May  7 09:43:30 myservername dockerd[1278]: time="2020-05-07T09:43:30.343131962+01:00" level=warning msg="Health check for container bc5076cf52d68cb45dda10cccd9bac4f89798f0533761f11ac2301bb62a8f20f error: context deadline exceeded: unknown"
May  7 09:43:30 myservername dockerd[1278]: time="2020-05-07T09:43:30.997820175+01:00" level=error msg="Handler for GET /images/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
May  7 09:43:33 myservername systemd[1]: Stopped A high performance web server and a reverse proxy server.
May  7 09:43:33 myservername systemd[1]: nginx.service: Unit entered failed state.
May  7 09:43:33 myservername systemd[1]: nginx.service: Failed with result 'timeout'.
May  7 09:43:34 myservername wsdd.py[1254]: ERROR: error in main loop
May  7 09:43:34 myservername wsdd.py[1254]: Traceback (most recent call last):
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 812, in serve_wsd_requests
May  7 09:43:34 myservername wsdd.py[1254]:     key.data.handle_request()
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 447, in handle_request
May  7 09:43:34 myservername wsdd.py[1254]:     msg = wsd_handle_message(msg, self.interface, address)
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 396, in wsd_handle_message
May  7 09:43:34 myservername wsdd.py[1254]:     msg_id = header.find('./wsa:MessageID', namespaces).text
May  7 09:43:34 myservername wsdd.py[1254]: AttributeError: 'NoneType' object has no attribute 'text'
May  7 09:43:34 myservername wsdd.py[1254]: ERROR: error in main loop
May  7 09:43:34 myservername wsdd.py[1254]: Traceback (most recent call last):
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 812, in serve_wsd_requests
May  7 09:43:34 myservername wsdd.py[1254]:     key.data.handle_request()
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 447, in handle_request
May  7 09:43:34 myservername wsdd.py[1254]:     msg = wsd_handle_message(msg, self.interface, address)
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 396, in wsd_handle_message
May  7 09:43:34 myservername wsdd.py[1254]:     msg_id = header.find('./wsa:MessageID', namespaces).text
May  7 09:43:34 myservername wsdd.py[1254]: AttributeError: 'NoneType' object has no attribute 'text'
May  7 09:43:34 myservername wsdd.py[1254]: ERROR: error in main loop
May  7 09:43:34 myservername wsdd.py[1254]: Traceback (most recent call last):
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 812, in serve_wsd_requests
May  7 09:43:34 myservername wsdd.py[1254]:     key.data.handle_request()
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 447, in handle_request
May  7 09:43:34 myservername wsdd.py[1254]:     msg = wsd_handle_message(msg, self.interface, address)
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 396, in wsd_handle_message
May  7 09:43:34 myservername wsdd.py[1254]:     msg_id = header.find('./wsa:MessageID', namespaces).text
May  7 09:43:34 myservername wsdd.py[1254]: AttributeError: 'NoneType' object has no attribute 'text'
May  7 09:43:34 myservername wsdd.py[1254]: ERROR: error in main loop
May  7 09:43:34 myservername wsdd.py[1254]: Traceback (most recent call last):
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 812, in serve_wsd_requests
May  7 09:43:34 myservername wsdd.py[1254]:     key.data.handle_request()
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 447, in handle_request
May  7 09:43:34 myservername wsdd.py[1254]:     msg = wsd_handle_message(msg, self.interface, address)
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 396, in wsd_handle_message
May  7 09:43:34 myservername wsdd.py[1254]:     msg_id = header.find('./wsa:MessageID', namespaces).text
May  7 09:43:34 myservername wsdd.py[1254]: AttributeError: 'NoneType' object has no attribute 'text'
May  7 09:43:34 myservername wsdd.py[1254]: ERROR: error in main loop
May  7 09:43:34 myservername wsdd.py[1254]: Traceback (most recent call last):
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 812, in serve_wsd_requests
May  7 09:43:34 myservername wsdd.py[1254]:     key.data.handle_request()
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 447, in handle_request
May  7 09:43:34 myservername wsdd.py[1254]:     msg = wsd_handle_message(msg, self.interface, address)
May  7 09:43:34 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 396, in wsd_handle_message
May  7 09:43:34 myservername wsdd.py[1254]:     msg_id = header.find('./wsa:MessageID', namespaces).text
May  7 09:43:34 myservername wsdd.py[1254]: AttributeError: 'NoneType' object has no attribute 'text'
May  7 09:43:35 myservername wsdd.py[1254]: ERROR: error in main loop
May  7 09:43:35 myservername wsdd.py[1254]: Traceback (most recent call last):
May  7 09:43:35 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 812, in serve_wsd_requests
May  7 09:43:35 myservername wsdd.py[1254]:     key.data.handle_request()
May  7 09:43:35 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 447, in handle_request
May  7 09:43:35 myservername wsdd.py[1254]:     msg = wsd_handle_message(msg, self.interface, address)
May  7 09:43:35 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 396, in wsd_handle_message
May  7 09:43:35 myservername wsdd.py[1254]:     msg_id = header.find('./wsa:MessageID', namespaces).text
May  7 09:43:35 myservername wsdd.py[1254]: AttributeError: 'NoneType' object has no attribute 'text'
May  7 09:43:35 myservername wsdd.py[1254]: ERROR: error in main loop
May  7 09:43:35 myservername wsdd.py[1254]: Traceback (most recent call last):
May  7 09:43:35 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 812, in serve_wsd_requests
May  7 09:43:35 myservername wsdd.py[1254]:     key.data.handle_request()
May  7 09:43:35 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 447, in handle_request
May  7 09:43:35 myservername wsdd.py[1254]:     msg = wsd_handle_message(msg, self.interface, address)
May  7 09:43:35 myservername wsdd.py[1254]:   File "/usr/sbin/wsdd.py", line 396, in wsd_handle_message
May  7 09:43:35 myservername wsdd.py[1254]:     msg_id = header.find('./wsa:MessageID', namespaces).text
May  7 09:43:35 myservername wsdd.py[1254]: AttributeError: 'NoneType' object has no attribute 'text'
May  7 09:43:35 myservername wsdd.py[1254]: ERROR: error in main loop
May  7 09:43:35 myservername wsdd.py[1254]: Traceback (most recent call last):

Display More

conorlap · May 7th 2020

Update:

I've performed the following commands in the hope it will fix my issues:

omv-update
omv-mkconf nginx
omv-mkconf php-fpm
service nginx restart
service php7.0-fpm restart

and finally ran 'omv-firstaid' and reconfigured web-gui.

Yield79 · May 8th 2020

I have similar problem from today.:

Code

May  8 17:56:18 NAS monit[1684]: 'nginx' failed protocol test [HTTP] at [127.0.0.1]:80 [TCP/IP] -- HTTP: Error receiving data -- Resource temporarily unavailable
May  8 17:57:04 NAS monit[1684]: 'nginx' failed protocol test [HTTP] at [127.0.0.1]:80 [TCP/IP] -- HTTP: Error receiving data -- Resource temporarily unavailable
May  8 17:57:33 NAS monit[1684]: 'nginx' trying to restart
May  8 17:57:33 NAS monit[1684]: 'nginx' stop: '/bin/systemctl stop nginx'
May  8 17:57:33 NAS systemd[1]: Stopping A high performance web server and a reverse proxy server...
May  8 17:57:34 NAS systemd[1]: nginx.service: Succeeded.
May  8 17:57:34 NAS systemd[1]: Stopped A high performance web server and a reverse proxy server.
May  8 17:57:34 NAS monit[1684]: 'nginx' start: '/bin/systemctl start nginx'
May  8 17:57:34 NAS systemd[1]: Starting A high performance web server and a reverse proxy server...
May  8 17:57:35 NAS systemd[1]: Started A high performance web server and a reverse proxy server.
May  8 17:58:05 NAS monit[1684]: 'nginx' connection succeeded to [127.0.0.1]:80 [TCP/IP]

Display More

i removed lines about email notification.

conorlap · May 8th 2020

Yeah I'm still having issues, not sure how to fix it! Thinking about upgrading to OMV 5 but seems like it could be a lot of hassle Will need to set aside time to do it!

Yield79 · May 8th 2020

Quote from conorlap

Yeah I'm still having issues, not sure how to fix it! Thinking about upgrading to OMV 5

I'm on OMV5

conorlap · May 8th 2020

Quote from Yield79

I'm on OMV5

Oh damn Hopefully someone knowledgeable can help out!

conorlap · May 9th 2020

I found a post in relation to our error:

Error Connection failed Service nginx

I made the edit as per that post to 20sec for the timeout, lets see if that fixes

conorlap · May 11th 2020

Haven't see any more errors in the log! So hopefully we're all good!

Any ideas what's going on here? Possible Nginx issue

conorlap Jun 29th 2020

Participate now!

Tags