OMV completely hangs

Hey people!

My OMV installation has been buzzing along nicely for nearly two years now, but in the recent days it has completely frozen on me 4 times. The system is completely unresponsive (webgui en local) and requires a hard reboot.
After a reboot I find lots of errors in the syslog, see below for a small selection (it's usually several pages).

Sep  4 21:04:29 belzultra collectd[3872]: rrdcached plugin: rrdc_update (/var/lib/rrdcached/db/localhost/load/load.rrd, [1473015869:0.180000:0.800000:0.520000], 1) failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: rrdcached plugin: rrdc_update (/var/lib/rrdcached/db/localhost/cpu-0/cpu-user.rrd, [1473015869:8546], 1) failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: rrdcached plugin: rrdc_update (/var/lib/rrdcached/db/localhost/cpu-0/cpu-nice.rrd, [1473015869:6], 1) failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: rrdcached plugin: rrdc_update (/var/lib/rrdcached/db/localhost/nut-ups/percent-charge.rrd, [1473015869:100.000000], 1) failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: rrdcached plugin: rrdc_update (/var/lib/rrdcached/db/localhost/interface-eth1/if_octets.rrd, [1473015869:358971033:10101493], 1) failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: rrdcached plugin: rrdc_update (/var/lib/rrdcached/db/localhost/rrdcached/queue_length.rrd, [1473015869:0.000000], 1) failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: rrdcached plugin: rrdc_update (/var/lib/rrdcached/db/localhost/memory/memory-used.rrd, [1473015869:1184849920.000000], 1) failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: rrdcached plugin: rrdc_update (/var/lib/rrdcached/db/localhost/memory/memory-buffered.rrd, [1473015869:36487168.000000], 1) failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: rrdcached plugin: rrdc_update (/var/lib/rrdcached/db/localhost/interface-eth1/if_packets.rrd, [1473015869:245269:64791], 1) failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: rrdcached plugin: rrdc_update (/var/lib/rrdcached/db/localhost/cpu-0/cpu-system.rrd, [1473015869:409], 1) failed with status -1.
Sep  4 21:04:29 belzultra collectd[3872]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.

How would I start to troubleshoot this? My fear is a piece of hardware is starting to fail but I see no indication as to what it could be. Searching online gives me mixed results. The NAS is used as file storage en Plex station.

Thanks for any advice!

J.

Thank you for the reply!

I ran omv-firstaid and selected the RRD database fix, below the output.

root@belzultra:~# omv-firstaid
Checking all RRD files. Please wait ...
All RRD database files are valid.

Last night I had another hard crash. While watchin a movie the system suddenly stopped and became completely unresponsive. No webgui, no shell access. I left it like that but it never recovered. After a reboot all is well again and this time there are no errors in syslog.

Does anyone have any idea as to what I can do before/during/after such a crash to troubleshoot this issue?

Hmm, the Curse of the Black Sun (or my inability to stop messing with the wrong things) continues.

The last few days my system ran fine. Today I installed the new OMV 2.2.7 release and the Plex update. After the installation Plex refused to run. Uninstalled/reinstalled Plex using the web interface to no avail.

Went to check the logs and saw alot lot of collectd errors again. Used omv-firstaid to fix the db. Erros showed up saying future dates were in the logs. Agreed to delete them.

After this the webgui stopped working saying config.xml couldn't be found. I found another thread about the error message with a solution: reinstalling OMV using these commands:

cp /usr/share/openmediavault/templates/config.xml /etc/openmediavault/
apt-get install --reinstall openmediavault

Webgui came back online but now I receive errors for every plugin I have installed:

Failed to get configuration (xpath=//services/pluginname)

Tried reinstalling them but it doesn't help. It might be time to do a complete reinstallation maybe? I feel like I'm a kid with adhd in quicksand.

Thank you for the advice!

My Debian knowledge is too limited to attempt anything then what I already did. I was hoping I could stall a fresh install until OMV 3 but I'll give it a go tomorrow.

As for the OS drive, I did an extended test yesterday and from what I can make of it it passed without errors.

Great news! I'm using the following, but wouldn't be too bothered if not all are working 100% yet.

- Plex
- Minidlna
- extplorer
- sabnzbd
- transmissionbt
- nut
- virtualbox

I'll install OMV 3 tomorrow then! Awesome!

Not sure if I should make a new thread about this. The installation went fine but after installing OMV-extras the webgui shows the following error when trying to log in (it never lets you log in). I found

Error #0:
exception 'OMV\Rpc\Exception' with message 'Failed to connect to socket: No such file or directory' in /usr/share/php/openmediavault/rpc/rpc.inc:138
Stack trace:
#0 /var/www/openmediavault/rpc/session.inc(56): OMV\Rpc\Rpc::call('UserMgmt', 'authUser', Array, Array, 2, true)
#1 [internal function]: OMVRpcServiceSession->login(Array, Array)
#2 /usr/share/php/openmediavault/rpc/serviceabstract.inc(124): call_user_func_array(Array, Array)
#3 /usr/share/php/openmediavault/rpc/rpc.inc(84): OMV\Rpc\ServiceAbstract->callMethod('login', Array, Array)
#4 /usr/share/php/openmediavault/rpc/proxy/json.inc(95): OMV\Rpc\Rpc::call('Session', 'login', Array, Array, 3)
#5 /var/www/openmediavault/rpc.php(45): OMV\Rpc\Proxy\Json->handle()
#6 {main}

If found this thread with a similar problem but it seems I'm locked out of root access for some reason. I'll try reinstalling again.

One error was returned:

PHP Fatal error: require_once(): Failed opening required 'openmediavault/object.inc' (include_path='.:/usr/share/php:/usr/share/pear') in /usr/share/openmediavault/engined/rpc/omvextrasorg.inc on line 22

I can't SSH into the machine using Putty (access denied, probably something that's not on by default?), but I can work fine directly on the machine.

Bah, sorry for messing up. Removed old version, installed new one. Everything works fine now! Thanks for the help!

My problem is back. I'm running 3.0.40 and it has been working perfectly.

Yesterday night however I came back to a completely unresponsive NAS. No webgui, no screen output.

Attached my logs. Hang occured as the last event on friday (the saturday log entries are after a hard reboot), but I see no obvious errors. I'm thinking this has to be a hardware problem but I'd like to hear your opinions.

Edit: After the reboot Plex was unresponsive. Tried to disable it in the webgui, but after clicking save it came back with an error after a long wait. After this the systems seems to have crashed again.

Update:

- updated bios
- stress tested cpu

Those went fine. Then I did what I should have done a lot earlier: Memtest86. It found a huge amount of errors; I'm actually amazed it still ran.

Thanks for the support!