OMV Nightly Hang

    • OMV 4.x
    • Resolved

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • OMV Nightly Hang

      Hi Everyone,

      I recently built a new server for OMV and have been experiencing Nightly hangs. The server is unresponsive and needs a manual reboot in order to come back up. I've swapped out several components and performed a fresh install of OMV every time. The hangs don't seem to happen if I reboot the server in the afternoon/evening. For example. I rebooted the server yesterday around 19:30 last night and didn't see the behavior. My server setup is as follows:

      CPU: AMD Ryzen 5 1600
      Motherboard: B450I Gaming Plus AC (also tried using a ASRock Mobo)
      Memory: G.SKILL FORTIS Series 16GB (8GB x 2) (also tried Crucial and Patriot memory)
      SSD Drive: ADATA SU800 M.2 2280
      SATA Drives: Hitachi HDS72303 x 4
      USB Drive: WD My Passport 2599

      Last time the hang happened I saw the following logs:

      daemon.log

      Source Code

      1. Apr 10 06:26:13 booknas systemd[1]: Starting Daily apt upgrade and clean activities...
      2. Apr 10 06:26:14 booknas systemd[1]: Started Daily apt upgrade and clean activities.
      3. Apr 10 06:26:14 booknas systemd[1]: apt-daily-upgrade.timer: Adding 22min 22.439752s random time.
      4. Apr 10 06:26:14 booknas systemd[1]: apt-daily-upgrade.timer: Adding 40min 16.960270s random time.


      debug

      Source Code

      1. Apr 10 05:21:26 booknas rrdcached[1118]: started new journal /var/lib/rrdcached/journal/rrd.journal.1554891686.354734
      2. Apr 10 05:21:26 booknas rrdcached[1118]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1554884486.354732
      3. Apr 10 06:21:26 booknas rrdcached[1118]: flushing old values
      4. Apr 10 06:21:26 booknas rrdcached[1118]: rotating journals
      5. Apr 10 06:21:26 booknas rrdcached[1118]: started new journal /var/lib/rrdcached/journal/rrd.journal.1554895286.354742
      6. Apr 10 06:21:26 booknas rrdcached[1118]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1554888086.354727
      syslog

      Source Code

      1. Apr 10 06:21:26 booknas rrdcached[1118]: flushing old values
      2. Apr 10 06:21:26 booknas rrdcached[1118]: rotating journals
      3. Apr 10 06:21:26 booknas rrdcached[1118]: started new journal /var/lib/rrdcached/journal/rrd.journal.1554895286.354742
      4. Apr 10 06:21:26 booknas rrdcached[1118]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1554888086.354727
      5. Apr 10 06:25:01 booknas CRON[23598]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
      6. Apr 10 06:26:13 booknas systemd[1]: Starting Daily apt upgrade and clean activities...
      7. Apr 10 06:26:14 booknas systemd[1]: Started Daily apt upgrade and clean activities.
      8. Apr 10 06:26:14 booknas systemd[1]: apt-daily-upgrade.timer: Adding 22min 22.439752s random time.
      9. Apr 10 06:26:14 booknas systemd[1]: apt-daily-upgrade.timer: Adding 40min 16.960270s random time.
      10. Apr 10 06:30:01 booknas CRON[23795]: (root) CMD (/usr/sbin/omv-mkrrdgraph >/dev/null 2>&1)

      Does anyone have any ideas on where else to look or something else to try?

      Thanks!
    • Fully updated OMV 4?
      Did you try the proxmox kernel that you can install from the Kernel tab of omv-extras?
      Did you run memtest to make sure the memory is ok?
      omv 4.1.22 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.15
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • y2kdread wrote:

      what's different about that kernel?
      It is Ubuntu 18.04's 4.15 LTS kernel. Seems a bit more stable than the backports kernel in Debian Stretch.
      omv 4.1.22 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.15
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • ryecoaaron wrote:

      Fully updated OMV 4?
      Did you try the proxmox kernel that you can install from the Kernel tab of omv-extras?
      Did you run memtest to make sure the memory is ok?
      I ran memtest, switched to the proxmox kernel, and finally was able done making changes. Sadly, the issue reoccured. I did have the terminal up and got the following error:

      Source Code

      1. Message from syslogd@booknas at Apr 16 02:59:28 ...
      2. kernel:[209987.536806] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u64:0:23400]
      Would this be due to a bad CPU or unrelated?
    • y2kdread wrote:

      Would this be due to a bad CPU or unrelated?
      It seems to be a hardware stability problem. Is the cpu overclocked? Have you updated to the latest bios? Maybe the power supply is having issues under high load? Hard to say.
      omv 4.1.22 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.15
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • New

      ryecoaaron wrote:

      It seems to be a hardware stability problem. Is the cpu overclocked? Have you updated to the latest bios? Maybe the power supply is having issues under high load? Hard to say.
      Turns out it was the power supply! I wasn't overclocking and I don't think it had to deal with a high load. I hooked up a kill-a-watt to the machine and the max I saw it pulling was 85 watts and the supply is rated for 450 watts. I ended up buying a higher end power supply and the problem went away. It's strange that the hanging happened with a specific set of criteria. I would think a power supply issue would be random.

      Either way, I hope this thread helps people out of they see the same thing in the future.

      Thanks for your help!