OMV crashing/rebooting

Hendrik1 · Aug 16th 2023

Hi,

I am trying to figure out why OMV is randomly rebooting. I'm not sure if it is rebooting or crashing and then restarting. It happened on August 12th and 15th at midnight for no apparent reason. It appears that the system is offline for 6 minutes each time, or at least there are no logs in that time.

OMV version 6.4.8-1 (Shaitan) | 64 bit | Kernel: Linux 6.1.0-0.deb11.7-amd64

System running with i7-4770K and 16 GB RAM.

Syslog for first reboot looks like:

syslog

Code

Aug 12 23:58:54 node tailscaled[3940809]: control: doLogin(regen=false, hasUrl=true)
Aug 12 23:58:54 node tailscaled[3940809]: control: RegisterReq: onode=[BYWCH] node=[aOwGV] fup=true nks=false
Aug 12 23:58:54 node tailscaled[3940809]: Received error: register request: http 400: unknown followup url
Aug 12 23:59:10 node tailscaled[3940809]: control: doLogin(regen=false, hasUrl=true)
Aug 12 23:59:10 node tailscaled[3940809]: control: RegisterReq: onode=[BYWCH] node=[aOwGV] fup=true nks=false
Aug 12 23:59:10 node tailscaled[3940809]: Received error: register request: http 400: unknown followup url
Aug 12 23:59:50 node tailscaled[3940809]: control: doLogin(regen=false, hasUrl=true)
Aug 12 23:59:50 node tailscaled[3940809]: control: RegisterReq: onode=[BYWCH] node=[aOwGV] fup=true nks=false
Aug 12 23:59:50 node tailscaled[3940809]: Received error: register request: http 400: unknown followup url
Aug 13 00:00:00 node systemd[1]: Starting Podman auto-update service...
Aug 13 00:00:00 node systemd[1]: Starting Rotate log files...
Aug 13 00:00:00 node systemd[1]: Started Update a database for mlocate.
Aug 13 00:00:00 node systemd[1]: rsyslog.service: Sent signal SIGHUP to main process 3954689 (rsyslogd) on client request.


Aug 13 00:00:00 node systemd[1]: logrotate.service: Succeeded.
Aug 13 00:00:00 node systemd[1]: Finished Rotate log files.
Aug 13 00:00:00 node systemd[1775058]: var-lib-containers-storage-overlay.mount: Succeeded.
Aug 13 00:00:00 node systemd[1321583]: var-lib-containers-storage-overlay.mount: Succeeded.
Aug 13 00:00:00 node systemd[1]: var-lib-containers-storage-overlay.mount: Succeeded.
Aug 13 00:00:00 node systemd[1]: podman-auto-update.service: Succeeded.
Aug 13 00:00:00 node systemd[1]: Finished Podman auto-update service.
Aug 13 00:00:01 node CRON[1841718]: (root) CMD (/var/lib/openmediavault/cron.d/userdefined-57e13a3c-5a94-11ec-8153-3f587eab8e1f 2>&1 | mail -E -s "Cron" -a "From: Cron Daemon <root>" root >/dev/null 2>&1)
Aug 13 00:00:01 node CRON[1841719]: (root) CMD (sudo mount -a)
Aug 13 00:00:01 node CRON[1841720]: (root) CMD (/usr/sbin/omv-mkrrdgraph >/dev/null 2>&1)
Aug 13 00:00:19 node tailscaled[3940809]: control: doLogin(regen=false, hasUrl=true)
Aug 13 00:00:19 node tailscaled[3940809]: control: RegisterReq: onode=[BYWCH] node=[aOwGV] fup=true nks=false
Aug 13 00:00:19 node tailscaled[3940809]: Received error: register request: http 400: unknown followup url
Aug 13 00:01:10 node collectd[3977011]: plugin_read_thread: read-function of the `interface' plugin took 11.267 seconds, which is above its read interval (10.000 seconds). You might want to adjust the `Interval' or `ReadThreads' settings.
Aug 13 00:03:26 node collectd[3977011]: plugin_read_thread: read-function of the `memory' plugin took 17.204 seconds, which is above its read interval (10.000 seconds). You might want to adjust the `Interval' or `ReadThreads' settings.
Aug 13 00:10:17 node systemd-modules-load[355]: Inserted module 'macvlan'
Aug 13 00:10:17 node blkmapd[373]: open pipe file /run/rpc_pipefs/nfs/blocklayout failed: No such file or directory
Aug 13 00:10:17 node systemd-modules-load[355]: Inserted module 'coretemp'
Aug 13 00:10:17 node systemd-modules-load[355]: Inserted module 'nct6775'
Aug 13 00:10:17 node kernel: [    0.000000] microcode: microcode updated early to revision 0x28, date = 2019-11-12
Aug 13 00:10:17 node systemd-modules-load[355]: Inserted module 'softdog'
Aug 13 00:10:17 node kernel: [    0.000000] Linux version 6.1.0-0.deb11.7-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-2~bpo11+1 (2023-04-23)
Aug 13 00:10:17 node kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.1.0-0.deb11.7-amd64 root=UUID=e3f751f8-a0bf-4a6d-b6d3-65283a6aaf28 ro quiet

Display More

And for the second reboot (shortened for brevity):

syslog2

Code

Aug 15 23:40:19 node smartd[1483]: Device: /dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_S598NJ0NB02829P [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 70 to 71
Aug 15 23:45:01 node CRON[2546503]: (root) CMD (sudo mount -a)
Aug 15 23:45:01 node CRON[2546504]: (root) CMD (/usr/sbin/omv-mkrrdgraph >/dev/null 2>&1)
Aug 15 23:50:01 node CRON[2548298]: (root) CMD (sudo mount -a)
Aug 15 23:55:01 node CRON[2549485]: (root) CMD (sudo mount -a)
Aug 16 00:00:00 node systemd[1]: Starting Podman auto-update service...
Aug 16 00:00:00 node systemd[1]: Starting Rotate log files...
Aug 16 00:00:00 node systemd[1]: Started Update a database for mlocate.
Aug 16 00:00:00 node systemd[1]: logrotate.service: Succeeded.
Aug 16 00:00:00 node systemd[1]: Finished Rotate log files.
Aug 16 00:00:00 node systemd[1]: var-lib-containers-storage-overlay.mount: Succeeded.
Aug 16 00:00:00 node systemd[220921]: var-lib-containers-storage-overlay.mount: Succeeded.
Aug 16 00:00:00 node systemd[1]: podman-auto-update.service: Succeeded.
Aug 16 00:00:00 node systemd[1]: Finished Podman auto-update service.
Aug 16 00:00:01 node CRON[2550631]: (root) CMD (sudo mount -a)
Aug 16 00:00:01 node CRON[2550632]: (root) CMD (/usr/sbin/omv-mkrrdgraph >/dev/null 2>&1)
Aug 16 00:06:13 node systemd-modules-load[357]: Inserted module 'macvlan'
Aug 16 00:06:13 node kernel: [    0.000000] microcode: microcode updated early to revision 0x28, date = 2019-11-12
Aug 16 00:06:13 node blkmapd[375]: open pipe file /run/rpc_pipefs/nfs/blocklayout failed: No such file or directory
Aug 16 00:06:13 node kernel: [    0.000000] Linux version 6.1.0-0.deb11.7-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-2~bpo11+1 (2023-04-23)
Aug 16 00:06:13 node kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.1.0-0.deb11.7-amd64 root=UUID=e3f751f8-a0bf-4a6d-b6d3-65283a6aaf28 ro quiet
Aug 16 00:06:13 node kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Aug 16 00:06:13 node kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Aug 16 00:06:13 node kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Aug 16 00:06:13 node kernel: [    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
Aug 16 00:06:13 node systemd-modules-load[357]: Inserted module 'coretemp'
Aug 16 00:06:13 node kernel: [    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
Aug 16 00:06:13 node kernel: [    0.000000] signal: max sigframe size: 1776
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-provided physical RAM map:
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d7ff] usable
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x000000000009d800-0x000000000009ffff] reserved
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
Aug 16 00:06:13 node systemd-modules-load[357]: Inserted module 'nct6775'
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000c4ff3fff] usable
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000c4ff4000-0x00000000c4ffafff] ACPI NVS
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000c4ffb000-0x00000000c5452fff] usable
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000c5453000-0x00000000c58d6fff] reserved
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000c58d7000-0x00000000d861afff] usable
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000d861b000-0x00000000d86a4fff] reserved
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000d86a5000-0x00000000d86bdfff] ACPI data
Aug 16 00:06:13 node systemd-modules-load[357]: Inserted module 'softdog'
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000d86be000-0x00000000d8bfefff] ACPI NVS
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000d8bff000-0x00000000d9ffefff] reserved
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000d9fff000-0x00000000d9ffffff] usable
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000dc000000-0x00000000de1fffff] reserved
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
Aug 16 00:06:13 node kernel: [    0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed03fff] reserved
Aug 16 00:06:13 node systemd[1]: Starting Flush Journal to Persistent Storage...

Display More

Are there other logs which hold more info? I don't see anything being logged as to why the system is restarting.

Thank you for any help you can provide.

macom · Aug 16th 2023

You should check the logs in journalctl just before the boot.

In the journal the boot process starts with something like

Code

-- Boot 27844393d3ad45d3b9c679a6c241968c --

Here some pages that explain how to use journalctl

Using journalctl - The Ultimate Guide To Logging

Journalctl is a utility for querying and displaying logs from journald, systemd’s logging service. Since journald stores log data in a binary format instead of…

www.loggly.com

journalctl

Hendrik1 · Aug 21st 2023

Quote from macom
You should check the logs in journalctl just before the boot.

In the journal the boot process starts with something like
Code
-- Boot 27844393d3ad45d3b9c679a6c241968c --
Here some pages that explain how to use journalctl
https://www.loggly.com/ultimate-guide/using-journalctl/
https://www.freedesktop.org/so…stemd/man/journalctl.html
Display More

Unfortunately I don't see any more information with journalctl logs. It just restarted at 23.00 again today, and this is what the journalctl log looks like:

Code

Aug 21 22:55:01 node sudo[3525231]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Aug 21 22:55:01 node sudo[3525231]: pam_unix(sudo:session): session closed for user root
Aug 21 22:55:01 node CRON[3525228]: pam_unix(cron:session): session closed for user root
Aug 21 23:00:01 node CRON[3541858]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Aug 21 23:00:01 node CRON[3541857]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Aug 21 23:00:01 node CRON[3541860]: (root) CMD (sudo mount -a)
Aug 21 23:00:01 node CRON[3541861]: (root) CMD (/usr/sbin/omv-mkrrdgraph >/dev/null 2>&1)
Aug 21 23:00:01 node sudo[3541863]:     root : PWD=/root ; USER=root ; COMMAND=/usr/bin/mount -a
Aug 21 23:00:01 node sudo[3541863]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Aug 21 23:00:01 node sudo[3541863]: pam_unix(sudo:session): session closed for user root
Aug 21 23:00:02 node CRON[3541858]: pam_unix(cron:session): session closed for user root
Aug 21 23:00:16 node CRON[3541857]: pam_unix(cron:session): session closed for user root
Aug 21 23:05:57 node kernel: microcode: microcode updated early to revision 0x28, date = 2019-11-12
Aug 21 23:05:57 node kernel: Linux version 6.1.0-0.deb11.7-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-2~bpo11+1 (2023-04-23)
Aug 21 23:05:57 node kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.1.0-0.deb11.7-amd64 root=UUID=e3f751f8-a0bf-4a6d-b6d3-65283a6aaf28 ro quiet
Aug 21 23:05:57 node kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'

crashtest Aug 16th 2023