My OMV 6.9.16-1 running as a VM on proxmox-ve: 7.3-1 (running kernel: 5.15.83-1-pve) has shutdown inexplicably at about 02:00 two days in a row. Historically OMV (and Promox, and all the other VMs & CTs) run forever without issue. I've gone up to a year without a restart.
Several days ago I applied all available outstanding OMV updates. It had been a while since I updated OMV. I beleive there were MANY updates. Dozens? I didnt really comit it to memory. I YOLOd it. I feel very confident in my daily backups SnapRAID, etc so... Maybe an update did something.
Anyway, I'm very fresh to linux & homelabbing. I did search and search, but this is one of those once per 5 years where I need help.
I did check the OMV Diagnostics > System Logs > Logs, but dont see anything.
It doesnt appear that Promox commanded the shutdown either.
Can someone please point me in the right direct for where & which logs to check? Any ideas? I looked at every log in /var/log and they all just end by 02:00 and start back up at 16:00 when I started the VM after noticing it off again.
Syslog:
Aug 19 01:52:55 omv qemu-ga: info: guest-ping called
Aug 19 01:53:06 omv qemu-ga: info: guest-ping called
Aug 19 01:53:17 omv qemu-ga: info: guest-ping called
Aug 19 01:53:24 omv monit[1798]: 'filesystem_srv_dev-disk-by-uuid-771b0622-8f6d-49ed-b34c-3d9c7c5ac620' space usage 86.7% matches resource limit [space usage > 85.0%]
Aug 19 01:53:24 omv monit[1798]: 'filesystem_srv_dev-disk-by-uuid-bcb8a1ed-f415-450e-b95d-98299634f575' space usage 87.0% matches resource limit [space usage > 85.0%]
Aug 19 01:53:24 omv monit[1798]: 'filesystem_srv_dev-disk-by-uuid-d7dd1fe7-28af-46a1-8bc3-2e45dcde9c20' space usage 86.6% matches resource limit [space usage > 85.0%]
Aug 19 01:53:24 omv monit[1798]: 'filesystem_srv_dev-disk-by-uuid-76c5c4d6-159f-4a2f-aae3-690d7283a3da' space usage 87.1% matches resource limit [space usage > 85.0%]
Aug 19 01:53:24 omv monit[1798]: 'filesystem_srv_dev-disk-by-uuid-a39e8369-d5bd-4713-84ed-f352d155fe2f' space usage 87.6% matches resource limit [space usage > 85.0%]
Aug 19 01:53:24 omv monit[1798]: 'filesystem_srv_dev-disk-by-uuid-f99ea0da-f67b-40bd-9562-8cfcfe366ea6' space usage 95.9% matches resource limit [space usage > 85.0%]
Aug 19 01:53:28 omv qemu-ga: info: guest-ping called
Aug 19 01:53:39 omv qemu-ga: info: guest-ping called
Aug 19 16:09:46 omv systemd-modules-load[303]: Inserted module 'softdog'
Aug 19 16:09:46 omv blkmapd[326]: open pipe file /run/rpc_pipefs/nfs/blocklayout failed: No such file or directory
Aug 19 16:09:46 omv systemd[1]: Starting Flush Journal to Persistent Storage...
Aug 19 16:09:46 omv systemd[1]: Finished Coldplug All udev Devices.
Aug 19 16:09:46 omv systemd[1]: Starting Helper to synchronize boot up for ifupdown...
Aug 19 16:09:46 omv systemd[1]: Starting Wait for udev To Complete Device Initialization...
Aug 19 16:09:46 omv systemd[1]: Finished Helper to synchronize boot up for ifupdown.
Aug 19 16:09:46 omv systemd[1]: Finished Flush Journal to Persistent Storage.
Messages:
Aug 19 01:51:38 omv qemu-ga: info: guest-ping called
Aug 19 01:51:49 omv qemu-ga: info: guest-ping called
Aug 19 01:52:00 omv qemu-ga: info: guest-ping called
Aug 19 01:52:11 omv qemu-ga: info: guest-ping called
Aug 19 01:52:22 omv qemu-ga: info: guest-ping called
Aug 19 01:52:33 omv qemu-ga: info: guest-ping called
Aug 19 01:52:44 omv qemu-ga: info: guest-ping called
Aug 19 01:52:55 omv qemu-ga: info: guest-ping called
Aug 19 01:53:06 omv qemu-ga: info: guest-ping called
Aug 19 01:53:17 omv qemu-ga: info: guest-ping called
Aug 19 01:53:28 omv qemu-ga: info: guest-ping called
Aug 19 01:53:39 omv qemu-ga: info: guest-ping called
Aug 19 16:09:46 omv rsyslogd: warning: ~ action is deprecated, consider using the 'stop' statement instead [v8.2102.0 try https://www.rsyslog.com/e/2307 ]
Aug 19 16:09:46 omv rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2102.0]
Aug 19 16:09:46 omv rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="1738" x-info="https://www.rsyslog.com"] start
Aug 19 16:09:46 omv kernel: [ 0.000000] Linux version 6.1.0-0.deb11.21-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1~bpo11+1 (2024-05-06)
Aug 19 16:09:46 omv kernel: [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.1.0-0.deb11.21-amd64 root=UUID=bd360c80-658e-4d24-9256-82de6f47cd2b ro quiet
Aug 19 16:09:46 omv kernel: [ 0.000000] BIOS-provided physical RAM map:
Kern.log
Aug 18 14:33:20 omv kernel: [ 122.279955] Attempt to set a LOCK_MAND lock via flock(2). This support has been removed and the request ignored.
Aug 18 14:34:21 omv kernel: [ 183.565153] BTRFS warning (device sdb1): block group 5610331439104 has wrong amount of free space
Aug 18 14:34:21 omv kernel: [ 183.565166] BTRFS warning (device sdb1): failed to load free space cache for block group 5610331439104, rebuilding it now
Aug 18 14:34:55 omv kernel: [ 217.605776] BTRFS info (device sdk1): the free space cache file (8908866322432) is invalid, skip it
Aug 18 14:37:50 omv kernel: [ 392.405464] BTRFS info (device sdl1): device stats zeroed by btrfs (8262)
Aug 18 14:37:50 omv kernel: [ 392.420982] BTRFS info (device sdt1): device stats zeroed by btrfs (8267)
Aug 18 14:37:50 omv kernel: [ 392.431654] BTRFS info (device sdn1): device stats zeroed by btrfs (8272)
Aug 18 14:37:50 omv kernel: [ 392.442111] BTRFS info (device sdm1): device stats zeroed by btrfs (8277)
Aug 18 14:37:50 omv kernel: [ 392.460627] BTRFS info (device sdo1): device stats zeroed by btrfs (8282)
Aug 18 14:37:50 omv kernel: [ 392.468143] BTRFS info (device sdk1): device stats zeroed by btrfs (8287)
Aug 18 14:37:50 omv kernel: [ 392.475575] BTRFS info (device sdb1): device stats zeroed by btrfs (8292)
Aug 19 16:09:46 omv kernel: [ 0.000000] Linux version 6.1.0-0.deb11.21-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1~bpo11+1 (2024-05-06)
Aug 19 16:09:46 omv kernel: [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.1.0-0.deb11.21-amd64 root=UUID=bd360c80-658e-4d24-9256-82de6f47cd2b ro quiet
Aug 19 16:09:46 omv kernel: [ 0.000000] BIOS-provided physical RAM map:
Aug 19 16:09:46 omv kernel: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Aug 19 16:09:46 omv kernel: [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
Aug 19 16:09:46 omv kernel: [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
Aug 19 16:09:46 omv kernel: [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable
Aug 19 16:09:46 omv kernel: [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved
Aug 19 16:09:46 omv kernel: [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
Aug 19 16:09:46 omv kernel: [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
Aug 19 16:09:46 omv kernel: [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000083fffffff] usable