syslog stops logging at end of startup sequence

I know that logging uses journald rather than rsyslog in omv7 but I have an issue where there are no further logs in the GUI or using journalctl from the console. Logging seems to stop at the end of a startup sequence but I do not know how to trouble shoot this.

My rpi4 system is upgraded from omv6 to omv7 using omv-release-upgrade and this worked fine. Everything is working well (except logs i think).

If I check the syslog in /var/log/syslog I can see recent logs so it is just journald / journalctl that is not working.

Any ideas?

I restarted journald and all seems fine now. Will check the situation after a reboot to see if this was just a little glitch.

systemctl restart systemd-journald

Update: After reboot - same situation with logs stopping at this point in the syslog.

12/26/2023, 5:48:49 PM
systemd[1]: Mounted export-docs.mount - /export/docs.
12/26/2023, 5:48:49 PM
systemd[1]: Mounting export-docs.mount - /export/docs...
12/26/2023, 5:48:48 PM
systemd[1]: Mounted srv-dev\x2ddisk\x2dby\x2duuid\x2d31995d48\x2d98d2\x2d41e6\x2d91ad\x2deb9b6f6ce10f.mount - /srv/dev-disk-by-uuid-31995d48-98d2-41e6-91ad-eb9b6f6ce10f.
12/26/2023, 5:48:48 PM
(udev-worker)[712]: regulatory.0: Process '/lib/crda/crda' failed with exit code 254.
12/26/2023, 5:48:46 PM
systemd[1]: Reached target remote-fs.target - Remote File Systems.

Status of systemd-journald after system reboot - see below. At this point there is no logging to syslog after the log entries above.

Dec 26 17:48:40 rpi4 systemd-journald[170]: Journal started
Dec 26 17:48:40 rpi4 systemd-journald[170]: Runtime Journal (/run/log/journal/80ec49fe88324c98a44d189232c2bfe0) is 8.0M, max 75.8M, 67.8M free.
Dec 26 17:48:40 rpi4 systemd-journald[170]: Time spent on flushing to /var/log/journal/80ec49fe88324c98a44d189232c2bfe0 is 32.662ms for 435 entries.
Dec 26 17:48:40 rpi4 systemd-journald[170]: System Journal (/var/log/journal/80ec49fe88324c98a44d189232c2bfe0) is 104.0M, max 1.7G, 1.6G free.
Dec 26 17:48:40 rpi4 systemd-journald[170]: Received client request to flush runtime journal.
Dec 26 17:48:40 rpi4 systemd-journald[170]: File /var/log/journal/80ec49fe88324c98a44d189232c2bfe0/system.journal corrupted or uncleanly shut down, renaming and replacing.

After restarting the systemd-journald, the status messages are slightly different but not sure if this is an indicator of the issue.

Dec 26 17:56:18 rpi4 systemd-journald[3326]: File /var/log/journal/80ec49fe88324c98a44d189232c2bfe0/system.journal corrupted or uncleanly shut down, renaming and replacing.
Dec 26 17:56:18 rpi4 systemd-journald[3326]: Journal started
Dec 26 17:56:18 rpi4 systemd-journald[3326]: System Journal (/var/log/journal/80ec49fe88324c98a44d189232c2bfe0) is 112.0M, max 189.6M, 77.6M free.

I was also able to reproduce with a clean install of omv7 using the install script on a deb12 raspberry pi lite base install.

Zitat von votdev

That is strange because OMV does not do any changes to the systemd journal configuration.

I thought that too.

I also had the end0 ethernet name even though I disabled predictable names.

Happy to have another test to see if I can reproduce either or both issues if that is helpful.

One thing… I didn’t download the install script and use the -b flag as I read on another thread that it can be run in the usual way.

I am able to reproduce these 2 issues with a clean install of OMV 7 or when using omv-release-upgrade from OMV6 to OMV7 on a raspberry pi 4.

It would be good to see if others can reproduce as if not resolved before OMV7 beta completes, I think there could be a load of people with rpi4's that have these issues...

I am using a clean/new OMV7 install from latest 64bit lite rpi4 image and running the omv install script for testing/troubleshooting.

Issue 1:

The LAN interface is created with the name end0 even if I have disabled predictable network names in raspi-config - I don't think this is a problem for those setting up a new OMV7 system - but it is if you are using omv-release-upgrade (e.g. docker networks linked to eth0 will be broken)

Issue 2:

System logging stops at some point late in the boot sequence. There are no clues why (for me). The work-a-round is to restart journal logging that resolves the issue until the next restart.

systemctl restart systemd-journald

ryecoaaron - bringing the discussion back to the correct thread. Thanks.

Summary is this:

reboot or startup rpi4 logging in GUI and journal begins then stops logging anything further.

at cli running the commands to create a log entry and then display it fails.

If I then restart the logging service and repeat - it then works and logging is working again - see below

in the syslog in the GUI. I can see the last line that is logged before it just stops at 12:16pm (bottom line highlighted).

when I restart logging using systemctl restart systemd-journald logging comes back to life

and you can now see the second test using logger testomv

The final bit of info is that if i check /var/log/syslog - I can see it has not been updated since I installed the OMV test system 2 days ago.

It is very weird!

Thought the syslog might be helpful. Last entry relates to rsyslog restarting. Maybe this is a clue?

2024-01-15T14:17:01.014716+11:00 rpi4 systemd[1]: Started run-r17b1a44560294562b55a143292ac6634.scope - /usr/bin/systemctl restart rsyslog.service.

I’m using brand new ssd usb disk. Tried 2 of them. Then a new SD in case it was related to usb boot.

Also tried new v omv6 upgrade using my prod omv6 system.

Same result every time.

Maybe it’s a rpi4 hardware version or the rpi is dying.

It’s weird as all I need to do is restart journald after booting and everything is fine. Could just add a scheduled task to do this I guess.

I have a SSK SSD - that I use in prod omv6. works fine. I use brand new scandisk SD cards.

I will do one last set of tests as follows using a new SD card and usb ssd.

1. install 64b raspberry lite on SD card & usb ssd

2. check if journald logging is working

3. run OMV install script

4. check if journald logging is working

5. login to omv gui and make a couple of minor config changes

6. reboot and check if journald logging is working

I will report back results from these tests.

I feel like this is a sign that I need a new rpi5 -

Zitat von ryecoaaron

Is it a 8GB RPi4?

4GB

Here are the results of my first test using a brand new 64gb scandisk SD card.

lite 64b - new (pass)

root@rpi4:~# logger testomv1
root@rpi4:~# journalctl --quiet --output=short-iso --no-pager | grep testomv
2024-01-18T14:03:58+1100 rpi4 root[904]: testomv1

update/upgrade - restart (pass)

root@rpi4:~# logger testomv2
root@rpi4:~# journalctl --quiet --output=short-iso --no-pager | grep testomv
2024-01-18T14:03:58+1100 rpi4 root[904]: testomv1
2024-01-18T14:09:43+1100 rpi4 root[789]: testomv2

install omv from script - restart (fail)

root@rpi4:~# logger testomv3
root@rpi4:~# journalctl --quiet --output=short-iso --no-pager | grep testomv
2024-01-18T14:03:58+1100 rpi4 root[904]: testomv1
2024-01-18T14:09:43+1100 rpi4 root[789]: testomv2

See above that testomv3 is not included. On my rpi4 there is something going wrong with the OMV install script that causes this issue.

I also have the issue with network interface eth0 / end0 naming so I don't think it is properly fixed. OMV GUI say eth0 but actual interface name is end0. This happens even after disabling predictable names.

Strange is that this issue only seems to impact my raspberry pi

I think there is a clue to the issue in /var/log/syslog that stops logging during the omv install (via the script).

Any thoughts on why this might be happening and where in the omv install process is this happening? ryecoaaron

Zitat von ryecoaaron

I don't see anything in that syslog file.

What are you seeing that makes you think it stops logging?

There is no further logging to var/log/syslog after the last entry that happens to be systemctl restart rsyslog.service.

The event/timestamp is while the omv install script was still running. I think that is strange or does omv intentionally stop/disable rsyslog?