Monitoring alerts, related to rrdcached and collectd

yyovm · 10. Dezember 2021

I recently setup an OMV 5 file server. I enabled email notifications, and in every boot I received 6 emails: 4 related to rrdcached and 2 related to collectd. The emails I receive are the following (in order of time received, although they are all received at almost the same time, so the order is not necessarily representative):

Code

=== 1 ===
The system monitoring needs your attention.

Host:        \openmediavault
Date:        Fri, 10 Dec 2021 07:23:24
Service:     rrdcached
Event:       Exists
Description: process is running with pid 741

This triggered the monitoring system to: alert

=== 2 ===
The system monitoring needs your attention.

Host:        \openmediavault
Date:        Fri, 10 Dec 2021 07:22:25
Service:     rrdcached
Event:       Execution failed
Description: failed to start (exit status -1) -- Program '/bin/systemctl start rrdcached' timed out after 30 s

This triggered the monitoring system to: alert

=== 3 ===
The system monitoring needs your attention.

Host:        \openmediavault
Date:        Fri, 10 Dec 2021 07:22:25
Service:     collectd
Event:       Does not exist
Description: process is not running

This triggered the monitoring system to: restart

=== 4 ===
The system monitoring needs your attention.

Host:        \openmediavault
Date:        Fri, 10 Dec 2021 07:21:55
Service:     rrdcached
Event:       Does not exist
Description: process is not running

This triggered the monitoring system to: restart

=== 5 ===
The system monitoring needs your attention.

Host:        \openmediavault
Date:        Fri, 10 Dec 2021 07:23:25
Service:     collectd
Event:       Exists
Description: process is running with pid 753

This triggered the monitoring system to: alert

=== 6 ===
The system monitoring needs your attention.

Host:        \openmediavault
Date:        Fri, 10 Dec 2021 07:23:25
Service:     rrdcached
Event:       Execution succeeded
Description: process is running after previous exec error (slow starting or manually recovered?)

This triggered the monitoring system to: alert

Alles anzeigen

Regarding the rrdcached messages, I read here that it is not something to worry about. After reading this tread (rrdcached service, running/not running, does not exist?), I made the changes indicated there (about OMV_MONIT_DELAY_SECONDS variable), and I will see again tomorrow morning if I get the related messages again. Although I do not understand why I get the same 2 messages twice about rrdcached.

What about the 2 collectd messages? Why are there at the first place? Is it something to worry about and how could I get rid of them?

macom · 10. Dezember 2021

Both might be related to monit starting too soon.

yyovm · 10. Dezember 2021

I rebooted the server after making the changes to the monit starting, and now I get only the 2 emails related to collectd. I setup the monit delay at 60 seconds.

macom · 10. Dezember 2021

What is the output of systemctl status collectd.service

yyovm · 10. Dezember 2021

Zitat von macom

What is the output of systemctl status collectd.service

I do not have access to the OMV server right now, it is shut down (every Friday, after the daily backup, I get a Clonezilla image of the boot disk, and the server goes to sleep ). I will be back on Monday with the report. Thank you very much!

macom · 10. Dezember 2021

Ok. If the first two lines read

Code

● collectd.service - Statistics collection and monitoring daemon
   Loaded: loaded (/lib/systemd/system/collectd.service; disabled; vendor preset: enabled)

please run

omv-salt deploy run collectd and post the output

and then systemctl status collectd.service again and post the output

yyovm · 13. Dezember 2021

Zitat von macom

What is the output of systemctl status collectd.service

Code

● collectd.service - Statistics collection and monitoring daemon
Loaded: loaded (/lib/systemd/system/collectd.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/collectd.service.d
└─openmediavault.conf
Active: active (running) since Mon 2021-12-13 08:07:13 EET; 2h 40min ago
Docs: man:collectd(1)
man:collectd.conf(5)
https://collectd.org
Main PID: 753 (collectd)
Tasks: 12 (limit: 4598)
Memory: 2.8M
CGroup: /system.slice/collectd.service
└─753 /usr/sbin/collectd


Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "interface" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "load" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "memory" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "rrdcached" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "syslog" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "unixsock" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "uptime" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: Systemd detected, trying to signal readyness.
Dec 13 08:07:13 openmediavault systemd[1]: Started Statistics collection and monitoring daemon.
Dec 13 08:07:13 openmediavault collectd[753]: Initialization complete, entering read-loop.

Alles anzeigen

yyovm · 13. Dezember 2021

Zitat von macom
Ok. If the first two lines read
Code
● collectd.service - Statistics collection and monitoring daemon
   Loaded: loaded (/lib/systemd/system/collectd.service; disabled; vendor preset: enabled)
please run

omv-salt deploy run collectd and post the output

and then systemctl status collectd.service again and post the output
Alles anzeigen

Yes, this is the output. Here are the outputs of these 2 commands:

command1.txt (too long to copy it here)

command2:

Code

● collectd.service - Statistics collection and monitoring daemon
Loaded: loaded (/lib/systemd/system/collectd.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/collectd.service.d
└─openmediavault.conf
Active: active (running) since Mon 2021-12-13 08:07:13 EET; 3h 0min ago
Docs: man:collectd(1)
man:collectd.conf(5)
https://collectd.org
Main PID: 753 (collectd)
Tasks: 12 (limit: 4598)
Memory: 2.8M
CGroup: /system.slice/collectd.service
└─753 /usr/sbin/collectd


Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "interface" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "load" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "memory" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "rrdcached" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "syslog" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "unixsock" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: plugin_load: plugin "uptime" successfully loaded.
Dec 13 08:07:13 openmediavault collectd[753]: Systemd detected, trying to signal readyness.
Dec 13 08:07:13 openmediavault systemd[1]: Started Statistics collection and monitoring daemon.
Dec 13 08:07:13 openmediavault collectd[753]: Initialization complete, entering read-loop.

Alles anzeigen

macom · 13. Dezember 2021

Zitat von yyovm

Loaded: loaded (/lib/systemd/system/collectd.service; enabled; vendor preset: enabled)

So collectd is now enabled. It will start during a reboot.

macom · 13. Dezember 2021

From the text file:

Zitat

Summary for debian

-------------

Succeeded: 16 (changed=3)

Failed: 0

should be all good now

yyovm · 14. Dezember 2021

Thank you very much for your help!

Yes, it seems to be solved. But, this morning upon startup, I received these 2 emails notifications, related to rrdcached again:

Code

The system monitoring needs your attention.

Host:        \openmediavault
Date:        Tue, 14 Dec 2021 07:20:22
Service:     rrdcached
Event:       Exists
Description: process is running with pid 749

This triggered the monitoring system to: alert

and

Code

The system monitoring needs your attention.

Host:        \openmediavault
Date:        Tue, 14 Dec 2021 07:19:50
Service:     rrdcached
Event:       Does not exist
Description: process is not running

This triggered the monitoring system to: restart

Any ideas?

macom · 14. Dezember 2021

Post the output of systemctl status rrdcached

yyovm · 15. Dezember 2021

Here it is:

Code

● rrdcached.service - LSB: start or stop rrdcached
   Loaded: loaded (/etc/init.d/rrdcached; generated)
   Active: active (running) since Wed 2021-12-15 07:21:50 EET; 1h 20min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 659 ExecStart=/etc/init.d/rrdcached start (code=exited, status=0/SUCCESS)
    Tasks: 8 (limit: 4598)
   Memory: 17.1M
   CGroup: /system.slice/rrdcached.service
           └─725 /usr/bin/rrdcached -B -F -f 3600 -w 900 -b /var/lib/rrdcached/db/ -j /var/lib/rrdcached/journal/ -p /run/rrdcached.pid -l unix:/run/rrdcache

Dec 15 07:21:49 openmediavault systemd[1]: Starting LSB: start or stop rrdcached...
Dec 15 07:21:50 openmediavault rrdcached[659]: rrdcached started.
Dec 15 07:21:50 openmediavault systemd[1]: Started LSB: start or stop rrdcached.

Alles anzeigen

macom · 15. Dezember 2021

run omv-salt deploy run rrdcached and check if the issue is solved after that

yyovm · 15. Dezember 2021

Zitat von macom

run omv-salt deploy run rrdcached and check if the issue is solved after that

I run the command, reboot, and I did not get any email alert this time !!!!

Thank you very much macom!!

I did not understand if these warnings reflected any malfunction in OVM, since everything works fine!

macom · 15. Dezember 2021

Your welcome

yyovm · 21. Dezember 2021

Today I received again the 2 emails regarding rrdcached. Strange isn't it?

Do I have to run the command

Code

omv-salt deploy run rrdcached

in every reboot?

macom · 21. Dezember 2021

What is the output of

omv-env get OMV_MONIT_DELAY_SECONDS

and

systemctl is-enabled rrdcached?

yyovm · 21. Dezember 2021

root@openmediavault:~# omv-env get OMV_MONIT_DELAY_SECONDS

OMV_MONIT_DELAY_SECONDS=60

root@openmediavault:~# systemctl is-enabled rrdcached

rrdcached.service is not a native service, redirecting to systemd-sysv-install.

Executing: /lib/systemd/systemd-sysv-install is-enabled rrdcached

enabled

macom · 21. Dezember 2021

Mhm. Last idea:

omv-salt stage run deploy

Monitoring alerts, related to rrdcached and collectd

Jetzt mitmachen!

Tags