[solved] Help! High load and lots of syslog entries

toppi · 26. Januar 2014

Hm,

it´s back

Code

root@intrepid:/media# cat /etc/mtab
/dev/sde1 / ext3 rw,errors=remount-ro 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid,mode=0755 0 0
proc /proc proc rw,noexec,nosuid,nodev 0 0
sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0
udev /dev tmpfs rw,mode=0755 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
/dev/sdd1 /media/6629d1a2-277f-4fbe-b660-6422d0836314 ext4 rw,noexec,_netdev,acl  ,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 0 0
/dev/sdc1 /media/879f7b87-4915-441d-9445-82f73d46d6cf ext4 rw,noexec,_netdev,acl  ,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 0 0
/dev/md127 /media/86ecc63e-317d-4fe3-bc42-bbe0e5531916 ext4 rw,noexec,_netdev,ac  l,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 0 0
/media/6629d1a2-277f-4fbe-b660-6422d0836314/recordings /export/recordings none r  w,bind 0 0
/media/6629d1a2-277f-4fbe-b660-6422d0836314/download /export/download none rw,bi  nd 0 0
/media/6629d1a2-277f-4fbe-b660-6422d0836314/mp3 /export/mp3 none rw,bind 0 0
rpc_pipefs /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0

Alles anzeigen

Code

root@intrepid:/media# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/sda1 during installation
UUID=0f1a6e7d-b84d-4d42-98f8-b8bcbb3d40f8 /               ext3    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=1d6e0320-eef4-4a67-bd23-e7e45c5b5e8e none            swap    sw              0       0
/dev/sdb1       /media/usb0     auto    rw,user,noauto  0       0
# >>> [openmediavault]
UUID=6629d1a2-277f-4fbe-b660-6422d0836314 /media/6629d1a2-277f-4fbe-b660-6422d0836314 ext4 defaults,nofail,acl,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 0 2
UUID=879f7b87-4915-441d-9445-82f73d46d6cf /media/879f7b87-4915-441d-9445-82f73d46d6cf ext4 defaults,nofail,acl,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 0 2
UUID=86ecc63e-317d-4fe3-bc42-bbe0e5531916 /media/86ecc63e-317d-4fe3-bc42-bbe0e5531916 ext4 defaults,nofail,acl,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 0 2
/media/6629d1a2-277f-4fbe-b660-6422d0836314/recordings /export/recordings none bind 0 0
/media/6629d1a2-277f-4fbe-b660-6422d0836314/download /export/download none bind 0 0
/media/6629d1a2-277f-4fbe-b660-6422d0836314/mp3/ /export/mp3 none bind 0 0
# <<< [openmediavault]

Alles anzeigen

davidh2k · 27. Januar 2014

Zitat von "tekkbebe"

All looks fine there. Can you explain this path:

/df/df-media-6629d1a2-277f-4fbe-b660-6422d0836314

Seems like the RRD graph failed to update - df media UUID for use of this specific hard disk.

Either the hard disk is missing, which i doubt, or your system time jumped way too much for the RRD graph to keep up.

Greetings
David

toppi · 27. Januar 2014

Hi,

but what can I do now?

I´ve deleted the ntp server and set the time manually -> Problem still exists.

I´ve added an other ntp Server -> Problem still exists.

Between the steps I rebootet my OMV. And the system time is always correct.

davidh2k · 27. Januar 2014

Dunno. The time must jumped inbetween. Have to look later or tomorrow at this.

Greetings
David

overflow68 · 28. Januar 2014

Hi,

Same problem here :

Code

Jan 28 17:50:53 openmediavault collectd[1910]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Jan 28 17:50:53 openmediavault collectd[1910]: uc_update: Value too old: name = localhost/df/df-media-25893b91-d5af-4bd8-8a9c-46a1a892aa77; value time = 1390927853; last cache update = 1390927853;
Jan 28 17:50:53 openmediavault collectd[1910]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Jan 28 17:50:53 openmediavault collectd[1910]: uc_update: Value too old: name = localhost/df/df-media-f36aa0cd-f5da-4209-bf3a-434e10ef49d9; value time = 1390927853; last cache update = 1390927853;
Jan 28 17:50:53 openmediavault collectd[1910]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Jan 28 17:50:53 openmediavault collectd[1910]: uc_update: Value too old: name = localhost/df/df-media-ac497453-920e-4954-928c-d105605d1e72; value time = 1390927853; last cache update = 1390927853;
Jan 28 17:50:53 openmediavault collectd[1910]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Jan 28 17:51:03 openmediavault collectd[1910]: uc_update: Value too old: name = localhost/df/df-media-9ca64ce5-0f66-4403-a8d2-50ddb516b20c; value time = 1390927863; last cache update = 1390927863;
Jan 28 17:51:03 openmediavault collectd[1910]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Jan 28 17:51:03 openmediavault collectd[1910]: uc_update: Value too old: name = localhost/df/df-media-25893b91-d5af-4bd8-8a9c-46a1a892aa77; value time = 1390927863; last cache update = 1390927863;
Jan 28 17:51:03 openmediavault collectd[1910]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Jan 28 17:51:03 openmediavault collectd[1910]: uc_update: Value too old: name = localhost/df/df-media-f36aa0cd-f5da-4209-bf3a-434e10ef49d9; value time = 1390927863; last cache update = 1390927863;
Jan 28 17:51:03 openmediavault collectd[1910]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.
Jan 28 17:51:03 openmediavault collectd[1910]: uc_update: Value too old: name = localhost/df/df-media-ac497453-920e-4954-928c-d105605d1e72; value time = 1390927863; last cache update = 1390927863;
Jan 28 17:51:03 openmediavault collectd[1910]: Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.

Alles anzeigen

Don't know exactly when it started.

Regard

Stefano

davidh2k · 29. Januar 2014

Quoting from http://collectd.org/faq.shtml

Zitat
I get frequent errors that a “value is too old”. What's this about?
The complete error message usually looks like this:
Code
[2009-05-06 14:03:05] uc_update: Value too old: name = device.domain.tld/snmp/frequency-output; value time = 1241611385; last cache update = 1241611385;
When adding a new value to the internal cache, the timestamp on that value is checked against the timestamp on the last value with the same name that was added to the cache. The error message informs you, that the value already in the cache was newer or as new as the value that should have been added. In the example above, a value for device.domain.tld/snmp/frequency-output should be added, but the current timestamp (1241611385) is the same as the timestamp already present in the cache, i.e. a duplicate.

The most common source of this is that somehow two values with the same identifier (name) are reported. One frequent reason for this is that two hosts report data using the same host name and send it to a central server. If the “last cache update time” increases with each message, this is very likely that case. You can use Wireshark (1.4 or later) to analyze and filter the collectd network traffic and find out from which IP addresses the duplicate values originate. The second most common reason is a misconfiguration of generic plugins, such as the SNMP plugin.

A similar variant of the above problem is that the daemon is running twice on the same host. You can use the ps command to check if this is the case.

These errors may also be caused by a plugin being loaded twice. You can check if each plugin is loaded only once by checking the LoadPlugin lines:
Code
grep -i LoadPlugin /etc/collectd/collectd.conf | egrep -v '^[[:space:]]*#' | sort | uniq -c
Another common cause is that time on the client jumps backwards. This may happen due to a weekly ntpdate forcefully setting the time, for example. Virtual hosts often have problems providing a steady wallclock time, but usually they have jumps forward (causing gaps). It might be worth investigating nonetheless.
Alles anzeigen

Whats the output of the following command:

Code

ps aux | grep collectd

You may also try the grep command in the above quote.

Greetings
David

toppi · 29. Januar 2014

Hi David,

it really seems that df is loaded twice:

Code

root@intrepid:~# grep -i LoadPlugin /etc/collectd/collectd.conf | egrep -v '^[[:space:]]*#' | sort | uniq -c
      1 LoadPlugin cpu
      2 LoadPlugin df
      1 LoadPlugin interface
      1 LoadPlugin load
      1 LoadPlugin memory
      1 LoadPlugin rrdtool
      1 LoadPlugin syslog
      1 LoadPlugin unixsock

Code

root@intrepid:~# ps aux | grep collectd
root      1521  0.0  0.0 231500  1980 ?        Ssl  Jan26   1:09 /usr/sbin/collectd -C /etc/collectd/collectd.conf -P /var/run/collectd.pid
root     29667  0.0  0.0   9664   844 pts/0    S+   06:49   0:00 grep collectd

davidh2k · 29. Januar 2014

Im at work now, without good internet. Either google yourself for how to stop the plugin that was startes twice or wait for me to come back later with a solution.

Greetings

toppi · 29. Januar 2014

Hi,

me too. I´ve deletetd the duplicate entry in the collectd.conf and for the short time before I want to office I saw no new entires

tekkb · 29. Januar 2014

It is very weird error. We have never seen it and in less than a week 2 people have it. Can both of you please tell us what you had installed on your OMV? I got feeling we may see this against and it we be nice if we can figure out where it is coming from.

toppi · 29. Januar 2014

Hi,

here is my actual list of the installed plugins:

Code

root@intrepid:~# dpkg -l | grep openmediavault
ii  openmediavault                      0.5.31                       Open network attached storage solution
ii  openmediavault-backup               0.5.1                        backup plugin for OpenMediaVault.
ii  openmediavault-downloader           0.5.3                        OpenMediaVault downloader plugin
ii  openmediavault-extplorer            0.5.5                        OpenMediaVault eXtplorer plugin
ii  openmediavault-keyring              0.2                          GnuPG archive keys of the OpenMediaVault archive
ii  openmediavault-omvextrasorg         0.5.38                       OMV-Extras.org Package Repositories for OpenMediaVault
ii  openmediavault-supportinfo          0.5.9                        OpenMediaVault support panel
ii  openmediavault-transmissionbt       0.5.18                       OpenMediaVault Transmission (BitTorrent client) plugin
ii  openmediavault-usbbackup            0.5.8                        OpenMediaVault USB backup plugin
root@intrepid:~#

Alles anzeigen

For about two weeks I installed vdr. But I´ve already removed it. Therefor I installed only a driver for my Sundtek DVB-C card.

davidh2k · 29. Januar 2014

Zitat von "tekkbebe"

It is very weird error. We have never seen it and in less than a week 2 people have it. Can both of you please tell us what you had installed on your OMV? I got feeling we may see this against and it we be nice if we can figure out where it is coming from.

Wait a minute! Volker changed some of the RRD graph code, maybe its a bug in the code change? Feel free to report it on the bugtracker so Volker has a look at it or I will do it later on.

Greetings
David

tekkb · 29. Januar 2014

If saw that David it should be reported. I'm back to sleep.....

overflow68 · 29. Januar 2014

Hi,
Same problem here:

Code

LoadPlugin syslog
LoadPlugin rrdtool
LoadPlugin unixsock
LoadPlugin cpu
LoadPlugin df
LoadPlugin df
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin nut

I deleted the double entry in collectd.conf and the flood seems finished ..

Greetings

tekkb · 29. Januar 2014

I think everyone has this problem now. I have same issue on my syslog of my vm. Did you check your system David?

davidh2k · 29. Januar 2014

Can check it later when I'm home. My NAS is not often active these days, only two or three times or so.

Greetings
David

WastlJ · 29. Januar 2014

My NAS is not powered on 24/7. But i cannot confirm. sys logs seems normal. None of those errors.

ryecoaaron · 29. Januar 2014

I have three OMV boxes that are on 24/7 and syslog is normal.

davidh2k · 29. Januar 2014

Volker fixed it. However he has not added it to the changelog for 0.5.33, yet.

See:
http://sourceforge.net/p/openmediavault/code/869/
and
http://sourceforge.net/p/openmediavault/code/870/

Greetings
David

tekkb · 29. Januar 2014

I had made a clone of my vm just before this happened. It is caused by a double entry in the /etc/collectd/collectd.conf

You will have to entries like this

Code

LoadPlugin df
<Plugin df>
  MountPoint "/media/185ebf0e-6b50-42f3-9ebf-9596c27c38b5"
  MountPoint "/media/6f7b4aef-782d-47e5-940d-1650f94806b5"
  IgnoreSelected false
</Plugin>


LoadPlugin df
<Plugin df>
  MountPoint "/media/185ebf0e-6b50-42f3-9ebf-9596c27c38b5"
  MountPoint "/media/6f7b4aef-782d-47e5-940d-1650f94806b5"
  IgnoreSelected false
</Plugin>

Alles anzeigen

You should only have 1 entry. Now here is a list of upgrades my clone shows. So the problem has to be from one of these:

Code

openmediavault-clamav
openmediavault-git
openmediavault-greyhole
openmediavault-nut
openmediavault-rsnapshot
openmediavault-snapraid
openmediavault-supportinfo
openmediavault-vdr
openmediavault-website

Jetzt mitmachen!