What causes Monit Alerts

  • Hello all,


    this week I received 20 Emails with Monit Alerts one morning between 6:06 am and 7:30 am.
    All the notification emails either stated "Recource limit matched Service localhost (loadavg(5min) of 2.6 matches resource limit [loadavg(5min)>2.0]) or "Resource limit suceeded Service localhost ('localhost' loadavg(1min) check succeeded [current loadavg(1min)=4.0]).


    Now I'm wondering what caused those alerts as there where no activities initialised by me during that period (like backups or whatever). Does anyone have a clue what might be responsible for the alerts or does anybody know a tool to trace the origin for such a bunch of notofications?


    Thanks in advance.


    McFly3025.

  • That is probably when you system checks for updates?

    omv 5.6.13 usul | 64 bit | 5.11 proxmox kernel | omvextrasorg 5.6.2 | kvm plugin 5.1.6
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • Could be collectd running amok again.


    Greetings
    David

    "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"


    Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.

    Upload Logfile via WebGUI/CLI
    #openmediavault on freenode IRC | German & English | GMT+1
    Absolutely no Support via PM!

  • Hi there,
    I observe more or less similar mail notifications since I dared to upgrade form Sardaukar to Kralizec 1.9 last week. I usually receive a pair of monit notification within one minute:

    Quote


    1. notification: monit alert -- Resource limit matched localhost: cpu wait usage of 100.0% matches resource limit [cpu wait usage>95.0%]
    2. notification: monit alert -- Resource limit succeeded localhost: 'localhost' cpu wait usage check succeeded [current cpu wait usage=1.5%]


    What makes me wonder:
    A. On Sardaukar I did not receive such messages
    B. Messages are sent sometimes when the OMV server starts up
    C Messages are sent sometimes when the OMV server resumes from hibernation (via AutoShutDown plugin)
    I could not find a pattern up to now, except that the messages were only sent on start up or resuming conditions.


    When I watch the processes on my OMV server with "top" I cant find any process which may be the cause for such a load. So, as the original poster wrote, I also do not have a clue what is causing thins load not do I have an idea how to tackle the issue. Perhaps someone has a hint?


    Thanks in advance,
    topi

    OpenMediaVault 5.x (most recent stable version) -- 64 bit -- OMV-Extras 5.x (most recent stable version) -- Default Kernel

  • For me its collectd that sometims clogs up the CPU really heavy. Wouldn't wonder me if it does this too for you guys when you resume from hibernation.


    Greetings
    David

    "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"


    Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.

    Upload Logfile via WebGUI/CLI
    #openmediavault on freenode IRC | German & English | GMT+1
    Absolutely no Support via PM!

  • Thanks David for the hint.
    Would there be a tool or a way to trace the resource consumption of collectd while resuming or starting up?
    Thanks,
    topi

    OpenMediaVault 5.x (most recent stable version) -- 64 bit -- OMV-Extras 5.x (most recent stable version) -- Default Kernel

  • You may check if this always happen at the same time if you are running 24/7. I assume that it is cron-apt that is checking for new updates. You may have a look into /var/log/cron-apt/log to check when the last execution was. Maybe this correlates with your emails.

  • Thank you for your replies.


    Actually my system has been up for almost 40 days without any interruptions. I recieived a couple of notifications during that time, but it was only last Wednesday that I received that amount for appr. 90 minutes.
    I was checking that apt-logfile and there was nothing suspicious catching my eyes. So maybe it was really this collectd.


    Best regards.


    McFly

  • Hi votdev,


    I checked the situation now for a few days when the mail notifications appear. I can say for sure that they appear only when my system starts up or when it resumes from hibernation. It shows alwas the same pattern, I usually receive a pair of monit notifications within one minute:


    Code
    1. notification: monit alert -- Resource limit matched localhost: cpu wait usage of 100.0% matches resource limit [cpu wait usage>95.0%]
    2. notification: monit alert -- Resource limit succeeded localhost: 'localhost' cpu wait usage check succeeded [current cpu wait usage=1.5%]


    So, would there be a way to make monit somehow more resilient with this cpu usage peaks?


    Thanks,
    topi

    OpenMediaVault 5.x (most recent stable version) -- 64 bit -- OMV-Extras 5.x (most recent stable version) -- Default Kernel

  • You can customize the settings. Have a look into /usr/share/openmediavault/mkconf/monit.d/system. Set the environment variables to your prefered valies in /etc/default/openmediavault. To apply the new values you have to execute


    Code
    # omv-mkconf monit
  • I keep getting the same notifications when my system resumes from suspended mode.


    votdev: checking those config-files but I don't understand what parameters I have to change to avoid those notifications...

  • I'd like to modify the MAIL-FORMAT statement. But seems that the changes I make get overwritten by omv-mkconf monit. So where can I define MAIL-FORMAT that omv-mkconf monit will use? Primarily, I'd like to change the subject line. (I'm still fighting the distinction of Resource limit matched vs. Resource limit succeeded).


    You mention in another post that I can modify:
    OMV_MONIT_SERVICE_SYSTEM_LOADAVG_1MIN_MULT
    OMV_MONIT_SERVICE_SYSTEM_LOADAVG_5MIN_MULT


    but I'm not sure how or what I may modify it to be.


    I event tried creating my own injection as /etc/monit/conf.d/openmediavualt-mailformat.conf, but it get's wiped out with omv-mkconf monit

  • This happen to me too. My server has more than enough power & ram to run OMV.


    At first I thought the ClamAV, antivirus, using too much resource.
    But after I uninstalled it, I still get notification about "Resource limit succeeded Service localhost".
    It happened every time the server is wake up/resumed from hibernation or shutdown.
    Can't find the solution.


    OMV v2.1.1

    OMV v5.0
    Asus Z97-A/3.1; i3-4370
    32GB RAM Corsair Vengeance Pro

  • I've had that noisy "loadavg" event in syslog twice per minute or more.
    I've fixed that by adding extra lines to /etc/default/openmediavault and reloading.

    Code
    # Fix monit issue with reporting high loadavg
    OMV_MONIT_SERVICE_SYSTEM_LOADAVG_1MIN_MULT=50
    OMV_MONIT_SERVICE_SYSTEM_LOADAVG_5MIN_MULT=50
    Bash
    omv-mkconf monit
    monit restart all

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!