[Solved: turn off monitoring in web GUI] monit keeps complaining about rrdcached service (emails, syslog spam)

1st Official Post

    Since the omv-writecache plugin updated from 8.2.2 to 8.2.3 on both my AMD64 and arm64 OMV servers yesterday, monit seems to be unhappy for some reason.


    Got two notification emails today (Sep. 6) at 2:39 pm, one saying that the rrdcached service does not exist (prompting monit to restart), the other one saying the restart failed.


    First email:


    Second email:


    My WriteCache settings on backserv (the arm64 system) are as follows. Note that it's set up for daily flush/rotate at ... yep, 2:39 pm.



    Output of systemctl --full --no-pager status monit:

    (This started around the 14:39 time stamp and just keeps going on)


    Output of systemctl --full --no-pager status rrdcached:



    This never happened before, and I didn't change any settings in the writecache plugin. Everything else seems to be working just fine, it's just the endless error logs.

    It's also interesting to note that this only affects my arm64 server. My main AMD64 system flushes/rotates both hourly and daily at 5:33 am, and has not complained at all.


    What's going on, and what's the fix?

    Addendum - I'm also seeing a TON of this in syslog:



    Further addendum - After seeing another thread about a similar issue, I ran omv-salt deploy run monit on both of my systems.


    That seemingly didn't do much, except now my main system - the one that's configured to flush/rotate hourly - is now sending me the same monit and rrdcached emails every hour.


    🫣

    • Official Post
    Quote from cubemin

    What's going on, and what's the fix?

    Are you just blaming writecache because it was updated? The change was pretty minor and I have been dogfooding it on four of my servers including an arm64 for a while. If you don't see any errors in the writecache log, I fail to see how it could've caused it.

    omv 8.5.11-1 synchrony | 7.0 proxmox kernel

    plugins:

    omvextrasorg 8.0.3 | kvm 8.3.4 | compose 8.2.2 | cterm 8.0.4 | borgbackup 8.2.1 | tempmon 8.0.5 |

    mergerfs 8.0.2 | scripts 8.0.6 | writecache 8.3.0 | backup 8.1 | kernel 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • cubemin

    Changed the title of the thread from “Possible bug in WriteCache plugin? monit keeps complaining about rrdcached service (emails, syslog spam)” to “monit keeps complaining about rrdcached service (emails, syslog spam)”.

    Not blaming it at all. I simply thought there might be an issue with it since the monit trouble started after updating the plugin. And you have to admit that when this happens right at flush/rotate time, that seems a bit more than a coincidence (at least to someone with limited knowledge, like yours truly).


    You say it's not writecache, and of course I'll take your word for it. Having said that, I was hoping for some advice as to how to stop this syslog spam and the email notifications.


    In the meantime, though, I've followed a suggestion to disable monitoring in the OMV GUI, apply the changes, re-enable it, and apply the changes again. That seems to have worked for now... no more syslog spam. If the emails stop as well, I will mark this thread resolved (and even wear a dunce hat).


    Update... nope, I got the same two emails again just now.

    • Official Post
    Quote from cubemin

    I simply thought there might be an issue with it since the monit trouble started after updating the plugin. And you have to admit that when this happens right at flush/rotate time, that seems a bit more than a coincidence

    And it still could be. rrdcache is very sensitive to any changes. It has been recommended to disable monitoring when using flashmemory/writecache since the beginning. I still add rrdcache's directories because I know people won't follow that advice.

    Quote from cubemin

    You say it's not writecache, and of course I'll take your word for it.

    I didn't say it wasn't writecache. I was just saying it is frustrating that it is blamed for every problem especially when I push out an update. I don't know what else I can do. I have hundreds of automated tests and run it on all of my systems before pushing out updates but obviously I can't replicate every omv setup. I even have AI look for weaknesses in writecache every time a new model comes out. Yet, people still have problems. It gets old...

    Quote from cubemin

    I was hoping for some advice as to how to stop this syslog spam and the email notifications.

    This might be one of the hardest things to diagnose and to do it through forum posts takes time and patience I don't have.


    Quote from cubemin

    nope, I got the same two emails again just now.

    I'm not sure how you can get rrdcache emails when it is disabled. Maybe the flush restored the emails?

    omv 8.5.11-1 synchrony | 7.0 proxmox kernel

    plugins:

    omvextrasorg 8.0.3 | kvm 8.3.4 | compose 8.2.2 | cterm 8.0.4 | borgbackup 8.2.1 | tempmon 8.0.5 |

    mergerfs 8.0.2 | scripts 8.0.6 | writecache 8.3.0 | backup 8.1 | kernel 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Quote from ryecoaaron

    And it still could be. rrdcache is very sensitive to any changes. It has been recommended to disable monitoring when using flashmemory/writecache since the beginning. I still add rrdcache's directories because I know people won't follow that advice.

    I wasn't even aware of this recommendation. What, exactly, will I lose by turning off monitoring?


    I don't have a problem doing so as long as I still get notified of potential issues (like cron job results or SMART errors, for example). With monitoring disabled, would monit still catch non-running services and restart as needed? Just trying to get an idea of what the on/off toggle really entails.


    Quote

    I didn't say it wasn't writecache. I was just saying it is frustrating that it is blamed for every problem especially when I push out an update. I don't know what else I can do. I have hundreds of automated tests and run it on all of my systems before pushing out updates but obviously I can't replicate every omv setup. I even have AI look for weaknesses in writecache every time a new model comes out. Yet, people still have problems. It gets old...

    That makes a lot of sense - I'm sorry to add to the frustration. Know that I definitely wasn't trying to assign any blame - rather, I felt that there might be a bug and I should bring it to someone's (your) attention. It's all good.


    Quote

    This might be one of the hardest things to diagnose and to do it through forum posts takes time and patience I don't have.

    Absolutely fair.


    Quote

    I'm not sure how you can get rrdcache emails when it is disabled. Maybe the flush restored the emails?

    I wouldn't know. However, since my last post there have been no more emails. They just... stopped. The syslog spam is still happening, though.


    I'm going to go ahead and turn off monitoring in the GUI, but please put my mind at ease and let me know what exactly it is I'm turning off. :)


    EDIT ... oh, I see. It's just the performance statistics that get turned off, right? Works for me - I have no need of them.

  • cubemin

    Added the Label resolved

  • cubemin

    Changed the title of the thread from “monit keeps complaining about rrdcached service (emails, syslog spam)” to “[Solved: turn off monitoring in web GUI] monit keeps complaining about rrdcached service (emails, syslog spam)”.

    Looks like it failed to umount /var/lib/rrdcached due to rrdcached is using the directory, which makes rrdcached was killed, and monit somehow failed to restart rrdcached (timing?).

    So I guess we have to options:

    1. Add /var/lib/rrdcached to LAZY_UNMOUNT_PATHS, and accept the potential risk of data loss.
    2. Config rrdcached as a 'stopstart' service, but monit is watching it, maybe we will still receive the warning.

    I will add /var/lib/rrdcached to LAZY_UNMOUNT_PATHS and see what happens tonight

    • Official Post
    Quote from nekoosuki

    2. Config rrdcached as a 'stopstart' service, but monit is watching it, maybe we will still receive the warning.

    At least one user is having good luck with that:

    Post
    RE: No rrdgraph for CPU Temp
    […]

    Try adding to the writecache services field.

    (Code, 2 lines)

    ryecoaaron
    Post
    RE: No rrdgraph for CPU Temp
    I restarted the NAS to make sure data is coming in for CPU etc., then I added the parameters to Write Cache settings and performed the flush via UI. Data is coming in afterwards and rrdcached is still running. That might be the solution. Thanks!

    (Gemini had the same idea after I fed it the last two meaningful changes on the writecache script. You were faster and I trust people with knowledge more then AI and me prompting without too much knowledge. Shxt in, shxt out.)
    erbsenzaehler

    omv 8.5.11-1 synchrony | 7.0 proxmox kernel

    plugins:

    omvextrasorg 8.0.3 | kvm 8.3.4 | compose 8.2.2 | cterm 8.0.4 | borgbackup 8.2.1 | tempmon 8.0.5 |

    mergerfs 8.0.2 | scripts 8.0.6 | writecache 8.3.0 | backup 8.1 | kernel 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Thanks 1
    • New
    • Official Post
    Quote from dacbarbos

    it looks like systemd keeps throwing deprecation warnings at you, and us. This one is about rrdcached.

    That is Debian's problem to fix.

    omv 8.5.11-1 synchrony | 7.0 proxmox kernel

    plugins:

    omvextrasorg 8.0.3 | kvm 8.3.4 | compose 8.2.2 | cterm 8.0.4 | borgbackup 8.2.1 | tempmon 8.0.5 |

    mergerfs 8.0.2 | scripts 8.0.6 | writecache 8.3.0 | backup 8.1 | kernel 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login