Why is my HDD waking up?

  • Hi!


    In a previous post , I was trying to understand why my FS was regurlary corrupted. It was said that I could be caused by an unsupported sleep mode, but I didn't look further.


    Now, I noticed another issue : the hard-drive correctly go to sleep mode after a 10 minutes or by issuing the command 'hdparm -y /dev/sda', but it awakes by itself, without any reason (no access to the hard drive at that time).
    I couldn't find the reason of that... It seems to occur event if the system is totally disconnected from the network.


    For information, I'm running OMV3 (yeah, I know, it's beta, but I want to run on Debian 8 ) on an Odroid XU4 installed with DietPi (based on Debian 8). I'm not sure that this issue is directly linked to OpenMediaVault, but I think that there are a lot of experts out there that could help me. The OS is running from an SD card, and the hard-drive is a Lacie external hard drive connected on USB 2.


    I've read the Guide to debugging disks spin up with no success : hdparm -C /dev/sda always return the state "standby" (even if it is actually running), so, this script cannot detect the drive spinning up.
    I've monitoried dmesg when the drive restarted, but I didn't see any messages about sda, only READ/WRITE blocks on mmcblk0p2.


    In the man page of hdparm, I found the param '-Z', that is supposed to disable automatic power saving modes for some Seagate hard-drive. The reference "ST3xxx models" is mentionned. When I saw that, I thought it should solve the issue, as my hard drive is a Seagate ST3500820AS. I ran hdparm -Z /dev/sda but it still spins up automatically.


    Here are some more information about the hard drive:


    I don't know how to diagnose further this issue. I'm looking for new ideas :)


    Thanks!

  • Hope you have the flashmemory plugin installed.


    Do you smart tests enabled? That could be waking the drive up.

    omv 5.5.17-3 usul | 64 bit | 5.4 proxmox kernel | omvextrasorg 5.4.2
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • Quote

    Hope you have the flashmemory plugin installed.


    It was installed on my previous installed (based on a ready made image but running Debian7). This installed on Dietpi is rather new, and I haven't installed it yet. But I think that DietPi already has some mecanism to reduce wear leveling.


    Here is my FSTAB, by the way:


    Quote

    Do you smart tests enabled? That could be waking the drive up.


    SMART tests are disabled in the administration panel of OMV. Is there any way to check that in the command line to be sure?


    Can I check if there is any scheduled job running of the system? It seems to spin up every 15 or 30 minutes...

  • Here are more informations : I was monitoring the file /var/log/daemon.log while the disk restarted. Here are the lines that appears just at that time :

    Code
    Jun 25 15:44:18 Odroid rrdcached[1229]: flushing old values
    Jun 25 15:44:18 Odroid rrdcached[1229]: rotating journals
    Jun 25 15:44:18 Odroid rrdcached[1229]: started new journal /var/lib/rrdcached/journal/rrd.journal.1466862258.461152
    Jun 25 15:44:18 Odroid rrdcached[1229]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1466855058.461991
    Jun 25 15:45:01 Odroid rrdcached[1229]: Received FLUSHALL


    Could this 'rrdcached' need to read/write to the external hard drive?

  • You have /var/log in tmpfs but you really need /var/lib/rrdcached and /var/lib/monit in tmpfs.


    rrdcached is the statistics/graphs in OMV. You can disable them to see. They shouldn't be causing the drives to spin up since their files live in /var/lib/.


    I would look at cron jobs as well.

    omv 5.5.17-3 usul | 64 bit | 5.4 proxmox kernel | omvextrasorg 5.4.2
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • You have /var/log in tmpfs but you really need /var/lib/rrdcached and /var/lib/monit in tmpfs.


    rrdcached is the statistics/graphs in OMV. You can disable them to see. They shouldn't be causing the drives to spin up since their files live in /var/lib/.


    I would look at cron jobs as well.


    Thanks for the tip, I'll add these directories in tmpfs.


    I disabled monitoring in the web interface, without any result on my issue...


    I analyzed all files in /etc/cron.d.
    The only entries that are executed regularly are

    • openmediavault-rrdtoolgraph, run every 15 minutes
    • php5, run at h09 and h39 (every 30 minutes)

    I moved these files away from cron to check if they cause the issue.
    Then, there are all the files in cron.daily, cron.weekly,... but I assume they are launched only once a day, week...


    Is there any way to log the HDD spinup/down in order to debug this easily?


    Thanks!

  • Don't know. I don't even spin my drives down.


    In my case, the HD is idle most of the time... it doesn't need to run all day when I don't need it.


    I did several tests with inotifywait on various location (/dev/sda, /dev/sda1, /media/mountpoint ,...). It detects an event on /dev/sda just when the HD spins up, but not on other locations. Here is the output :

    Code
    root@odroid:~# inotifywait -m /dev/sda
    Setting up watches.
    Watches established.
    /dev/sda OPEN
    /dev/sda CLOSE_NOWRITE,CLOSE


    So, it seems that something reads "something" on /dev/sda. But I assume that it is lower level than the filesystem (which is EXT4, btw).
    Any idea on what could access the device directly every 1/2h ?


    Thanks!

  • Try watching the mountpoint (/media/REPLACE_WITH_UUID) instead the drive itself. I think it should show you the file/folder being accessed then.

    omv 5.5.17-3 usul | 64 bit | 5.4 proxmox kernel | omvextrasorg 5.4.2
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • Unfortunately, watching the mountpoint shows nothin:

    Code
    root@odroid:~# inotifywait -m /media/36bc1b34-a3a9-4f53-ab75-c7d449e5f5e4
    Setting up watches.
    Watches established.


    But, inotifywait launched at the same time of the previous one shows this, just at the time the drive wakes up:

    Code
    root@odroid:~# inotifywait -m /dev/sda
    Setting up watches.
    Watches established.
    /dev/sda OPEN
    /dev/sda CLOSE_NOWRITE,CLOSE


    But it doesn't show any clue about the file or folder that is accessed :/


    I'll see if lsof can be of any help...


    EDIT : no, lsod doesn't see anything...

  • I don't think a file or folder is being accessed. Something seems to be just querying the drive. Not sure what that would be.

    omv 5.5.17-3 usul | 64 bit | 5.4 proxmox kernel | omvextrasorg 5.4.2
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • Yep, I came to the same conclusion. But I still don't know how to find what is querying the drive....


    On the previous installation, based on the official OMV image for the XU4, this didn't happen, so, it's possible to prevent these wake-ups... I just don't know how!

  • Well, I was digging a little bit deeper into the system and disabled temporarily the services collectd and monit. But this wasn't the clue.
    Then I read the great how to including test script in the following thread:
    My Guide to Debugging Disk Spin-ups


    So I have found, that "parted" was checking my devices over and over again. Unfortunately I was not able to discover which process was triggering parted. I've simply disabled the system tool via renaming it inside /sbin.


    I'm still curious, what's behind this process. So if anybody has an idea I'd be glad to read it here..

  • You renamed parted? That isn't the direction I would go. parted is not a service. Something is calling it and it isn't being called on my systems. I can't find any calls to parted in the OMV source code.

    omv 5.5.17-3 usul | 64 bit | 5.4 proxmox kernel | omvextrasorg 5.4.2
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • I totally agrre that's renaming parted is not the solution. But for a test if the hard drives stay spinned down, it was a good idea. Next thing is to find out, what process is calling parted over and over. And it is good to know, that it isn't a part of omv.
    When I've found out what is going on, I'll write it here.

  • Meanwhile I was abble to find out, which process has started parted every 5 minutes. It is an internal cron-job of webmin. :-( I've installed it a year ago. I dont't know if my problem with the hard drives spinning up was caused by an update of webmin. Anyway. If you go to the webmin configuration and there to the entry "Hintergrund-Status Sammlung" / background status collection, then you will find the reason for all the stuff: "Sammelt Festplattentemperaturen?" / Collecting hard drive temperatures. Changing this to "no" and the drives keep in power safe mode.


    How did I found out, what process causes my troubles? I changend /sbin/parted to a little script:
    #!/bin/sh
    ps -f $PPID >> /var/log/partedcall.txt
    sleep 300


    The sleep command was useful to keep the process alive which was calling parted. With the parent process id I was able to get the calling process name:
    ps -p <ppid from partedcall.txt> -o comm=


    And here ends my story - finally. :-)

  • Good to know. I don't install webmin on OMV boxes because it seems redundant in my opinion. Glad you found the issue. I'm sure it will help some of the other webmin users.

    omv 5.5.17-3 usul | 64 bit | 5.4 proxmox kernel | omvextrasorg 5.4.2
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • I had exactly the same issue with external disks waking up every 30 minutes. Both of them at once.
    I'm using "Armbian Stretch 4.14.y" with the newest version of OMV on Odroid XU4.


    I was trying to detect what process accesses my external drives, using the following commands:
    inotifywait -r -m /srv/dev-disk-by-label-EXTERNAL8TB/ /srv/dev-disk-by-label-EXTERNAL4TB/
    lsof -r1 | grep -E '(EXTERNAL8TB|EXTERNAL4TB)'


    But it didn't show anything when disks started spinning up for the next time.
    In OMV Control Panel -> Storage -> S.M.A.R.T, "Enable" switch was off.


    However, smartd was running:
    sudo service smartd status


    So I stopped and disabled it:
    sudo systemctl disable smartd.service
    sudo service smartd stop


    And no more wake-ups :)


    I know it's an old topic, but I hope someone will find this information useful.

  • Similar problem as well, thanks for the tip ! I'm fairly confident its started since i updated to 4.1.19-1 (using a Pi & external USB HDD)


    My SMART is also disabled in the Web UI but when I checked the status it says "Active (running)". My setup is very low demand (Plex & a daily backup @ night) so I've noticed the HDD hum on during the day when usually its spun down. Advanced power management is set to 1. with a 5min spin down time.


    Is this the same sort of activity you were seeing @Moan before you disabled SMART ?



    root@OMV:~# sudo service smartd status
    ● smartd.service - Self Monitoring and Reporting Technology (SMART) Daemon
    Loaded: loaded (/lib/systemd/system/smartd.service; enabled; vendor preset: enabled)
    Active: active (running) since Sun 2019-02-10 08:58:17 EST; 1h 19min ago
    Docs: man:smartd(8)
    man:smartd.conf(5)
    Main PID: 850 (smartd)
    CPU: 370ms
    CGroup: /system.slice/smartd.service
    └─850 /usr/sbin/smartd -n
    Feb 10 08:58:18 OMV smartd[850]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
    Feb 10 08:58:18 OMV smartd[850]: Device: /dev/sda [SAT], opened
    Feb 10 08:58:18 OMV smartd[850]: Device: /dev/sda [SAT], WDC WD40NMZW-59GX6S1, S/N:WD-WXT1EC76UW5M, WWN:5-0014ee-65df357bb, FW:01.01A01, 4.00 TB
    Feb 10 08:58:18 OMV smartd[850]: Device: /dev/sda [SAT], found in smartd database: Western Digital Elements / My Passport (USB, AF)
    Feb 10 08:58:18 OMV smartd[850]: Device: /dev/sda [SAT], is SMART capable. Adding to "monitor" list.
    Feb 10 08:58:18 OMV smartd[850]: Device: /dev/sda [SAT], state read from /var/lib/smartmontools/smartd.WDC_WD40NMZW_59GX6S1-WD_WXT1EC76UW5M.ata.state
    Feb 10 08:58:18 OMV smartd[850]: Monitoring 1 ATA/SATA, 0 SCSI/SAS and 0 NVMe devices
    Feb 10 08:58:18 OMV smartd[850]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 118 to 117
    Feb 10 08:58:18 OMV smartd[850]: Device: /dev/sda [SAT], state written to /var/lib/smartmontools/smartd.WDC_WD40NMZW_59GX6S1-WD_WXT1EC76UW5M.ata.state
    Feb 10 09:28:26 OMV smartd[850]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 117 to 118

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!