Pinned My Guide to Debugging Disk Spin-ups

    • OMV 2.x

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • My Guide to Debugging Disk Spin-ups

      I was having some difficulty with a disk spinning up periodically and in the process of debugging it I created a small shell script which others might find useful if they have the same problem.

      What it does:
      - It writes out buffered data to the disk
      - Enables Block Dump Reporting
      - Puts the Drive into Standby (spin-down)
      - Then sits in a while loop waiting for the drive to spin back up.
      - When it does spin up it turns off the reporting and displays the messages relating to the drive you are watching.


      Run: find_culprit <drive>
      Example: find_culprit sdc
      (/dev/sdc is my spin disk)

      Shell-Script

      1. #!/bin/sh
      2. #
      3. # Discover what is writing to disk
      4. #
      5. set -e
      6. DISK=${1-sdc}
      7. HDPARM_CMD="/sbin/hdparm -C /dev/$DISK"
      8. SLEEP_TIME=45
      9. # Sync to flush the cache to disk
      10. sync
      11. # Enable Block Dump Reporting
      12. echo 1 > /proc/sys/vm/block_dump
      13. # Clear old messages
      14. dmesg -c > /dev/null
      15. echo "Putting the disk $DISK into standby..."
      16. hdparm -y /dev/$DISK > /dev/null
      17. sleep 3;
      18. # Check the drive/sleep until it is awake
      19. echo "Checking the status of $DISK."
      20. HDPARM_RES=$($HDPARM_CMD|grep drive|cut -c 19-25)
      21. while [ "$HDPARM_RES" = "standby" ]; do
      22. echo "Drive is still in standby. Sleeping $SLEEP_TIME seconds..."
      23. sleep $SLEEP_TIME;
      24. HDPARM_RES=$($HDPARM_CMD|grep drive|cut -c 19-25)
      25. done
      26. echo "Drive is awake!"
      27. # Turn off block dump
      28. echo 0 > /proc/sys/vm/block_dump
      29. # Display the accesses that awoke the disk
      30. dmesg | grep $DISK
      Display All


      Output looks like this:

      Source Code

      1. root@openmediavault:~# ./find_culprit
      2. Putting the disk sdc into standby...
      3. Checking the status of sdc.
      4. Drive is still in standby. Sleeping 45 seconds...
      5. Drive is still in standby. Sleeping 45 seconds...
      6. Drive is still in standby. Sleeping 45 seconds...
      7. ...
      8. Drive is still in standby. Sleeping 45 seconds...
      9. Drive is still in standby. Sleeping 45 seconds...
      10. Drive is still in standby. Sleeping 45 seconds...
      11. Drive is awake!
      12. [24818.951861] cp(27268): dirtied inode 19005626 (Plex Media Server.log) on sdc1
      13. [24818.951925] cp(27268): dirtied inode 19005626 (Plex Media Server.log) on sdc1
      14. [24818.953546] cp(27268): WRITE block 278272 on sdc1 (256 sectors)
      15. [24818.954386] cp(27268): dirtied inode 19005465 (com.plexapp.system.log) on sdc1
      16. [24818.954441] cp(27268): dirtied inode 19005465 (com.plexapp.system.log) on sdc1
      17. [24818.954962] cp(27268): WRITE block 608649568 on sdc1 (64 sectors)
      18. [24818.956155] cp(27268): dirtied inode 19005451 (PMS Plugin Logs) on sdc1
      19. [24818.956458] cp(27268): dirtied inode 19005448 (Logs) on sdc1
      20. [24824.467173] jbd2/sdc1-8(1464): WRITE block 625215344 on sdc1 (8 sectors)
      21. [24829.539190] flush-8:32(27274): WRITE block 0 on sdc1 (8 sectors)
      22. [24829.539248] flush-8:32(27274): WRITE block 8 on sdc1 (8 sectors)
      23. [24829.539279] flush-8:32(27274): WRITE block 296 on sdc1 (8 sectors)
      24. [24829.539311] flush-8:32(27274): WRITE block 8808 on sdc1 (8 sectors)
      25. [24829.539428] flush-8:32(27274): WRITE block 608174424 on sdc1 (8 sectors)
      Display All


      So now I can see that the files:
      - Plex Media Server.log (in Logs)
      - com.plexapp.system.log (in PMS Plugin Logs)

      Were modified.

      In this case it was actually sync'ing the RAM directories back to the physical disk that woke it up (daily cron job). But this helps automate the work of finding what is accessing your disk and waking it up.
    • I tested your script, which is a great idea, as I have disks waking up for no apparent reason, but the only thing it says when the disk spins up is "Drive is awake"! There are no further messages. Could it be that the script doesn't work on a system also running the flashmemory-plugin?

      The post was edited 1 time, last by villeneuve ().

    • villeneuve wrote:

      Hm, ok, so any ideas why the script just ends with "Drive is awake"? BTW I'm running OMV 2.2.13 with the backport kernel.
      hdparm either isn't putting the drive to sleep or it wakes back up within the 3 second delay that the script waits before it starts checking if the drive is in standby. I would say something is constantly using your drive.
      omv 4.0.11 arrakis | 64 bit | 4.13 backports kernel | omvextrasorg 4.1.0
      omv-extras.org plugins source code and issue tracker - github.com/OpenMediaVault-Plugin-Developers

      Please don't PM for support... Too many PMs!
    • Oh, you made it sound like it went to Drive is awake right away. What is the output of dmesg | grep -E "WRITE|dirtied"
      omv 4.0.11 arrakis | 64 bit | 4.13 backports kernel | omvextrasorg 4.1.0
      omv-extras.org plugins source code and issue tracker - github.com/OpenMediaVault-Plugin-Developers

      Please don't PM for support... Too many PMs!
    • The output is:

      Source Code

      1. [79302.183422] smbd(31521): dirtied inode 141885503 (index) on sdb1
      2. [79302.324836] smbd(31521): dirtied inode 141885504 (info) on sdb1
      3. [79302.335551] smbd(31521): dirtied inode 141885505 (marks) on sdb1
      4. [79302.344334] smbd(31521): dirtied inode 141885506 (resume) on sdb1

      sda is the drive I'm investigating why it's waking up. sda and sdb are both 8 TB Seagate Archive HDDs, sdc is the drive OMV is installed on and that is a USB-stick.
    • Yes, I have set "Check interval" to the default 1800 and "Power Mode" to "Standby", but I have not scheduled tests.
      However there is a problem with the Seagate Archive disks that often results in hdparm -C /dev/sdx giving the result "drive state is: unknown". Maybe that is the cause for my problem, so that smartmontools wakes up the drive regardless of what is set as "Power Mode". The output of hdparm -C /dev/sdx is the same on both drives, but the frequent spin-ups seem to only happen on sda, which makes me wonder about a firmware-problem, because sda runs AR13 firmware and sdb is equipped with AR15. Seagate seem to offer updates to AR17 now, so I'll try to get it for both drives and see if the problem persists. Thanks!