My Guide to Debugging Disk Spin-ups

Reed · Dec 29th 2015

I was having some difficulty with a disk spinning up periodically and in the process of debugging it I created a small shell script which others might find useful if they have the same problem.

What it does:
- It writes out buffered data to the disk
- Enables Block Dump Reporting
- Puts the Drive into Standby (spin-down)
- Then sits in a while loop waiting for the drive to spin back up.
- When it does spin up it turns off the reporting and displays the messages relating to the drive you are watching.

Run: find_culprit <drive>
Example: find_culprit sdc
(/dev/sdc is my spin disk)

Bash

#!/bin/sh
#
# Discover what is writing to disk
#


set -e


DISK=${1-sdc}
HDPARM_CMD="/sbin/hdparm -C /dev/$DISK"
SLEEP_TIME=45


# Sync to flush the cache to disk
sync


# Enable Block Dump Reporting
echo 1 > /proc/sys/vm/block_dump


# Clear old messages
dmesg -c > /dev/null


echo "Putting the disk $DISK into standby..."
hdparm -y /dev/$DISK > /dev/null
sleep 3;


# Check the drive/sleep until it is awake
echo "Checking the status of $DISK."
HDPARM_RES=$($HDPARM_CMD|grep drive|cut -c 19-25)
while [ "$HDPARM_RES" = "standby" ]; do
  echo "Drive is still in standby.  Sleeping $SLEEP_TIME seconds..."
  sleep $SLEEP_TIME;
  HDPARM_RES=$($HDPARM_CMD|grep drive|cut -c 19-25)
done
echo "Drive is awake!"


# Turn off block dump
echo 0 > /proc/sys/vm/block_dump


# Display the accesses that awoke the disk
dmesg | grep $DISK

Display More

Output looks like this:

Code

root@openmediavault:~# ./find_culprit
Putting the disk sdc into standby...
Checking the status of sdc.
Drive is still in standby.  Sleeping 45 seconds...
Drive is still in standby.  Sleeping 45 seconds...
Drive is still in standby.  Sleeping 45 seconds...
...
Drive is still in standby.  Sleeping 45 seconds...
Drive is still in standby.  Sleeping 45 seconds...
Drive is still in standby.  Sleeping 45 seconds...
Drive is awake!
[24818.951861] cp(27268): dirtied inode 19005626 (Plex Media Server.log) on sdc1
[24818.951925] cp(27268): dirtied inode 19005626 (Plex Media Server.log) on sdc1
[24818.953546] cp(27268): WRITE block 278272 on sdc1 (256 sectors)
[24818.954386] cp(27268): dirtied inode 19005465 (com.plexapp.system.log) on sdc1
[24818.954441] cp(27268): dirtied inode 19005465 (com.plexapp.system.log) on sdc1
[24818.954962] cp(27268): WRITE block 608649568 on sdc1 (64 sectors)
[24818.956155] cp(27268): dirtied inode 19005451 (PMS Plugin Logs) on sdc1
[24818.956458] cp(27268): dirtied inode 19005448 (Logs) on sdc1
[24824.467173] jbd2/sdc1-8(1464): WRITE block 625215344 on sdc1 (8 sectors)
[24829.539190] flush-8:32(27274): WRITE block 0 on sdc1 (8 sectors)
[24829.539248] flush-8:32(27274): WRITE block 8 on sdc1 (8 sectors)
[24829.539279] flush-8:32(27274): WRITE block 296 on sdc1 (8 sectors)
[24829.539311] flush-8:32(27274): WRITE block 8808 on sdc1 (8 sectors)
[24829.539428] flush-8:32(27274): WRITE block 608174424 on sdc1 (8 sectors)

Display More

So now I can see that the files:
- Plex Media Server.log (in Logs)
- com.plexapp.system.log (in PMS Plugin Logs)

Were modified.

In this case it was actually sync'ing the RAM directories back to the physical disk that woke it up (daily cron job). But this helps automate the work of finding what is accessing your disk and waking it up.

villeneuve · Jan 13th 2017

I tested your script, which is a great idea, as I have disks waking up for no apparent reason, but the only thing it says when the disk spins up is "Drive is awake"! There are no further messages. Could it be that the script doesn't work on a system also running the flashmemory-plugin?

ryecoaaron · Jan 14th 2017

flashmemory plugin shouldn't cause issues with that script.

villeneuve · Jan 14th 2017

Hm, ok, so any ideas why the script just ends with "Drive is awake"? BTW I'm running OMV 2.2.13 with the backport kernel.

ryecoaaron · Jan 14th 2017

Quote from villeneuve

Hm, ok, so any ideas why the script just ends with "Drive is awake"? BTW I'm running OMV 2.2.13 with the backport kernel.

hdparm either isn't putting the drive to sleep or it wakes back up within the 3 second delay that the script waits before it starts checking if the drive is in standby. I would say something is constantly using your drive.

villeneuve · Jan 14th 2017

No, no, it definitely puts the drive to sleep. Before the script gave the message "Drive is awake" it's saying "Drive is still in standby. Sleeping 45 seconds..." multiple times for half an hour or so.

ryecoaaron · Jan 14th 2017

Oh, you made it sound like it went to Drive is awake right away. What is the output of dmesg | grep -E "WRITE|dirtied"

villeneuve · Jan 17th 2017

The output is:

Code

[79302.183422] smbd(31521): dirtied inode 141885503 (index) on sdb1
[79302.324836] smbd(31521): dirtied inode 141885504 (info) on sdb1
[79302.335551] smbd(31521): dirtied inode 141885505 (marks) on sdb1
[79302.344334] smbd(31521): dirtied inode 141885506 (resume) on sdb1

sda is the drive I'm investigating why it's waking up. sda and sdb are both 8 TB Seagate Archive HDDs, sdc is the drive OMV is installed on and that is a USB-stick.

ryecoaaron · Jan 17th 2017

No idea why it is waking up then. Do you have smart tests enabled?

villeneuve · Jan 17th 2017

Yes, I have set "Check interval" to the default 1800 and "Power Mode" to "Standby", but I have not scheduled tests.
However there is a problem with the Seagate Archive disks that often results in hdparm -C /dev/sdx giving the result "drive state is: unknown". Maybe that is the cause for my problem, so that smartmontools wakes up the drive regardless of what is set as "Power Mode". The output of hdparm -C /dev/sdx is the same on both drives, but the frequent spin-ups seem to only happen on sda, which makes me wonder about a firmware-problem, because sda runs AR13 firmware and sdb is equipped with AR15. Seagate seem to offer updates to AR17 now, so I'll try to get it for both drives and see if the problem persists. Thanks!

mrperfektone · Sep 18th 2017

Hi and thanks for your script - If I have two hardrive in a dual usb case with the Sda / Sdb name, how should scripted be? - So it has an impact on both hard drives?

Sorry for my bad English

Thanks in advance

Mvh Rene

Raistlin666 · Nov 15th 2019

Quote from Reed

Run: find_culprit <drive>
Example: find_culprit sdc
(/dev/sdc is my spin disk)

dummy question here:

how do I run this script?
entering it on the omv console at my pi 4 only shows "Could not find the database of available applications, run update-command-not-found as root to fix this"

I'm new to this (windows user)

macom · Nov 15th 2019

https://www.cyberciti.biz/faq/howto-run-a-script-in-linux/

Raistlin666 · Nov 15th 2019

cant even figure out how to start?
what do i have to do? I have to enter the scripts name. Soo do I have to save your post and upload it to the omv or what?^^

macom · Nov 15th 2019

Have a look at the guide which is linked in the signature. You need to use the CLI. If you are on Windows that means you need to use Putty (or powershell) to ssh into your server. Hope this becomes clearer after you read the section about Putty in the guide.

Then you need to ssh into the server. Create a file using a editor called nano. It is already installed on your RPi. Copy the content of the script into that file. Safe the file. Then continue with the link I posted.

OMV is based on Debian which is a standard Linux distribution. You find a lot support using google (or startpage or yahoo or whatever).

Raistlin666 · Nov 15th 2019

Ok can you please tell me where to look at the pdf?
I used putty before but it dont know how to use nano

macom · Nov 15th 2019

ok, might be nano is not included in the guide
https://linuxize.com/post/how-to-use-nano-text-editor/

Raistlin666 · Nov 15th 2019

Ah nano is an editor..
Got your code in there, but what do i enter after pressing save? some directory?

Raistlin666 · Nov 15th 2019

ok so i named it spindown.sh now
when i follow your guide on how to run a script, the first 2 options dont do anything and after that its all
"Could not find the database of available applications, run update-command-not-found as root to fix this"
again

Sorry to bother you but I dont get it..

villeneuve · Nov 15th 2019

The file is saved in that directory where you were when you started nano unless you specified a directory when starting nano.
So for example launching nano with

Code

nano /tmp/test.txt

would put that text-file into the "/tmp"-directory.
Beware that /tmp usually gets deleted on shutdown/reboots of the system.

Participate now!