(HowTo) avoid Autoshutdown while a ZFS scub is running

Hi folks,

I have the problem that my NAS is executing a shutdown while a ZFS scrub is running. I tried to change several settings in the autoshutdown-plugin, but I wasn´t successful.

• I have activated 'load average'. But in the autoshutdown log I get 'load average is lower than (40)' => Shutdown
  BTW: I have no idea what '40' (the default setting) means.
• I have activated 'HDD-IO', because a ZFS scrub generates a heavy disk load. But in the autoshutdown log only /dev/sda (the boot drive) is mentioned. It seems that autoshutdown isn´t respecting a ZFS scrub as HDD-IO. This activity is not "seen" by the plugin.
• Then I wanted to check for a running process. But there is no special process which is responsible for the scrubbing. It is started with zpool scrub <poolname> which terminates immediately.

So I didn´t find a solution to avoid a shutdown while a zfs scrub is running.

I would be happy If someone has an idea how to configure autoshutdown to avoid this.

Thanks in advance.

###################################################################################

Edit 2017-08-17:
Meanwhile I found a solution and wrote a small HowTo, which can be found in post no. 8 to no. 10.

Quote from subzero79

How do you check the status of a running scrub?

I did it by TEMPPROCNAMES="txg_sync" in the autoshutdown-plugin settings. txg_sync was shown with a high CPU utilization in 'top'. But later on I saw that this process is active too, when no scrub is running. Therefore the NAS then was never going into standby at all.

Quote from subzero79

That in combination with the ZED (zfs events daemon) should give you a formula for preventing as to trigger while a scrub is running.

I have read a little bit about ZED. I have understood, that if some specfic ZFS events are coming, a script can be executed. But I do not see, how to use ZED in the autoshutdown settings. Unfortunately I am not a great linux shell programmer.

Quote from subzero79

you should start learning, reading and practicing.

For personal computers I am doing this since 1987 And I did a lot of reading here in this forum too - and the process is ongoing. However I am not a giftet coder - and I will never get one!
Linux shell programming is great: All this nested commands, where the output of one command is piped into the next command in one line! Wonderfull, but beyond of my abilities.

Quote from subzero79

So you create the file when the scrub starts (via zed event), then remove the file when the scrub ends.

Thank you for your advice. I will try to implement this.

Ok, first step is done.

The relevant ZFS events are:
Aug 6 2017 17:43:21.192843043 ereport.fs.zfs.scrub.start
Aug 6 2017 17:45:50.296982936 ereport.fs.zfs.scrub.finish

For the second event there is already a so called "ZEDLET" available: /etc/zfs/zed.d/scrub.finish-notify.sh
For the first event I have created a new one /etc/zfs/zed.d/scrub.start-notify.sh, which is a copy of the other one with some minor modifcations.

Currently both scripts are generating an email when the scrub begins and finishes.

Next step will be to modify the scripts to handle a lock file, which shall be evaluated by the autoshutdown plugin.

Ok, I did it! Now there is no more autoshutdown, while a ZFS scrub is running. It is based on the ZED (ZFS event daemon).

Therefore I will write a small Howto with my installation notes.

Some documentation:

• Anyone managed to get ZFS Event Daemon up and running?
• zed man page
• The ZFS Event Daemon on Linux - Louwrentius
• List of zfs-events

Approach: Create a lockfile by zed while a ZFS scrub is running. Use autoshutdown-plugin 'plugincheck'-option to detect the lockfile and prevent shutdown.

The next steps are describing, what I did. I am not denying that there may be a better way.

Part 1: Configuring zed to start a script each at scrub start and scrub finish.

• Edit ZED resource file /etc/zfs/zed.d/zed.rc
  Add a new line:
  ZED_NOTIFY_VERBOSE=1
  In verbose mode an email is sent, when a scrub starts and when it finishes. Not absolutely neccessary, but maybe useful for troubleshooting.
• A list of major zfs-events (ZEVENT) can be found in that link above. The relevant ZEVENT for this use case is 'scrub.start' and 'scrub.finish'.
• ZETLET
  ZEDLETs are executables invoked by the ZED in response to a given ZEVENT. There are two important folders:
  'Installed' ZETLET folder: /usr/lib/x86_64-linux-gnu/zfs/zed.d/
  'Enabled' ZETLET folder: /etc/zfs/zed.d/
  Hint: 'Enabled' ZETLETs are created by a symlink to files in the 'Installed' ZETLET folder.
• By default there is only a ZETLET for the scrub.finish zevent: /usr/lib/x86_64-linux-gnu/zfs/zed.d/scrub.finish-notify.sh
  To get a ZETLET for the scrub.start zevent, I made a copy with a new name first: /usr/lib/x86_64-linux-gnu/zfs/zed.d/scrub.start-notify.sh
• Then it is necessary to create a symlink in the 'Enabled' ZETLET folder:
  ln -s /usr/lib/x86_64-linux-gnu/zfs/zed.d/scrub.start-notify.sh /etc/zfs/zed.d/
• To be on the safe side restart zed
  systemctl restart zed
  

  
  

  => Now when a scrub starts the script 'scrub.start-notify.sh' is executed, when it has finished then 'scrub.finish-notify.sh'.

  
  

  End of part 1
  
  

Part 2: Modify 'scrub.start-notify.sh' and 'scrub.finish-notify.sh' to handle a lockfile

• Modifications on 'scrub.start-notify.sh'
  - This file was derived from 'scrub.start-notify.sh'. Therefore all strings "resilver.finish" and "scrub.finish" were replaced by "resilver.start" and "scrub.start"
  - Add some lines of code which creates a lockfile in /tmp according to the file name pattern '<ZEVENT>.<poolname>' eg. 'scrub.start.datapool'
• Modifications on 'scrub.finish-notify.sh'
  - Add some lines of code which removes the lockfile '<ZEVENT>.<poolname>' from /tmp when the ZEVENt has finished.

End of Part 2

Part 3: Configure autoshutdown-plugin to detect the lockfile

In the 'Expert Settings' of the autoshutdown-plugin the 'plugincheck' has to be activated.

• Create a so-called plugin file in /etc/autoshutdown.d
  In this folder there are still several examples. I have used 'autounrarstatus" as a template to detect the lockfile with the content 'ZFS has started a action!'.
  I named the copied file 'scrub.datapool". The name is not relevant, only the content of the file. Three lines are important:
  folder="/tmp"
  file="scrub.start.datapool"
  content="ZFS has started a action!"
  

  
  

  Note: The entry for 'file=' must match with the used ZFS pool name. In the example the ZEVENT is 'scrub.start' and the poolname is 'datapool'. Replace it by your poolname.
  

• Last step:
  In the autoshutdown-plugin in the 'Expert Settings -> Extra options' add a new line:
  PLUGINCHECK="true"
  

  
  

  When the autoshutdown-plugin is detecting the file /tmp/scrub.start.datapool and the content is 'ZFS has started a action!' then no shutdown is initiated as long as the file is present. If the scrub has finished then the ZED removes the file and the autoshutdown can be initiated some times later.