(HowTo) avoid Autoshutdown while a ZFS scub is running

  • Hi folks,


    I have the problem that my NAS is executing a shutdown while a ZFS scrub is running. I tried to change several settings in the autoshutdown-plugin, but I wasn´t successful.

    • I have activated 'load average'. But in the autoshutdown log I get 'load average is lower than (40)' => Shutdown
      BTW: I have no idea what '40' (the default setting) means.
    • I have activated 'HDD-IO', because a ZFS scrub generates a heavy disk load. But in the autoshutdown log only /dev/sda (the boot drive) is mentioned. It seems that autoshutdown isn´t respecting a ZFS scrub as HDD-IO. This activity is not "seen" by the plugin.
    • Then I wanted to check for a running process. But there is no special process which is responsible for the scrubbing. It is started with zpool scrub <poolname> which terminates immediately.

    So I didn´t find a solution to avoid a shutdown while a zfs scrub is running.


    I would be happy If someone has an idea how to configure autoshutdown to avoid this.


    Thanks in advance.


    ###################################################################################


    Edit 2017-08-17:
    Meanwhile I found a solution and wrote a small HowTo, which can be found in post no. 8 to no. 10.

    OMV 3.0.100 (Gray style)

    ASRock Rack C2550D4I C0-stepping - 16GB ECC - 6x WD RED 3TB (ZFS 2x3 Striped RaidZ1) - Fractal Design Node 304 -

    3x WD80EMAZ Snapraid / MergerFS-pool via eSATA - 4-Bay ICYCube MB561U3S-4S with fan-mod

    2 Mal editiert, zuletzt von cabrio_leo ()

  • How do you check the status of a running scrub?

    I did it by TEMPPROCNAMES="txg_sync" in the autoshutdown-plugin settings. txg_sync was shown with a high CPU utilization in 'top'. But later on I saw that this process is active too, when no scrub is running. Therefore the NAS then was never going into standby at all.



    That in combination with the ZED (zfs events daemon) should give you a formula for preventing as to trigger while a scrub is running.

    I have read a little bit about ZED. I have understood, that if some specfic ZFS events are coming, a script can be executed. But I do not see, how to use ZED in the autoshutdown settings. Unfortunately I am not a great linux shell programmer.

    OMV 3.0.100 (Gray style)

    ASRock Rack C2550D4I C0-stepping - 16GB ECC - 6x WD RED 3TB (ZFS 2x3 Striped RaidZ1) - Fractal Design Node 304 -

    3x WD80EMAZ Snapraid / MergerFS-pool via eSATA - 4-Bay ICYCube MB561U3S-4S with fan-mod

    Einmal editiert, zuletzt von cabrio_leo ()

    • Offizieller Beitrag

    Unfortunately I am not a great linux shell programmer

    You're managing a zfs server, you should start learning, reading and practicing. Basically the as plugin can check for a file (is called a lock file) if the file is present, won't go down, if the file doesn't exist it will go down.


    So you create the file when the scrub starts (via zed event), then remove the file when the scrub ends.

  • you should start learning, reading and practicing.

    For personal computers I am doing this since 1987 :) And I did a lot of reading here in this forum too - and the process is ongoing. However I am not a giftet coder - and I will never get one!
    Linux shell programming is great: All this nested commands, where the output of one command is piped into the next command in one line! Wonderfull, but beyond of my abilities.


    So you create the file when the scrub starts (via zed event), then remove the file when the scrub ends.

    Thank you for your advice. I will try to implement this.

    OMV 3.0.100 (Gray style)

    ASRock Rack C2550D4I C0-stepping - 16GB ECC - 6x WD RED 3TB (ZFS 2x3 Striped RaidZ1) - Fractal Design Node 304 -

    3x WD80EMAZ Snapraid / MergerFS-pool via eSATA - 4-Bay ICYCube MB561U3S-4S with fan-mod

  • Ok, first step is done.


    The relevant ZFS events are:
    Aug 6 2017 17:43:21.192843043 ereport.fs.zfs.scrub.start
    Aug 6 2017 17:45:50.296982936 ereport.fs.zfs.scrub.finish


    For the second event there is already a so called "ZEDLET" available: /etc/zfs/zed.d/scrub.finish-notify.sh
    For the first event I have created a new one /etc/zfs/zed.d/scrub.start-notify.sh, which is a copy of the other one with some minor modifcations.


    Currently both scripts are generating an email when the scrub begins and finishes.


    Next step will be to modify the scripts to handle a lock file, which shall be evaluated by the autoshutdown plugin.

    OMV 3.0.100 (Gray style)

    ASRock Rack C2550D4I C0-stepping - 16GB ECC - 6x WD RED 3TB (ZFS 2x3 Striped RaidZ1) - Fractal Design Node 304 -

    3x WD80EMAZ Snapraid / MergerFS-pool via eSATA - 4-Bay ICYCube MB561U3S-4S with fan-mod

  • Ok, I did it! Now there is no more autoshutdown, while a ZFS scrub is running. It is based on the ZED (ZFS event daemon).


    Therefore I will write a small Howto with my installation notes.


    Some documentation:

    Approach: Create a lockfile by zed while a ZFS scrub is running. Use autoshutdown-plugin 'plugincheck'-option to detect the lockfile and prevent shutdown.


    The next steps are describing, what I did. I am not denying that there may be a better way.


    Part 1: Configuring zed to start a script each at scrub start and scrub finish.

    • Edit ZED resource file /etc/zfs/zed.d/zed.rc
      Add a new line:
      ZED_NOTIFY_VERBOSE=1
      In verbose mode an email is sent, when a scrub starts and when it finishes. Not absolutely neccessary, but maybe useful for troubleshooting.
    • A list of major zfs-events (ZEVENT) can be found in that link above. The relevant ZEVENT for this use case is 'scrub.start' and 'scrub.finish'.
    • ZETLET
      ZEDLETs are executables invoked by the ZED in response to a given ZEVENT. There are two important folders:
      'Installed' ZETLET folder: /usr/lib/x86_64-linux-gnu/zfs/zed.d/
      'Enabled' ZETLET folder: /etc/zfs/zed.d/
      Hint: 'Enabled' ZETLETs are created by a symlink to files in the 'Installed' ZETLET folder.
    • By default there is only a ZETLET for the scrub.finish zevent: /usr/lib/x86_64-linux-gnu/zfs/zed.d/scrub.finish-notify.sh
      To get a ZETLET for the scrub.start zevent, I made a copy with a new name first: /usr/lib/x86_64-linux-gnu/zfs/zed.d/scrub.start-notify.sh
    • Then it is necessary to create a symlink in the 'Enabled' ZETLET folder:
      ln -s /usr/lib/x86_64-linux-gnu/zfs/zed.d/scrub.start-notify.sh /etc/zfs/zed.d/
    • To be on the safe side restart zed
      systemctl restart zed


      => Now when a scrub starts the script 'scrub.start-notify.sh' is executed, when it has finished then 'scrub.finish-notify.sh'.


      End of part 1

    OMV 3.0.100 (Gray style)

    ASRock Rack C2550D4I C0-stepping - 16GB ECC - 6x WD RED 3TB (ZFS 2x3 Striped RaidZ1) - Fractal Design Node 304 -

    3x WD80EMAZ Snapraid / MergerFS-pool via eSATA - 4-Bay ICYCube MB561U3S-4S with fan-mod

    8 Mal editiert, zuletzt von cabrio_leo ()

  • Part 2: Modify 'scrub.start-notify.sh' and 'scrub.finish-notify.sh' to handle a lockfile



    • Modifications on 'scrub.start-notify.sh'
      - This file was derived from 'scrub.start-notify.sh'. Therefore all strings "resilver.finish" and "scrub.finish" were replaced by "resilver.start" and "scrub.start"
      - Add some lines of code which creates a lockfile in /tmp according to the file name pattern '<ZEVENT>.<poolname>' eg. 'scrub.start.datapool'
    • Modifications on 'scrub.finish-notify.sh'
      - Add some lines of code which removes the lockfile '<ZEVENT>.<poolname>' from /tmp when the ZEVENt has finished.


    End of Part 2


  • Part 3: Configure autoshutdown-plugin to detect the lockfile

    In the 'Expert Settings' of the autoshutdown-plugin the 'plugincheck' has to be activated.


    • Create a so-called plugin file in /etc/autoshutdown.d
      In this folder there are still several examples. I have used 'autounrarstatus" as a template to detect the lockfile with the content 'ZFS has started a action!'.
      I named the copied file 'scrub.datapool". The name is not relevant, only the content of the file. Three lines are important:
      folder="/tmp"
      file="scrub.start.datapool"
      content="ZFS has started a action!"


      Note: The entry for 'file=' must match with the used ZFS pool name. In the example the ZEVENT is 'scrub.start' and the poolname is 'datapool'. Replace it by your poolname.

    • Last step:
      In the autoshutdown-plugin in the 'Expert Settings -> Extra options' add a new line:
      PLUGINCHECK="true"


      When the autoshutdown-plugin is detecting the file /tmp/scrub.start.datapool and the content is 'ZFS has started a action!' then no shutdown is initiated as long as the file is present. If the scrub has finished then the ZED removes the file and the autoshutdown can be initiated some times later.

    Dateien

    OMV 3.0.100 (Gray style)

    ASRock Rack C2550D4I C0-stepping - 16GB ECC - 6x WD RED 3TB (ZFS 2x3 Striped RaidZ1) - Fractal Design Node 304 -

    3x WD80EMAZ Snapraid / MergerFS-pool via eSATA - 4-Bay ICYCube MB561U3S-4S with fan-mod

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!