SnapRaid "z-parity" instead of 3-parity

    • OMV 4.x
    • SnapRaid "z-parity" instead of 3-parity

      Hello,

      I apologize in advance if my question has been answered in some form before, or is not located in the correct section.

      The problem I am trying to solve is that, I have an old Supermicro chassis with AMD Opteron processor, and this CPU does not support SSSE3 instructions. That is a major problem for 3 or more parity drives, as SnapRaid uses it to accelerate calculation. At the moment my system with 20 Data drives and 4 parity HDDs takes 4 days to do scrubs etc...

      There is one option for poor schmucks like me, that is to get down to 3 parity drives, and select "z-parity" for the 3rd drive. This "z-parity" option was added in SnapRaid for old CPUs to accelerate 3 parity drive calculation.

      I have not seen this option available on the OMV SnapRaid plugin, maybe I missed it, but I am sure that would be a bit of a headache to implement only to support ancient hardware. Although I wouldn't complain if someone as this to the plug-in.

      So my question is: how can I go about to modify snapraid.conf without making a mess with the plug-in database? And make sure my change won't be reverted by OMV behind my back?

      Thank you!
    • New

      Looking further into the Shell script in charge of creating the SnapRAID configuration file, I didn't see anything about z-parity. So I am planning to modify the script like follows on my instance of server. This is simpler for me to do it this way as I am familiar with Shell script, but not so much with whatever language is used for the GUI. And also z-parity is supposed to be faster than 3-parity no matter what, so I figured I may as well force it when there are only 3 parity drives total.

      path: /usr/share/openmediavault/mkconf/snapraid
      The modification is done in the IF statement creating the lines for the parity files (should be the correct line numbers here under)

      Shell-Script: snapraid (modified)

      1. #EDIT - Something is wrong with this code and I'll update with correction when time allows.
      2. if [ "${countp}" != "1" ]; then
      3. if [ "${parity}" -eq "3" -a "${countp}" -eq "3"]; then
      4. echo "z-parity ${path}/snapraid.z-parity" >> ${SNAPRAID_CONFIG}
      5. else
      6. echo "${countp}-parity ${path}/snapraid.${countp}-parity" >> ${SNAPRAID_CONFIG}
      7. fi
      8. else
      9. echo "parity ${path}/snapraid.parity" >> ${SNAPRAID_CONFIG}
      10. fi
      Display All

      For reference the original version of the file used on my system is as follows:

      Shell-Script: snapraid (original)

      1. #!/bin/sh
      2. #
      3. # @author Volker Theile <volker.theile@openmediavault.org>
      4. # @author OpenMediaVault Plugin Developers <plugins@omv-extras.org>
      5. # @copyright Copyright (c) 2009-2013 Volker Theile
      6. # @copyright Copyright (c) 2013-2017 OpenMediaVault Plugin Developers
      7. #
      8. # This program is free software: you can redistribute it and/or modify
      9. # it under the terms of the GNU General Public License as published by
      10. # the Free Software Foundation, either version 3 of the License, or
      11. # any later version.
      12. #
      13. # This program is distributed in the hope that it will be useful,
      14. # but WITHOUT ANY WARRANTY; without even the implied warranty of
      15. # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
      16. # GNU General Public License for more details.
      17. #
      18. # You should have received a copy of the GNU General Public License
      19. # along with this program. If not, see <http://www.gnu.org/licenses/>.
      20. set -e
      21. . /etc/default/openmediavault
      22. . /usr/share/openmediavault/scripts/helper-functions
      23. OMV_name=$(omv_product_info "name")
      24. OMV_packagename=$(omv_product_info "packagename")
      25. OMV_versionname=$(omv_product_info "versionname")
      26. OMV_version=$(dpkg-query -W -f='${Version}' ${OMV_packagename})
      27. SNAPRAID_FILENAME="snapraid.conf"
      28. SNAPRAID_CONFIG="/etc/${SNAPRAID_FILENAME}"
      29. DIFF_SCRIPT_FILENAME="snapraid-diff.conf"
      30. DIFF_SCRIPT_CONFIG="/etc/${DIFF_SCRIPT_FILENAME}"
      31. SERVICE_XPATH_NAME="snapraid"
      32. SERVICE_XPATH="/config/services/${SERVICE_XPATH_NAME}"
      33. DRIVES="${SERVICE_XPATH}/drives/drive"
      34. RULES="${SERVICE_XPATH}/rules/rule"
      35. PLUGIN_VERSION="$(dpkg -l | grep $OMV_packagename-snapraid | awk '{print $3}')"
      36. # Create snapraid config file
      37. cat <<EOF > ${SNAPRAID_CONFIG}
      38. # this file was automatically generated from
      39. # ${OMV_name} ${OMV_versionname} ${OMV_version}
      40. # and 'openmediavault-snapraid' ${PLUGIN_VERSION}
      41. EOF
      42. xmlstarlet sel -t -m "${SERVICE_XPATH}" \
      43. -i "blocksize != ''" -v "concat('block_size ', blocksize)" -n -b \
      44. -i "autosave != ''" -v "concat('autosave ', autosave)" -n -b \
      45. -i "nohidden[. = '1']" -o "nohidden" -n -b \
      46. ${OMV_CONFIG_FILE} | xmlstarlet unesc >> ${SNAPRAID_CONFIG}
      47. # Add default content
      48. #echo "content /var/snapraid/content" >> ${SNAPRAID_CONFIG}
      49. # Process drives
      50. count=$(omv_config_get_count "${DRIVES}")
      51. index=1
      52. countp=0
      53. while [ ${index} -le ${count} ]; do
      54. content=$(omv_config_get "${DRIVES}[position()=${index}]/content")
      55. data=$(omv_config_get "${DRIVES}[position()=${index}]/data")
      56. parity=$(omv_config_get "${DRIVES}[position()=${index}]/parity")
      57. path=$(omv_config_get "${DRIVES}[position()=${index}]/path")
      58. name=$(omv_config_get "${DRIVES}[position()=${index}]/name")
      59. label=$(omv_config_get "${DRIVES}[position()=${index}]/label")
      60. if [ "${path}" != "" ]; then
      61. echo "#####################################################################" >> ${SNAPRAID_CONFIG}
      62. echo "# OMV-Name: ${name} Drive Label: ${label}" >> ${SNAPRAID_CONFIG}
      63. if [ "${content}" -eq "1" ]; then
      64. echo "content ${path}/snapraid.content" >> ${SNAPRAID_CONFIG}
      65. fi
      66. if [ "${data}" -eq "1" ]; then
      67. echo "disk ${name} ${path}" >> ${SNAPRAID_CONFIG}
      68. fi
      69. if [ "${parity}" -eq "1" ]; then
      70. countp=$(( ${countp} + 1 ))
      71. if [ "${countp}" -le 6 ]; then
      72. if [ "${countp}" != "1" ]; then
      73. echo "${countp}-parity ${path}/snapraid.${countp}-parity" >> ${SNAPRAID_CONFIG}
      74. else
      75. echo "parity ${path}/snapraid.parity" >> ${SNAPRAID_CONFIG}
      76. fi
      77. fi
      78. fi
      79. echo >> ${SNAPRAID_CONFIG}
      80. fi
      81. index=$(( ${index} + 1 ))
      82. done
      83. # Add default excludes
      84. cat <<EOF >> ${SNAPRAID_CONFIG}
      85. exclude *.bak
      86. exclude *.unrecoverable
      87. exclude /tmp/
      88. exclude lost+found/
      89. exclude .content
      90. exclude aquota.group
      91. exclude aquota.user
      92. exclude ${SNAPRAID_FILENAME}*
      93. EOF
      94. # Process rules
      95. count=$(omv_config_get_count "${RULES}")
      96. index=1
      97. while [ ${index} -le ${count} ]; do
      98. rtype=$(omv_config_get "${RULES}[position()=${index}]/rtype")
      99. rule=$(omv_config_get "${RULES}[position()=${index}]/rule1")
      100. rulefolder=$(omv_config_get "${RULES}[position()=${index}]/rule-folder")
      101. if [ "${rule}" != "" ]; then
      102. if [ "${rulefolder}" != "" ]; then
      103. echo "# Rule selected from ${rulefolder}" >> ${SNAPRAID_CONFIG}
      104. fi
      105. if [ "${rtype}" != "0" ]; then
      106. echo "include ${rule}" >> ${SNAPRAID_CONFIG}
      107. else
      108. echo "exclude ${rule}" >> ${SNAPRAID_CONFIG}
      109. fi
      110. fi
      111. index=$(( ${index} + 1 ))
      112. done
      113. # backup snapraid.conf
      114. count=$(omv_config_get_count "${DRIVES}")
      115. index=1
      116. while [ ${index} -le ${count} ]; do
      117. data=$(omv_config_get "${DRIVES}[position()=${index}]/data")
      118. path=$(omv_config_get "${DRIVES}[position()=${index}]/path")
      119. # do backups - keep the last 3
      120. if [ "${path}" != "" ]; then
      121. if [ "${data}" -eq "1" ]; then
      122. if [ -f "${path}/${SNAPRAID_FILENAME}.bak" ]; then
      123. rm -f "${path}/${SNAPRAID_FILENAME}.bak"
      124. fi
      125. if [ -f "${path}/${SNAPRAID_FILENAME}" ]; then
      126. mv "${path}/${SNAPRAID_FILENAME}" "${path}/${SNAPRAID_FILENAME}.bak"
      127. fi
      128. if [ -f "${SNAPRAID_CONFIG}" ]; then
      129. if [ -d "${path}" ]; then
      130. cp ${SNAPRAID_CONFIG} ${path}/
      131. fi
      132. fi
      133. fi
      134. fi
      135. index=$(( ${index} + 1 ))
      136. done
      137. # create snapraid diff script config file
      138. cat <<EOF > ${DIFF_SCRIPT_CONFIG}
      139. # this file was automatically generated from
      140. # ${OMV_name} ${OMV_versionname} ${OMV_version}
      141. # and 'openmediavault-snapraid' ${PLUGIN_VERSION}
      142. EOF
      143. xmlstarlet sel -t -m "${SERVICE_XPATH}" \
      144. -i "syslog = 0" -o "SYSLOG=\"false\"" -n -b \
      145. -i "syslog = 1" -o "SYSLOG=\"true\"" -n -b \
      146. -i "debug = 0" -o "DEBUG=\"false\"" -n -b \
      147. -i "debug = 1" -o "DEBUG=\"true\"" -n -b \
      148. -i "sendmail = 0" -o "SENDMAIL=\"false\"" -n -b \
      149. -i "sendmail = 1" -o "SENDMAIL=\"true\"" -n -b \
      150. -i "runscrub = 0" -o "RUN_SCRUB=\"false\"" -n -b \
      151. -i "runscrub = 1" -o "RUN_SCRUB=\"true\"" -n -b \
      152. -i "scrubfreq != ''" -v "concat('SCRUB_FREQUENCY_IN_DAYS=', scrubfreq)" -n -b \
      153. -i "delthreshold != ''" -v "concat('DEL_THRESHOLD=', delthreshold)" -n -b \
      154. -i "scrubpercent != ''" -v "concat('SCRUB_PERCENT=', scrubpercent)" -n -b \
      155. ${OMV_CONFIG_FILE} | xmlstarlet unesc >> ${DIFF_SCRIPT_CONFIG}
      Display All


      I have not tested it yet, and if someone more familiar with the system has a remark/correction etc let me know.

      Usual disclaimer as it is not tested, and I can't guarantee that it won't wreck your machine... Hell I can't guarantee it won't wreck MY server

      The post was edited 1 time, last by Husker: Issue with script edit, will be updated later ().

    • New

      Husker wrote:

      I have not tested it yet, and if someone more familiar with the system has a remark/correction etc let me know.
      Usual disclaimer as it is not tested, and I can't guarantee that it won't wreck your machine... Hell I can't guarantee it won't wreck MY server
      At least that's a fair warning.

      Husker wrote:

      At the moment my system with 20 Data drives and 4 parity HDDs takes 4 days to do scrubs etc...
      Since I'm sure you've been over the options and have probably tested a few, perhaps you could help me understand why you went this route? What would be wrong with breaking the 20 data to 4 parity down to 4 each sets of 5 data to 1 parity? The 20 to 4 setup would be more fault tolerant, allowing more than 1 failed drive, but scrubs would have to faster in multiple 5 to 1 setups where scrubs might be set to run and finish afterhours. (Or so it would seem.)
    • New

      Hey,

      I simply didn't think of that possibility, you are making a good point. My main concern originally was not to stray far from the comfort of the GUI, and not having too many manual changes... I really wanted to have one tidy and simple system with its glorious 24 drives. But I guess that's a reality check for me.

      Following up on your message I looked a bit around and saw a few subjects on the SnapRAID forum referring to that:
      sourceforge.net/p/snapraid/dis…/1677233/thread/58d5a134/
      sourceforge.net/p/snapraid/dis…/1677233/thread/15ecf531/

      At the moment I am going to try this setup with 3 parity drives with the last one using z-parity, and see how it performs. And if it's not good enough I'll see with just loosing the SnapRAID plugin and doing multiple config file/process instances as you suggested. I'll keep updating the thread with my findings.

      Thank you for throwing some ideas in there, I appreciated it!