Erasmus Low Power ZFS NAS based on Skylake/Fujitsu D3417-B

    • OMV 3.x
    • Erasmus Low Power ZFS NAS based on Skylake/Fujitsu D3417-B

      Last year I could successfully build a low power but high performance NAS based on Haswell and Openmediavault Kralizec running below 12 watts (idle) with 3 x 4TB disks Raid5 installed.
      http://forums.openmediavault.org/index.php/Thread/4593-Low-Power-but-High-Performance-Haswell-based-OMV-NAS

      One of the disadvantages with this config is that the used Intel DH87RL motherboard does not support ECC memory, which is e.g. beneficial to run ZFS. Comparable server configurations during this time were much above in the range of 20 watts idle /wo disks.

      Now since the availability of Skylake CPUs as well as the ZFS plugin in Openmediavault, I was wondering when new low power server boards will become available. Ct magazine 5/16 published a first test with several skylake server boards featuring ECC RAM. Power /wo disks was mentioned to be as low as 9 watts for the Fujitsu D3417-B.

      So my decision was to give it a try to achieve again below 15 watts idle power on an Openmediavault NAS with 3 x 4TD disks, but this time with ECC and ZFS. The focus here is again on low power idle and not so much on the ZFS config and performance.

      Hardware
      • CPU Intel Pentium G3900 2.8 GHz (with boxed cooler)
      • Mainboard Fujitsu D3417-B
      • ECC RAM 2 x Samsung DDR4 8GB DIMM 288-PIN 2133 MHz / PC4-17000 M391A1G43DB0-CPB ECC
      • Case Corsair Carbide 200R (need to exchange the fans as the D3417 only supports PWM fans)
      • PSU Be quiet pure power 300 Watt L8
      • SSD Samsung SM951 NVMe 128GB M.2
      • Disks 3 x WD Red 4 TB

      Since Skylake CPUs are only supported well in kernels 4.3+ I decided to do some testing (NO productive setup) on OMV 3.x Erasmus.

      To get it working I had to manually install Debian Jessie instead of the OMV 3.0.2 image, to work around a grub bug that prevents from install grub on the M.2 SSD. Jessie comes with 3.16 kernel and 4.3 is available as backport.
      poweredbyjeff.com/2015/10/29/Intel-750-SSD-in-Debian-Jessie/
      (had to change line 2: mount –-rbind /proc proc/ )

      Then there was an issue with the backport kernel that the intel i915 firmware was missing.
      bugs.debian.org/cgi-bin/bugreport.cgi?bug=800835
      The specific firmware is part of the firmware-misc-nonfree package (justinstall it and the error will go away).

      The OMV Erasmus install on Jessie was straight forward. Powertop was used to do power optimization but I expect some kernel issues are still to be solved in upcoming backports.

      Power figures are measured at the AC side 230V /w KD 302
      35 watts - SMB large file copy to/from OMV NAS maxing out GiB speed
      30 watts - incremental file access/backup
      26 watts - OMV idle / 3x4TB disks spinning
      14 watts - OMV idle/ 3x4TB disks spun down


      The power consumption measured in this setup (still non-optimized) is already very good although it shows slightly higher power consumption in idle modes compared with my optimized Haswell build.
      More fine tuning will NOT be performed before next kernel backports 4.4+ and official Erasmus releases will become available as several issues are pending to be resolved.

      update June 2016:
      With backport Kernel 4.6.x still no change in power savings. Fujitsu indicated in their technote update 1.4 June/2016 that NVMe support will be improved Q3/2016. Up to now ASPM cannot be properly used with the Samsung NVMe SSD. Will update the power figures once the new BIOS is available.
      latest OMV Erasmus | 64 bit | backport kernels | latest omvextrasorg
      low power Skylake NAS build <= 10 watts idle (4 disks), Fujitsu D3417-B, 16GB ECC RAM, 4x4TB WD Red - RAIDZ1, 128GB M.2 NVMe SSD boot.
      backup NAS HP Proliant N54L, 4GB ECC RAM, 4x4TB WD Red - RAID 5, 30GB SATA boot.

      The post was edited 4 times, last by techtom ().

    • While I don't think linux skylake support is anywhere near as optimized as haswell, ecc memory does use a bit more power than non-ecc. With two sticks, I don't think it would be 2 watts though. Back in the fb-dimm days, those ECC controllers on each stick got hot and used quite a bit of power.
      omv 4.0.5 arrakis | 64 bit | 4.12 backports kernel | omvextrasorg 4.0.2
      omv-extras.org plugins source code and issue tracker - github.com/OpenMediaVault-Plugin-Developers

      Please don't PM for support... Too many PMs!
    • @ryecoaaron: 1 x ECC UDIMM idle power is 13.7 watts / compared with 2 x ECC UDIMM in idle is only 300mwatts less
      latest OMV Erasmus | 64 bit | backport kernels | latest omvextrasorg
      low power Skylake NAS build <= 10 watts idle (4 disks), Fujitsu D3417-B, 16GB ECC RAM, 4x4TB WD Red - RAIDZ1, 128GB M.2 NVMe SSD boot.
      backup NAS HP Proliant N54L, 4GB ECC RAM, 4x4TB WD Red - RAID 5, 30GB SATA boot.
    • UPDATE - after BIOS update of D3417-B which enabled ASPM for the NVMe SSD the idle POWER of the system is now below 10 watts
      (Erasmus 3.0.25 and backport kernel 4.6.0)

      Power figures updated - measured at the AC side 230V /w KD 302
      32 watts - SMB large file copy to/from OMV NAS maxing out GiB speed
      28 watts - incremental file access/backup
      22 watts - OMV idle / 3x4TB disks spinning
      9.5 watts - OMV idle/ 3x4TB disks spun down


      subtracting the power needed for the disks in standby then the basic system with SSD is running at 8 watts

      Overall this is excellent result for this ZFS NAS build with 3x4TB and still possible to expand by 3 more SATA disks.

      Now I am looking forward to the first official Openmediavault Erasmus release to use/test this config in next productive environment builds.
      latest OMV Erasmus | 64 bit | backport kernels | latest omvextrasorg
      low power Skylake NAS build <= 10 watts idle (4 disks), Fujitsu D3417-B, 16GB ECC RAM, 4x4TB WD Red - RAIDZ1, 128GB M.2 NVMe SSD boot.
      backup NAS HP Proliant N54L, 4GB ECC RAM, 4x4TB WD Red - RAID 5, 30GB SATA boot.
    • UPDATE - after fresh install of openmedivault with latest backport kernel and power optimizations the system is running just below 10 watts now with 4 x 4TB WD Red being in stand-by (spin down).
      (Erasmus 3.0.59 and backport kernel 4.8)

      I have now completely switched (OMV eramus seems to be stable) from RAID5 to ZFS on all online and backup servers - no issues since 5 weeks. Very efficient. I am using ZREP -ZFS replication and failover script to backup the online zfs servers which contain different ZFS data sets to zfs backup servers. Therefore the backup servers are waked up every night using wol from the routers then zrep is called via cron at the backup server to replicate changed data on the zfs online servers to the zfs backup servers.
      example cron from backup server: ssh <name of zfs server> zrep sync <zfs volume>/<data set>
      This generates consistent snaphots on both servers with option of fail over switching.
      Next step is to investigate the handling of the backups e.g via zfsbackup – ZFS backup handling to keep the backups similar to rsnapshot which I used before in Raid5 environments.
      latest OMV Erasmus | 64 bit | backport kernels | latest omvextrasorg
      low power Skylake NAS build <= 10 watts idle (4 disks), Fujitsu D3417-B, 16GB ECC RAM, 4x4TB WD Red - RAIDZ1, 128GB M.2 NVMe SSD boot.
      backup NAS HP Proliant N54L, 4GB ECC RAM, 4x4TB WD Red - RAID 5, 30GB SATA boot.
    • Thumb up to skylake pentium, ECC, low power, ZFS
      Look cool, competive with other commercial NAS solution for me
      Considering power consumption, expandability, stability, I think it is more wise to choose Intel platform rather than ARM based board except price

      Do you have any specific reason to choose Fujitsu D3417-B?

      C232 chipset might be good choice since omv doesn't need gpu
      or turning off intel intergated gpu saves power AFAIK
      OMV3 on Proxmox
      Intel E3-1245 v5 | 32GB ECC RAM | 4x3TB RAID10 HDD
      omv-zfs | omv-nginx | omv-letsencrypt | omv-openvpn
      Click link for more details
    • @luxflow of course C232 is also nice but D3417 was lowest power server mainboard I have found so far also because of NO dedicated remote management chipset which take quite some power. Built-in Gfx of the used Celeron G3900 is per default using highest RC states. And C236 has more SATA ports which I am going to use soon ;)
      latest OMV Erasmus | 64 bit | backport kernels | latest omvextrasorg
      low power Skylake NAS build <= 10 watts idle (4 disks), Fujitsu D3417-B, 16GB ECC RAM, 4x4TB WD Red - RAIDZ1, 128GB M.2 NVMe SSD boot.
      backup NAS HP Proliant N54L, 4GB ECC RAM, 4x4TB WD Red - RAID 5, 30GB SATA boot.