SMB encryption with OMV using AES-NI?

    • OMV 4.x
    • SMB encryption with OMV using AES-NI?

      Hi folks,

      Is there any way I can install Samba 4.7.0+ so as to get hardware offloas AES encryption of SMB? I'm using an Intel NUC with a Celeron 3050 for OpenMediaVault, and get 112MB/s without SMB encryption, and only 40MB/s with it enabled.
    • tkaiser wrote:

      Out of curiosity:
      • With which SMB version is the client connecting? (might matter)
      • Is the bottleneck the server or maybe (also) the client?
      Can you provide testparm and smbstatus command output with the client connected?

      Have you checked with htop for example whether one CPU core on the server is maxing out?
      Clients are all Windows 10 and connecting with SMB3. I have a Samba server running on a VM which has four i7-6950X cores attached to it and it's maxing out the line (102MB/s).

      The bottleneck is not the client as it's getting the aforementioned 102MB/s from the VM server.

      Testparm output:

      Source Code

      1. Load smb config files from /etc/samba/smb.conf
      2. rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
      3. Processing section "[backups]"
      4. Loaded services file OK.
      5. Server role: ROLE_STANDALONE
      6. Press enter to see a dump of your service definitions
      7. # Global parameters
      8. [global]
      9. server string = %h server
      10. workgroup = MELNED
      11. local master = No
      12. log file = /var/log/samba/log.%m
      13. logging = syslog
      14. max log size = 1000
      15. panic action = /usr/share/samba/panic-action %d
      16. disable spoolss = Yes
      17. load printers = No
      18. printcap name = /dev/null
      19. disable netbios = Yes
      20. server min protocol = SMB3
      21. smb ports = 445
      22. pam password change = Yes
      23. passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
      24. passwd program = /usr/bin/passwd %u
      25. restrict anonymous = 2
      26. socket options = TCP_NODELAY IPTOS_LOWDELAY
      27. dns proxy = No
      28. idmap config * : backend = tdb
      29. printing = bsd
      30. create mask = 0777
      31. directory mask = 0777
      32. aio read size = 16384
      33. aio write size = 16384
      34. use sendfile = Yes
      35. [backups]
      36. path = /srv/dev-disk-by-label-backups/backups
      37. hide special files = Yes
      38. create mask = 0664
      39. directory mask = 0775
      40. force create mode = 0664
      41. force directory mode = 0775
      42. inherit acls = Yes
      43. read only = No
      Display All


      smbstatus output (with encryption disabled/not enforced):


      Brainfuck Source Code

      1. Samba version 4.5.16-Debian
      2. PID Username Group Machine Protocol Version Encryption Signing
      3. ----------------------------------------------------------------------------------------------------------------------------------------
      4. 16952 eluck users 192.168.0.100 (ipv4:192.168.0.100:59493) SMB3_11 - partial(AES-128-CMAC)
      5. Service pid Machine Connected at Encryption Signing
      6. ---------------------------------------------------------------------------------------------
      7. IPC$ 16952 192.168.0.100 Tue May 21 16:55:27 2019 AEST - -
      8. backups 16952 192.168.0.100 Tue May 21 16:55:26 2019 AEST - -
      9. Locked files:
      10. Pid Uid DenyMode Access R/W Oplock SharePath Name Time
      11. --------------------------------------------------------------------------------------------------
      12. 16952 1000 DENY_NONE 0x100081 RDONLY NONE /srv/dev-disk-by-label-backups/backups . Tue May 21 16:55:28 2019
      13. 16952 1000 DENY_NONE 0x100081 RDONLY NONE /srv/dev-disk-by-label-backups/backups . Tue May 21 16:55:28 2019
      Display All

      HTOP with SMB encryption disabled:



      HTOP with SMB encryption enabled (note, it does peak and hold at 100% for much of the time):
    • Kaldek wrote:

      The bottleneck is not the client as it's getting the aforementioned 102MB/s from the VM server
      Well, with encryption enabled it could've been the client too (since both ends of the connection need to use crypto functions) but your htop output points to the server's single-threaded CPU performance being the bottleneck (and no AES-NI in use of course).

      Since you're experienced it might be an idea to set up another VM with OMV5 (relying on Debian Buster and Samba 4.9) and give this a try. I wanted to test this already days ago but am running a bit out of time right now (and can't test directly since away from my OMV5 test VM and accessing through VPN):

      Brainfuck Source Code

      1. root@buster:/home/tk# smbstatus
      2. Samba version 4.9.5-Debian
      3. PID Username Group Machine Protocol Version Encryption Signing
      4. ----------------------------------------------------------------------------------------------------------------------------------------
      5. 6089 tk users 10.0.64.4 (ipv4:10.0.64.4:62921) SMB3_02 AES-128-CCM partial(AES-128-CMAC)
      6. Service pid Machine Connected at Encryption Signing
      7. ---------------------------------------------------------------------------------------------
      8. btrfs 6089 10.0.64.4 Tue May 21 09:59:36 2019 CEST AES-128-CCM AES-128-CMAC
      9. Locked files:
      10. Pid Uid DenyMode Access R/W Oplock SharePath Name Time
      11. --------------------------------------------------------------------------------------------------
      12. 6089 1000 DENY_NONE 0x100081 RDONLY NONE /srv/dev-disk-by-path-pci-0000-03-00.0-scsi-0-0-1-0-part2/btrfs . Tue May 21 09:59:38 2019
      13. root@buster:/home/tk# testparm
      14. rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
      15. Registered MSG_REQ_POOL_USAGE
      16. Registered MSG_REQ_DMALLOC_MARK and LOG_CHANGED
      17. Load smb config files from /etc/samba/smb.conf
      18. rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
      19. Processing section "[ext4]"
      20. Processing section "[btrfs]"
      21. Loaded services file OK.
      22. Server role: ROLE_STANDALONE
      23. Press enter to see a dump of your service definitions
      24. # Global parameters
      25. [global]
      26. disable spoolss = Yes
      27. dns proxy = No
      28. load printers = No
      29. log file = /var/log/samba/log.%m
      30. logging = syslog
      31. map to guest = Bad User
      32. max log size = 1000
      33. pam password change = Yes
      34. panic action = /usr/share/samba/panic-action %d
      35. passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
      36. passwd program = /usr/bin/passwd %u
      37. printcap name = /dev/null
      38. server min protocol = SMB2
      39. server string = %h server
      40. socket options = TCP_NODELAY IPTOS_LOWDELAY
      41. fruit:model = MacPro
      42. fruit:aapl = yes
      43. idmap config * : backend = tdb
      44. aio read size = 16384
      45. aio write size = 16384
      46. create mask = 0777
      47. directory mask = 0777
      48. printing = bsd
      49. smb encrypt = required
      50. use sendfile = Yes
      51. [ext4]
      52. comment = ext4 partition on omv5 dataset
      53. create mask = 0664
      54. directory mask = 0775
      55. ea support = No
      56. force create mode = 0664
      57. force directory mode = 0775
      58. guest ok = Yes
      59. hide special files = Yes
      60. inherit acls = Yes
      61. path = /srv/dev-disk-by-path-pci-0000-03-00.0-scsi-0-0-1-0-part1/ext4/
      62. read only = No
      63. store dos attributes = No
      64. vfs objects = full_audit catia fruit streams_xattr
      65. fruit:time machine = yes
      66. fruit:resource = file
      67. fruit:metadata = netatalk
      68. fruit:locking = none
      69. fruit:encoding = private
      70. full_audit:priority = NOTICE
      71. full_audit:facility = local7
      72. full_audit:failure = none
      73. full_audit:success = mkdir rename unlink rmdir pwrite
      74. full_audit:prefix = %u|%I|%m|%P|%S
      75. [btrfs]
      76. comment = btrfs partition on omv5 dataset
      77. create mask = 0664
      78. directory mask = 0775
      79. ea support = No
      80. force create mode = 0664
      81. force directory mode = 0775
      82. guest ok = Yes
      83. hide special files = Yes
      84. inherit acls = Yes
      85. path = /srv/dev-disk-by-path-pci-0000-03-00.0-scsi-0-0-1-0-part2/btrfs/
      86. read only = No
      87. store dos attributes = No
      88. vfs objects = catia fruit streams_xattr
      89. fruit:time machine = yes
      90. fruit:resource = file
      91. fruit:metadata = netatalk
      92. fruit:locking = none
      93. fruit:encoding = private
      Display All

      The post was edited 1 time, last by tkaiser ().

    • tkaiser wrote:

      Kaldek wrote:

      The bottleneck is not the client as it's getting the aforementioned 102MB/s from the VM server
      Well, with encryption enabled it could've been the client too (since both ends of the connection need to use crypto functions) but your htop output points to the server's single-threaded CPU performance being the bottleneck (and no AES-NI in use of course).
      Since you're experienced it might be an idea to set up another VM with OMV5 (relying on Debian Buster and Samba 4.9) and give this a try.
      Better not be my client - it's an i7-8700k running at 5Ghz! :)

      Anyway, I'm happy to try OMV5 as this host isn't really in production use yet.
    • tkaiser wrote:

      Kaldek wrote:

      I'm happy to try OMV5 as this host isn't really in production use yet
      Please report back whether you can see transport encryption benefitting from AES-NI. Personally interested in this since in 2019 we have to switch with a lot of installations away from AFP/Netatalk/Helios to Samba (even dealing with Windows clients again after a decade).
      Yeesh. OMV5 kernel panics on the NUC 3050 hardware when I boot the ISO. :-/
    • Just did a quick test with macOS 10.14.5 as SMB client (SMB3_02). AES-128-CCM first vs. no encryption:

      Brainfuck Source Code

      1. Service pid Machine Connected at Encryption Signing
      2. ---------------------------------------------------------------------------------------------
      3. ext4 11004 192.168.21.91 Wed May 22 11:55:55 2019 CEST AES-128-CCM AES-128-CMAC
      4. ext4 12864 192.168.21.91 Wed May 22 12:08:08 2019 CEST - -

      Encrypted:



      vs. unencrypted:



      Clearly a client side issue. I need to find a Windows box there... the server setup is not the bottleneck (up to 500 MB/s with 10GbE)

      Edit: Found the culprit: not encryption is the bottleneck but SMB packet signing. As test I created /etc/nsmb.conf on the Mac with enforcing signing by setting signing_required=yes and now get similar slow speeds as with encryption/signing:

      Brainfuck Source Code

      1. Service pid Machine Connected at Encryption Signing
      2. ---------------------------------------------------------------------------------------------
      3. ext4 14549 192.168.21.91 Wed May 22 12:23:20 2019 CEST - AES-128-CMAC

      The post was edited 1 time, last by tkaiser ().

    • Now a test with OMV5 against a virtualized Win10 Pro system running on the same vSphere server:


      Both OMV5 and the Win10 VM have 2 vCPUs configured. Connection is established with signing and encryption enabled (SMB3_11 version):

      Brainfuck Source Code

      1. Service pid Machine Connected at Encryption Signing
      2. ---------------------------------------------------------------------------------------------
      3. ext4 6038 192.168.21.116 Wed May 22 16:07:38 2019 CEST AES-128-CCM AES-128-CMAC


      Samba --> Windows:


      Windows --> Samba:


      On the OMV server CPU utilization was identical both times, I use the values from an iostat 30 call:

      Source Code

      1. avg-cpu: %user %nice %system %iowait %steal %idle
      2. 4.97 0.00 1.83 0.84 0.00 92.36
      3. 45.67 0.00 12.70 6.49 0.00 35.15
      4. 40.10 0.00 10.59 9.19 0.00 40.12
      5. 42.78 0.00 11.76 4.99 0.00 40.47
      6. 32.21 0.00 7.98 3.58 0.00 56.23
      7. 1.15 0.00 1.02 0.18 0.00 97.64
      So we're talking about 60% CPU utilization with 2 CPU cores. To me this looks like at least on the Samba side encryption is making use of AES-NI while with that high CPU utilization on the Win10 Pro machine I'm not entirely sure but according to Coreinfo AES-NI support is there:

      The post was edited 1 time, last by tkaiser ().

    • And another test again with Win10 but this time without encryption/signing:

      Samba --> Windows:

      Still high CPU utilization in Windows but better performance. CPU utilization of the dual-core Samba server below:

      Source Code

      1. avg-cpu: %user %nice %system %iowait %steal %idle
      2. 0.99 0.00 0.95 0.03 0.00 98.03
      3. 1.39 0.00 6.72 3.10 0.00 88.79
      4. 1.77 0.00 9.97 2.49 0.00 85.77
      5. 2.01 0.00 22.18 8.74 0.00 67.08
      6. 1.59 0.00 17.77 11.20 0.00 69.44
      7. 1.43 0.00 12.78 8.66 0.00 77.12
      8. 1.39 0.00 1.10 0.42 0.00 97.09


      Windows --> Samba:


      And CPU utilization at the Samba server:

      Source Code

      1. avg-cpu: %user %nice %system %iowait %steal %idle
      2. 0.54 0.00 0.54 0.20 0.00 98.72
      3. 2.47 0.00 15.08 4.49 0.00 77.95
      4. 3.78 0.00 36.35 11.35 0.00 48.52
      5. 1.58 0.00 4.71 1.28 0.00 92.43
      6. 1.19 0.00 0.94 0.00 0.00 97.87


      TL;DR: at least with OMV5 (Samba 4.9) SMB encryption runs AES-NI accelerated if the CPU supports it.

      But there is a price to pay: with encryption enabled the whole crypto stuff triggers constant switching between kernel and userspace, see the huge %user percentage with encryption vs. no encryption below where the whole network <--> storage thing runs inside the kernel.

      Source Code

      1. crypto: %user %nice %system %iowait %steal %idle
      2. 45.67 0.00 12.70 6.49 0.00 35.15
      3. 40.10 0.00 10.59 9.19 0.00 40.12
      4. 42.78 0.00 11.76 4.99 0.00 40.47
      5. none: %user %nice %system %iowait %steal %idle
      6. 1.77 0.00 9.97 2.49 0.00 85.77
      7. 2.01 0.00 22.18 8.74 0.00 67.08
      8. 1.59 0.00 17.77 11.20 0.00 69.44

      The post was edited 2 times, last by tkaiser ().

    • Thanks so much for your testing effort here. I'll look at the Debian install with OMV installed on top, because whilst I won't be copying files using SMB every day, when I do need to it's generally large files and being an InfoSec guy I'm a bit of a stickler for transport encryption.
    • Kaldek wrote:

      being an InfoSec guy I'm a bit of a stickler for transport encryption
      Well, 'best practices' in certain environments demand transport encryption. I'm about to migrate a few older AFP/SMB server installs to Samba this year and most probably we'll decide to differentiate between shares with sensitive data and others that don't need encryption. With Samba (and OMV) it's possible to define smb encrypt = auto in the global section and then smb encrypt = required per share so it's possible to adjust behavior on a per share and on a per client basis.