10G iscsi tweaking

    • OMV 2.x
    • 10G iscsi tweaking

      Hi All,
      This thread is continue of a topic i opened recently regrading Using 10g iscsi over xenserver Sr iscsi
      last time i did tested iscsi with OMV , i got very poor performance , two day ago i reinstalled OMV and started to test again same scenario and the result got little bit better
      as shown below , the volume i have exported is raiz1 based ZFS with following below settings, I also tweaked the network card of both sides xen host and omv to following below
      Now my question is there iscsi specific setting to max out the 10G network?
      Thanks please advice


      ZFS OPTIONS

      Source Code

      1. NAME PROPERTY VALUE SOURCE
      2. po01 type filesystem -
      3. po01 creation Sun Nov 1 11:15 2015 -
      4. po01 used 619G -
      5. po01 available 1.50T -
      6. po01 referenced 19K -
      7. po01 compressratio 1.01x -
      8. po01 mounted yes -
      9. po01 quota none default
      10. po01 reservation none default
      11. po01 recordsize 128K default
      12. po01 mountpoint /po01 default
      13. po01 sharenfs off default
      14. po01 checksum on default
      15. po01 compression lz4 local
      16. po01 atime off local
      17. po01 devices on default
      18. po01 exec on default
      19. po01 setuid on default
      20. po01 readonly off default
      21. po01 zoned off default
      22. po01 snapdir hidden default
      23. po01 aclinherit restricted default
      24. po01 canmount on default
      25. po01 xattr on default
      26. po01 copies 1 default
      27. po01 version 5 -
      28. po01 utf8only off -
      29. po01 normalization none -
      30. po01 casesensitivity sensitive -
      31. po01 vscan off default
      32. po01 nbmand off default
      33. po01 sharesmb off default
      34. po01 refquota none default
      35. po01 refreservation none default
      36. po01 primarycache all default
      37. po01 secondarycache all default
      38. po01 usedbysnapshots 0 -
      39. po01 usedbydataset 19K -
      40. po01 usedbychildren 619G -
      41. po01 usedbyrefreservation 0 -
      42. po01 logbias latency default
      43. po01 dedup off default
      44. po01 mlslabel none default
      45. po01 sync standard default
      46. po01 refcompressratio 1.00x -
      47. po01 written 19K -
      48. po01 logicalused 8.13G -
      49. po01 logicalreferenced 9.50K -
      50. po01 filesystem_limit none default
      51. po01 snapshot_limit none default
      52. po01 filesystem_count none default
      53. po01 snapshot_count none default
      54. po01 snapdev hidden default
      55. po01 acltype off default
      56. po01 context none default
      57. po01 fscontext none default
      58. po01 defcontext none default
      59. po01 rootcontext none default
      60. po01 relatime off default
      61. po01 redundant_metadata all default
      62. po01 overlay off default
      63. po01/iscsivol type volume -
      64. po01/iscsivol creation Mon Nov 2 3:40 2015 -
      65. po01/iscsivol used 619G -
      66. po01/iscsivol available 2.09T -
      67. po01/iscsivol referenced 8.07G -
      68. po01/iscsivol compressratio 1.01x -
      69. po01/iscsivol reservation none default
      70. po01/iscsivol volsize 600G local
      71. po01/iscsivol volblocksize 8K -
      72. po01/iscsivol checksum on default
      73. po01/iscsivol compression lz4 inherited from po01
      74. po01/iscsivol readonly off default
      75. po01/iscsivol copies 1 default
      76. po01/iscsivol refreservation 619G local
      77. po01/iscsivol primarycache all default
      78. po01/iscsivol secondarycache all default
      79. po01/iscsivol usedbysnapshots 0 -
      80. po01/iscsivol usedbydataset 8.07G -
      81. po01/iscsivol usedbychildren 0 -
      82. po01/iscsivol usedbyrefreservation 611G -
      83. po01/iscsivol logbias latency default
      84. po01/iscsivol dedup off default
      85. po01/iscsivol mlslabel none default
      86. po01/iscsivol sync standard default
      87. po01/iscsivol refcompressratio 1.01x -
      88. po01/iscsivol written 8.07G -
      89. po01/iscsivol logicalused 8.12G -
      90. po01/iscsivol logicalreferenced 8.12G -
      91. po01/iscsivol snapshot_limit none default
      92. po01/iscsivol snapshot_count none default
      93. po01/iscsivol snapdev hidden default
      94. po01/iscsivol context none default
      95. po01/iscsivol fscontext none default
      96. po01/iscsivol defcontext none default
      97. po01/iscsivol rootcontext none default
      98. po01/iscsivol redundant_metadata all default
      99. po01/iscsivol shareiscsi off default
      Display All


      last tested performance

      Brainfuck Source Code

      1. ----------------------------------------------------------------------
      2. CrystalDiskMark 5.0.2 x64 (C) 2007-2015 hiyohiyo
      3. Crystal Dew World : http://crystalmark.info/
      4. -----------------------------------------------------------------------
      5. * MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
      6. * KB = 1000 bytes, KiB = 1024 bytes
      7. Sequential Read (Q= 32,T= 1) : 411.012 MB/s
      8. Sequential Write (Q= 32,T= 1) : 351.445 MB/s
      9. Random Read 4KiB (Q= 32,T= 1) : 122.886 MB/s [ 30001.5 IOPS]
      10. Random Write 4KiB (Q= 32,T= 1) : 115.293 MB/s [ 28147.7 IOPS]
      11. Sequential Read (T= 1) : 332.592 MB/s
      12. Sequential Write (T= 1) : 225.649 MB/s
      13. Random Read 4KiB (Q= 1,T= 1) : 9.992 MB/s [ 2439.5 IOPS]
      14. Random Write 4KiB (Q= 1,T= 1) : 7.127 MB/s [ 1740.0 IOPS]
      15. Test : 1024 MiB [E: 0.0% (0.2/500.0 GiB)] (x1) [Interval=5 sec]
      16. Date : 2015/11/03 12:58:26
      17. OS : Windows Server 2012 R2 Server Standard (full installation) [6.3 Build 9600] (x64)
      Display All


      Source Code

      1. # -- tuning -- #
      2. # Increase system file descriptor limit
      3. fs.file-max = 65535
      4. # Increase system IP port range to allow for more concurrent connections
      5. net.ipv4.ip_local_port_range = 1024 65000
      6. # -- 10gbe tuning from Intel ixgb driver README -- #
      7. # turn off selective ACK and timestamps
      8. net.ipv4.tcp_sack = 0
      9. net.ipv4.tcp_timestamps = 0
      10. # memory allocation min/pressure/max.
      11. # read buffer, write buffer, and buffer space
      12. net.ipv4.tcp_rmem = 10000000 10000000 10000000
      13. net.ipv4.tcp_wmem = 10000000 10000000 10000000
      14. net.ipv4.tcp_mem = 10000000 10000000 10000000
      15. net.core.rmem_max = 524287
      16. net.core.wmem_max = 524287
      17. net.core.rmem_default = 524287
      18. net.core.wmem_default = 524287
      19. net.core.optmem_max = 524287
      20. net.core.netdev_max_backlog = 300000
      Display All
    • just to add local storage perf test i did show results as follows

      Source Code

      1. root@openmediavault:~# dd if=/dev/zero of=/po01/temp.dat bs=4M count=10000
      2. 10000+0 records in
      3. 10000+0 records out
      4. 41943040000 bytes (42 GB) copied, 37.1105 s, 1.1 GB/s
      5. root@openmediavault:~# dd if=/dev/zero of=/po01/temp.dat bs=1024k count=25k
      6. 25600+0 records in
      7. 25600+0 records out
      8. 26843545600 bytes (27 GB) copied, 16.5653 s, 1.6 GB/s
      9. root@openmediavault:~# dd if=/dev/zero of=/po01/temp.dat bs=4k count=250000
      10. 250000+0 records in
      11. 250000+0 records out
      12. 1024000000 bytes (1.0 GB) copied, 3.19075 s, 321 MB/s
      13. root@openmediavault:~# dd if=/dev/zero of=/po01/mytestfile.out bs=1000 count=1000000
      14. 1000000+0 records in
      15. 1000000+0 records out
      16. 1000000000 bytes (1.0 GB) copied, 10.6731 s, 93.7 MB/s
      17. root@openmediavault:~# dd if=/po01/temp.dat of=/dev/null bs=64k count=100024
      18. 15625+0 records in
      19. 15625+0 records out
      20. 1024000000 bytes (1.0 GB) copied, 0.470513 s, 2.2 GB/s
      21. root@openmediavault:~#
      Display All