File transfers to OMV drop in speed after ~1GB

    • OMV 4.x

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • File transfers to OMV drop in speed after ~1GB

      Setting up OMV on a Rock64 1GB board using the precompiled image. Setup went fine and configured a USB drive w/ shares, users, and access. Connecting to share w/ Win 10 PC works as expected. Moved a 2GB file to the share and write speeds start at around 75MBps and after 1GB or so of the file transferring the speeds drops to about 15MBps for the rest of the file. Reading files from the drive gives me a steady 75MBps the whole way through. Thought it might be a SAMBA issue so I tried the suggestions in the SAMBA tuning threads but it did not help. I tried moving the file over w/ FTP and saw the same issue. The drop off was slower but after about 1GB of the file moving it get slower and slower until it ended up at around 20MBps when the file finished. Any thoughts on what I could be looking for?
    • wolf8auer wrote:

      Maybe it's an MacOS issue
      It's not. And to nail the problem down the procedure is always the same: test storage and network individually. If you search the forum for 'iperf iozone' or 'iperf3 iozone3' you most probably find some posts by my elaborating on this.

      If the storage is the bottleneck then writes through the network will be fast until filesystem caches/buffers fill up and then the storage performance bottlenecks.

      On the ARM images both tools are installed by default, no idea about x86. And on macOS for iperf3 all you do is to install homebrew and then a sudo brew install iperf3.
    • htop



      iperf test to and from

      Brainfuck Source Code

      1. Connecting to host 192.168.1.123, port 5201
      2. [ 4] local 192.168.1.238 port 60277 connected to 192.168.1.123 port 5201
      3. [ ID] Interval Transfer Bandwidth
      4. [ 4] 0.00-1.00 sec 112 MBytes 112 MBytes/sec
      5. [ 4] 1.00-2.00 sec 112 MBytes 112 MBytes/sec
      6. [ 4] 2.00-3.00 sec 113 MBytes 113 MBytes/sec
      7. [ 4] 3.00-4.00 sec 113 MBytes 113 MBytes/sec
      8. [ 4] 4.00-5.00 sec 113 MBytes 113 MBytes/sec
      9. [ 4] 5.00-6.00 sec 113 MBytes 113 MBytes/sec
      10. [ 4] 6.00-7.00 sec 113 MBytes 113 MBytes/sec
      11. [ 4] 7.00-8.00 sec 113 MBytes 113 MBytes/sec
      12. [ 4] 8.00-9.00 sec 113 MBytes 113 MBytes/sec
      13. [ 4] 9.00-10.00 sec 113 MBytes 113 MBytes/sec
      14. - - - - - - - - - - - - - - - - - - - - - - - - -
      15. [ ID] Interval Transfer Bandwidth
      16. [ 4] 0.00-10.00 sec 1.10 GBytes 113 MBytes/sec sender
      17. [ 4] 0.00-10.00 sec 1.10 GBytes 113 MBytes/sec receiver
      18. iperf Done.
      19. Accepted connection from 192.168.1.238, port 60276
      20. [ 5] local 192.168.1.123 port 5201 connected to 192.168.1.238 port 60277
      21. [ ID] Interval Transfer Bandwidth
      22. [ 5] 0.00-1.00 sec 110 MBytes 110 MBytes/sec
      23. [ 5] 1.00-2.00 sec 111 MBytes 111 MBytes/sec
      24. [ 5] 2.00-3.00 sec 113 MBytes 113 MBytes/sec
      25. [ 5] 3.00-4.00 sec 113 MBytes 113 MBytes/sec
      26. [ 5] 4.00-5.00 sec 113 MBytes 113 MBytes/sec
      27. [ 5] 5.00-6.00 sec 113 MBytes 113 MBytes/sec
      28. [ 5] 6.00-7.00 sec 113 MBytes 113 MBytes/sec
      29. [ 5] 7.00-8.00 sec 113 MBytes 113 MBytes/sec
      30. [ 5] 8.00-9.00 sec 113 MBytes 113 MBytes/sec
      31. [ 5] 9.00-10.00 sec 113 MBytes 113 MBytes/sec
      32. [ 5] 10.00-10.02 sec 2.72 MBytes 113 MBytes/sec
      33. - - - - - - - - - - - - - - - - - - - - - - - - -
      34. [ ID] Interval Transfer Bandwidth
      35. [ 5] 0.00-10.02 sec 0.00 Bytes 0.00 MBytes/sec sender
      36. [ 5] 0.00-10.02 sec 1.10 GBytes 113 MBytes/sec receiver
      37. Server listening on 5201
      38. -----------------------------------------------------------
      39. Accepted connection from 192.168.1.123, port 39638
      40. [ 5] local 192.168.1.238 port 5201 connected to 192.168.1.123 port 39640
      41. [ ID] Interval Transfer Bandwidth
      42. [ 5] 0.00-1.00 sec 96.4 MBytes 96.4 MBytes/sec
      43. [ 5] 1.00-2.00 sec 100 MBytes 100 MBytes/sec
      44. [ 5] 2.00-3.00 sec 96.2 MBytes 96.2 MBytes/sec
      45. [ 5] 3.00-4.00 sec 100 MBytes 100 MBytes/sec
      46. [ 5] 4.00-5.00 sec 98.4 MBytes 98.4 MBytes/sec
      47. [ 5] 5.00-6.00 sec 100 MBytes 100 MBytes/sec
      48. [ 5] 6.00-7.00 sec 100 MBytes 100 MBytes/sec
      49. [ 5] 7.00-8.00 sec 101 MBytes 101 MBytes/sec
      50. [ 5] 8.00-9.00 sec 100 MBytes 100 MBytes/sec
      51. [ 5] 9.00-10.00 sec 100 MBytes 100 MBytes/sec
      52. [ 5] 10.00-10.03 sec 3.45 MBytes 99.9 MBytes/sec
      53. - - - - - - - - - - - - - - - - - - - - - - - - -
      54. [ ID] Interval Transfer Bandwidth
      55. [ 5] 0.00-10.03 sec 0.00 Bytes 0.00 MBytes/sec sender
      56. [ 5] 0.00-10.03 sec 997 MBytes 99.4 MBytes/sec receiver
      57. Connecting to host 192.168.1.238, port 5201
      58. [ 4] local 192.168.1.123 port 39640 connected to 192.168.1.238 port 5201
      59. [ ID] Interval Transfer Bandwidth Retr Cwnd
      60. [ 4] 0.00-1.00 sec 102 MBytes 102 MBytes/sec 1 212 KBytes
      61. [ 4] 1.00-2.00 sec 100 MBytes 101 MBytes/sec 0 212 KBytes
      62. [ 4] 2.00-3.00 sec 96.2 MBytes 96.1 MBytes/sec 94 212 KBytes
      63. [ 4] 3.00-4.00 sec 100 MBytes 100 MBytes/sec 0 212 KBytes
      64. [ 4] 4.00-5.00 sec 98.0 MBytes 98.2 MBytes/sec 111 212 KBytes
      65. [ 4] 5.00-6.00 sec 101 MBytes 101 MBytes/sec 0 212 KBytes
      66. [ 4] 6.00-7.00 sec 101 MBytes 100 MBytes/sec 0 212 KBytes
      67. [ 4] 7.00-8.00 sec 101 MBytes 100 MBytes/sec 0 212 KBytes
      68. [ 4] 8.00-9.00 sec 100 MBytes 100 MBytes/sec 0 212 KBytes
      69. [ 4] 9.00-10.00 sec 100 MBytes 100 MBytes/sec 0 212 KBytes
      70. - - - - - - - - - - - - - - - - - - - - - - - - -
      71. [ ID] Interval Transfer Bandwidth Retr
      72. [ 4] 0.00-10.00 sec 1000 MBytes 100 MBytes/sec 206 sender
      73. [ 4] 0.00-10.00 sec 997 MBytes 99.7 MBytes/sec receiver
      74. iperf Done.
      Display All



      Still trying to figure out how to use iozone though. I found a command in another thread but not sure this is what you need? It's supposed to be writing to a usb 3.0 thumb drive.

      Source Code

      1. Auto Mode
      2. Using maximum file size of 8192000 kilobytes.
      3. File size set to 8192000 kB
      4. Record Size 1024 kB
      5. Command line used: iozone -a -g 8000m -s 8000m -i 0 -i 1 -r 1024K
      6. Output is in kBytes/sec
      7. Time Resolution = 0.000001 seconds.
      8. Processor cache size set to 1024 kBytes.
      9. Processor cache line size set to 32 bytes.
      10. File stride size set to 17 * record size.
      11. random random bkwd record stride
      12. kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
      13. 8192000 1024 207891 122426 254448 277796
      14. iozone test complete.
      Display All
      EDIT: Found another thread where you said to do cd $share first then run iozone. Did that and here are the results. Still not sure if this is what you need though. not sure im using iozone correctly.


      Source Code

      1. Auto Mode
      2. Using maximum file size of 8192000 kilobytes.
      3. File size set to 8192000 kB
      4. Record Size 1024 kB
      5. Command line used: iozone -a -g 8000m -s 8000m -i 0 -i 1 -r 1024K
      6. Output is in kBytes/sec
      7. Time Resolution = 0.000001 seconds.
      8. Processor cache size set to 1024 kBytes.
      9. Processor cache line size set to 32 bytes.
      10. File stride size set to 17 * record size.
      11. random random bkwd record stride
      12. kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
      13. 8192000 1024 194685 128177 276941 266232
      14. iozone test complete.
      Display All

      The post was edited 3 times, last by Munkee915 ().

    • Thanks @tkaiser!

      Source Code

      1. iperf3 -c 192.168.0.64
      2. Connecting to host 192.168.0.64, port 5201
      3. [ 5] local 192.168.0.84 port 50191 connected to 192.168.0.64 port 5201
      4. [ ID] Interval Transfer Bitrate
      5. [ 5] 0.00-1.00 sec 111 MBytes 933 Mbits/sec
      6. [ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec
      7. [ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec
      8. [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec
      9. [ 5] 4.00-5.00 sec 112 MBytes 940 Mbits/sec
      10. [ 5] 5.00-6.00 sec 112 MBytes 940 Mbits/sec
      11. [ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec
      12. [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec
      13. [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec
      14. [ 5] 9.00-10.00 sec 112 MBytes 942 Mbits/sec
      15. - - - - - - - - - - - - - - - - - - - - - - - - -
      16. [ ID] Interval Transfer Bitrate
      17. [ 5] 0.00-10.00 sec 1.09 GBytes 940 Mbits/sec sender
      18. [ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec receiver
      19. iperf Done.
      Display All


      Source Code

      1. Server listening on 5201
      2. -----------------------------------------------------------
      3. Accepted connection from 192.168.0.84, port 50190
      4. [ 5] local 192.168.0.64 port 5201 connected to 192.168.0.84 port 50191
      5. [ ID] Interval Transfer Bandwidth
      6. [ 5] 0.00-1.00 sec 109 MBytes 917 Mbits/sec
      7. [ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec
      8. [ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec
      9. [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec
      10. [ 5] 4.00-5.00 sec 112 MBytes 939 Mbits/sec
      11. [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec
      12. [ 5] 6.00-7.00 sec 112 MBytes 939 Mbits/sec
      13. [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec
      14. [ 5] 8.00-9.00 sec 112 MBytes 942 Mbits/sec
      15. [ 5] 9.00-10.01 sec 112 MBytes 931 Mbits/sec
      16. [ 5] 10.01-10.01 sec 0.00 Bytes 0.00 bits/sec
      17. - - - - - - - - - - - - - - - - - - - - - - - - -
      18. [ ID] Interval Transfer Bandwidth
      19. [ 5] 0.00-10.01 sec 0.00 Bytes 0.00 bits/sec sender
      20. [ 5] 0.00-10.01 sec 1.09 GBytes 937 Mbits/sec receiver
      21. -----------------------------------------------------------
      22. Server listening on 5201
      23. -----------------------------------------------------------
      Display All
      I'm not sure how to interpret htop though...

      connected via AFP, as user rocky while copying 7 GB files.

      The post was edited 1 time, last by wolf8auer ().

    • Did I understand correctly, that the target medium is a thumb drive? This would be totally expected speeds with long copies than.
      Your network seems fine.
      You can test the drive using dd.
      dd if=/dev/zero of=path/inside/thumdrive/test.img bs=100M count=20 oflag=direct
      I am confident this will output the write speeds in which you saturated in long copies. This will create a 2gb file test.img, you can delete it after.

      The post was edited 1 time, last by getName() ().

    • Munkee915 wrote:

      It's supposed to be writing to a usb 3.0 thumb drive
      These things can start to throttle based on access pattern. Your two iozone tests both seemed to test an USB drive since reported values are a bit too high for SD card or eMMC (or maybe you run off a really good high capacity eMMC module)?

      Anyway: the mode to test with iozone is

      Source Code

      1. cd /path/to/mountpoint
      2. iozone -e -I -a -s 1000M -r 128k -r 1024k -r 16384k -i 0 -i 1
      This will test sequential transfer speeds with three different block sizes (128KB, 1MB, 16MB) and will also show throttling effects if an USB thumb drives starts to heat up and then slows down after some time. That's why we test with several block sizes of increasing size since normally the larger the block size the higher the transfer speeds. If performance decreases you know the thumb drive throttles to prevent overheating.
    • wolf8auer wrote:

      I'm not sure how to interpret htop though...
      CPU utilization of the afpd task seems ok to me but on average every CPU core is utilized with 40% which seems to indicate that there's a lot of kernel background activity. As suggested in your other thread I would call iostat 10 in the background to get the bigger picture (armbianmonitor -m 10 should also suffice but that's not available on x86 installations)
    • getName() wrote:

      Did I understand correctly, that the target medium is a thumb drive? This would be totally expected speeds with long copies than.
      Your network seems fine.
      You can test the drive using dd.
      dd if=/dev/zero of=path/inside/thumdrive/test.img bs=100M count=20 oflag=direct
      I am confident this will output the write speeds in which you saturated in long copies. This will create a 2gb file test.img, you can delete it after.

      Source Code

      1. root@rock64:~# dd if=/dev/zero of=/srv/dev-disk-by-label-HD01/HD01/test.img bs=100M count=20 oflag=direct
      2. 20+0 records in
      3. 20+0 records out
      4. 2097152000 bytes (2.1 GB, 2.0 GiB) copied, 20.4388 s, 103 MB/s
    • tkaiser wrote:

      wolf8auer wrote:

      FANTEC QB-X8US3
      O well, those unreliable external USB3 disk graveyards. Are you able to access all your disks via SMART (it works via eSATA but with USB3 it's always gambling)?
      When I tried to activate SMART monitoring on Device /dev/sda and dev/sdb, I get "communication failure" for each.

      However at the next attempt it suddenly worked:


      FANTEC has an eSATA port.

      Do you think an eSATA to USB cable would make any sense?
    • tkaiser wrote:

      Munkee915 wrote:

      It's supposed to be writing to a usb 3.0 thumb drive
      These things can start to throttle based on access pattern. Your two iozone tests both seemed to test an USB drive since reported values are a bit too high for SD card or eMMC (or maybe you run off a really good high capacity eMMC module)?
      Anyway: the mode to test with iozone is

      Source Code

      1. cd /path/to/mountpoint
      2. iozone -e -I -a -s 1000M -r 128k -r 1024k -r 16384k -i 0 -i 1
      This will test sequential transfer speeds with three different block sizes (128KB, 1MB, 16MB) and will also show throttling effects if an USB thumb drives starts to heat up and then slows down after some time. That's why we test with several block sizes of increasing size since normally the larger the block size the higher the transfer speeds. If performance decreases you know the thumb drive throttles to prevent overheating.
      cd /srv/dev-disk-by-label-HD01

      Source Code

      1. iozone -e -I -a -s 1000M -r 128k -r 1024k -r 16384k -i 0 -i 1
      2. Iozone: Performance Test of File I/O
      3. Version $Revision: 3.429 $
      4. Compiled for 32 bit mode.
      5. Build: linux
      6. Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
      7. Al Slater, Scott Rhine, Mike Wisner, Ken Goss
      8. Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
      9. Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
      10. Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
      11. Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
      12. Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
      13. Vangel Bojaxhi, Ben England, Vikentsi Lapa.
      14. Run began: Sun May 5 17:16:00 2019
      15. Include fsync in write timing
      16. O_DIRECT feature enabled
      17. Auto Mode
      18. File size set to 1024000 kB
      19. Record Size 128 kB
      20. Record Size 1024 kB
      21. Record Size 16384 kB
      22. Command line used: iozone -e -I -a -s 1000M -r 128k -r 1024k -r 16384k -i 0 -i 1
      23. Output is in kBytes/sec
      24. Time Resolution = 0.000001 seconds.
      25. Processor cache size set to 1024 kBytes.
      26. Processor cache line size set to 32 bytes.
      27. File stride size set to 17 * record size.
      28. random random bkwd record stride
      29. kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
      30. 1024000 128 33658 36342 36771 37017
      31. 1024000 1024 38906 38972 38676 38955
      32. 1024000 16384 39763 39877 39832 39767
      33. iozone test complete.
      Display All
    • wolf8auer wrote:

      Source Code

      1. kB reclen write rewrite read reread
      2. 1024000 128 33658 36342 36771 37017
      3. 1024000 1024 38906 38972 38676 38955
      4. 1024000 16384 39763 39877 39832 39767
      That smells like USB 2.0 (less than 40 MB/s). You could provide output of armbianmonitor -u but since you're using an ayufan image and not an Armbian based OMV image amount of information may be limited. At least you should check lsusb and lsusb -t

      wolf8auer wrote:

      Do you think an eSATA to USB cable would make any sense?
      Nope. In those eSATA/USB drive enclosures there's one SATA port multiplier (with USB3 usually a JMicron JMB575) combined with one USB3-to-SATA bridge (most probably a JMS567). So all you would do is exchanging one USB3-to-SATA bridge with another which usually makes things even worse.
    • wolf8auer wrote:

      Do you think an eSATA to USB cable would make any sense?
      No! You need an eSATA-port or at least an SATA-port on the other side. And if you want to address more than one disk in the enclosure the sata/esata-port on the motherboard must support port-multiplier. It´s a game of luck to find a combination that works.

      Edit: time overlap ;)
      OMV 3.0.90 (Gray style)
      ASRock Rack C2550D4I - 16GB ECC - 6x WD RED 3TB (ZFS 2x3 Striped RaidZ1)- Fractal Design Node 304
    • Source Code

      1. lsusb
      2. Bus 005 Device 002: ID 152d:0551 JMicron Technology Corp. / JMicron USA Technology Corp.
      3. Bus 005 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
      4. Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
      5. Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
      6. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
      7. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

      Source Code

      1. lsusb -t
      2. /: Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
      3. |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
      4. /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M
      5. /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=ohci-platform/1p, 12M
      6. /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-platform/1p, 480M
      7. /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc2/1p, 480M