2.5 gigabit nic - slow speed from PC to NAS

pa67 · 16. März 2022

Hi all,

I've a weird speed performance with a 2.5gbps ethernet pcie card mounted on an Asrock j4125-itx. The test configuration is made with a ssd sata disk and a smb share and iperf3. I used a Windows 11 laptop for client, with a usb-c to 2.5gbps ethernet adapter. When a test file is copied from nas to pc the speed is about 295 MB/s and in the windows 11 task manager the receiving is 2.5 gbps, but when I upload the test file from pc to nas the speed is about 130MB/s and in the windows 11 task manager the sending is only 1.1 gbps. The iperf3 tests have the same behavour. I set the jumbo frame to 9014. I used a 2.5 gbps switch. I'm trying to isolate the problem so a I changed only a component\software (motherboard, omv5 to omv6, etc) at a time and I retested the speed, but the result is the same.

pa67 · 21. März 2022

I changed the usb-c to 2.5gbps ethernet adapter with a new one. Now the testings with iperf3 are quite good with a average speed, in both direction, of 2.3 Gbits/s. The problem is with the files transfer from pc to nas. The test file transfer begins with 280 - 290 Mb/s, but after few seconds, it drops to 140 - 130 Mb/s until the end of transfer. I tried to add extra options in samba configuration with no particularly improvements.

donh · 21. März 2022

Are you using OMV6 or 5? 6 is built on debian bullseye as opposed to 5 on buster. Maybe the drivers and such have improved?

Good Luck

sbocquet · 23. März 2022

Hi,

I have in some ways the same problem as you with my setup...

I have 4 NICs on my OMV5 NAS:

- 2 x Intel i210 1Gb on the ASROCK X470D4U motherboard,

- 2 x Intel x550T2 10Gb on a PCie card which is on a correct PCIe extension slot (PCIe3.0 x8)).

They are all linked to the LAN with a 1Gb Netgear GS108E switch (works fine but plan to upgrade very soon to a 10Gb one).

I'm facing some strange drops in transfert rates with the x550T2 card whereas the same copy operations on the same disks run perfect with the i210 NICs.

1Gb NIC transferts

10Gb NIC transferts

To be short:
- NAS to PC is rock solid with both NIC,
- PC to NAS 1Gb NIC is rock solid
- PC to NAS 10Gb NIC is... strange

I've already checked that SSDs (PC & NAS) are not overheating (throttling), and tested also on my RAID 6 array to eliminate a disk issue.
NAS isn't CPU or RAM limited.
Copy operations are done by SMB protocol between my PC SSD (Windows 10) and the NAS, but seems to act the same way with SFTP protocol.
SSD and RAID6 array are capable of 300MB/s reads and writes speeds.

Seems like a NIC buffer problem to me, but I have already upgraded my buffers to max

Code

root@home-server:~# ethtool -g enp1s0f1
Ring parameters for enp1s0f1:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096

root@home-server:~# ethtool -g enp35s0
Ring parameters for enp35s0:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096

Alles anzeigen

I finished to do some testing with perf3 between the PC and the NAS, and it doesn't seems to be the NIC apparently...

Code

root@home-server:~# iperf3 -s -B 192.168.1.100
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.31, port 52113
[  5] local 192.168.1.100 port 5201 connected to 192.168.1.31 port 52114
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   110 MBytes   925 Mbits/sec    0    220 KBytes
[  5]   1.00-2.00   sec   113 MBytes   952 Mbits/sec    0    220 KBytes
[  5]   2.00-3.00   sec   112 MBytes   940 Mbits/sec    0    220 KBytes
[  5]   3.00-4.00   sec   113 MBytes   949 Mbits/sec    0    220 KBytes
[  5]   4.00-5.00   sec   114 MBytes   953 Mbits/sec    0    220 KBytes
[  5]   5.00-6.00   sec   113 MBytes   946 Mbits/sec    0    220 KBytes
[  5]   6.00-7.00   sec   112 MBytes   943 Mbits/sec    0    220 KBytes
[  5]   7.00-8.00   sec   113 MBytes   949 Mbits/sec    0    220 KBytes
[  5]   8.00-9.00   sec   112 MBytes   939 Mbits/sec    0    220 KBytes
[  5]   9.00-10.00  sec   113 MBytes   951 Mbits/sec    0    220 KBytes
[  5]  10.00-11.00  sec   113 MBytes   945 Mbits/sec    0    220 KBytes
[  5]  11.00-12.00  sec   113 MBytes   951 Mbits/sec    0    220 KBytes
[  5]  12.00-13.00  sec   113 MBytes   948 Mbits/sec    0    220 KBytes
[  5]  13.00-14.00  sec   113 MBytes   951 Mbits/sec    0    220 KBytes
[  5]  14.00-15.00  sec   113 MBytes   947 Mbits/sec    0    220 KBytes
[  5]  15.00-16.00  sec   113 MBytes   944 Mbits/sec    0    220 KBytes
[  5]  16.00-17.00  sec   112 MBytes   940 Mbits/sec    0    220 KBytes
[  5]  17.00-18.00  sec   113 MBytes   949 Mbits/sec    0    220 KBytes
[  5]  18.00-19.00  sec   113 MBytes   951 Mbits/sec    0    220 KBytes
[  5]  19.00-20.00  sec   107 MBytes   894 Mbits/sec    0    220 KBytes
[  5]  20.00-20.04  sec  4.29 MBytes   977 Mbits/sec    0    220 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.04  sec  2.20 GBytes   943 Mbits/sec    0             sender
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated

root@home-server:~# iperf3 -s -B 192.168.1.200
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.31, port 52121
[  5] local 192.168.1.200 port 5201 connected to 192.168.1.31 port 52122
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   109 MBytes   916 Mbits/sec    0    218 KBytes
[  5]   1.00-2.00   sec   113 MBytes   949 Mbits/sec    0    218 KBytes
[  5]   2.00-3.00   sec   113 MBytes   949 Mbits/sec    0    218 KBytes
[  5]   3.00-4.00   sec   112 MBytes   942 Mbits/sec    0    218 KBytes
[  5]   4.00-5.00   sec   112 MBytes   943 Mbits/sec    0    218 KBytes
[  5]   5.00-6.00   sec   113 MBytes   950 Mbits/sec    0    218 KBytes
[  5]   6.00-7.00   sec   113 MBytes   949 Mbits/sec    0    218 KBytes
[  5]   7.00-8.00   sec   104 MBytes   875 Mbits/sec    0    218 KBytes
[  5]   8.00-9.00   sec   113 MBytes   945 Mbits/sec    0    218 KBytes
[  5]   9.00-10.00  sec   111 MBytes   934 Mbits/sec    0    218 KBytes
[  5]  10.00-11.00  sec   111 MBytes   935 Mbits/sec    0    218 KBytes
[  5]  11.00-12.00  sec   112 MBytes   937 Mbits/sec    0    218 KBytes
[  5]  12.00-13.00  sec   113 MBytes   948 Mbits/sec    0    218 KBytes
[  5]  13.00-14.00  sec   113 MBytes   946 Mbits/sec    0    218 KBytes
[  5]  14.00-15.00  sec   112 MBytes   939 Mbits/sec    0    218 KBytes
[  5]  15.00-16.00  sec   113 MBytes   945 Mbits/sec    0    218 KBytes
[  5]  16.00-17.00  sec   113 MBytes   952 Mbits/sec    0    218 KBytes
[  5]  17.00-18.00  sec   112 MBytes   938 Mbits/sec    0    218 KBytes
[  5]  18.00-19.00  sec   112 MBytes   938 Mbits/sec    0    218 KBytes
[  5]  19.00-20.00  sec   113 MBytes   949 Mbits/sec    0    218 KBytes
[  5]  20.00-20.04  sec  4.96 MBytes   985 Mbits/sec    0    218 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.04  sec  2.19 GBytes   939 Mbits/sec    0             sender
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

Alles anzeigen

But I still cannot explain why the copies are working so badly with the 10Gb card... and not with the 1Gb one !

Any idea or help is welcome.

mi-hol · 23. März 2022

Zitat von sbocquet

Seems like a NIC buffer problem to me

unfortunately not an issue OMV could fix because this are OS & driver related issues.

I suggest to upgrade to a later kernel version.

5.17 has many networking improvements see https://www.phoronix.com/scan.…&px=Linux-5.17-Networking

sbocquet · 24. März 2022

Unfortunately, only the 5.10 kernel is present in backports of OMV5 (Debian 10)... and 5.17 will probably never be available.

donh · 24. März 2022

You could try the proxmox kernel plugin. I think it is up to 5.15. Proxmox is very network connected so there may be some tweaks to networking.

mi-hol · 24. März 2022

sbocquet I assumed that you tried already drivers from manufacturer. Is that a correct assumption?

https://www.intel.in/content/w…work-adapter-x550-t2.html

sbocquet · 25. März 2022

Zitat von mi-hol

sbocquet I assumed that you tried already drivers from manufacturer. Is that a correct assumption?
https://www.intel.in/content/w…work-adapter-x550-t2.html

Hi,

Not at the moment... I need more free time to try, but I'm going to...

EDIT: I just tried the latest driver version 5.14.6 (Intel Ethernet Drivers and Utilities - Browse /ixgbe stable/5.14.6 at SourceForge.net). It seems to improve stability of the PC to NAS transfers by a little bit... It needs more testing.

sbocquet · 29. März 2022

Little update with my 10Gb Card...

I ran some tests using qbittorrent to download some big files on my NVME disk, and the tranfert rates is stable at 110/112MB/s aka 1Gb/s...

So I think that maybe it's more a SMB protocol performance problem than a disk or NIC one when I'm copying from my PC (Windows) to the NAS (Linux)...

mi-hol · 30. März 2022

Zitat von sbocquet

hey are all linked to the LAN with a 1Gb Netgear GS108E switch (works fine but plan to upgrade very soon to a 10Gb one).

Zitat von sbocquet

the tranfert rates is stable at 110/112MB/s aka 1Gb/s.

did you rule out hardware (i.e CAT5 cables) as the root cause?

pa67 · 31. März 2022

Zitat von donh

Are you using OMV6 or 5? 6 is built on debian bullseye as opposed to 5 on buster. Maybe the drivers and such have improved?
Good Luck

Hi,

I'm using OMV6. I made a lot of tests: I used three different nics, I changed kernels, I tuned samba but with the same results, a random dropping from ~290Mb/s to ~140Mb/s, from pc to nas. Is there a bottleneck somewhere, maybe the mobo or the cpu? I made some tests, using htop.. During the pc-to-Nas test there are a significant presence of state "D" in the "process state" and before that (htop_pc_to_nas) the cpu had an high activity. The nas-to-pc test is quite good.

pa67 · 4. April 2022

Hi,

I added the result of stack info (/proc/[pid]/stack) from the pid in "D" state. The tests are made with this kernel: Debian GNU/Linux, with Linux 5.15.30-1-pve.

sbocquet · 12. April 2022

Hi,

Sorry for this late answer.
In fact, I've just rebuilt a new PC for my needs, and just done some new tests with my NAS and... surprise !

The transfert rate is now stable and constant.

The previous motherboard had a REALTEK 1Gb/s NIC, and a Win 10 up to date OS. My new PC has now a REALTEK 2.5Gb/s NIC and Win 11 OS.

I can't explain why it wasn't working nice between the different NAS NIC Intel i210 vs Intel x550) before, but the fact is that it works now without a problem with my new PC motherboard !

Sometimes, you don't need answers, you just need it to work ! lol

Jetzt mitmachen!