Very slow transfer with 2.5 GBit/s link

    Hi,


    I am running OMV 7 (latest updates) on an Asrock Deskmini X300. I have added an A/E Key M.2 ethernet card for 2.5 GbE (Realtek RTL8125 2.5GbE Controller rev 05). This was working as expected with my GbE switch - transfers being limited by the switch to ~100 MByte/s.


    Yesterday I got myself a 2.5 GBit NBase-T switch (TEM2007X), and things got weird. My PC also uses a an RTL8125 2.5GbE Controller (rev 05) chip. When I hooked both to the new switch, transfer speed from SMB share were somewhere between 3 and 22 MByte/s. A fraction of what I had with the GbE switch. Still, the LED indicates a 2.5 GbE link.


    I tried the following:

    • Swap the cable between my switch and the server (tried three different ones). No change
    • Attached a USB Realtek RTL8153 Gigabit Ethernet Adapter to my PC. No change
    • Plugged the cable into the X300 server's onboard GbE port instead (Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller rev 15). Full GbE speed
    • Changed back to the GbE switch. Full GbE speed
    • Connected the server directly to the PC with the both RTL8125 adapters. Full 2.5 GbE speed (~230 MByte/s)

    So it looks like the connection between the TEM2007X and the RTL8125 in my OMV server is the combo that does not work together well. Any ideas what might be the reason for the hiccup? The driver seems to be r8169 for all adapters.


    This is the ethtool output on my OMV machine:



    And on my PC:


    Okay, a little addendum. Well, two. No, three.

    1. I booted my PC into Windows 11, but copying a large file from my OMV SMB share to the PC was super slow under Windows too.
    2. I tried copying a file from my PC to my OMV server. Lo and behold, it was as fast as hoped for.
    3. And I just tried iperf3. With my OMV machine as the server, it shows the expected speed. With my PC as the iperf server, it shows a fraction of this.

    OMV as iperf3 server:



    My PC as iperf3 server:



    Is this a duplex issue?

    I’d say it’s most likely a Realtek driver issue on Linux. Realtek cards usually don’t perform well on Linux. You may be able to improve it a bit if the driver supports adjusting its buffers.


    Linux : How to tune up receive (TX) and transmit (RX) buffers on network interface | 24x7serversupport Blog
    tuneup TX and RX bufferns on network interface, Tweak settings to increase TX and RX buffers on network interface cards. Linux network interface tweaks.
    www.24x7serversupport.com

    That looks low:



    Same on the other machine.


    But it still worries me that the issue only exists with the "fast" switch and that speeds are 10-20% of GbE...

    try to change to 64KB , 256B is clearly too low:

    Code
    ethtool -g eno1
Ring parameters for eno1:
Pre-set maximums:
RX: 256

    Sounds like the Realtek card/driver doesn’t support buffer adjustments. The onboard Realtek in my system has the same restriction but my intel addin card does allow adjustments.

    Quote from BernH

    I’d say it’s most likely a Realtek driver issue on Linux. Realtek cards usually don’t perform well on Linux. You may be able to improve it a bit if the driver supports adjusting its buffers.

    I don't understand why there is a perception of Realtek drivers not working well under Linux. I have three systems with Realtek 8152, two are the much maligned USB dongles. I rail out the ethernet at 2.33 Gbps iPerf and even running through a 2.5G switch. The disc transfer from a Windows on SSD to an RPI4 with dongle will max out at 280 MB/s. I had played around with SMB buffer sizes in the past but noted no effect.


    I can confirm there was an issue with Bullseye and OMV6 (stuck in half duplex) which did not allow maximum transfer but it's no longer a problem after Bookworm/OMV7. Maybe the problem statement is no longer valid.

    7.4.16-1 (Sandworm)

    Raspberry Pi 4 Model B Rev 1.5

    Linux 6.6.62+rpt-rpi-v8

    Edited once, last by Barney ().

    Quote from Barney

    I don't understand why there is a perception of Realtek drivers not working well under Linux. I have three systems with Realtek 8152, two are the much maligned USB dongles. I rail out the ethernet at 2.33 Gbps iPerf and even running through a 2.5G switch. The disc transfer from a Windows on SSD to an RPI4 with dongle will max out at 280 MB/s. I had played around with SMB buffer sizes in the past but noted no effect.


    I can confirm there was an issue with Bullseye and OMV6 (stuck in half duplex) which did not allow maximum transfer but it's no longer a problem after Bullseye/OMV7. Maybe the problem statement is no longer valid.

    Tradionally they have not performed as well as intel cards, and the fact the you still can't tweak their buffers continues to support that idea in my mind.

    Quote from BernH

    Tradionally they have not performed as well as intel cards, and the fact the you still can't tweak their buffers continues to support that idea in my mind.

    Yeah I'm not buying that you can't change the buffer setting. I've played with it before and noted that I could degrade the transmission speed. I'll revisit this.

    7.4.16-1 (Sandworm)

    Raspberry Pi 4 Model B Rev 1.5

    Linux 6.6.62+rpt-rpi-v8

    This is most likely totally unrelated however I have experienced bad performance with my RTL8125B when ASPM L1.2 is enabled. 1.1 is ok but 1.2 absolutely murders performance. Might be worth a check

    Interesting observation, but I think it is disabled on mine (not sure if I interpret that correctly):


    Code
    04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

    I think the weird-ass internal flat band cable between M.2 card and ethernet port might not be ideal either: https://i.imgur.com/htyh10l.jpeg


    I got the new switch, and I now get the expected transfer rates. Yay! iperf3 shows lots of re-transmits in one direction though, so it is not all perfect:



    The other direction shows 0 retransmits. So maybe I should try getting another of the little M.2 cards at some point. They are not so expensive.

    Can you change that ugly cable? Ribbon cables are so yesterday and not good for high speed transmission (if not laid out correctly.) I don't know what options exist though.

    7.4.16-1 (Sandworm)

    Raspberry Pi 4 Model B Rev 1.5

    Linux 6.6.62+rpt-rpi-v8

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login