Slow Network download speed - upload fine

blublub · 2. November 2016

Hi

I have a strange issue that I just noticed yesterday.

When I upload files to OMV I get decent speeds around 70-110 MB/s but when I download files from OMV to my PCs the speed is stuck at roughly 30 MB/s.

Config:
OMV 2.2.x
1 GBE Intel dual Nic
16 GB Ecc RAM
ZFS Raid z2

What I have tried:

Downloaded from ZFS shares and ext4 shares (latter is on a SSD) - so not a disk/ZFS issue

Changed Nic and switched switch ports

Changed cables

Tried different Kernels (3.2, 3.16 and 4.4)

Tried SMB shares and FTP download

Tested different PCs

...all to no avail - stuck at 30 MB/s

Is there any setting in OMV or debian that could limit the send speed of OMV ?

Last time I transferred large files from OMV to my PCs I did not have this issue, though I usually push to OMV and hence I can't remember when every was ok before...

Any thoughts here?

ryecoaaron · 2. November 2016

Zitat von blublub

Is there any setting in OMV or debian that could limit the send speed of OMV ?

No.

Have you checked the read speed of the zfs array directly on the system? I would test that by timing the copy of a large file to another fast drive in the system. I would test the raw networking with iperf as well.

blublub · 2. November 2016

I also downloaded from the SSD which is a ext4 volume.
And as I said upload us 110mb/s

So altogether I think array speed and network speed in general can't be the issue as it is an isolated uploaded/sending problem from the NAS

ryecoaaron · 2. November 2016

Zitat von blublub

So altogether I think array speed and network speed in general can't be the issue as it is an isolated uploaded/sending problem from the NAS

Actually I would say the networking is fine but the array read speed is suspect. Just because the write speed is fast does not mean the read speed will be. You haven't shown me any evidence the read speed is fast.

blublub · 2. November 2016

Yeah fine I will test it when I get home

But why should the read speed of the SSD - which is not in any array - and is not ZFS (it's ext4) be bad as well then - that's why I think it is unrelated to disk speeds

ryecoaaron · 2. November 2016

I read your post wrong. I thought you were saying download from the ssd was fine. So, that means it could be the networking. Once again, good upload doesn't mean good download. I would check with iperf.

blublub · 2. November 2016

Zitat von ryecoaaron

I read your post wrong. I thought you were saying download from the ssd was fine. So, that means it could be the networking. Once again, good upload doesn't mean good download. I would check with iperf.

NP

Nevertheless I did some testing:

Copy speed from ZFS Array to SSD: roughly 230-280MB/s - could be better, but I think this is ok

iperf:

OMV server as client:

Code

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   335 MBytes   281 Mbits/sec

Windows PC as client:

Code

[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.08 GBytes   926 Mbits/sec

Switch port info:

Code

Port Type 	Normal 	
Port Channel ID 	Disable 	
Port Role 		
STP Mode 	Enable 	
STP State 		
Admin Mode 	Enable 	
Flow Control Mode 	Disable 	
LACP Mode 	Enable 	
Physical Mode 	Auto 	
Physical Status 	1000 Mbps 	
Link Status 	Link Up 	
Link Trap 	Enable 	
Packets RX and TX 64 Octets 	405336 	
Packets RX and TX 65-127 Octets 	38812 	
Packets RX and TX 128-255 Octets 	19158 	
Packets RX and TX 256-511 Octets 	32712 	
Packets RX and TX 512-1023 Octets 	2611 	
Packets RX and TX 1024-1518 Octets 	397939 	
Packets RX and TX 1519-2047 Octets 	0 	
Packets RX and TX 2048-4095 Octets 	0 	
Packets RX and TX 4096-9216 Octets 	0 	
Octets Received 	607866713 	
Packets Received 64 Octets 	2870 	
Packets Received 65-127 Octets 	2648 	
Packets Received 128-255 Octets 	7585 	
Packets Received 256-511 Octets 	8886 	
Packets Received 512-1023 Octets 	905 	
Packets Received 1024-1518 Octets 	397107 	
Packets Received > 1518 Octets 	0 	
Total Packets Received Without Errors 	420001 	
Unicast Packets Received 	419565 	
Multicast Packets Received 	288 	
Broadcast Packets Received 	148 	
Receive Packets Discarded 	0 	
Total Packets Received with MAC Errors 	0 	
Jabbers Received 	0 	
Fragments Received 	0 	
Undersize Received 	0 	
Alignment Errors 	0 	
Rx FCS Errors 	0 	
Overruns 	0 	
Total Received Packets Not Forwarded 	0 	
802.3x Pause Frames Received 	0 	
Unacceptable Frame Type 	0 	
Total Packets Transmitted (Octets) 	43703933 	
Packets Transmitted 64 Octets 	402466 	
Packets Transmitted 65-127 Octets 	36164 	
Packets Transmitted 128-255 Octets 	11573 	
Packets Transmitted 256-511 Octets 	23826 	
Packets Transmitted 512-1023 Octets 	1706 	
Packets Transmitted 1024-1518 Octets 	832 	
Packets Transmitted > 1518 Octets 	0 	
Maximum Frame Size 	9216 	
Total Packets Transmitted Successfully 	476567 	
Unicast Packets Transmitted 	448902 	
Multicast Packets Transmitted 	18898 	
Broadcast Packets Transmitted 	8767 	
Transmit Packets Discarded 	0 	
Total Transmit Errors 	0 	
Total Transmit Packets Discarded 	0 	
Single Collision Frames 	0 	
Multiple Collision Frames 	0 	
Excessive Collision Frames 	0 	
Dropped Transmit Frames 	0 	
STP BPDUs Received 	0 	
STP BPDUs Transmitted 	0 	
RSTP BPDUs Received 	0 	
RSTP BPDUs Transmitted 	2020 	
MSTP BPDUs Received 	0 	
MSTP BPDUs Transmitted 	0 	
802.3x Pause Frames Transmitted 	0 	
EAPOL Frames Received 	0 	
EAPOL Frames Transmitted 	0

Alles anzeigen

So there is definitively something wrong with OMV NICs sending data - and I have no clue were to look...

Edit:

Some more info:

ethtool:

Code

Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: on
        Supports Wake-on: pumbg
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

Alles anzeigen

and:

Code

mii-tool
eth0: negotiated 1000baseT-FD flow-control, link ok

So takeaway here: NIC is set to auto negotiate and is running 1GB Full duplex - thought it might be a duplex issue here, but doesn't seem to be

Ok some more stuff I tested:

disabled TLP ( so no power management)
set Set PCIe Links to x3 (I need the slow x1 speed because my DVB card wont be detected in BIOS otherwise)
checked my switch - all ports are set to autonegotiate, so the same as the NIC
rest OMV network via firstaid

And the issue still persists and I am totally out of ideas.....

ryecoaaron · 2. November 2016

This is rare with Intel NICs. Normally, I would say upgrade the kernel but you already did that. Maybe a switch issue? Not sure what the issue is.

blublub · 2. November 2016

Yeah, I did that.

I updated the switch info on the connected port.

The switch being the issue doesn't make much sense to me neither the make/model nor firmware changed recently

ryecoaaron · 2. November 2016

Boot systemrescuecd on the system and try iperf again. I know this isn't an OMV issue but that will eliminate a few things.

blublub · 2. November 2016

OK will do that, but might take till friday

ryecoaaron · 2. November 2016

Zitat von blublub

OK will do that, but might take till friday

Trust me... I'm not in a rush

blublub · 2. November 2016

Ok,

I just updated the firmware of the switch and connected my notebook directly to the NAS - same issue.

But what I noticed was that if I disconnect the cable and reconnect it the speed is set to 10mbs FD !? - only after I hard-reboot of the NAS it gets back to 1Gbs.
This happened with the switch and my notebook.

I will still boot with the System rescue disc tomorrow or on friday an re-check iperf but somehow it seems to me like a broken onboard NIC - right?

EDIT:

According to this page:
http://noahdavids.org/self_published/duplex_mismatch.html

It doesn't seem to be a duplex mismatch as all my counters are "0"

ryecoaaron · 3. November 2016

This definitely sounds like a NIC and/or switch issue.

blublub · 3. November 2016

Zitat von ryecoaaron

This definitely sounds like a NIC and/or switch issue.

Can't be the switch because the issue persists when I connect my Notebook directly to the NAS.

I will try another LiveOS and if it persists I will contact the Asrockrack support

blublub · 5. November 2016

OK I solved the mystery for the main part:

It was my Antivirus scanner on the Windows machine limiting inbound speed to max 40Mbs. After de-installation it is back to eral GbE speed - honestly WTF !
I still suffer from slower than expected speed between Linux and Linux systems (50Mb/s) but that is likely a firmware issue on the Linux machine with its Realtek card..

Thx for helping me out here

Jetzt mitmachen!