Network errors: Rx_missed ?

sbocquet · 6. Mai 2018

Hi,

I have just noticed that my NAS network card is getting some rx_missed errors, increasing quickly under heavy load from my PC to the NAS (1 Gb/s link through a Netgear switch between the 2).

Code

root@home-server:~# ethtool -S enp3s0
NIC statistics:
     tx_packets: 3766795
     rx_packets: 17095452
     tx_errors: 0
     rx_errors: 0
     rx_missed: 35234
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 17092316
     broadcast: 1875
     multicast: 1261
     tx_aborted: 0
     tx_underrun: 0

Alles anzeigen

But when I check the stats with ifconfig or other... no errors

Code

root@home-server:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1a:4d:4d:6e:14 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.10/24 brd 192.168.0.255 scope global enp3s0
       valid_lft forever preferred_lft forever
root@home-server:~# ifconfig
enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.10  netmask 255.255.255.0  broadcast 192.168.0.255
        ether 00:1a:4d:4d:6e:14  txqueuelen 1000  (Ethernet)
        RX packets 17094589  bytes 25299659869 (23.5 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3766419  bytes 238437954 (227.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Boucle locale)
        RX packets 1007  bytes 231948 (226.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1007  bytes 231948 (226.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Alles anzeigen

Network chipset on the motherboard is a Realtec

Code

root@home-server:~# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller (rev 02)
00:02.0 VGA compatible controller: Intel Corporation 82G33/G31 Express Integrated Graphics Controller (rev 02)
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 02)
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 02)
00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 02)
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02)
00:1c.4 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 5 (rev 02)
00:1c.5 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 6 (rev 02)
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02)
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02)
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02)
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IR (ICH9R) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 01)
04:07.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller (PHY/Link)

Alles anzeigen

Any idea ?

Cheers

sbocquet · 7. Mai 2018

So after some more tests, the rx_missed increase under heavy load (file copy through SMB for example), or PC backup with UrBackup.
If the NAS network isn't stressed (SSH, SFTP transfert...), the number stay stable. That's logical as the rx_missed is define as:

"Total of RX packets dropped by the HW, because there are no available buffer (i.e. RX queues are full).
The main reason for full RX queues is a "slow" application, which is not able to process packets in a rate they arrive on interface."

I tried to tweak the network buffers but it doesn't seems to have an effect...

Code

# sysctl -p
net.core.rmem_max = 12582912
net.core.wmem_max = 12582912
net.ipv4.tcp_rmem = 10240 87380 12582912
net.ipv4.tcp_wmem = 10240 87380 12582912
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.core.netdev_max_backlog = 5000

I don't know much about that, so if someone has a better understanding or knowledge, any help will be greatly appreciated.

Thx

sbocquet · 8. Mai 2018

My NIC is a Realtek 8111b and it seems that this NIC could have some driver problems... so I follow this guide

https://unixblogger.com/2011/1…111rtl8168-ethernet-card/

The fact is that I have an error when compiling the kernel...

Code

root@home-server:/lib/modules/4.14.0-0.bpo.3-amd64# apt-get install r8168-dkms
Lecture des listes de paquets... Fait
Construction de l'arbre des dépendances
Lecture des informations d'état... Fait
Le paquet suivant a été installé automatiquement et n'est plus nécessaire :
  dmd-bin
Veuillez utiliser « apt autoremove » pour le supprimer.
Les NOUVEAUX paquets suivants seront installés :
  r8168-dkms
0 mis à jour, 1 nouvellement installés, 0 à enlever et 3 non mis à jour.
Il est nécessaire de prendre 88,0 ko dans les archives.
Après cette opération, 1 135 ko d'espace disque supplémentaires seront utilisés.
Réception de:1 http://ftp.fr.debian.org/debian stretch/non-free amd64 r8168-dkms all 8.043.02-1 [88,0 kB]
88,0 ko réceptionnés en 0s (786 ko/s)
Sélection du paquet r8168-dkms précédemment désélectionné.
(Lecture de la base de données... 72719 fichiers et répertoires déjà installés.)
Préparation du dépaquetage de .../r8168-dkms_8.043.02-1_all.deb ...
Dépaquetage de r8168-dkms (8.043.02-1) ...
Paramétrage de r8168-dkms (8.043.02-1) ...
Loading new r8168-8.043.02 DKMS files...
Building for 4.14.0-0.bpo.3-amd64
Building initial module for 4.14.0-0.bpo.3-amd64
Error! Bad return status for module build on kernel: 4.14.0-0.bpo.3-amd64 (x86_64)
Consult /var/lib/dkms/r8168/8.043.02/build/make.log for more information.

Alles anzeigen

Here is the log

Code

root@home-server:/lib/modules/4.14.0-0.bpo.3-amd64# cat /var/lib/dkms/r8168/8.043.02/build/make.log
DKMS make.log for r8168-8.043.02 for kernel 4.14.0-0.bpo.3-amd64 (x86_64)
lundi 7 mai 2018, 23:53:50 (UTC+0200)
make : on entre dans le répertoire « /usr/src/linux-headers-4.14.0-0.bpo.3-amd64 »
  AR      /var/lib/dkms/r8168/8.043.02/build/built-in.o
  CC [M]  /var/lib/dkms/r8168/8.043.02/build/r8168_n.o
  CC [M]  /var/lib/dkms/r8168/8.043.02/build/r8168_asf.o
  CC [M]  /var/lib/dkms/r8168/8.043.02/build/rtl_eeprom.o
/var/lib/dkms/r8168/8.043.02/build/r8168_n.c: In function ‘rtl8168_rx_interrupt’:
/var/lib/dkms/r8168/8.043.02/build/r8168_n.c:25418:28: error: ‘struct net_device’ has no member named ‘last_rx’
                         dev->last_rx = jiffies;
                            ^~
/usr/src/linux-headers-4.14.0-0.bpo.3-common/scripts/Makefile.build:319 : la recette pour la cible « /var/lib/dkms/r8168/8.043.02/build/r8168_n.o » a échouée
make[3]: *** [/var/lib/dkms/r8168/8.043.02/build/r8168_n.o] Erreur 1
make[3]: *** Attente des tâches non terminées....
/usr/src/linux-headers-4.14.0-0.bpo.3-common/Makefile:1525 : la recette pour la cible « _module_/var/lib/dkms/r8168/8.043.02/build » a échouée
make[2]: *** [_module_/var/lib/dkms/r8168/8.043.02/build] Erreur 2
Makefile:146 : la recette pour la cible « sub-make » a échouée
make[1]: *** [sub-make] Erreur 2
Makefile:8 : la recette pour la cible « all » a échouée
make: *** [all] Erreur 2
make : on quitte le répertoire « /usr/src/linux-headers-4.14.0-0.bpo.3-amd64 »

Alles anzeigen

And some infos... The drivers is added but not installed

Code

root@home-server:/lib/modules/4.14.0-0.bpo.3-amd64# grep r816 /etc/modprobe.d/*
/etc/modprobe.d/r8168-dkms.conf:# settings for r8168-dkms
/etc/modprobe.d/r8168-dkms.conf:# map the specific PCI IDs instead of blacklisting the whole r8169 module
/etc/modprobe.d/r8168-dkms.conf:alias   pci:v00001186d00004300sv00001186sd00004B10bc*sc*i*      r8168
/etc/modprobe.d/r8168-dkms.conf:alias   pci:v000010ECd00008168sv*sd*bc*sc*i*                    r8168
/etc/modprobe.d/r8168-dkms.conf:# to blacklist the whole r8169 module
/etc/modprobe.d/r8168-dkms.conf:#blacklist r8169
root@home-server:/lib/modules/4.14.0-0.bpo.3-amd64# cat /etc/modprobe.d/r8168-dkms.conf
# settings for r8168-dkms


# map the specific PCI IDs instead of blacklisting the whole r8169 module
alias   pci:v00001186d00004300sv00001186sd00004B10bc*sc*i*      r8168
alias   pci:v000010ECd00008168sv*sd*bc*sc*i*                    r8168


# if the aliases above do not work, uncomment the following line
# to blacklist the whole r8169 module
#blacklist r8169
root@home-server:/lib/modules/4.14.0-0.bpo.3-amd64# dkms status
r8168, 8.043.02: added
spl, 0.7.6, 4.14.0-0.bpo.3-amd64, x86_64: installed
zfs, 0.7.6, 4.14.0-0.bpo.3-amd64, x86_64: installed

Alles anzeigen

How can I make it work ?

Cheers,

sbocquet · 8. Mai 2018

That is a known bug...

Took the one from buster

https://packages.debian.org/buster/r8168-dkms

... and nothing change, except the transfert rate is more stable at max speed.

sbocquet · 16. Mai 2018

So after upgraded the network card driver, I've made a lot of tests with Iperf for that problem recently (screenshots 1 & 2).

The 9339 missed were from a previous test.

The conclusion is that I have no error even at full Gb/s speed, even in multiple ways at the same time! I was thinking of perhaps a MTU problem but no... MTU is 1500 on both hardware (Win PC & NAS)

Code

root@home-server:~# dkms status
r8168, 8.045.08, 4.14.0-0.bpo.3-amd64, x86_64: installed
spl, 0.7.6, 4.14.0-0.bpo.3-amd64, x86_64: installed
zfs, 0.7.6, 4.14.0-0.bpo.3-amd64, x86_64: installed


root@home-server:~# ifconfig
enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.10  netmask 255.255.255.0  broadcast 192.168.0.255
        ether 00:1a:4d:4d:6e:14  txqueuelen 1000  (Ethernet)
        RX packets 59949869  bytes 89097667373 (82.9 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11276232  bytes 17929411660 (16.6 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  base 0x5000


lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Boucle locale)
        RX packets 34290  bytes 8240627 (7.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 34290  bytes 8240627 (7.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


On my Win 10 PC :
C:\Users\Stéphane\Desktop\iperf-2.0.9-win64>netsh interface ipv4 show subinterfaces
   MTU  État détect supp O entrant  O sortant  Interface
------  ---------------  ---------  ---------  -------------
  1500                1          0    3733212  VirtualBox Host-Only Network #2
4294967295                1          0     227540  Loopback Pseudo-Interface 1
  1500                1  7269188119  16013239872  Connexion au réseau local


Even a test in bidirectionnal is ok…
C:\Users\Stéphane\Desktop\iperf-2.0.9-win64>iperf -c home-server -d -t 30
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to home-server, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.11 port 51869 connected with 192.168.0.10 port 5001
[  5] local 192.168.0.11 port 5001 connected with 192.168.0.10 port 55662
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  3.27 GBytes   937 Mbits/sec
[  5]  0.0-30.0 sec  5.31 GBytes  1.52 Gbits/sec

Alles anzeigen

The thing is that if I try to copy a 16G test file with FileZilla through SFTP, no errors (see attached screenshots 3) at near the same ‘full’ speed...

But if I copy this same file to the NAS with a SMB share, there is a lot of rx_missed... From the NAS to the PC, no error at all.

Could it be some sort of samba problem ???

I'm totally lost with this...

Jetzt mitmachen!