Help troubleshooting Issues pulling Docker images and SSH dropping

2424 · 20. November 2018

I'm hoping someone can help me troubleshoot an issue with failed checksums when pulling docker images:

# docker pull linuxserver/plex
Using default tag: latest
latest: Pulling from linuxserver/plex
32802c0cfa4d: Verifying Checksum
1643fa16260a: Download complete
5768751b4c70: Download complete
05b8a8268766: Download complete
cef14ece6765: Download complete
47930d69bbc1: Download complete
c21e238ddbe4: Verifying Checksum
8294a1b762c6: Download complete
4007aef5d48e: Verifying Checksum
3e2d225ae398: Download complete
filesystem layer verification failed for digest sha256:4007aef5d48e666701d5e2608e6f8ce77525bc37563e05bbd4c66b878de500f0

and

#docker pull linuxserver/plex
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Amazon Root CA 1")

I am also having SSH drop my connection intermittently with "Incorrect MAC received on packet".

Looking at the above i thought it was a network issue but i have installed OMV on another system connected to the same LAN and configured the same way but it does not have these issues.

Currently OMV is installed as a VM on ESXi 6.7 but i have also tried installing it directly on the host and installing OMV 3 but i have the same issue. I have used different NIC, RAM, HDD, changed BIOS settings, disabled TCP offloading and RX checksumming then i thought it might be too many devices on the PCI bus so i tried installing with the min config (no PCI cards) but nothing seems to change the issue!

Hardware is
Motherboard: Z79-WS
CPU: Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz
RAM: 24GB DDR3
NIC: Intel PRO/1000 PT Quad Port LP and onboard Intel I218-LM and Intel I210
Storage 3x HBA: LSI Logic / Symbios Logic LSI2008
Drives: 24 WD Red 3TB

Any help would be greatly appreciated!!!

Stramm · 21. November 2018

You could try to remove the image you downloaded and stop and restart docker.

On cli (with root) check if a plex image is existent

Code

//check only for the plex image
docker images -a |  grep "plex"


//or list all images 
docker images

I don't use Plex but I can show you the output for Emby

Code

REPOSITORY          TAG                 IMAGE ID            CREATED           SIZE
emby/embyserver     latest              835050fb3fae        2 months ago      213MB

To delete the image (for my Emby example, you would have to adapt this to fit your Plex image)

Code

docker rmi emby/embyserver

then stop and restart the docker service either using the web gui or cli. For latter:

Code

service docker stop
service docker start

Now retry and download the Plex image again... good luck.

2424 · 22. November 2018

Thanks for the reply.

It never actually installs the image so i can't remove it (no output from the grep above). When i try and pull Emby i get the same issue as above:

# docker pull emby/embyserver
Using default tag: latest
latest: Pulling from emby/embyserver
f70adabe43c0: Pull complete
bf44fd9e1723: Downloading [================> ] 26.33MB/80.7MB
3d2016cdab4a: Verifying Checksum
accefb972ef0: Verifying Checksum
1bf977e07153: Verifying Checksum
filesystem layer verification failed for digest sha256:1bf977e071533b6317a6f6bfd50c33582b9675546f05ecb32848db7936a1bf62

I have got it to install "Hello World" which seems to work as it is a single image and only small.

# docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
d1725b59e92d: Pull complete
Digest: sha256:0add3ace90ecb4adbf7777e9aacf18357296e799f81cabc9fde470971e499788
Status: Downloaded newer image for hello-world:latest

Whatever the issue is seems to occur with larger files and i can only assume its because something is messing with the file/packets causing them to fail checksum. as you can see bellow i am not seeing errors in Ethtool, the bellow shows tx_tcp_seg_good and rx_csum_offload_good but these dont seem to be incrementing anymore since i disabled them with ethtool.

Code

# ethtool -S ens192f0


NIC statistics:


     rx_packets: 1590641512


     tx_packets: 5121883032


     rx_bytes: 184054406439


     tx_bytes: 7719161096022


     rx_broadcast: 409165


     tx_broadcast: 1293


     rx_multicast: 10425


     tx_multicast: 33


     rx_errors: 0


     tx_errors: 0


     tx_dropped: 0


     multicast: 10425


     collisions: 0


     rx_length_errors: 0


     rx_over_errors: 0


     rx_crc_errors: 0


     rx_frame_errors: 0


     rx_no_buffer_count: 119


     rx_missed_errors: 12


     tx_aborted_errors: 0


     tx_carrier_errors: 0


     tx_fifo_errors: 0


     tx_heartbeat_errors: 0


     tx_window_errors: 0


     tx_abort_late_coll: 0


     tx_deferred_ok: 0


     tx_single_coll_ok: 0


     tx_multi_coll_ok: 0


     tx_timeout_count: 0


     tx_restart_queue: 15873330


     rx_long_length_errors: 0


     rx_short_length_errors: 0


     rx_align_errors: 0


     tx_tcp_seg_good: 54944040


     tx_tcp_seg_failed: 0


     rx_flow_control_xon: 8063880


     rx_flow_control_xoff: 8063880


     tx_flow_control_xon: 3


     tx_flow_control_xoff: 15


     rx_csum_offload_good: 640798714


     rx_csum_offload_errors: 0


     rx_header_split: 0


     alloc_rx_buff_failed: 0


     tx_smbus: 0


     rx_smbus: 0


     dropped_smbus: 0


     rx_dma_failed: 0


     tx_dma_failed: 0


     rx_hwtstamp_cleared: 0


     uncorr_ecc_errors: 0


     corr_ecc_errors: 0


     tx_hwtstamp_timeouts: 0


     tx_hwtstamp_skipped: 0

Alles anzeigen

Code

# ethtool -S ens192f1


NIC statistics:


     rx_packets: 806479


     tx_packets: 1355060


     rx_bytes: 108908150


     tx_bytes: 2010427793


     rx_broadcast: 410520


     tx_broadcast: 0


     rx_multicast: 10436


     tx_multicast: 48


     rx_errors: 0


     tx_errors: 0


     tx_dropped: 0


     multicast: 10436


     collisions: 0


     rx_length_errors: 0


     rx_over_errors: 0


     rx_crc_errors: 0


     rx_frame_errors: 0


     rx_no_buffer_count: 0


     rx_missed_errors: 0


     tx_aborted_errors: 0


     tx_carrier_errors: 0


     tx_fifo_errors: 0


     tx_heartbeat_errors: 0


     tx_window_errors: 0


     tx_abort_late_coll: 0


     tx_deferred_ok: 0


     tx_single_coll_ok: 0


     tx_multi_coll_ok: 0


     tx_timeout_count: 0


     tx_restart_queue: 3


     rx_long_length_errors: 0


     rx_short_length_errors: 0


     rx_align_errors: 0


     tx_tcp_seg_good: 128668


     tx_tcp_seg_failed: 0


     rx_flow_control_xon: 14


     rx_flow_control_xoff: 14


     tx_flow_control_xon: 0


     tx_flow_control_xoff: 0


     rx_csum_offload_good: 503902


     rx_csum_offload_errors: 0


     rx_header_split: 0


     alloc_rx_buff_failed: 0


     tx_smbus: 0


     rx_smbus: 0


     dropped_smbus: 0


     rx_dma_failed: 0


     tx_dma_failed: 0


     rx_hwtstamp_cleared: 0


     uncorr_ecc_errors: 0


     corr_ecc_errors: 0


     tx_hwtstamp_timeouts: 0


     tx_hwtstamp_skipped: 0

Alles anzeigen

Stramm · 22. November 2018

In the VM try setting the network type to NAT... after that I'm out of ideas

https://forums.docker.com/t/al…for-digest-sha256/26518/4

2424 · 22. November 2018

I found a similar post for Virtualbox issues and tried to do that but ESXi does not include a NAT option (only on VMware desktop). In my case i have the NIC configured as PCI pass-through so ESXi should not be doing much with the NIC. In case it is in some way i tried installing directly on the machine without ESXi but got the same problem

I'm pretty much out of ideas as well... At this point all i can think of is some strange incompatibility with the CPU or motherboard as i have swapped out every other component and the issue remains.

In case someone is aware of issues with one of the motherboard components or CPU i will include the bellow outputs from ESXi:

Code

lspci


0000:00:00.0 Bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller


0000:00:01.0 Bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [PCIe RP[0000:00:01.0]]


0000:00:02.0 Display controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller


0000:00:03.0 Multimedia controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller


0000:00:14.0 Serial bus controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller


0000:00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1


0000:00:19.0 Network controller: Intel Corporation Ethernet Connection (2) I218-LM [vmnic0]


0000:00:1a.0 Serial bus controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2


0000:00:1b.0 Multimedia controller: Intel Corporation 9 Series Chipset Family HD Audio Controller


0000:00:1c.0 Bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [PCIe RP[0000:00:1c.0]]


0000:00:1c.3 Bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [PCIe RP[0000:00:1c.3]]


0000:00:1c.4 Bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 [PCIe RP[0000:00:1c.4]]


0000:00:1c.6 Bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 [PCIe RP[0000:00:1c.6]]


0000:00:1d.0 Serial bus controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1


0000:00:1f.0 Bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller


0000:00:1f.3 Serial bus controller: Intel Corporation 9 Series Chipset Family SMBus Controller


0000:01:00.0 Bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch


0000:02:08.0 Bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch


0000:02:09.0 Bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch


0000:02:10.0 Bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch


0000:02:11.0 Bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch


0000:03:00.0 Mass storage controller: LSI Logic / Symbios Logic LSI2008 [vmhba0]


0000:04:00.0 Mass storage controller: LSI Logic / Symbios Logic LSI2008 [vmhba1]


0000:05:00.0 Mass storage controller: LSI Logic / Symbios Logic LSI2008 [vmhba2]


0000:06:00.0 Mass storage controller: LSI / Symbios Logic MegaRAID SAS GEN2 Controller [vmhba3]


0000:08:00.0 Bridge:


0000:09:01.0 Bridge:


0000:09:02.0 Bridge:


0000:09:03.0 Bridge:


0000:09:04.0 Bridge:


0000:09:05.0 Bridge:


0000:09:06.0 Bridge:


0000:09:07.0 Bridge:


0000:0c:00.0 Mass storage controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller [vmhba4]


0000:0f:00.0 Network controller: Intel Corporation I210 Gigabit Network Connection [vmnic1]


0000:10:00.0 Serial bus controller: VIA Technologies, Inc. P8P67 Deluxe Motherboard


0000:11:00.0 Bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch


0000:12:00.0 Bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch


0000:12:01.0 Bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch


0000:13:00.0 Network controller: Intel Corporation PRO/1000 PT Quad Port LP Server Adapter [vmnic2]


0000:13:00.1 Network controller: Intel Corporation PRO/1000 PT Quad Port LP Server Adapter [vmnic3]


0000:14:00.0 Network controller: Intel Corporation PRO/1000 PT Quad Port LP Server Adapter [vmnic4]


0000:14:00.1 Network controller: Intel Corporation PRO/1000 PT Quad Port LP Server Adapter [vmnic5]


0000:15:00.0 Serial bus controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller

Alles anzeigen

Code

esxcli hardware cpu global get
   CPU Packages: 1
   CPU Cores: 4
   CPU Threads: 8
   Hyperthreading Active: true
   Hyperthreading Supported: true
   Hyperthreading Enabled: true
   HV Support: 3
   HV Replay Capable: false
   HV Replay Disabled Reasons: incompatible CPU

Code

esxcli hardware cpu list
CPU:0
   Id: 0
   Package Id: 0
   Family: 6
   Model: 60
   Type: 0
   Stepping: 3
   Brand: GenuineIntel
   Core Speed: 3497972602
   Bus Speed: 99942069
   APIC ID: 0x0
   Node: 0
   L2 Cache Size: 262144
   L2 Cache Associativity: 8
   L2 Cache Line Size: 64
   L2 Cache CPU Count: 2
   L3 Cache Size: 8388608
   L3 Cache Associativity: 16
   L3 Cache Line Size: 64
   L3 Cache CPU Count: 2


CPU:1
   Id: 1
   Package Id: 0
   Family: 6
   Model: 60
   Type: 0
   Stepping: 3
   Brand: GenuineIntel
   Core Speed: 3497972602
   Bus Speed: 99942069
   APIC ID: 0x1
   Node: 0
   L2 Cache Size: 262144
   L2 Cache Associativity: 8
   L2 Cache Line Size: 64
   L2 Cache CPU Count: 2
   L3 Cache Size: 8388608
   L3 Cache Associativity: 16
   L3 Cache Line Size: 64
   L3 Cache CPU Count: 2
...etc to CPU 7

Alles anzeigen

I did also find this this and this which is a pretty intense read but given i still get the issue when passing through or installing with the on-board NIC (and i have tried to disable TCP offload) i assume it cant be it... but it does seems to be close to the issues i am having.... I might have to see if i can find a non Intel NIC to test... I have another motherboard and CPU from my old desktop but it does not have on-board graphics so does not have sufficient PCI slots once i install a single slot GPU.

The only other ideas i can think of are to try some ECC DIMMs, non Intel NIC, Buy a new Motherboard/ CPU (Ughhh...) or maybe taking the PCI slots down to gen 1 or 2...

It's very hard when i cant really tell if the cheksums are being incorrectly computed/data being corrupted by:
CPU (No issues when i run a SSL speed tests; "openssl speed")
Memory fault (Have tried different DIMMS)
LAN issue (Have another OMV machine on the LAN with the same network config without issues)
NIC Issue (Have tried multiple NICs)
OS Issue (OMV 3 has the same problem)
PCI Address conflict (Same issue when PCI cards removed)
etc...

Any other suggestions for tests to isolate would be appreciated... I don't want to spend more cash on hardware just to have the same problem!

Jetzt mitmachen!