I have two servers connected using a direct connect 40gb ethernet (2 Chelsio T580’s). Iperf3 shows about 38gb transfer rates either direction. I’m am trying to read off a samba share on the omv machine and direct it to /dev/null on the client machine. The omv machine is running debian 11 bullseye and the client is running ubuntu 21.10 with a 5.10 kernel (chelsio drivers won’t compile with 5.11+).
For storage, I’m using a raidz2 on the omv machine with 8 nvme 2tb drives (pcie-3). Benchmarking of this array with fio gives throughput of around 4GB/s (bytes, not bits). Reads from this array over the 40gbe direct connect are maxing out at about 200 MB/s. I am using encryption on the array, gzip, but I have 2 Intel 8960 QAT cards that handle it in hardware, and the above benchmarking includes these cards and the compression settings for ZFS. Regular RAID performs better on tests using fio, but both far outperform transfer rates.
The omv machine is an epyc 7401 w/ 128gb ddr4-3200 with 24/48 cores. The client machine is a supermicro 4 cpu, 12 core/cpu, xeon 4657L2 (48/96 cores total) with 512 GB DDR3-1600 RAM.
I was expecting performance much closer to 4GB/s and I’m off by a factor of about a factor of 20. I’m not sure where the bottleneck is, but I’m thinking it may be CPU. The chelsio cards are capable of full TCP offload, and dmesg shows that the driver is offload capable.
Any ideas where to start troubleshooting this?
Thanks in advance.