Poor speed with SMB over 40gbe

I have two servers connected using a direct connect 40gb ethernet (2 Chelsio T580’s). Iperf3 shows about 38gb transfer rates either direction. I’m am trying to read off a samba share on the omv machine and direct it to /dev/null on the client machine. The omv machine is running debian 11 bullseye and the client is running ubuntu 21.10 with a 5.10 kernel (chelsio drivers won’t compile with 5.11+).

For storage, I’m using a raidz2 on the omv machine with 8 nvme 2tb drives (pcie-3). Benchmarking of this array with fio gives throughput of around 4GB/s (bytes, not bits). Reads from this array over the 40gbe direct connect are maxing out at about 200 MB/s. I am using encryption on the array, gzip, but I have 2 Intel 8960 QAT cards that handle it in hardware, and the above benchmarking includes these cards and the compression settings for ZFS. Regular RAID performs better on tests using fio, but both far outperform transfer rates.

The omv machine is an epyc 7401 w/ 128gb ddr4-3200 with 24/48 cores. The client machine is a supermicro 4 cpu, 12 core/cpu, xeon 4657L2 (48/96 cores total) with 512 GB DDR3-1600 RAM.

I was expecting performance much closer to 4GB/s and I’m off by a factor of about a factor of 20. I’m not sure where the bottleneck is, but I’m thinking it may be CPU. The chelsio cards are capable of full TCP offload, and dmesg shows that the driver is offload capable.

Any ideas where to start troubleshooting this?

Thanks in advance.

After a whole day of benchmarking and testing, the problem doesn’t seem to be samba. I can’t copy any file faster than about 800 MB/s, even from one ramdisk to another. I have no idea what is causing this.

Motherboard is fully up to date (asrock epyc d8).

I get higher rates using fio with 6 streams - but any single copy of a file remains really, really slow. I’m still lost as to why. It’s not the network card, as it occurs locally on the machine.

Using nautilus, I can get faster transfers than with smbnetfs (by about a factor of 2, about 125 vs 250 or so MB/s).

I even set up a RAID array on some pcie gen4 2tb gigabyte aorus in an x16 gen 3 slot (so slightly bottlenecked there), and no improvement. The cpu is not maxed out although some cores do hit 75% use - making me think this is somehow connected.

I’ve tried dd, cp, and rsync so far and all are slow.

fio on the RAID array can get close to 100 gbps (10 gBps) - but I can’t seem to realize that in any actual copying, making me wonder if it’s a parallelism issue. Still, I’ve seen demos online on youtube with people copying files much, much faster over a network (single files). Something is not right.