I'm having a nightmare of a time here. I have an ESXi 6.5 all-in-one with one VM for my media and one VM for my OMV. It's all running on a J3455B-ITX with 16 GB of RAM and four 3 TB WD Reds through an LSI 9211-8i. Since the LSI HBA (flashed to IT mode) is passed through to the OMV VM, I have 4 GB of RAM dedicated to it. Both VM's are set to use 4 CPUs of the quad core. The MTU is set to 9000 on every NIC in ESXi as well as in the interfaces settings for each VM OS (both Debian).
The drives are setup using SnapRAID and mergerfs.
The test is to copy a 1 GB file from my non-VM Windows 10 desktop workstation to my OMV VM. Another test is to copy a 1 GB file from my media box VM to my OMV VM.
These are the baffling results:
- SMB shares will write a 1 GB file at around 105-110 MB/s using Windows copy
- SMB shares will read a 1 GB file at around 110-115 MB/s using Windows copy
- NFS shares will write a 1 GB file at around 40-45 MB/s using dd
- NFS shares will read a 1 GB file at around 120-125 MB/s using dd
NFS server options are secure,async. I'm have tried sync,no_subtree_check,insecure,no_acl,no_root_squash,wdelay,crossmnt,fsid=1, and every combination thereof.
The client is mounting as r/wsize=524288. I have tried vers=3, 4, and 4.1. I have tried r/wsize=65536 and 1048576,hard,intr,sync,actimeo=0,fsc,nosharecache,nolock,noatime,nodiratime, and every combination thereof.
I have tried creating a separate port group for the OMV NIC. I've tried the E1000E instead of the VMXNET3 driver.
CPU and RAM loads on OMV are minimal in all cases. They are not the bottleneck.
I've run iperf tests and they look fine:
------------------------------------------------------------
Client connecting to 192.168.1.13, TCP port 5001
TCP window size: 1.84 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.11 port 55193 connected with 192.168.1.13 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 2.36 GBytes 2.03 Gbits/sec
I just can't figure out why a separate machine on the network would have much faster writes over Samba than another VM in the same box over NFS. Note that the OMV NFS share isn't even mounted within ESXi yet; this is straight VM to VM. When I do mount the OMV NFS in ESXi, I get the same, slow write speed.
I've also tried OMV 2.1.
Any help would be greatly appreciated.