Poor NFS performance (50% or more overhead in OMV6 vs. TrueNAS Scale)

TheLinuxGuy · 20. Oktober 2022

I'm doing a few experiments at home using proxmox virtual machines with PCIe passthrough NVME disks to the VM. I am seeing very poor NFS performance in OpenMediaVault 6.0 when I compare the same setup with different operating systems.

The VMs have the following hardware settings (all same):

- 4 vCPU

- 2GB RAM

- MTU 9000 / jumbo frames

- Virtio NIC driver

- PCI passthru NVME drive 256GB Samsung PRO 950

I use fio to test SEQUENTIAL writes using these settings

Code

fio --name=fiotest --filename=/samsung/test1 --size=16Gb --rw=write --bs=1M --direct=1 --numjobs=8 --ioengine=libaio --iodepth=8 --group_reporting --runtime=60 --startdelay=60

NVME write tests from OMV 6.0 OS show expected raw-disk performance:

Code

Run status group 0 (all jobs):
  WRITE: bw=897MiB/s (941MB/s), 897MiB/s-897MiB/s (941MB/s-941MB/s), io=52.6GiB (56.5GB), run=60062-60062msec

Run status group 0 (all jobs):
  WRITE: bw=895MiB/s (938MB/s), 895MiB/s-895MiB/s (938MB/s-938MB/s), io=52.5GiB (56.4GB), run=60072-60072msec

The proxmox host has 10GB SFP+ uplinks to a client in LAN switch doing these NFS tests. We mount the share via

Code

 mount -o nfsvers=4.2 192.168.1.190:/sammy /mnt/derp/

But the performance is very poor in OpenMediaVault 6.0 - 3 separate attempts output

Code

  WRITE: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=16.0GiB (17.2GB), run=60151-60151msec

  WRITE: bw=208MiB/s (218MB/s), 208MiB/s-208MiB/s (218MB/s-218MB/s), io=12.8GiB (13.8GB), run=63341-63341msec

  WRITE: bw=282MiB/s (296MB/s), 282MiB/s-282MiB/s (296MB/s-296MB/s), io=16.7GiB (17.9GB), run=60685-60685msec

Shutting down the OMV 6.0 VM and starting up an UNRAID VM with the same hardware settings; then mounting NFS share on client with same settings result in these speed results (very close to max performance of raw disk/fio result)

Code

  WRITE: bw=747MiB/s (784MB/s), 747MiB/s-747MiB/s (784MB/s-784MB/s), io=43.8GiB (47.1GB), run=60057-60057msec

A different OS, TrueNAS Scale with same hardware and using NFS share (only difference here is no brtfs support in TNS - so I had to wipe disk and make it ZFS which uses DDR cache and speeds up the performance but skews the numbers as you see here):

Code

WRITE: bw=1468MiB/s (1539MB/s), 1468MiB/s-1468MiB/s (1539MB/s-1539MB/s), io=91.1GiB (97.8GB), run=63547-63547msec

I think there's some performance optimizations possibly missing in OMV for NFS? Given UNRAID is able to provide 90% of the raw-disk performance over NFS some configuration setting feels amiss to me. The only variables in my tests are operating system (and ZFS exception case for truenas as noted before but still I can almost max out my 10Gbit network with all other OS but not with OMV over NFS).

votdev · 20. Oktober 2022

Zitat von TheLinuxGuy

I think there's some performance optimizations possibly missing in OMV for NFS?

Sure, optimizations are possible. Why not contributing them to the openmediavault project? You are already testing it, so you can also test various settings and if you think they are fine, contribute them to the project so that everyone can participate from your tests and experience. This is how open source works.

Krisbee · 20. Oktober 2022

How do I know this is not comparing apples to oranges, or this data is simply a reflection of the sustained write characteristics of a consumer SSD? Have you used pci-e passthrough for the nvme data drive or is it passed as raw disk?

You can dismiss the TrueNAS Scale data in this context and you only mention btrfs is in the mix towards the bottom of your post.

How do I know the mount params for the data drive are the same in each VM? Are the nfs export params the same in each VM? Are the mount characteristics on your unamed client the same in all cases (check output of nfsstat -m on client) ? Did you check iperf3 both client to server, server to client in all cases? Did you monitor the writes in the VMs and or the network traffic ?

OMV6 is just debian 11 under the hood, so what do you think these “optimisations” might be?

siteswap423 · 20. Oktober 2022

There is an old post on this forum with some fine tunings about Nfs.

Check This

"https://forum.openmediavault.org/index.php?thread/31295-10gbe-nfs-tuning/"

Best regards,

Daniele.

Krisbee · 20. Oktober 2022

siteswap423 That's a great post by micdon showing how NB TCP stack optimisation can be in 10Gbe setups, especially when virtualised. The export params used look like the OMV6 defaults to me, you don't have to explicitly state async. Whether that's applicable in Proxmox is another question.

TheLinuxGuy · 20. Oktober 2022

Zitat von votdev

Sure, optimizations are possible. Why not contributing them to the openmediavault project? You are already testing it, so you can also test various settings and if you think they are fine, contribute them to the project so that everyone can participate from your tests and experience. This is how open source works.

I don't mind contributing and sharing what may fix the performance issues I am seeing, Sure. That's the purpose of this post actually.

Zitat von Krisbee

Have you used pci-e passthrough for the nvme data drive or is it passed as raw disk?

The VMs are directly 'passed thru' to VMs - it is not "mapped as raw disk" (aka there is no overhead from proxmox OS here since its passthru). More info: https://pve.proxmox.com/wiki/PCI(e)_Passthrough

Zitat von siteswap423

Check This
"https://forum.openmediavault.org/index.php?thread/31295-10gbe-nfs-tuning/"

Thank you I will check and play with these settings this weekend.

TheLinuxGuy · 22. Oktober 2022

Update: I have been able to improve NFS performance on OpenMediaVault 6 to double what it was; the key to this improvement is the settings on the NFS share. UNRAID off the shelf still performs much faster than OMV6 with default settings; the tweak below did improve things 50%+

I noticed "async" is used in unraid and also mentioned in https://wiki.archlinux.org/title/NFS#Performance_tuning

I configured my OMV share with no_subtree_check,no_root_squash,async settings.

I found OMV's default settings use 'subtree_check' setting which the manual of NFS exports explicitly call it as a bad choice:
"From release 1.1.0 of nfs-utils onwards, the default will be no_subtree_check as subtree_checking tends to cause more problems than it is worth. If you genuinely require subtree checking, you should explicitly put that option in the exports file. If you put neither option, exportfs will warn you that the change is pending." -- source: https://linux.die.net/man/5/exports

Code

root@pve:~# fio --name=fiotest --size=16Gb --rw=write --bs=1M --direct=1 --numjobs=8 --ioengine=libaio --iodepth=8 --group_reporting --runtime=60 --startdelay=60 --filename=/mnt/derp/filemeow
fiotest: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
...
fio-3.25
Starting 8 processes
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
Jobs: 8 (f=8): [W(8)][100.0%][eta 00m:00s]                       
fiotest: (groupid=0, jobs=8): err= 0: pid=302679: Sat Oct 22 01:24:19 2022
  write: IOPS=535, BW=536MiB/s (562MB/s)(31.4GiB/60050msec); 0 zone resets
    slat (usec): min=111, max=1052, avg=242.86, stdev=47.70
    clat (msec): min=9, max=6748, avg=119.14, stdev=446.78
     lat (msec): min=9, max=6748, avg=119.38, stdev=446.78
    clat percentiles (msec):
     |  1.00th=[   55],  5.00th=[   62], 10.00th=[   64], 20.00th=[   67],
     | 30.00th=[   69], 40.00th=[   71], 50.00th=[   72], 60.00th=[   74],
     | 70.00th=[   78], 80.00th=[   82], 90.00th=[   92], 95.00th=[  106],
     | 99.00th=[ 1586], 99.50th=[ 1620], 99.90th=[ 6678], 99.95th=[ 6745],
     | 99.99th=[ 6745]
   bw (  KiB/s): min=98304, max=964608, per=100.00%, avg=782881.49, stdev=27190.90, samples=672
   iops        : min=   96, max=  942, avg=764.52, stdev=26.55, samples=672
  lat (msec)   : 10=0.01%, 20=0.03%, 50=0.16%, 100=93.40%, 250=4.82%
  lat (msec)   : 2000=1.19%, >=2000=0.40%
  cpu          : usr=0.47%, sys=1.40%, ctx=32724, majf=0, minf=82
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32166,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=536MiB/s (562MB/s), 536MiB/s-536MiB/s (562MB/s-562MB/s), io=31.4GiB (33.7GB), run=60050-60050msec
root@pve:~# mount -o nfsvers=4.2 192.168.1.35:/export/shared /mnt/derp/ -v^C
root@pve:~# fio --name=fiotest --filename=/mnt/derp/file123 --size=16Gb --rw=write --bs=1M --direct=1 --numjobs=8 --ioengine=libaio --iodepth=8 --group_reporting --runtime=60 --startdelay=60 
fiotest: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
...
fio-3.25
Starting 8 processes
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
Jobs: 8 (f=8): [W(8)][100.0%][w=926MiB/s][w=925 IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=8): err= 0: pid=318524: Sat Oct 22 01:27:33 2022
  write: IOPS=509, BW=510MiB/s (535MB/s)(29.9GiB/60066msec); 0 zone resets
    slat (usec): min=107, max=991, avg=242.41, stdev=46.42
    clat (msec): min=4, max=13436, avg=125.24, stdev=659.36
     lat (msec): min=4, max=13436, avg=125.49, stdev=659.36
    clat percentiles (msec):
     |  1.00th=[   57],  5.00th=[   61], 10.00th=[   63], 20.00th=[   65],
     | 30.00th=[   67], 40.00th=[   69], 50.00th=[   71], 60.00th=[   73],
     | 70.00th=[   77], 80.00th=[   82], 90.00th=[   95], 95.00th=[  109],
     | 99.00th=[ 1603], 99.50th=[ 3205], 99.90th=[13355], 99.95th=[13355],
     | 99.99th=[13355]
   bw (  KiB/s): min=79872, max=991232, per=100.00%, avg=802506.54, stdev=29243.04, samples=624
   iops        : min=   78, max=  968, avg=783.72, stdev=28.55, samples=624
  lat (msec)   : 10=0.02%, 20=0.06%, 50=0.13%, 100=91.78%, 250=6.97%
  lat (msec)   : 2000=0.21%, >=2000=0.84%
  cpu          : usr=0.47%, sys=1.31%, ctx=31123, majf=0, minf=116
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,30621,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=510MiB/s (535MB/s), 510MiB/s-510MiB/s (535MB/s-535MB/s), io=29.9GiB (32.1GB), run=60066-60066msec

Run status group 0 (all jobs):
  WRITE: bw=529MiB/s (554MB/s), 529MiB/s-529MiB/s (554MB/s-554MB/s), io=32.2GiB (34.6GB), run=62401-62401msec

Alles anzeigen

Then decided to experiment with Google's TCP BRR driver.

Code

root@openmediavault:~# sysctl net.ipv4.tcp_available_congestion_control
net.ipv4.tcp_available_congestion_control = reno cubic
root@openmediavault:~# sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic
root@openmediavault:~# vi /etc/sysctl.conf.
root@openmediavault:~# nano /etc/sysctl.conf 
root@openmediavault:~# sysctl -p
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
root@openmediavault:~# sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = bbr
root@openmediavault:~#

Alles anzeigen

and the tweaks suggested

Code

- # Disable timestamps
sysctl -w net.ipv4.tcp_timestamps=0



# Selective acks
sysctl -w net.ipv4.tcp_sack=1



# Increase maximum length of processor input queues
sysctl -w net.core.netdev_max_backlog=250000



# Increase the TCP maximum and default buffer sizes
sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_max=4194304
sysctl -w net.core.rmem_default=4194304
sysctl -w net.core.wmem_default=4194304
sysctl -w net.core.optmem_max=4194304



# Increase memory thresholds to prevent packet dropping:
sysctl -w net.ipv4.tcp_rmem="4096 87380 4194304"
sysctl -w net.ipv4.tcp_wmem="4096 65536 4194304"



# Enable low latency mode for TCP:
sysctl -w net.ipv4.tcp_low_latency=1



# Buffer split evenly between TCP Window and Applications
sysctl -w net.ipv4.tcp_adv_win_scale=1

Alles anzeigen

Results after all this stuff... better than original (OMV6 defaults) but still not as great as Unraid "default" - compared to where we started sub 200Mb/s this is much better.

Code

root@pve:~# fio --name=fiotest --filename=/mnt/derp/file123 --size=16Gb --rw=write --bs=1M --direct=1 --numjobs=8 --ioengine=libaio --iodepth=8 --group_reporting --runtime=60 --startdelay=60 
fiotest: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
...
fio-3.25
Starting 8 processes
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
fiotest: Laying out IO file (1 file / 16384MiB)
Jobs: 8 (f=8): [W(8)][100.0%][eta 00m:00s]                         
fiotest: (groupid=0, jobs=8): err= 0: pid=522621: Sat Oct 22 02:04:43 2022
  write: IOPS=674, BW=675MiB/s (708MB/s)(39.6GiB/60044msec); 0 zone resets
    slat (usec): min=122, max=1257, avg=239.43, stdev=44.04
    clat (msec): min=4, max=6829, avg=94.53, stdev=423.19
     lat (msec): min=4, max=6830, avg=94.77, stdev=423.19
    clat percentiles (msec):
     |  1.00th=[   49],  5.00th=[   52], 10.00th=[   53], 20.00th=[   54],
     | 30.00th=[   55], 40.00th=[   55], 50.00th=[   56], 60.00th=[   57],
     | 70.00th=[   58], 80.00th=[   59], 90.00th=[   62], 95.00th=[   66],
     | 99.00th=[  305], 99.50th=[ 3205], 99.90th=[ 6745], 99.95th=[ 6745],
     | 99.99th=[ 6812]
   bw (  KiB/s): min=73728, max=1243452, per=100.00%, avg=1023331.40, stdev=34330.27, samples=648
   iops        : min=   72, max= 1214, avg=999.32, stdev=33.52, samples=648
  lat (msec)   : 10=0.01%, 20=0.06%, 50=2.69%, 100=94.65%, 250=1.49%
  lat (msec)   : 500=0.15%, 2000=0.32%, >=2000=0.63%
  cpu          : usr=0.60%, sys=1.73%, ctx=41504, majf=0, minf=102
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,40529,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=675MiB/s (708MB/s), 675MiB/s-675MiB/s (708MB/s-708MB/s), io=39.6GiB (42.5GB), run=60044-60044msec

Alles anzeigen

Krisbee · 22. Oktober 2022

TheLinuxGuy

Glad to see you’ve got closer to comparing apples with apples by paying attention to the basics of configuring NFS shares. But I’d suggest that anyone making serious use of NFS, especially on a 10Gbe network, would not blindly apply default NFS share options whatever product they were using. You seem to be downvoting OMV6 as it didn’t do your thinking for you.

Talking of thinking, here’s an example of a default NFS share on OMV6 and the underlying exports file:

Code

root@omv6:~# cat /etc/exports
# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.

# /etc/exports: the access control list for filesystems which may be exported
#               to NFS clients.  See exports(5).
/export/example_A 192.168.0.0/24(fsid=7d724a89-a243-4d51-a6e1-fccbf5994712,rw,subtree_check,insecure)

# NFSv4 - pseudo filesystem root
/export 192.168.0.0/24(ro,fsid=0,root_squash,no_subtree_check,hide)

How do you explain the use of “no_subtree_check” here? In any case, what makes the biggest difference, using async or no_subtree_check and it what sort of application other than just streaming data? OMV6 defaults to a sane safe choice. While you've not said what unRAID's export settings are, my guess is it defaults to the potentially unsafe async mode. Also unRAID offers a rather mysterious setting of "use cache" on its NFS share settings UI. I've no idea what the does under the hood.

In any case you’ve still only looked at half the NFS story as you’ve made no attempt to compare the NFS server configuration on OMV6 and unRaid. Nor did you show whether you are even making a NFSv4 in both cases as you didn’t want to check the output of “nfsstat –m” on your linux client mounts or add a -vvv option to your mount command.

OMV6 is configured to allow both nfsv3 and nfsv4 connections, is unRaid?

Code

root@kubuntu:~# mount -vvv -t nfs 192.168.0.61:/example_A  /home/chris/NFS
mount.nfs: timeout set for Sat Oct 22 09:11:26 2022
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.0.61,clientaddr=192.168.0.201'
root@kubuntu:~# nfsstat -m 
/home/chris/NFS from 192.168.0.61:/example_A
 Flags: rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.201,local_lock=none,addr=192.168.0.61

root@kubuntu:~# umount  /home/chris/NFS
root@kubuntu:~# mount -vvv -t nfs 192.168.0.61:/export/example_A  /home/chris/NFS
mount.nfs: timeout set for Sat Oct 22 09:12:07 2022
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.0.61,clientaddr=192.168.0.201'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'addr=192.168.0.61'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.0.61 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.0.61 prog 100005 vers 3 prot UDP port 55752
root@kubuntu:~# nfsstat -m 
/home/chris/NFS from 192.168.0.61:/export/example_A
 Flags: rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.61,mountvers=3,mountport=55752,mountproto=udp,local_lock=none,addr=192.168.0.61

root@kubuntu:~# umount  /home/chris/NFS

Alles anzeigen

ryecoaaron · 22. Oktober 2022

unraid can also focus on one architecture. OMV runs on armhf, arm64, i386, and amd64. So, OMV's defaults have to be a little more general than other NAS. But the fact that you can tweak the settings shows OMV's flexibility.

Jetzt mitmachen!