OMV5: read/write performance for SMB shares hosted on RPi4

mi-hol · Dec 30th 2020

Last update: 4Feb2022

To allow for a comparison of "apples to apples" and answer questions like "read or write rate is only #MB/sec on my hardware".

Short summary:

A RPi4 using below SMB 'Extra Options' will max out a 1Gbit/s wired network (common maximum speed in home and office networks)
SMB 'Extra Options' are platform independent
performance bottleneck for clients accessing data on the NAS is the network hardware
- best case for wireless network access via WiFi-4/801.11n is for Write ~ 10 MByte/s & Read ~ 13.5 MByte/s
- best case for wireless network access via WiFi-5/801.11ac is Write ~ 18 MByte/s & Read ~ 20 MByte/s
- random access of data is much slower between 1,3 MByte/s to 2 MByte/s for WiFi-4/801.11n & WiFi-5/801.11ac
investing in fast NAS hardware (i.e. SSD data disk, CPU, RAM) has low to no benefit to clients accessing the data, because network hardware is the real bottleneck
valid comparisons can only be performed by using the same tools. Details are documented below

SMB 'Extra Options' modified to:

min receivefile size = 16384

write cache size = 524288

getwd cache = yes

socket options = TCP_NODELAY IPTOS_LOWDELAY

read raw = yes

write raw = yes

These are measured values from a:

server: RPi4 with 4GB RAM running
- omv 5.5.19-1 (usul)
- Debian 5.4.79
storage: 2x 6TB Toshiba HDWT360 HDD in Icy Box IB-RD3662-C31 /RAID1
- HDDs use Conventional Magnetic Recording (CMR)
- IB-RD3662-C31 uses ASM1061R chipset providing hardware supported RAID1
- connected via original USB3.1 cable of IB-RD3662-C31
- formatted with ext4 file system
further configuration details at bottom

Measurements:

Performance of SMB share hosted on OMV RPi4 server connected via wired 1Gb Ethernet versus 2 WLAN routers using (802.11n) & (802.11.ac = WiFi5) protocol to a Windows 10 client.

Measurement done with CrystalDiskMark 8 on a Windows 10 client.

Many more benchmarks have been published. Worth mentioning are:

- Jeff Geerling's blog titled "Building the fastest Raspberry Pi NAS, with SATA RAID"

- optimum encrypted RAID5 settings for NAS with RaspberryPI 4 (here)

1) Client and server connected via 1Gbit switch & wired Ethernet performance

[Read Performance]

Measurement	MB/s	IOPS
SEQ 1MiB (Q= 8, T= 1)	93.0	88.7
SEQ 1MiB (Q= 1, T= 1)	95.4	91.0
RND 4KiB (Q= 32, T= 1)	43.2	10557.4
RND 4KiB (Q= 1, T= 1)	8.4	2054.4

[Write Performance]

Measurement	MB/s	IOPS
SEQ 1MiB (Q= 8, T= 1)	117.3	111.8
SEQ 1MiB (Q= 1, T= 1)	111.4	106.2
RND 4KiB (Q= 32, T= 1)	26.7	6518.8
RND 4KiB (Q= 1, T= 1)	9.6	2337.6

2) Connected via WiFi-4 /WLAN(802.11n) (WiFi router was old single core model)

[Read]

SEQ 1MiB (Q= 8, T= 1): 6.1 MB/s [ 5.8 IOPS]

SEQ 1MiB (Q= 1, T= 1): 5.9 MB/s [ 5.6 IOPS]

RND 4KiB (Q= 32, T= 1): 5.7 MB/s [ 1395.3 IOPS]

RND 4KiB (Q= 1, T= 1): 1.3 MB/s [ 313.5 IOPS]

[Write]

SEQ 1MiB (Q= 8, T= 1): 11.7 MB/s [ 11.2 IOPS]

SEQ 1MiB (Q= 1, T= 1): 10.1 MB/s [ 9.6 IOPS]

RND 4KiB (Q= 32, T= 1): 9.7 MB/s [ 2356.0 IOPS]

RND 4KiB (Q= 1, T= 1): 1.3 MB/s [ 317.6 IOPS]

3) Connected via WiFi-4 /WLAN (802.11n) (WiFi router was new dual core model)

[Read]

SEQ 1MiB (Q= 8, T= 1): 14.2 MB/s [ 13.6 IOPS]

SEQ 1MiB (Q= 1, T= 1): 13.6 MB/s [ 13.0 IOPS]

RND 4KiB (Q= 32, T= 1): 11.3 MB/s [ 2765.9 IOPS]

RND 4KiB (Q= 1, T= 1): 2.0 MB/s [ 485.6 IOPS]

[Write]

SEQ 1MiB (Q= 8, T= 1): 12.2 MB/s [ 11.6 IOPS]

SEQ 1MiB (Q= 1, T= 1): 11.3 MB/s [ 10.8 IOPS]

RND 4KiB (Q= 32, T= 1): 10.9 MB/s [ 2655.3 IOPS]

RND 4KiB (Q= 1, T= 1): 1.9 MB/s [ 458.0 IOPS]

3) Connected via WiFi-5 /WLAN (802.11ac) (WiFi router was new dual core model)

[Read]

SEQ 1MiB (Q= 8, T= 1): 38.4 MB/s [ 36.6 IOPS]

SEQ 1MiB (Q= 1, T= 1): 19.9 MB/s [ 19.0 IOPS]

RND 4KiB (Q= 32, T= 1): 21.1 MB/s [ 5140.6 IOPS]

RND 4KiB (Q= 1, T= 1): 1.7 MB/s [ 426.0 IOPS]

[Write]

SEQ 1MiB (Q= 8, T= 1): 31.0 MB/s [ 29.6 IOPS]

SEQ 1MiB (Q= 1, T= 1): 17.8 MB/s [ 17.0 IOPS]

RND 4KiB (Q= 32, T= 1): 15.0 MB/s [ 3672.4 IOPS]

RND 4KiB (Q= 1, T= 1): 1.7 MB/s [ 423.3 IOPS]

------------------------------------------------------------------------------

Profile: Default

Test: 64 MiB (x5) [Z: 22% (1237/5544GiB)]

Mode: [Admin]

Time: Measure 5 sec / Interval 5 sec

Date: 2020/12/20 15:53:48

Client OS: Windows 10 Professional [10.0 Build 19042] (x64)

------------------------------------------------------------------------------

For reference and troubleshooting below information is needed

Server:

Code

uname -a

lsusb -t && lsusb -v

output from tested system:

Code

root@nas:~# uname -a
Linux nas 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l GNU/Linux

root@nas:~# lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
|__ Port 2: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M

Good performance is provided by SATA-to-USB adapters supported by "Driver=uas"

------------------------------------------------------------------------------

Crystal Dew World: https://crystalmark.info/

------------------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes

metRo_ · Jan 2nd 2021

Did you change any settings of samba?

mi-hol · Jan 3rd 2021

Good you asked, yes I did and updated first post

mi-hol · Jan 4th 2021

@metRo_ you may want to try an early version of a benchmark script to allow for comparison of results.

https://github.com/mi-hol/DiskSpdAuto in case of issues please open a github issue there

metRo_ · Jan 4th 2021

I need to get an ethernet adapter since my laptop doesn't have one.

Right now I'll try to get the same speeds of windows on osx and linux since linux is the OS I use most.

mi-hol · Jan 6th 2021

Reaching "same speeds of Windows on osx" (MacOS) seems difficult according to some comments made on other sites

MarcS · Jan 10th 2021

Quote from mi-hol

- optimum encrypted RAID5 settings for NAS with RaspberryPI 4 (here)

I saw your cryptsetup benchmarks and noticed that I get almost 50% lower results on my Pi4. How can this be if no disks are even considered?

Code

cryptsetup benchmark @1,5 GHz

# Tests are approximate using memory only (no storage IO).

The second thing I noticed was that my Pi4 actually stays at 1.5GHz although I overclock in config.txt

Code

over_voltage=6
arm_freq=2147
gpu_freq=750

mi-hol · Jan 11th 2021

Quote from mi-hol

CrystalDiskMark 8.

I've never done cryptsetup benchmarks , because only IO benchmarks are relevant for NAS

mi-hol · Feb 3rd 2021

positive feedback from using above detailed settings is in this forum post

mi-hol · Feb 10th 2021

Kernel 5.10.11 has NOT introduced a SMB performance regression as was reported on RPi Forum & https://github.com/raspberrypi/linux/issues/4133 !

My initial comment was a false conclusion due to Anti-Malware software that had changed between measurements on the Windows client!

Repeated measurement on a Windows 10 Pro client.

server: RPi4 with 4GB RAM running

omv 5.5.23-1 (usul)
Debian 5.10.11

------------------------------------------------------------------------------

Crystal Dew World: https://crystalmark.info/

------------------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes

[Read]

SEQ 1MiB (Q= 8, T= 1): 109.220 MB/s [ 104.2 IOPS] < 75982.32 us>

SEQ 1MiB (Q= 1, T= 1): 103.986 MB/s [ 99.2 IOPS] < 10069.66 us>

RND 4KiB (Q= 32, T= 1): 45.550 MB/s [ 11120.6 IOPS] < 2867.51 us>

RND 4KiB (Q= 1, T= 1): 8.629 MB/s [ 2106.7 IOPS] < 472.22 us>

[Write]

SEQ 1MiB (Q= 8, T= 1): 117.207 MB/s [ 111.8 IOPS] < 70708.24 us>

SEQ 1MiB (Q= 1, T= 1): 110.506 MB/s [ 105.4 IOPS] < 9466.85 us>

RND 4KiB (Q= 32, T= 1): 13.929 MB/s [ 3400.6 IOPS] < 3330.61 us>

RND 4KiB (Q= 1, T= 1): 6.246 MB/s [ 1524.9 IOPS] < 650.69 us>

Profile: Default

Test: 64 MiB (x5) [Z: 23% (1253/5544GiB)]

Mode: [Admin]

Time: Measure 5 sec / Interval 5 sec

Date: 2021/02/10 17:40:56

OS: Windows 10 Professional [10.0 Build 19042] (x64)

Performance impact of installed Anti-Malware software (Tanium and SentinelOne) on Windows Enterprise client:

------------------------------------------------------------------------------

Crystal Dew World: https://crystalmark.info/

------------------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes

[Read]

SEQ 1MiB (Q= 8, T= 1): 28.937 MB/s [ 27.6 IOPS] <280809.89 us>

SEQ 1MiB (Q= 1, T= 1): 28.101 MB/s [ 26.8 IOPS] < 37209.92 us>

RND 4KiB (Q= 32, T= 1): 16.972 MB/s [ 4143.6 IOPS] < 7706.54 us>

RND 4KiB (Q= 1, T= 1): 1.699 MB/s [ 414.8 IOPS] < 2403.04 us>

[Write]

SEQ 1MiB (Q= 8, T= 1): 43.206 MB/s [ 41.2 IOPS] <188604.44 us>

SEQ 1MiB (Q= 1, T= 1): 35.021 MB/s [ 33.4 IOPS] < 29890.23 us>

RND 4KiB (Q= 32, T= 1): 21.026 MB/s [ 5133.3 IOPS] < 6210.67 us>

RND 4KiB (Q= 1, T= 1): 1.458 MB/s [ 356.0 IOPS] < 2254.01 us>

Profile: Default

Test: 64 MiB (x5) [Z: 23% (1253/5544GiB)]

Mode:

Time: Measure 5 sec / Interval 5 sec

Date: 2021/02/10 16:28:37

OS: Windows 10 Enterprise [10.0 Build 18363] (x64)

sabamimi · Feb 10th 2021

Thanks a lot for the SMB Extra Options modified to:

min receivefile size = 16384

write cache size = 524288

getwd cache = yes

socket options = TCP_NODELAY IPTOS_LOWDELAY

read raw = yes

write raw = yes

It did improve my system !

MarcS · Feb 13th 2021

Didn’t improve anything on my system. I suspect that my bottleneck is the WIFI connection at home.

What routers do people recommend for high speed SMB file access performance?

mi-hol · Feb 13th 2021

Any cheap WiFi6 router (prices in Germany start at 35Euro) will do the trick if your client accessing the data has a WiFi6 card too

jorensanbar · Feb 15th 2021

Same bug with kernel 5.10.11-v71+ (last) Speed: 70 MB/s

Working on kernel 5.4.79-v71+ #1373, speed: 115 MB/s

Pi4b 2GB + WD Elements 5TB USB 3.0

Wanderer189 · Feb 19th 2021

Will this help any if HDD's are formatted NTFS? Or is there any tweak in your settings that will help the NTFS file system?

On a side note, if I swapped out a Pi 4 2GB to a 4GB would it really make enough difference to do?

mi-hol · Feb 19th 2021

Valid questions but as I don't have the hardware I'm unable to answer.

You are invited to provide benchmark results using your equipment.

The setup I used is documented, hence it should be easy to reproduce.

let me know if questions arise.

jorensanbar · Feb 25th 2021

Hello.

It has already been checked in different ways, and it is an error generated in the Raspberry Kernel.

Here you can follow up: https://github.com/raspberrypi/linux/issues/4133

It seems that the error is presented from Kernel 5.5, In my case, I left without updating the Kernel, so I am in version 5.4.83-v71 +

Birdybird · Mar 1st 2021

How to get the old kernel. Its not in plugins ?

I have the same problem transfer speed drops.

I have already installed the smb extra options.

MarcS · Mar 1st 2021

Quote from mi-hol

Any cheap WiFi6 router (prices in Germany start at 35Euro) will do the trick if your client accessing the data has a WiFi6 card too

Sadly there are no WiFi6 USB cards available yet.

mi-hol · Mar 1st 2021

Quote from MarcS

there are no WiFi6 USB cards available

Note: for important update see #24

For what client computer are you looking for a WiFi6 NIC?

~~I.e. laptops build since ~ 2006 have an internal mini PCIe slot for WiFi NIC and are easy to upgrade using for example a less than 20 Euro~~ ~~Intel AX200 card~~

OMV5: read/write performance for SMB shares hosted on RPi4

ryecoaaron Dec 30th 2020

Participate now!

Tags