does mergerfs need a high singlecore performance?

Niemand · 7. Dezember 2016

Hi,

Before I got a Opteron 3280 Octacore in my OMV3-Server anf mergerfs was quiet slow in writing speed (65mb/s; HDDs connected to SATA2 @AMD 760G). The A4-4000 now has 100mb/s at S-ATA3.
I dont think that it depends on the SATA-Ports, but what is the Problem? Is it the AMD 770G or the bad singlecore performance of the Opteron?

I'm thinking about to move my FX-4100 (@ the same Board as the Opteron) to my OMV3-Server after ZEN-Release because of it's higher CPU-Power and more threads (for Plex). But I would be mad if I have slow writing speeds after changing the hardware again

What do you think? Was the Problem the Opteron or is it the Motherboard (Asrock 980DE3/U3S3) ?

ryecoaaron · 7. Dezember 2016

mergerfs is a fuse filesystem. So, a faster cpu does help. You can also increase writes speeds with the direct_io at the cost of a bit of read speed.

Niemand · 7. Dezember 2016

direct_io only increased from 58mb/s to 60mb/s.

ryecoaaron · 7. Dezember 2016

I saw much better results on fast CPUs. Your cpu is probably cpu bound then. My celeron J1800 cpu can hit 167 MB/s (4tb WD Reds).

Code

dd if=/dev/zero of=test.dd bs=1M conv=fdatasync count=5000 && sync
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 31.3195 s, 167 MB/s

How fast can you write to the individual drive?

Huberer · 7. Dezember 2016

Sorry for hijacking this thread but I'm very interested in this case too.

@ryecoaaron
How did you perform this test with

Code

dd if=/dev/zero of=test.dd bs=1M conv=fdatasync count=5000 && sync

I tried it on my homeserver (see sig.) with this:

Code

root@HomeServer:/media/aac75702-c08f-4587-a08b-9fc72e60c614# dd if=/dev/zero of=test.dd bs=1M conv=fdatasync count=5000 && sync
5000+0 Datensätze ein
5000+0 Datensätze aus
5242880000 Bytes (5,2 GB) kopiert, 50,3806 s, 104 MB/s

/media/aac75702-c08f-4587-a08b-9fc72e60c614/ is my mergerfs-Pool (with 3x3TB WD Red) on a SnapRaid (3x3TB WD Red = mergerfs + 1x3TB WD Red parity-drive) and I use an Intel i5-4590s (Haswell Quadcore). Your Celeron is blowing away my i5... What the f*ck?????? Or does it mean that OMV3 is faster than OMV2?

Edit: and on my Test-Server (with an intel core2duo E8500@3,16ghz) it looks like this:

Code

root@OMV3-Server:/media/aaad84ac-fd19-4c0f-90a9-47f796e4609e# dd if=/dev/zero of=test.dd bs=1M conv=fdatasync count=5000 && sync
5000+0 Datensätze ein
5000+0 Datensätze aus
5242880000 Bytes (5,2 GB) kopiert, 48,8553 s, 107 MB/s

Quite the same speed with a single 1TB-SSHD

Edit2: and a test with just a single drive of my main OMV-server (the i5-version)

Code

root@HomeServer:/media/e409ca68-2351-410d-836a-1a2d26cdd099# dd if=/dev/zero of=test.dd bs=1M conv=fdatasync count=5000 && sync
5000+0 Datensätze ein
5000+0 Datensätze aus
5242880000 Bytes (5,2 GB) kopiert, 50,0082 s, 105 MB/s

ryecoaaron · 7. Dezember 2016

Zitat von Huberer

Your Celeron is blowing away my i5... What the f*ck?????? Or does it mean that OMV3 is faster than OMV2?

I think OMV 3.x is faster especially on a newer board like yours. I wouldn't think it would be that much faster though. I also have 4TB WD Red Pros which are 7200 rpm vs your 5400 rpm drives. So, that is probably more of the speed difference.

Huberer · 7. Dezember 2016

I see, this could also be the reason that the SSHD (1TB Seagate) with 5400rpm is quite as fast as the WD Red's.
Maybe the rpm could be the important factor in this matter.
Thanks for clearing me up.

trapexit · 7. Dezember 2016

Few things:

1. When benchmarking you should isolate any bottlenecks unrelated to the core metrics to be tested. In this case it's best to use a RAM fs. tmpfs is what I tend to use.
2. If CPU is not 100% then it's not likely to be the bottleneck.
3. mergerfs is built on top of FUSE and libfuse. Older versions of both can be quite a bit slower. Best to use a modern'ish kernel and libfuse 2.9.5+ when checking speeds.

I've used several tools (perf,prof,etc.) in an attempt to find possible bottlenecks and have yet to find anything obvious. The IO itself tends to be the bulk of time spent and while I'm not a fan of libfuse's design nothing popped up as a point of contention and replacing it would be a big undertaking.

I've been meaning to put together some tool to help benchmarking so as to maybe provide a document which indicates expected speeds. Perhaps I'll look into that again.

ryecoaaron · 7. Dezember 2016

Zitat von trapexit

If CPU is not 100% then it's not likely to be the bottleneck.

Just to clarify... Do you mean 100% of 1 cpu or all cpus?

Zitat von trapexit

I've been meaning to put together some tool to help benchmarking so as to maybe provide a document which indicates expected speeds.

I think this would be a great idea.

Huberer · 7. Dezember 2016

@trapexit

Thanks for the info and it would be great if you could post some tools with document to test

I also checked the cpu usage while performing the write test. My main-server (with the i5) was between 50-70% (while watching the system information on the main page of the webui) and the test-server (with the C2D) was mainly at 100% cpu usage. But I don't know if only one core or all cores were at 100%. This isn't shown on the system information.

trapexit · 7. Dezember 2016

Of 1 core. If there's CPU to spare then raw compute is not the bottleneck.

ryecoaaron · 7. Dezember 2016

On the J1800 Celeron, cpu percentage reported by top averaged about 35%. 1 min load was about 1.2.

$ dd if=/dev/zero of=test.dd bs=1M conv=fdatasync count=10000 && sync
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 60.4546 s, 173 MB/s

trapexit · 7. Dezember 2016

I've got a Core i7-3770T (4 core / 8 thread @ 2.50GHz) and using tmpfs I'm getting ~30 - ~50% CPU usage and transferring at 1.4 - 1.6GB/s. Raw to the tmpfs mount is ~4GB/s. The great difference in speed is probably due to the much increased overhead but that overhead is much greater due to the high performance of tmpfs vs a drive. Regardless... the fact I'm getting 1.6GB/s indicates that mergerfs in general should be able to at least artificially benchmark at high speeds.

And if you add `direct_io` I get speeds up to 2.8GB/s w/ ~55% - ~80% CPU usage. Clearly caching takes it toll in terms of raw throughput.

Code

# mkdir /tmp/tmpfs /tmp/mergerfs
# mount -t tmpfs -o size=2G tmpfs /tmp/tmpfs
# mergerfs -odefaults,allow_other /tmp/tmpfs /tmp/mergerfs
# for x in $(seq 1 10); do dd if=/dev/zero of=/tmp/mergerfs/zero.img bs=1M count=2000 conv=fdatasync; done

trapexit · 7. Dezember 2016

In single threaded mode: "mergerfs -s"

Getting more consistent 1.7 - 1.8GB/s without direct_io w/ ~55% CPU usage in top

With direct_io it seems about 2.7 - 2.9 GB/s w/ ~82% - 85% CPU

ryecoaaron · 7. Dezember 2016

On the same Celeron J1800, running your test, I get:

2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 4.08636 s, 527 MB/s

cpu for mergerfs is about 52% and dd is about 50%

trapexit · 7. Dezember 2016

BTW...

Code

$ mergerfs -v
mergerfs version: 2.17.0
FUSE library version: 2.9.4
fusermount version: 2.9.4
using FUSE kernel interface version 7.19


$ uname -a
Linux ion 4.4.0-52-generic #73~14.04.1-Ubuntu SMP Wed Nov 30 17:20:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Niemand · 7. Dezember 2016

okay, so it is very single Core dependent.

That could be the reason why my Opteron was slow as a snail.

But I think I'll upgrade the A4-4000 to an A8-5500 or something like that. Would be easier than changing Motherboard, CPU and reinstalling OMV.

trapexit · 7. Dezember 2016

http://cpuboss.com/cpus/Intel-…J1800-vs-AMD-Opteron-3280

Doesn't appear that simple. The single core performance according to this is similar to the J1800.

Niemand · 7. Dezember 2016

True, this is strange. But with my A4-4000 I got a better performance with mergerfs?

So it is probably the mainboard?
Slow: Asrock 980DE3/U3S3 (AMD 760G)
Fast: GA-F2A88XM-HD3 (AMD A88X)

ryecoaaron · 7. Dezember 2016

My mergerfs versions:

Code

~# mergerfs -v
mergerfs version: 2.17.0
FUSE library version: 2.9.3
fusermount version: 2.9.3
using FUSE kernel interface version 7.19

Jetzt mitmachen!