mergerfs bad performance with CIFS/SMB

  • Is anyone experiencing bad performance with unionfilesystems (mergerfs) using SMB?


    I am using snapRAID, unionfilesystems (mergerfs) and smb to access my files via network.


    After a clean setup, I was able to get a constant speed of about 110 MB/s.


    I did not check the performance after every change, but now I only get about 35 MB/s, sometimes 50 MB/s.


    Here is the top output:


    top - 16:07:16 up 22:15, 1 user, load average: 13,66, 10,63, 5,34
    Tasks: 163 total, 1 running, 162 sleeping, 0 stopped, 0 zombie
    %Cpu0 : 2,9 us, 8,2 sy, 0,0 ni, 54,7 id, 28,8 wa, 0,4 hi, 4,9 si, 0,0 st
    %Cpu1 : 2,4 us, 4,4 sy, 0,0 ni, 39,8 id, 53,4 wa, 0,0 hi, 0,0 si, 0,0 st
    %Cpu2 : 9,8 us, 4,7 sy, 0,0 ni, 37,8 id, 47,6 wa, 0,0 hi, 0,0 si, 0,0 st
    %Cpu3 : 1,3 us, 5,4 sy, 0,0 ni, 33,7 id, 59,3 wa, 0,0 hi, 0,3 si, 0,0 st
    %Cpu4 : 8,4 us, 7,0 sy, 0,0 ni, 30,2 id, 54,4 wa, 0,0 hi, 0,0 si, 0,0 st
    %Cpu5 : 2,0 us, 6,8 sy, 0,0 ni, 32,9 id, 58,3 wa, 0,0 hi, 0,0 si, 0,0 st
    %Cpu6 : 11,1 us, 6,4 sy, 0,0 ni, 33,2 id, 49,0 wa, 0,0 hi, 0,3 si, 0,0 st
    %Cpu7 : 7,8 us, 9,8 sy, 0,0 ni, 30,4 id, 52,0 wa, 0,0 hi, 0,0 si, 0,0 st
    KiB Mem: 8197284 total, 7940744 used, 256540 free, 966908 buffers
    KiB Swap: 901116 total, 68568 used, 832548 free. 5380440 cached Mem


    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    11339 root 10 -10 1848168 21992 908 S 26,2 0,3 58:24.41 mergerfs


    As you can see, the cpu has to wait a for I/O access ==> disk?! So I think the problem has to do something with mergerfs/unionfilesystems.


    If mergerfs isn't running anymore, it looks like this:
    top - 16:12:35 up 22:21, 1 user, load average: 0,14, 4,07, 4,03
    Tasks: 160 total, 1 running, 159 sleeping, 0 stopped, 0 zombie
    %Cpu0 : 8,9 us, 0,0 sy, 0,0 ni, 91,1 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
    %Cpu1 : 8,0 us, 0,7 sy, 0,0 ni, 91,4 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
    %Cpu2 : 0,0 us, 0,0 sy, 0,0 ni, 95,6 id, 0,0 wa, 0,0 hi, 4,4 si, 0,0 st
    %Cpu3 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
    %Cpu4 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
    %Cpu5 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
    %Cpu6 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
    %Cpu7 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
    KiB Mem: 8197284 total, 7927004 used, 270280 free, 967388 buffers
    KiB Swap: 901116 total, 68552 used, 832564 free. 5369676 cached Mem



    Sysinfo:
    Linux 3.16.0-4-amd64
    Host: ESXi 6

    Edited 4 times, last by mnui: additional info (top output) ().

  • According to this problem, I want also to ask if it's better to enable or disable write cache of the "data"-drives?

    OMV-Server-HW: MoBo Fujitsu D3417-B2 (Intel-LAN), Intel Xeon E3-1245 v6 Kaby Lake (4x3.70GHz), 16GB-Ram ECC UDIMM, 1x512GB SSD Samsung 850 Pro (sda2 - 30GB system, 4GB swap, sda5/rest - for work), 1x 10TB WD Red Pro, 1x 3TB WD Red (both basic setup) - Digibit R1 Sat-IP-Server with SatIP-Axe-Firmware


    OMV-Server-SW: Debian Buster with Proxmox kernel (always up-to-date), OMV v5 (always latest), omv-extras-plugin (always latests), AutoShutdown-Plugin, Docker with PlexMediaServer, TVHeadend, any many more


    BackupServer: Synology DS1010+ with 4GB Ram, 9TB@SHR (different hdd's), DSM 5.2-5967-2

  • I've never enabled write-cache on any of my OMV systems.

    omv 5.6.13 usul | 64 bit | 5.11 proxmox kernel | omvextrasorg 5.6.2 | kvm plugin 5.1.6
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • Thanks for the info. I also made a test with write-cache enabled and disabled of my data drives (4x3TB WD Red with snapraid and mergerfs) and realized that without write-cache the speeds are a little bit higher.
    So I will stay like that.

    OMV-Server-HW: MoBo Fujitsu D3417-B2 (Intel-LAN), Intel Xeon E3-1245 v6 Kaby Lake (4x3.70GHz), 16GB-Ram ECC UDIMM, 1x512GB SSD Samsung 850 Pro (sda2 - 30GB system, 4GB swap, sda5/rest - for work), 1x 10TB WD Red Pro, 1x 3TB WD Red (both basic setup) - Digibit R1 Sat-IP-Server with SatIP-Axe-Firmware


    OMV-Server-SW: Debian Buster with Proxmox kernel (always up-to-date), OMV v5 (always latest), omv-extras-plugin (always latests), AutoShutdown-Plugin, Docker with PlexMediaServer, TVHeadend, any many more


    BackupServer: Synology DS1010+ with 4GB Ram, 9TB@SHR (different hdd's), DSM 5.2-5967-2

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!