mergerfs bad performance with CIFS/SMB

    • OMV 3.x (stable)
    • Resolved
    • mergerfs bad performance with CIFS/SMB

      Is anyone experiencing bad performance with unionfilesystems (mergerfs) using SMB?

      I am using snapRAID, unionfilesystems (mergerfs) and smb to access my files via network.

      After a clean setup, I was able to get a constant speed of about 110 MB/s.

      I did not check the performance after every change, but now I only get about 35 MB/s, sometimes 50 MB/s.

      Here is the top output:

      top - 16:07:16 up 22:15, 1 user, load average: 13,66, 10,63, 5,34
      Tasks: 163 total, 1 running, 162 sleeping, 0 stopped, 0 zombie
      %Cpu0 : 2,9 us, 8,2 sy, 0,0 ni, 54,7 id, 28,8 wa, 0,4 hi, 4,9 si, 0,0 st
      %Cpu1 : 2,4 us, 4,4 sy, 0,0 ni, 39,8 id, 53,4 wa, 0,0 hi, 0,0 si, 0,0 st
      %Cpu2 : 9,8 us, 4,7 sy, 0,0 ni, 37,8 id, 47,6 wa, 0,0 hi, 0,0 si, 0,0 st
      %Cpu3 : 1,3 us, 5,4 sy, 0,0 ni, 33,7 id, 59,3 wa, 0,0 hi, 0,3 si, 0,0 st
      %Cpu4 : 8,4 us, 7,0 sy, 0,0 ni, 30,2 id, 54,4 wa, 0,0 hi, 0,0 si, 0,0 st
      %Cpu5 : 2,0 us, 6,8 sy, 0,0 ni, 32,9 id, 58,3 wa, 0,0 hi, 0,0 si, 0,0 st
      %Cpu6 : 11,1 us, 6,4 sy, 0,0 ni, 33,2 id, 49,0 wa, 0,0 hi, 0,3 si, 0,0 st
      %Cpu7 : 7,8 us, 9,8 sy, 0,0 ni, 30,4 id, 52,0 wa, 0,0 hi, 0,0 si, 0,0 st
      KiB Mem: 8197284 total, 7940744 used, 256540 free, 966908 buffers
      KiB Swap: 901116 total, 68568 used, 832548 free. 5380440 cached Mem

      PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
      11339 root 10 -10 1848168 21992 908 S 26,2 0,3 58:24.41 mergerfs

      As you can see, the cpu has to wait a for I/O access ==> disk?! So I think the problem has to do something with mergerfs/unionfilesystems.

      If mergerfs isn't running anymore, it looks like this:
      top - 16:12:35 up 22:21, 1 user, load average: 0,14, 4,07, 4,03
      Tasks: 160 total, 1 running, 159 sleeping, 0 stopped, 0 zombie
      %Cpu0 : 8,9 us, 0,0 sy, 0,0 ni, 91,1 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
      %Cpu1 : 8,0 us, 0,7 sy, 0,0 ni, 91,4 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
      %Cpu2 : 0,0 us, 0,0 sy, 0,0 ni, 95,6 id, 0,0 wa, 0,0 hi, 4,4 si, 0,0 st
      %Cpu3 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
      %Cpu4 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
      %Cpu5 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
      %Cpu6 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
      %Cpu7 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
      KiB Mem: 8197284 total, 7927004 used, 270280 free, 967388 buffers
      KiB Swap: 901116 total, 68552 used, 832564 free. 5369676 cached Mem


      Sysinfo:
      Linux 3.16.0-4-amd64
      Host: ESXi 6

      The post was edited 4 times, last by mnui: additional info (top output) ().

    • According to this problem, I want also to ask if it's better to enable or disable write cache of the "data"-drives?
      OMV-Server-HW: MoBo Asrock B250M-Pro4 (Intel-LAN), Intel Pentium G4600, 8GB-Ram, 1x512GB SSD Samsung 850 Evo M.2 (sda2 - 25GB system, sda4/rest - for work), 4x3TB WD Red's Snapraid w/ mergerfs, DVB-S card DVBSky s952v3,

      OMV-Server-SW: Debian Stretch with 4.14.x-Backports-Kernel (always up-to-date), OMV 4.0.x (always latest), omv-extras-plugin (always latests), AutoShutdown-Plugin, PlexMediaServer, Emby Server, SMB-Shares, TVHeadendServer (unstable release),

      BackupServer: Synology DS1010+ with 4GB Ram, 9TB@SHR (different hdd's), DSM 5.2-5967-2
    • Thanks for the info. I also made a test with write-cache enabled and disabled of my data drives (4x3TB WD Red with snapraid and mergerfs) and realized that without write-cache the speeds are a little bit higher.
      So I will stay like that.
      OMV-Server-HW: MoBo Asrock B250M-Pro4 (Intel-LAN), Intel Pentium G4600, 8GB-Ram, 1x512GB SSD Samsung 850 Evo M.2 (sda2 - 25GB system, sda4/rest - for work), 4x3TB WD Red's Snapraid w/ mergerfs, DVB-S card DVBSky s952v3,

      OMV-Server-SW: Debian Stretch with 4.14.x-Backports-Kernel (always up-to-date), OMV 4.0.x (always latest), omv-extras-plugin (always latests), AutoShutdown-Plugin, PlexMediaServer, Emby Server, SMB-Shares, TVHeadendServer (unstable release),

      BackupServer: Synology DS1010+ with 4GB Ram, 9TB@SHR (different hdd's), DSM 5.2-5967-2