less than 30 Mb/s ZFS Raidz2 - Analyze

    Hi,


    probably one topic alot of people have. Yes, its not the newest system. But still i think (and hope) there is something wrong:


    System:


    - Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz

    - 32GB ECC memory, no min/max ZFS settings

    - MSI C236A, latest BIOS

    - SATA Controller ASM1166 - also tried onboard connector only, but was the same



    I started with a ZFS consisting of SMR + CMR disks -> i thought this is/was the problem as a lot of people mention to get rid of SMR. So i replaced the SMR disks with CMR only

    Today:

    Now, the "error" is still the same

    - e.g. copying 37 Gig ( 3 Gb files each) form one to another dataset: starts fast, drop to 110 MB/s and the continues with 30 MB/s with some drops to << 10 MB/s. Sometimes it stops for multiple seconds. Overall it took 21 minutes (30 MB/s)

    - cant see any SMART Errors (all green)

    - SMART Temperature below 30 °C

    - i used a script to measure before (with 2 SMR disks ( WD 4 TB EFAX)). With them replaced by ST4000VN008... disk, it get better. But still a major drop after a few GB


    Code
       /usr/bin/time -f "%e" sh -c 'dd if=/dev/zero of=/srv/NAS-Daten/20GB.img bs=20M count=1000 2> /dev/null'
with SMR+CMR mix 196s, CMR only 64s

    - no dmesg errors

    - no compression activated, no dedup

    - no hot temperaturen using sensors

    - same happens if i copy a large amounts of data using 1Gb Ethernet

    - bad sata cables - dont think so because it starts good

    - bad cooling: HDD cages have fans, CPU + housing also


    -> how to determine the Problem?


    As the system is quite low in free space, i consider to by new disk soon**. But aslong i dont know what the problem with the system is, i postpone this

    - Mainboard issue: datatransfer starts promissing, but collaps after 10G or so: No Mainboard/CPU/Bridge Problem?

    - sata controller issue: also used onboard Sata without any major changes. Would be strange if both onboard and PCIe Controller do the same strange thing

    - temperature problem: Not the disks i think (SMART value ok)

    - replace raidz2.... no idea

    - **get rid or raidz2 and replace by mirror of two Seagate Exos X20, 18TB



    iostat with average over 60s when copying 37 gig test data: directly after i start the copy




    bonnie++ with the zfs arrangement above + iostat during bonnie++

  • votdev

    Hat das Thema freigeschaltet.

    try to google about ZFS tune, some can increase speed, and same for SMB Tune


    Chris's Wiki :: blog/solaris/ZFSScrubsOurSpeedup


    Performance Tuning for SMB File Servers
    learn.microsoft.com

    Hi,


    i did some further tests:

    copy from a nvme to a ssd using either onboard or pcie card: close to 500 Mbyte/s

    copy from nvme to a SMR Disk (the one i replaced)-> 130 MByte/s

    -> no HW issue

    -> yes, looks like wrong zfs settings


    I do a scrub right now after I did the replace+resilver

    After a few minutes it says

    Code
    2.02T scanned at 4.46G/s, 120G issued at 265M/s, 11.4T total

    265 M/s? Lets see later



    But for now

    Ashift:

    Code
    get disk value:
sudo smartctl -a /dev/sda | grep 'Sector Size'
for all members sda/sdb/sdd/sde same output
512 bytes logical, 4096 bytes physical
-> 2^12
zpool get all | grep ashift
NAS-Daten  ashift                         12                             local

    So the setting for the pool seems to be ok


    "Common" settings

    Code
    NAS-Daten  atime                 off                                   local
NAS-Daten  xattr                 sa                                    local


Settings user access,...
NAS-Daten  xattr                 sa                                    local
NAS-Daten  acltype               posix                                 local
NAS-Daten  aclinherit            passthrough                           local

    "Compression"

    Code
    root@omv-neu:~# zfs get all NAS-Daten | grep compress
NAS-Daten  compressratio         1.00x                                 -
NAS-Daten  compression           off                                   default
NAS-Daten  refcompressratio      1.00x                                 -

    I set it to off. With compression data transfer should be better. I am looking to get closer to 1GBit/s 100 MB/s. Aslong as i am as far away from that, I dont want to copy all data again so that it will be compressed. Also my Data are most Pictures and Vides. I dont expect that huge effect on compression


    Cache/Log: Not right now


    Kernel for OMV

    - i am using Debian GNU/Linux, with Linux 6.1.15-1-pve

    - once i wanted to use the server also for VMs, but i do this now on another PC


    Memory: The more the better. But does it help in that case?

    -> Arc Min/Max untouched

    Code
    /sys/module/zfs/parameters/zfs_arc_max -> 0


    ZFS Sync - often mentioned, untouched (not always what could slow down pool)

    Code
    NAS-Daten  sync                  standard                              default

  • maltejahn

    Hat das Label gelöst hinzugefügt.

    Hi,


    after a few days I wanted to tell about the solution for my performance problem. I tried what most people suggested..... Avoid onboard SATA. The problems where atleast because i mixed onboard an additional sata controller ports.

    After i connected all of the disks to the ASM1166 6 port expander, I get really got good write values. One issue was a "write" error on one Disk when writing alot of Data. With this, the ZFS was degraded. It seems it was the cable (as also mentioned using google).


    For now, with the 4 disks i get (only tested a few times, 8 mp4 files, total 37GB, no compression):

    NVME -> ZFS -> 250MB/s

    NVME-> ZFS with 1MB recordsize -> 370 MB/s

    SSD 870 -> ZFS with 1MB recordsize + NVME log (for testing, removed it then) -> 390 MB/s



    For the movie dataset i will probably set recordsize to 256kb as there are also alot of small files for every movie. And with 1MB i think there would be alot of overhead/wasted space

    For Pictures i will do the same, the rest stays the same.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!

Benutzerkonto erstellen Anmelden