Lower Btrfs scrub and file system write performance with backport kernel 6.17.13, compared to stable trixie kernel 6.12.57

  • Hi folks


    I find a fundamental degraded Btrfs performance (srub+filesystem write) with the bpo kernel 6.17.13. Compared with the stable 6.12.57 kernel

    on my Supermicro server. No change on system, just change the boot kernel in Grub. Filesystem is a Btrfs raid 1 over 3 Samsung SSDs


    Scrub with 6.17.13:



    Scrub with 6.12.57



    That´s double time for exact the same Btrfs data filesystem!

    Copying a large file from Windows 11 client to the Samba share is slower,too. But not that much.


    At this time, no idea what changed in Btrfs kernel driver, but on my old dual core Intel Atom D510 server, this is bad stuff with 6.17.13.


    If anybody noticed identical performance issues, please keep me informed.

    Ideas welcome!


    Best regards

    prtigger

    Old (2011) Supermicro 1U Rackserver X7SPA-HF, Intel Atom D510, 4GB RAM (maximum) | System: Supermicro SATA DOM 64GB SSD | Data: 3*Samsung 870 QVO 4TB SATA (Btrfs Raid1) | OMV 8.x services: SSH, SMB, NFS, MiniDLNA | 6.12.x kernel | Bare-metal

  • prtigger1

    Changed the title of the thread from “Lower Btrfs Scrub and file system write performace with backport kernel 6.17.13, compared to stable trixie kernel 6.12.57” to “Lower Btrfs scrub and file system write performance with backport kernel 6.17.13, compared to stable trixie kernel 6.12.57”.
    • Official Post

    At this time, no idea what changed in Btrfs kernel driver, but on my old dual core Intel Atom D510 server, this is bad stuff with 6.17.13.

    Newer kernels are usually likely to be better optimized on recent hardware. Your D510 is 17 years old.

    omv 8.0.10-2 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.6 | compose 8.1.3 | cterm 8.0 | borgbackup 8.1.5 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Hi Aaron

    That's a possible explanation...

    The load is higher and the cache usage is

    lower during scrub. That's what i can see.

    The scrub time difference is massive...


    Normally my system is performing pretty

    well with the older kernel...


    If there is any way to optimize this bad behavior, i don't know at this time.

    Switching to other file system may be the

    last step...


    That's the actual status...


    Thank's and best regards

    prtigger

    Old (2011) Supermicro 1U Rackserver X7SPA-HF, Intel Atom D510, 4GB RAM (maximum) | System: Supermicro SATA DOM 64GB SSD | Data: 3*Samsung 870 QVO 4TB SATA (Btrfs Raid1) | OMV 8.x services: SSH, SMB, NFS, MiniDLNA | 6.12.x kernel | Bare-metal

    • Official Post

    On a system that old, there is little reason to use the newer kernel. I am curious if you see the same results with the 6.14 and 6.17 proxmox kernel.

    omv 8.0.10-2 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.6 | compose 8.1.3 | cterm 8.0 | borgbackup 8.1.5 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • ryecoaaron


    Exact this was my idea today!


    I disabled backports, cleaned up the kernel stuff and repos. Installed the lastest 6.12.63 kernel meta package...

    After that, i installed the 6.17.4-2-pve kernel... I've both kernel to start.


    First test (Writing 1.8GB file to Samba share): PVE Kernel is slower than Debian bpo 6.17.13 kernel, yesterday.

    Compared to actual 6.12.63, the write performace is more than 20% slower... Worse!!


    Btrfs scrub is just running for 41 minutes...


    By the way, if the settings in:

    /etc/sysctl.d/99-openmediavault-nonrot.conf

    are useful for my hardware, i had not fully checked at this time... (-nonrot stands for ???)

    I`ve got an extra conf file, optimized for my network stack.

    At this time i override laptop.mode = 0 (Changed yesterday, after i post the information here)


    But both kernels run with this settings...


    Best regards

    prtigger




    Best regards

    prtigger

    Old (2011) Supermicro 1U Rackserver X7SPA-HF, Intel Atom D510, 4GB RAM (maximum) | System: Supermicro SATA DOM 64GB SSD | Data: 3*Samsung 870 QVO 4TB SATA (Btrfs Raid1) | OMV 8.x services: SSH, SMB, NFS, MiniDLNA | 6.12.x kernel | Bare-metal

  • The scrub is only a little bit slower than with 6.17.13 bpo:



    Best regards

    prtigger

    Old (2011) Supermicro 1U Rackserver X7SPA-HF, Intel Atom D510, 4GB RAM (maximum) | System: Supermicro SATA DOM 64GB SSD | Data: 3*Samsung 870 QVO 4TB SATA (Btrfs Raid1) | OMV 8.x services: SSH, SMB, NFS, MiniDLNA | 6.12.x kernel | Bare-metal

    • Official Post

    -nonrot stands for ???)

    non-rotational. So, an ssd/usb stick/sd, etc

    omv 8.0.10-2 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.6 | compose 8.1.3 | cterm 8.0 | borgbackup 8.1.5 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • ryecoaaron


    Hi Aaron

    Thank's for that info...


    Is is possible to deactivate the

    99-openmediavault*.conf files persistant?

    Where are they generated?


    I should not change this files, so

    i have to add an extra file with overrides and

    additions.


    Best regards

    prtigger

    Old (2011) Supermicro 1U Rackserver X7SPA-HF, Intel Atom D510, 4GB RAM (maximum) | System: Supermicro SATA DOM 64GB SSD | Data: 3*Samsung 870 QVO 4TB SATA (Btrfs Raid1) | OMV 8.x services: SSH, SMB, NFS, MiniDLNA | 6.12.x kernel | Bare-metal

    • Official Post

    Is is possible to deactivate the

    99-openmediavault*.conf files persistant?

    Those files are written by omv's saltstack. But you can change the values in environment variables - https://github.com/openmediava…loy/sysctl/nonrot.sls#L26


    Or you could create a file without openmediavault in the name and as long as it sorts alphanumerically after 99-openmediavault, it will override the value in the omv files.

    omv 8.0.10-2 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.6 | compose 8.1.3 | cterm 8.0 | borgbackup 8.1.5 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • ryecoaaron


    Hi Aaron

    Thank you, i just realized, that these settings

    exists for longer time and aren't new with

    OMV 8.

    Never noticed that before, because i was focused

    on /etc/sysctl.conf... ;)


    But it's not to late to keep an eye on the

    default changes...


    I guess that will not fix the performance

    problem with my Btrfs storage...


    I decide to stay with the very good performing

    6.12.xx stable kernel, until a newer backport

    kernel is coming up. 6.17.13 is EOL right now.


    Best regards

    Prtigger

    Old (2011) Supermicro 1U Rackserver X7SPA-HF, Intel Atom D510, 4GB RAM (maximum) | System: Supermicro SATA DOM 64GB SSD | Data: 3*Samsung 870 QVO 4TB SATA (Btrfs Raid1) | OMV 8.x services: SSH, SMB, NFS, MiniDLNA | 6.12.x kernel | Bare-metal

    • Official Post

    6.18.3 is in the unstable repo now. It shouldn't be long before it is in backports.

    omv 8.0.10-2 synchrony | 6.17 proxmox kernel

    plugins :: omvextrasorg 8.0.2 | kvm 8.0.6 | compose 8.1.3 | cterm 8.0 | borgbackup 8.1.5 | cputemp 8.0 | mergerfs 8.0 | scripts 8.0.1 | writecache 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • This may be the next LTS kernel... ;)

    Old (2011) Supermicro 1U Rackserver X7SPA-HF, Intel Atom D510, 4GB RAM (maximum) | System: Supermicro SATA DOM 64GB SSD | Data: 3*Samsung 870 QVO 4TB SATA (Btrfs Raid1) | OMV 8.x services: SSH, SMB, NFS, MiniDLNA | 6.12.x kernel | Bare-metal

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!