rsync 3.5.0 regression - skip this version

  • Hello.


    In the past few days, I have been struggling to find out why a rsync task that was running fine in 7-8 minutes started behaving strange and increased duration to more than 3 hours.

    Context:

    - NAS with gigabit ethernet connection and OMV 8 (latest version of all packages)

    - Desktop with gigabit connection and a shared folder

    - mount the desktop shared folder in a specific location on NAS

    - rsync a specific folder containing all kinds of files from one of the locations on NAS to the desktop shared folder (mount point)

    NOTE: there are more than 500.000 files: videos, photos, documents, etc. ranging in size from 0 bytes to 10 GB, but most of the files do not get updated, there are new files are created and from time to time, files smaller than 10 MB get updated.


    with version 3.4.x, the sync duration was 7-8 minutes

    after upgrading a few days ago to 3.5.0, this task duration increased to more than 3 hours


    since this is an old desktop (core 2 quad / DDR2), I blamed immediately the desktop and started to investigate it:

    done some speed tests and some benchmarks on components, all seemed to work as they should, but considering the age of the desktop, the results were as expected:

    - speed test (on ookla) - 780 mbps on download and upload

    - copy from NAS to desktop and from desktop to NAS of large files: 120 MB/s (initiated from both, desktop and NAS)

    - copy between internal HDDs of large files, 180 MB/s

    - memory tests did not show any problem

    - cpu did not overheat at all during transfers (never went over 60 C even during CPU benchmarks)


    since desktop was showing no degradation results, started investigating what changed on NAS.

    - ckecked the list of recently upgraded packages and found kernel and rsync upgrades
    - booted to older kernel (7.1.8-1~bpo13+1) did not solve the issue

    - downgraded rsync from 3.5.0+ds1-0+deb13u1 to 3.4.1+ds1-5+deb13u4 and this solved the issue.


    If there is someone else that observes a speed downgrade when running rsync tasks and wants to fix it, either install 3.5.1 (that theoretically contains the fix) or downgrade to 3.4.1:

    Code
    sudo apt install rsync=3.4.1+ds1-5+deb13u4

    Hope this helps someone.

    • New
    • Official Post

    The reason Debian moved to rsync 3.5 is that there were so many CVEs to fix that the maintainer did not want to backport all of the fixes to 3.4 and risk more issues.


    I'm curious what options you are using? I have a nightly rsync to an RPi that is about 410k files (not many change though) and it hasn't slowed down.

    omv 8.5.11-1 synchrony | 7.0 proxmox kernel

    plugins:

    omvextrasorg 8.0.3 | kvm 8.3.4 | compose 8.2.2 | cterm 8.0.4 | borgbackup 8.2.1 | tempmon 8.0.5 |

    mergerfs 8.0.2 | scripts 8.0.6 | writecache 8.3.0 | backup 8.1 | kernel 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • I am using it like this:

    Code
    rsync -avu ${SOURCE_FOLDER} ${DESTINATION_FOLDER}

    Nothing special with regard to options...

    But I am not rsync-ing over SSH, but to a mounted SMB share from the desktop.


    But I am 99% sure that the problem doesn't come from mounting the SMB share on OMV, it is exclusively a rsync problem, it got magically fixed after going back to previous version.

    • New
    • Official Post

    Might be worth reporting to Debian. Although they might call it a "fix" in the name of "security"

    omv 8.5.11-1 synchrony | 7.0 proxmox kernel

    plugins:

    omvextrasorg 8.0.3 | kvm 8.3.4 | compose 8.2.2 | cterm 8.0.4 | borgbackup 8.2.1 | tempmon 8.0.5 |

    mergerfs 8.0.2 | scripts 8.0.6 | writecache 8.3.0 | backup 8.1 | kernel 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • I'll wait for 3.5.1 to be available in debian sources and if the problem appears again, I will report it.

    I honestly don't want to report it to debian if debian doesn't provide the latest version that might contain the fix.


    I could try to get 3.5.1 before debian provides it officially, but I do not have the time right now to do it.

    Maybe next week...

    • New
    • Official Post

    3.5.1 is in forky. You could probably download the .deb file and try it - https://packages.debian.org/forky/rsync

    omv 8.5.11-1 synchrony | 7.0 proxmox kernel

    plugins:

    omvextrasorg 8.0.3 | kvm 8.3.4 | compose 8.2.2 | cterm 8.0.4 | borgbackup 8.2.1 | tempmon 8.0.5 |

    mergerfs 8.0.2 | scripts 8.0.6 | writecache 8.3.0 | backup 8.1 | kernel 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Hi there,


    just stumbled over that thread. My OMV instances have also "rsync=3.5.0+ds1-0+deb13u1" running, installed by GUI Update on 29th Sept. One instance does provide rsyncd service for various shares, so basically rsync server, and 2 other OMV instances are collecting the data by rsync client jobs for each share. Data is mostly pretty neat, its raspiBackup system Backups from various Raspberry Pi, with deep directory structures.


    I checked the 3 days before update, the day of update and the 3 days after the security update of rsync. There's no difference in the runtimes before and after. Its just seconds, basically just dispersion and well, maybe just a little data growth.


    So, can't confirm your issues here, sorry.

  • 3.5.1 is in forky. You could probably download the .deb file and try it - https://packages.debian.org/forky/rsync

    the forky version needs libacl1 (>= 2.4.0) and the version that I have installed on OMV is 2.3.2...

    I could try to install it and get the required dependencies too, but I am afraid not to break other software... and because for the moment I am not near the NAS, I don't plan to test things right now...

    probably depends on what you rsync (location of source and destination) and how huge the data is.

    on my side, on local drive it works as before.

    but when I sync from a local drive to a mounted SMB share from the desktop it is a lot slower.

    by only downgrading rsync to previous version fixed the problem so I am almost completely sure that the problem comes from rsync and not my setup.

    • New
    • Official Post

    I could try to install it and get the required dependencies too,

    I wouldn't do that then. I would wait for 3.5.1 since 3.4 is working for you.

    omv 8.5.11-1 synchrony | 7.0 proxmox kernel

    plugins:

    omvextrasorg 8.0.3 | kvm 8.3.4 | compose 8.2.2 | cterm 8.0.4 | borgbackup 8.2.1 | tempmon 8.0.5 |

    mergerfs 8.0.2 | scripts 8.0.6 | writecache 8.3.0 | backup 8.1 | kernel 8.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Apologies for hijacking the thread, I wasn't sure if a new one was appropriate.


    I haven't experienced any drop in speed or performance from Rsync, however, I've experienced a change in Rsync logging behavior from jobs since updating to 8.5.10.-1, though the issue may not even be Rsync itself. It's also possible that this is a "skill issue" as I am prone to tinkering with things that should be left alone.


    Observation of a task (/var/lib/openmediavault/cron.d/rsync-xxxx) shows the behavior that I'm using to seeing when the task completes normally:

    Code
    if [ ${status} -eq 0 ]; then
        omv_log "The synchronisation has completed successfully."

    The tasks are running and completing as expected, and even print the completion message to the terminal and GUI (if applicable). Shortly after running a task I run:

    Code
    journalctl --since "5 minutes ago" -o short-precise | grep '<rsync_task_id_number>'

    and see:

    Code
    Oct 05 08:02:21.024198 hostname rsync-xxxx[296086]: The synchronisation has completed successfully.

    However, I don't see this entry in the log file entry under Rsync-Jobs in the GUI nor in the remote logs being sent to the local syslog server. In the GUI I do see the other entries such as "total size is x speedup is y" etc.

    Please consider donating to the project if you rely on OMV like many of us do:


    OMV


    Plugins (scroll to the bottom)

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!