rsync to eSata is extremely slow

  • Hi,


    I'm running a daily backup job using something like


    rsync -avR --delete "${home}" "${target}/" --link-dest="${targetRoot}last/"


    which syncs 800GB from a Raid5 to an internal backup drive.


    Once a month I run the same script that takes another backup for same Raid5 to an external drive connected via eSata.


    The daily backup takes 6 minutes (not a lot of changes every day), but the monthly backup to eSata takes several hours (99,9% of the files still are unchanged).


    I checked the read/write speed of the disks - they are almost identically, so it should be some issue on how I use rsync.


    Any ideas?


    Mark.

  • Not sure this is going to help you but I've found rsync to be slower than anticipated (underlying compression or list compiling/diff ?). Granted I had nothing copied or mirrored to begin with, just attempted to copy to a new drive from an old external NAS. Have you tried any include/exclude lists for files/directories you know for a fact haven't changed? Based on what I've read compression is optional, also depending on the version of rsync it puts quite a bit of its list of files into memory. Check out the FAQ and information regarding the algorithm used by rsync.


    https://rsync.samba.org/FAQ.html


    http://everythinglinux.org/rsync/

    AMD Athlon 5350
    APUASRock AM1H-ITX mobo
    PNY CS1311 SSD 120GB 6GB/s TLC (for boot only)
    8GB Crucial Ballistix Sport DDR3 1600 x1
    Thermaltake Core V1 case
    External Toshiba 90w PSU brick ac adapter
    OMV version: Release: 2.2.1

  • but why is the backup to internal drive fast and only the external target gives such slow transfer speed?


    You'll have to see where the bottleneck resides. Could be network or the external drive itself. These types of speed issues are frustrating when you have a known 1GB network and know your transfers should be faster. But this is never true sustained speed because of the overhead of CPU/Memory and Drive Performance. This is what I did recently on the internal drive (any and all drives I guess):


    Create a 100M file using dd command (caution you could wack things if not careful). Can't recall exact syntax to create empty 100M but google something:
    dd if=/dev/uuid/random of=output.dat bs=1024 count=102400


    Then ran this to test amount of time to copy file1 to file2:
    sync;time (cp output1.dat output2.dat; sync)


    Results for me:
    real 0m0.700s
    user 0m0.000s
    sys 0m0.516s


    Then ran scp from external box to new internal nas drive of same file:


    output.dat 100% 100MB 5.3MB/s 00:19


    This told me it was network or rsync/scp and not the drive perf itself. I then went to ftp versus rsync or scp.
    What took me over a day to move took 30mins for a 226GB .rar file from my old nas to new.

    AMD Athlon 5350
    APUASRock AM1H-ITX mobo
    PNY CS1311 SSD 120GB 6GB/s TLC (for boot only)
    8GB Crucial Ballistix Sport DDR3 1600 x1
    Thermaltake Core V1 case
    External Toshiba 90w PSU brick ac adapter
    OMV version: Release: 2.2.1

  • Thanks for your posting....


    There is no network involved here - I'm syncing my NAS to an external eSata drive, so basically it should behave the same as the daily backup to the spare drive within the NAS.


    I tried ot disable the block checking (file size only) etc. no success...


    No network, no NFS, .... - I'm totally at loss here :(


  • Ah. My bad. You should be getting 1.5Gbps per specs for eSata. What if you do more frequently than a month? Don't wait so long. Incremental should be same as monthly. Try --progress to see what file(s) or if disruptions. Nothing else I can think of.


    Also, do you know what cpu on was is at? Must be some checksumming or maybe a switch to limit checks. I'm curious about cpu and progress.

    AMD Athlon 5350
    APUASRock AM1H-ITX mobo
    PNY CS1311 SSD 120GB 6GB/s TLC (for boot only)
    8GB Crucial Ballistix Sport DDR3 1600 x1
    Thermaltake Core V1 case
    External Toshiba 90w PSU brick ac adapter
    OMV version: Release: 2.2.1

    Edited once, last by Frogdaddy ().

  • CPU is a AMD Turion II Neo N54L with 8GB RAM, rsync fully uses both cores.


    Disk transfer rates on internal spare drive and external eSata are identical.


    I did some more tests, all with a complete backup in place which is used for the link-dest argument of rsync.


    My skript backups several directories, most of which contain a few hundred/thousand small/medium files with NO changes/modifications, but one directory contains 15000 photos of ~10MB each and another one keeps 80000 audio samples of similar size.


    Running my skript with the internal target is very fast - no changes, nothing to do.


    Running it with the eSata drive as target skips all files (since they are unmodified) in all source directories (including the photos) whith the exception of those in the Audio directory. So although those have NOT been modified either, rsync thinks it needs to process them anyway and unnecessarily shuffles around 800GB.


    Strange!

  • yes, both target drives are using ext4


    Code
    /dev/sdd3 on /media/ab855e11-d4b3-4d1e-8dd8-5487cb6d9518 type ext4 (rw,noexec,relatime,user_xattr,acl,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
    /dev/sdf1 on /media/9233e731-35e8-44a1-acda-494a8e782c5b type ext4 (rw,noexec,relatime,user_xattr,acl,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!