Fastest way to rsync large files between two local machines?

jfromeo · May 27th 2023

Hello all.

I need to rsync a big chunk of large files (around 5TB, mostly 10GB files) between 2 machines connected to my local network. I have done some tests and the maximum speed size I get is 5MBps, which is damn slow.

The command I run on a remote Putty session over Wireguard on the machine with the source folders is:

rsync -v -r /source_folder/ root@IP_destination_machine:/destination_folder/

The speed output was:

sent 11,020,343,517 bytes received 35 bytes 4,849,436.11 bytes/sec

Not even 5 MBps. Given the drives and interfaces I was expecting >60 MBps

I do not know where the bottleneck is here.

The source machine is a SBC Pi4B 8GB running OMV 6.4.0 and the destination one is a NAS based on Xeon E3-1230v5 32GB running OMV 6.4.0.

Source HDD is WD Black 5TB connected to USB3.0 port of the Pi4B and destination HDD is WD HC560 20TB connected to SATA3 port of the NAS. Both Pi4B and NAS are connected to the same 10/100/1000GB switch (Linksys WRT-SE4008EJ) on Cat6 cables (<25m).

What could be the problem? I cannot detach the WD Black from the Pi4B and attach it to the NAS as both are at a remote location far from where I live.

Thanks in advance.

chente · May 27th 2023

Quote from jfromeo

The command I run on a remote Putty session over Wireguard on the machine with the source folders is:

Maybe that's the problem? I would try setting up an rsync task in the OMV GUI from one server to another using samba.

jfromeo · May 27th 2023

Quote from chente

Maybe that's the problem? I would try setting up an rsync task in the OMV GUI from one server to another using samba.

Thank you for the idea.

So I would need to create a remote SMB share on my Pi4B (source) which points to a SMB share on my NAS (destination) and create a rsync job from source to destination right?.

I will try.

chente · May 27th 2023

Quote from jfromeo

Thank you for the idea.

So I would need to create a remote SMB share on my Pi4B (source) which points to a SMB share on my NAS (destination) and create a rsync job from source to destination right?.

I will try.

That is. That job should work locally without a doubt.

However, it is always preferable if it is possible for the job to be executed as a pull rather than a push. Then I would create the share on the PI. I would do a remote mount on the xeon with openmediavault-remotemount. And I would create the job in pull mode from the xeon.

chente · May 27th 2023

When I configure works like that, I usually add this in extra options.

--chmod=0775 --chown=root:users

That ensures that permissions won't cause problems.

This is only if they are data. If there are container or configuration files, don't do it.

jfromeo · May 27th 2023

Quote from chente

That is. That job should work locally without a doubt.
However, it is always preferable if it is possible for the job to be executed as a pull rather than a push. Then I would create the share on the PI. I would do a remote mount on the xeon with openmediavault-remotemount. And I would create the job in pull mode from the xeon.

Thank you so much for all the help.

I will try it as soon as I hit home.

crashtest · May 27th 2023

Since you're doing a big rsync job, it would be better to "pull" than to "push". This means that the machine that is running the command should be the destination and the source should be the remote server.

MarcS · May 30th 2023

Quote from jfromeo

Not even 5 MBps. Given the drives and interfaces I was expecting >60 MBps

The rsync command you are using uses encryption via SSH, which slows down your transfer considerably. Instead you can try unencrypted rsync:

Code

rsync -avhP --stats /source_folder/ user@LAN_host_IP::Rsync_Target

This should give you 10-20 MB/s instead of 5MB/s. Thats the maximum I can get on my internal LAN.

(You need to define RsyncTarget as a folder within the OMV rsync plugin though. There its called a Module)

Fastest way to rsync large files between two local machines?

Participate now!

Tags