Speed rsync raspberry pi 4b too slow

    • OMV 4.x

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • Speed rsync raspberry pi 4b too slow

      Hello everyone,

      i installed this image (sourceforge.net/projects/openm…/Raspberry%20Pi%20images/) on my raspberry pi 4b. I set up an rsyncjob using the GUI to copy files from a NAS (cifs) to a USB drive connected to the USB 3.0 port of the raspberry pi.
      When copying files I realised the speed is only around 20 MB/s.

      However if i connect the USB drive to my windows machine and copy from the same NAS i get speeds of around 110MB/s.

      Should the raspberry pi 4b not be able to achive something around at least 80 MB/s? It is connected via ethernet to the NAS.
      USB drive has its own power supply.

      (i m a newbie. If you need more information i can provide it. Please provide detailed explanation how to get it)

      Thanks for any help!
    • Rsync can be pretty slow copying files. Especially if it is many small files. But it also may depend on the direction and the filesystems used. Faster when using SATA, NFS and ext4 than SMB, NTFS, USB and/or SSH. In my experience it also seems that rsync is a little faster pulling data than pushing data.

      Pulling means you run rsync on the destination server, not on the source server. It might be better cache usage?

      It may also be possible to speed up rsync by running rsync using the rsync daemon and bypass network filesystems completely. Also you might speed up rsync by turning off some features and checksum stuff and/or running it on a more powerful server.

      But still rsync will be slower than plain metal to the metal filecopying.

      The benefit of using rsync becomes more apparent when you update the files later. Then only changed/added files need updating.
      OMV 4, 7 x ODROID HC2, 1 x ODROID HC1, 5 x 12TB, 1 x 8TB, 1 x 2TB SSHD, 1 x 2TB SSD, GbE, WiFi mesh
    • Adoby wrote:

      Rsync can be pretty slow copying files. Especially if it is many small files. But it also may depend on the direction and the filesystems used. Faster when using SATA, NFS and ext4 than SMB, NTFS, USB and/or SSH. In my experience it also seems that rsync is a little faster pulling data than pushing data.

      Pulling means you run rsync on the destination server, not on the source server. It might be better cache usage?

      It may also be possible to speed up rsync by running rsync using the rsync daemon and bypass network filesystems completely. Also you might speed up rsync by turning off some features and checksum stuff and/or running it on a more powerful server.

      But still rsync will be slower than plain metal to the metal filecopying.

      The benefit of using rsync becomes more apparent when you update the files later. Then only changed/added files need updating.
      Thanks for the quick reply. You really think rsync/the server is the bottle neck in this case? Interessting...

      rsync is running on the rasp. so i'm pulling the data from the NAS. So that won't help. The NAS has a NTFS file system (can't change that.) The USB hard disk is plugged into the rasp. and is also NTFS (can't change that, since windows needs to read that filesystem as well)

      In this case i can't bypass the network file systems. Which features do you recommend to turn off to speed up the process?
    • Try to google "rsync faster" for ideas. But most suggestions are about using rsync on the command line.

      You could try to activate the rsync daemon on the NAS and do the copying directly between rsync-rsync, bypassing SMB.

      Or just accept that rsync isn't very fast, and let it run over night or a few days. It might be much, much faster to move the USB drive to the NAS and copy locally from the NAS directly to the USB drive.

      But I suspect a more powerful computer and not using USB and NTFS would help the most.
      OMV 4, 7 x ODROID HC2, 1 x ODROID HC1, 5 x 12TB, 1 x 8TB, 1 x 2TB SSHD, 1 x 2TB SSD, GbE, WiFi mesh
    • Next week I intend to setup a OMV NAS based on a RPi4 and a dual drive external UASP USB 3 enclosure. I intend to use two old 8TB Archive SMR drives in the case. Not in any form of RAID! This NAS will only be used for versioned rsync snapshots of old/static data on other OMV NAS.

      Startech S352BU33HR

      It will be interesting to see what speeds I can get.
      OMV 4, 7 x ODROID HC2, 1 x ODROID HC1, 5 x 12TB, 1 x 8TB, 1 x 2TB SSHD, 1 x 2TB SSD, GbE, WiFi mesh

      The post was edited 2 times, last by Adoby ().

    • New

      Any idea how i can test the general copy speed without using rsync? Just to make sure that rsync is the bottel neck. Just copy a large file using cp command from the NAS (which btw is a drobo 5N, not a OMV) to the USB hard drive connected directly to the pi.

      or is there a better quick and easy way?
    • New

      Try something like:

      time cp source destination

      It is not only rsync that might be the problem. It may also be the USB interface, possibly in combination with rsync.

      There are external drives with UASP-interfaces that claim to be much faster and less processor demanding. That seem to imply that non-UASP-interfaces are much slower and mor processor demanding. This would show extra well when using rsync.

      I never buy any external USB storage that does not use UASP.

      en.wikipedia.org/wiki/USB_Attached_SCSI
      OMV 4, 7 x ODROID HC2, 1 x ODROID HC1, 5 x 12TB, 1 x 8TB, 1 x 2TB SSHD, 1 x 2TB SSD, GbE, WiFi mesh
    • New

      just copyied a 2950 MB file using cp from the NAS to the USB hard drive in 4 minutes and 30 sec (real: 4min 30sec, user 0, sys 26 sek). Thats really slow. It's just over 10,9 MB/s.

      That makes me think rsync is not the problem.

      Thanks for the wikipedia link. I don't think the external USB device is the problem here. I used that same USB HDD on my windows machine and achieved speeds way over 100 MB/s. e.g. NAS -- via Gigabit Eternert -- Windows Laptop -- on USB 3 port -- USB 3.0 HDD.

      Now the setup is NAS -- Gigabit Ethernet -- rasperry pi 4 b -- USB 3 port -- USB 3.0 HDD. In this setup i somehow loose the speed yet i don't know why. Any other ideas?

      (running htop i checked CPU is at around 33%. Most of that ist the mount command. the cp only makes up for around 5-7%)

      The post was edited 1 time, last by AlessBoe ().

    • New

      I just copied a bunch of big files from an SSD on the RPi4 to another NAS. I used Midnight Commander. And I got speeds of about 80-90 MB/s sustained. A bunch of 40-250 GB zip files. I did a "push" to a Odroid HC2 also running OMV.

      I used a UASP enclosure for the Crucial MX500 SSD. NFS4. EXT4. A GbE switch in between. No special optimizations, just my normal settings. 1 MB write buffers set in autofs. Seagate Ironwolf on the HC2.

      However, when I started to copy another set of big files simultaneously to another NAS, in another console, the transfer speed fell not to half as expected, but to around a quarter. The combined speed was perhaps 40 MB/s.

      Then I started unzipping in parallel using 7zip on the HC2s, and also using Emby to stream video, and the transfer speed fell further. Still streaming, copying and unzipping... Should be finished in a few hours. Streaming is fine...

      I have read somewhere that Linux don't recognize all chipsets for USB interfaces. So it might be possible that Windows can use the UASP interface correctly, but not OMV. Try another USB interface/enclosure?

      USB can be very pɹᴉǝʍ...
      OMV 4, 7 x ODROID HC2, 1 x ODROID HC1, 5 x 12TB, 1 x 8TB, 1 x 2TB SSHD, 1 x 2TB SSD, GbE, WiFi mesh

      The post was edited 2 times, last by Adoby ().