Duplicati and Rsync sucking resources dry

  • Hello all


    I backup data from my OMV 4 in 2 ways
    1 - via rsync push into a lan-based rsync daemon server. Work is carried by a number of rsync jobs defined and managed via OMV's rsync plugin.
    2 - via Duplicati (installed as a container), sending encrypted backups towards a OneDrive account.


    It all works, but not satisfactorily. The problem is that both job types seem to "suck all hw resources dry" when they work.
    In practical terms, what happens is that while either Duplicati or Rsync are doing their stuff
    - it's impossible to log into OMV's GUI
    - it's also impossible to SSH-in (a timeout happens)


    Underlaying hardware is Odroid HC2.
    Flashmemory plugin is indeed up and running.


    Is this to be considered "normal" (meaning, is the something one has to expect from an ARM-based platform etc) or is there something I need to check on my configuration?


    Thanks for any help


    A.

    • Offizieller Beitrag

    Is this to be considered "normal" (meaning, is the something one has to expect from an ARM-based platform etc) or is there something I need to check on my configuration?

    I see it on my lower powered x86 systems too. rsync only copies the changed blocks. This can use a lot of cpu as you have found. I don't think I've even been blocked from logging in via ssh though. But I run my syncs in the middle of the night.

    omv 7.0.4-2 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.10 | compose 7.1.2 | k8s 7.0-6 | cputemp 7.0 | mergerfs 7.0.3


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    • Offizieller Beitrag

    It is possible to throttle rsync jobs. But it shouldn't be needed.


    You could run duplicati on the backup server instead, after the backup has been sent to it. No reason to use the HC2.


    Are you using a "normal" filesystem? The fastest is most likely EXT4 without encryption. Add encryption and/or NTFS and things may slow down quite a bit.


    Use a switch.

  • So in a nutshell the answer to my question is "yes". Aright I'll live with it.


    @Adoby
    Yes I'll launch Duplicati on the backup box to oflload some work.
    Already thought about it, together, maybe, with "pulling" rsync from the remote machine instead of "pushing" into it from OMV.
    Yes I'm using a "normal" filesystem on OMV (EXT4 unencrypted, idd)
    And yes, I'm using a GbE switch. All involved servers have wired eth connections to it. Main router is an EdgeRouter X which elsewhere proved to be powerful enough to support my internet bandwith (980something mbits dowload, 390something upload - the latter being involved with Duplicati backup ops)

    • Offizieller Beitrag

    Well, I have never experienced any problems logging in to the GUI, even while running several rsync jobs. And I don't use duplicati. An I typically use WiFi to connect to my OMV servers. But the OMV servers are all connected using GbE and a switch.


    The network topology can also cause problems. For instance if you have more than one switch and several switches/routers/servers/devices are sending traffic to one port in one switch. Ideally you should have only one switch. And every device/router/server connected directly to that. No daisy chaining.


    I assume that you run rsync in "snapshot" mode so that you don't copy files that are unchanged? Instead hardlinking between snapshots? That makes rsync, for me, create a full backup in seconds or minutes instead of hours or even days.


    Duplicati can be run the same way I assume? No point in calculating checksums for the same blocks twice?


    At what times do you run rsync/duplicati? Can you do it at night or when you are at work? Then you don't have to notice heavy loads at all. I run all my backups daily at 03:00. Usually it just takes a couple of minutes to update a full backup of around 20TB. Half an hour, perhaps, if I have added new media/files.

  • Mmm. From what you write I supsect I do have some investigation to run on my OMV but I do not know where to dig.


    Net topology is totally straight in my case. Only 1 main switch. On 1 port I got the EdgeRouter. On 2 other ports I got 2 WAPs. Other ports for 3-4 ARM servers. All clients (PCs, tablets) connect via wifi.


    No I don't use snapshot mode (basically because I don't know how to). I just dumbly "push" from one folder onto Rsync server daemon. That said I'm quite positive only what has changed gets sent anyway, which is very little on a daily basis.


    Duplicati also sends only changed blocks (not even files).


    All tasks run of course very early morning like you mentioned (5AM or so), which is on one side OK as there is no one using the system anyway so who cares if it crawls for 20 minutes or so (all jobs included - yes each one goes for like 4 min, 8min, 3min... due to very little changes to be accounted for).
    But on the other hand this means that those sole tasks are able to flatten the server on their own, as nothing else is happening when they run idd, which, again, seems "odd" to me.

    • Offizieller Beitrag

    Duplicati also sends only changed blocks (not even files).

    As far as I know Duplicati is also checking some data on the backup site if they are correct.


    Location of the database might be another area to look at if it is on a slow SD card.

  • Yes Duplicati is checking on the target, which is OneDrive so it is relatively slow access. But this should have no impact on originating server resources. It should just "take longer" then it would if targeting a local server.


    Duplicati is run from within a Container and (of course) container data (as overall Docker data) are on HD, not on SD card.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!