[SOLVED] rsync stops at 4:30am

Hi all, I'm experiencing a problem with OMV.

I replicate my NASes over VPN using rsync. I use pretty much the same scripts I wrote 4 years ago on freenas. This script worked well on nas4free but now that I moved to OMV the transfer stops exactly at 4:30am.

I use bash scripts instead of the rsync GUI. I then call these scripts via the crontab GUI and have tried an output of the command to emai, the last three lines of which are:

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (197610 bytes received so far) [sender]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(601) [sender=3.0.7]

I checked the routers and the log but everything seems normal, so I was wondering if there might me anything OMV does with the crontab?

Not sure

Thanks!
rs232

As an example:

#!/bin/bash
RSYNC="/usr/bin/rsync"
KILLALL="/usr/bin/killall"


$KILLALL rsync 2>/dev/null
sleep 5
cd /mnt/raid5/tomato
$RSYNC -ratP --filter=-rs_/encrypted /mnt/raid5/DOCS/* 10.10.9.4:/mnt/raid5/DOCS/
#echo "DOCS done"
$RSYNC -ratP --inplace /mnt/raid5/DOCS/encrypted/* 10.10.9.4:/mnt/raid5/DOCS/encrypted/
#echo "Encrypted done"
$RSYNC -ratP /mnt/raid5/PRINTER/* 10.10.9.4:/mnt/raid5/PRINTER/
#echo "Printer done"
$RSYNC -ratP --filter="merge 36k-213w-rsync-filters" --safe-links /mnt/raid5/* 10.10.9.4:/mnt/raid5/

I did check and I also checked the crontab of the router but nothing at 4:30. I suspect the ISP is doing something at that point of time but strange, it never happened before.

I'll keep an eye on this

So after further investigation this seems to be an OMV issue. The VPN is up and running and according to the logs it didn't go down for days.

A further piece of info is: 4:30 is a fact... but rsync also brakes randomly. Let me explain, on the router mrtg I can see the traffic dropping to zero at 4:30, however when I run the rsync manually:

after 22minutes I got:

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (216789 bytes received so far) [sender]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(601) [sender=3.0.7]

Another attempt stopped after about 6 hours and 1/2 minutes

rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(543) [sender=3.0.7]
 
real    328m27.819s
user    0m36.058s
sys    0m13.853s

A third attempt:

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (92190 bytes received so far) [sender]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(601) [sender=3.0.7]
 
real    23m25.217s
user    0m10.297s
sys    0m3.432s

NOTE: did not stop the sync manually!

Not sure how to tackle this now...

RIght, another piece of info.

I've been running the rsync cron job every 30 minutes. This way it interrupts itself every 1/2 hour and restore the transfer. Not ideal of course but it works.

proceeding with the troubleshooting I then change the cronjob from every 30/min to every 60min and an interesting data appeared. Please see pic in attachment.

at 2am you see a little drop in the throughput as my rsync script starts with a killall rsync, so nothing strange, then at 2:30 rsync fails (by itself and restart again at 3am).

I have noticed that rsync fails every 2 hours! So 2:30, 4:30, 6:30 and 8:30 (it's 9:30 now that I'm writing).

Thus the 4:30 in the title is a fact but the reality is that rsync stops every two hours at specific intervals.

If anybody has any clue on how to solve this please help.

Regards

This is indeed an OMV problem!

I have run the rsync bedside a ping from the nas to the brother nas, the default gateway, the remote default gateway and google.com.

All the pings are successful where rsync still brakes every 2 hours. Please help!

RIght,
I got to the bottom of this but I still need your help if you don't mind

My script uses the "killall rsync" command at the beginning of the script. I had to install killall using apt-get btw.

Now to cut a long story short: OMV-A replicates some data to OMV-B and some data from OMV-B is replicated to OMV-A.

For some strange reason when run "killall rsync" on OMV-B this kills the transfer initiated by OMV-A.

I can't figure out why as (AFAIK) rysnc uses ssh and there should be no trace of rsync on the remote system at all. Once again the very same scripts on Nas4free and Freenas where working fine...

How come a kill all rsync kills a remote rsync transfer??

I did also try to add
-e ssh
to make sure rsync uses ssh; same music.

Any help?!

Thanks

Thanks for the input and glad I got to the bottom of this!

I've chosen a slightly different approach, no killall needs to be installed now

ps | grep rsync | grep -v server | grep -v grep | grep -v _rsync_ | cut -d " " -f1 | while read f; do kill -9 $f; done

Note the "_rsync_" is part of the script filenames I'm using, thus a grep -v is needed as I don't want it to kill themselves of course

It seems to be working now!

Thanks again!