Rsync from whatbox seedbox

looking to rsync from my seedbox to my OMV server

First I need to setup passwordless ssh login I found this guide https://www.debian.org/devel/passwordlessssh

I believe I did it right. I had OMV spit out they keys and I put the file in /.ssh/authorized_keys folder on the seedbox.

Is there a way to test this?

Here is the rest of the directions given by whatbox

Zitat

Regularly downloading files with rsyncWith a few extras.

Using rsync is by itself pretty straightforward, but you can also get a bit more out of it. Setting this up will download anything new when it is finished, and send you an email listing of what was downloaded since the last email, with usage information from Skynet. Using cron you decide how often it runs, and within the script you decide at what hour you want the mail.

RequirementsFor this to work properly you need to be using rTorrent and passwordless SSH authentication. Add the following line to your rtorrent.rc and create a directory called "finished" in the root of your home directory.

Code

system.method.set_key = event.download.finished,link3,"execute=ln,-s,$d.get_base_path=,/home/droptopfox90/finished"

This makes a symbolic link in ~/finished when whatever you are downloading is completed.

You will also need netcat, and msmtp on your local *nix server. netcat is needed to get info from Skynet, and msmtp needs to be configured separately for mail to work. This guide should be good for that.

The script below should be saved to a .sh file, and use cron to run it as often as you like.

The Script

Code

# This is a script which works for me, based on the previous script submitted here and then further tailored to suit my preferences. Use at your own risk etc.


#!/bin/sh


# Configuration, should be all set up.
remoteuser="droptopfox90"
remoteserver="taco.whatbox.ca"
sourcefolder="/home/droptopfox90/finished/"
targetfolder="/your/local/folder"
templog=`mktemp -t templog.XXXX`
templogfind=`mktemp -t templogfind.XXXX`
syncvar="-avzPL -e ssh --log-file=$templog --bwlimit=4096"


# Except this part.
mailto="receiving mail adress"
mailfrom="outgoing mail adress"


tempmail=".tempmail"


# Mail out an error report when called by a general signal trap
    cleanfail () {
    printf "To: $mailto\nFrom: $mailfrom\nSubject: Whatbox transfers - From `date +%d-%m-%Y-%H:%M:%S`\n\nWhatbox synchronisation has failed due to unspecified signal error.\n\n" > $tempmail
    cat $tempmail | msmtp -a default $mailto
    rm $tempmail
    rm $templog
    rm $templogfind
    exit 1
    }

# Mail out an error report when called by an rsync trap.
    cleanfail_rsync () {
    printf "To: $mailto\nFrom: $mailfrom\nSubject: Whatbox transfers - From `date +%d-%m-%Y-%H:%M:%S`\n\nWhatbox rsync has failed. Rsync log attached.\n\n" > $tempmail
    cat $templog >> $tempmail
    cat $tempmail | msmtp -a default $mailto
    rm $tempmail
    rm $templog
    rm $templogfind
    exit 1
    }


# Mail out an error report when called by a find trap.
    cleanfail_find () {
    printf "To: $mailto\nFrom: $mailfrom\nSubject: Whatbox transfers - From `date +%d-%m-%Y-%H:%M:%S`\n\nWhatbox symbolic link cleanup has failed. Find process log attached.\n\n" > $tempmail
    cat $templogfind >> $tempmail
    cat $tempmail | msmtp -a default $mailto
    rm $tempmail
    rm $templog
    rm $templogfind
    exit 1
    }


# Mail out an error report when called by an existing instance of rsync already running trap.
    cleanfail_rsync_active () {
    printf "To: $mailto\nFrom: $mailfrom\nSubject: Whatbox transfers - From `date +%d-%m-%Y-%H:%M:%S`\n\nrsync process is still active. Large download may still be in progress.\n\n" > $tempmail
    cat $tempmail | msmtp -a default $mailto
    rm $tempmail
    rm $templog
    rm $templogfind
    exit 1
    }


# Check if rsync is already running. If so, exit with an email notification reporting transfers still in progress.
pgrep rsync >/dev/null 2>&1 && cleanfail_rsync_active


# Touch timestamp file to prevent deleting links that are created while rsync is running
ssh $remoteuser@$remoteserver touch $sourcefolder/.rsync-timestamp


# Trap error signals
trap cleanfail 1 2 3 15


# If all is well so far, commence the rsync automated download from the seedbox. Run it twice in case if more files came in during the first rsync process.
rsync $syncvar $remoteuser@$remoteserver:$sourcefolder $targetfolder
rsync $syncvar $remoteuser@$remoteserver:$sourcefolder $targetfolder


# Trap any errors, log and email them if rsync does not terminate with a successful exit value (0). Otherwise, continue to delete symlinks now that we're done with them. If the deletion of symlinks fails, this error is also trapped, logged and emailed.
if [ $? != 0 ]; then
    cleanfail_rsync
else
    printf "\n\nTransfer successful. Deleted following symbolic links:\n" > $templogfind
    ssh $remoteuser@$remoteserver find $sourcefolder \! -newer $sourcefolder/.rsync-timestamp -type l -delete -print >> $templogfind
    if [ $? != 0 ]; then
        cleanfail_find
    else
    :
    fi
fi


# Mail out report for a successful transfer. List a trimmed down version of rsync log file, trimming down to parent folders and otherwise single files. If you prefer the full rsync log file, just uncomment the line after the following printf and comment out the cut line after it.
printf "To: $mailto\nFrom: $mailfrom\nSubject: Transfers - From `date +%d-%m-%Y-%H:%M:%S`\n\nDownloaded File List:\n" > $tempmail
# cat $templog >> $tempmail
# I believe this cut line is simpler than using awk with print arguments. This series of commands looks for the "+" in the rsync log file and removes everything up to and including the last "+". Then grep filters out the first supposed file in the list which is just "./". After this, sed takes over and strips all the subdirectories from the first "/" encountered after the parent directory name(s). All that's then left to do is to wipe out all the duplicate lines so that all that remains is each parent directory without any duplication and unique files. It essentially creates the same list as what's in the /finished/ folder but at least it's from a confirmed rsync log file rather than working on assumptions, just to keep the script robust.
cut -s -d + -f 8 $templog | grep -v "\./" | sed 's%/.*%%' | uniq >> $tempmail


# Add the symlink deletion results
cat $templogfind >> $tempmail


# Collect usage stats from Skynet IRC bot and add to email report
printf "\n$remoteserver Status:\n" >> $tempmail
printf "NICK $remoteuser\nUSER $remoteuser 8 * $remoteuser\nPRIVMSG Skynet !usage\nPRIVMSG Skynet !traffic\nPRIVMSG Skynet !expires\nQUIT\n" | netcat -i 3 irc.whatbox.ca 6667 | grep -w "Skynet" | grep -v "Checking...please" | awk -F":" '{ print $3 $4 $5 }' >> $tempmail


# Add local disk stats to email report
printf "\n`hostname`/mnt/sdb1 usage:\n" >> $tempmail; df -h | grep Filesystem >> $tempmail && df -h | grep sdb1 >> $tempmail


# Email the compiled report    
cat $tempmail | msmtp -a default $mailto


rm $templog
rm $templogfind
rm $tempmail
exit 0

Alles anzeigen

Alles anzeigen

Is there any reason this isn't doable? or an easier better way to do it?

I would like to do it through a tunneled /secure/encrypted connection so it can't be snooped on..

Thanks guys

actually is this able to be set up from the plugin? looks like it might be able to

Anyone have any experience with this?