How do I know if the rsynch copy is good and usable ...

  • I have followed the guides "Installing OMV5 on a R-PI - rev 2.2" and "Getting_Started-OMV5" and I have OMV running beautifully on a Raspberry Pi 4B.


    Using the instructions on page 60 of "Getting_Started-OMV5" titled "Full Disk Mirroring / Backup with Rsync" I have a second USB drive with an exact copy of the main drive.


    I have setup a scheduled task to run:


    rsync -av --delete /srv/dev-disk-by-label-HDD1/ /srv/dev-disk-by-label-HDD2/


    every Wednesday morning at 1:00am. All good, so far.


    Now, how can I check that HDD2 does in fact contain an exact copy of HDD1 and that I could use HDD2 to retrieve files to my Windows PCs if HDD1 ever failed?

  • If you run a second rsync operation and there is no files list or transferred data, the copy is complete. In that case, if no errors are reported and there's no changed files list, the drives are mirrored.


    If you have doubts, you can switch a shared folder from the source drive to HDD2, using the Guide's instructions for recovery. You'll find that the backup share on HDD2 is identical to the source in every detail and is immediately available. After the test, simply switch the shared folder back to the primary drive HDD1 in the same manner.


    If you'd be happy with a files comparison, install midnight commander per the Guide and check the contents of the drives in the left and right windows. But exercise some caution. MC doesn't have an undo button. Once confirmed, operations are immediately executed.

  • With

    diff -qr source-folder/ target-folder/

    you can check if there are different files on source and target.


    To check if the files were copied correctly you can run rsync with the -c option

    This can take a long time.

    Quote

    -c, --checksum
    This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a lqquick checkrq that (by default) checks if each file's size and time of last modification match between the sender and receiver. This option changes this to compare a 128-bit checksum for each file that has a matching size. Generating the checksums means that both sides will expend a lot of disk I/O reading all the data in the files in the transfer (and this is prior to any reading that will be done to transfer changed files), so this can slow things down significantly.


    Rsync is around for a loooooong time (since 1996) and very widely used. So I would trust it.

  • If you run a second rsync operation and there is no files list or transferred data, the copy is complete. In that case, if no errors are reported and there's no changed files list, the drives are mirrored.

    I ran a second rsync operation, expecting it to do a quick file compare and be done. However, it seemed to be copying all the files from HDD1 to HDD2 all over again ...


    In fact, when I aborted rsynch and checked the used space on HDD2, it was about 180GB more than the used space on HDD1. So, it looked like it was copying all the files across for a second time.


    Where do I find the "files list" that you refer to and where would the errors (if there were any) be reported?


    I know that rsynch has been around for a long time and I should trust it, but I don't trust my limited Linux knowledge and I hate flying blind. It would be good to see a log file of what files were copied across, and what files were deleted, so I know that I have it working correctly.

  • Log files are in /var/log/

    e.g. /var/log/rsync


    You can also see the content of the log file in the GUI of OMV. What is being logged depends on the log level set. With -v as used in the command in your first post the verbosity is increased.


    Did you run exactly the same command? A type or a missing or added / can make a big difference.

  • Thanks for your help macom. You were correct, it was a typo - somehow (probably in the copy/paste process) I ended up with a tiny minus sign at the end of the command line. I have no idea what the minus sign means, but removing that character solved the problem.


    However, I cannot find a rsync log file anywhere in /var/log/ .... :(


    Also, I cannot find a log file for rsync in the GUI of OMV. There is a real-time listing of the activity in OMV while I am logged in, but I cannot find anything after the event. Given that I am running this process at 1 am, I would like to be able to check the next morning that the process completed successfully by checking a log file?

  • In the GUI you find log files in


    Diagnostics >> System Logs >> Log


    use the drop down field at the top to select the log file you want to see.

    What is the output of

    ls -al /var/log/rsync*


    on my system it looks like this:

    Code
    -rw-r----- 1 root adm 0 Nov 13 00:00 /var/log/rsync.log
    -rw-r----- 1 root adm 1019 Nov 8 20:57 /var/log/rsync.log.1.gz
    -rw-r----- 1 root adm 266 Nov 1 20:57 /var/log/rsync.log.2.gz
    -rw-r----- 1 root adm 213 Oct 25 20:57 /var/log/rsync.log.3.gz
    -rw-r----- 1 root adm 216 Oct 18 20:57 /var/log/rsync.log.4.gz

    rsync.log is currently empty (0 byte). The log has been rotated to the compressed log files with .gz at the end

  • When I go to Diagnostics >> System Logs >> Log>> Rsync - jobs in the GUI, this is what I get:




    And the output of


    ls -al /var/log/rsync*


    is this:


    pi@openmediavault:~ $ ls -al /var/log/rsync*

    ls: cannot access '/var/log/rsync*': No such file or directory


    :(

  • Should I add --log-file=/var/log/rsync.log to the end of that command line to get a log produced?

    Correction, I guess that the command line should be this:

    rsync -av --delete --log-file=/var/log/rsync.log /srv/dev-disk-by-label-HDD1/ /srv/dev-disk-by-label-HDD2/


    Is that correct?

  • Yes, I have just confirmed that is the correct command line and it works now ... :D


    Thank you very much for your help macom ...


    Here's the result I was looking for in OMV:


    Given that there is an option in OMV to view the rsync logs, I am surprised that there is no mention in the document "Getting_Started-OMV5" that it is necessary to include --log-file=/var/log/rsync.log in the command line to produce the log files ... :huh:

  • Given that there is an option in OMV to view the rsync logs, I am surprised that there is no mention in the document "Getting_Started-OMV5" that it is necessary to include --log-file=/var/log/rsync.log in the command line to produce the log files ... :huh:

    crashtest might be worth a consideration to add this hint to the guide (though I would prefer to use the rsync capability from the GUI of OMV even if shared folders have to be created for this)

  • I would prefer to use the rsync capability from the GUI of OMV even if shared folders have to be created for this

    I am not sure what you mean by that?


    I have only ever been using the GUI of OMV and the "command line" that I refer to is the one in the Scheduled Job window:


    Why would shared folders have to be created for this?

  • crashtest might be worth a consideration to add this hint to the guide

    The idea was to help users create disk-to-disk backup, that's "dirt simple to restore", that could be setup and recovered from inside the GUI. (No command line required.) Of the thousands that have read the user guide, this is the first time I've heard a user doubt the outcome of an rsync operation and seek confirmation that it actually took place.


    In the interests of trying to keep the guide simple and a reasonable length, I'm a bit leery about adding more detail. The User Guide is a start for learning OMV and, to a much lesser degree, Linux. It's not all inclusive and it's already over 80 pages long. There's much to be learned from tutorials and manpages on rsync.


    I am not sure what you mean by that?


    I have only ever been using the GUI of OMV and the "command line" that I refer to is the one in the Scheduled Job window:

    macom is referring to rsync as it's found under Services, rsync. This implementation of rsync synchronizes two shared folders. It's useful but there are limitations, particularly when trying to describe how to mirror an entire data disk in a single operation.
    ((There are ways to do it, using Services - rsync, like sharing a root folder of a data drive with all SMB network shares added as sub-folders, but these are very specific setups that are not new user friendly.))
    _______________________________________________

    As an example of rsync under Services, in the GUI:

    This is a old backup server that I setup a long time ago and didn't change. It uses the remote mount plugin, to make a remote share appear to be local, and rsync to "pull" sync remote shares. This is a relatively simple method of replicating server to server, one SMB share at a time, thereby creating a fully functional backup server. It's something I've thought about adding to the User Guide.

    Perhaps it should be written as an addendum.


  • ((There are ways to do it, using Services - rsync, like sharing a root folder of a data drive with all SMB network shares added as sub-folders, but these are very specific setups that are not new user friendly.))

    It is possible to create a shared folder that is pointing at the complete drive, e.g.

  • It is possible to create a shared folder that is pointing at the complete drive, e.g.

    The other option is, just create a path that is the top directory, and everything I want backed up by rsync, goes in there...


    Code
    ken@openmediavault:~$ cd /srv/dev-disk-by-label-D1/Disk_1/
    ken@openmediavault:/srv/dev-disk-by-label-D1/Disk_1$ ls
    Data Media
    ken@openmediavault:/srv/dev-disk-by-label-D1/Disk_1$ cd /srv/dev-disk-by-id-ata-ST4000DM004-2CV104_ZFN1R4FX-part1/Disk_2/
    ken@openmediavault:/srv/dev-disk-by-id-ata-ST4000DM004-2CV104_ZFN1R4FX-part1/Disk_2$ ls
    Data Media
    ken@openmediavault:/srv/dev-disk-by-id-ata-ST4000DM004-2CV104_ZFN1R4FX-part1/Disk_2$

    in the example above, Disk_1, and Disk_2, are top directories. Only two directories I don't put under there, is AppData and Containers (Home is under Media). What i've found is if I add Containers and or AppData to the job, if a container is logging, it will constantly loop and cause a bit of a problem (this was a long time ago w/ omv 3.. .so I just stuck with it). My Disk_1 to Disk_2 sync is automatic at 0900 every morning. AppData and Containers, I have manual jobs set up.. usually I log in, stop docker, run those two manually, then start docker again when it's done


    Since my actual docker configs and containers do not change all that much, this works pretty well

    Air Conditioners are a lot like PC's... They work great until you open Windows.


  • For the docker stuff you could use rsnapshot-plugin for automation. The plugin allows you to run a command before and after the rsnapshot job runs. So sequence would be

    1. stop docker
    2. run rsnapshot
    3. start docker

    I considered it... but honestly I didn't see any reason to run something else when rsync seemed to work just as well. I usually only sync my containers maybe once a month... it's just not necessary for me to do it more frequently than that. Other people's work flow may require it more frequently, but how I've got it works fine.

    Air Conditioners are a lot like PC's... They work great until you open Windows.


Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!