Rsync frustration of doing the simplest of things

  • Ok, so it is all fine and working as expected. The other command df -hT had an additional space. I wrote this on an android device, where the keyboard adds spaces all the time...
    It is much simplier than anything else. You just ignored the case sensitive part. It is names Dir1sum, not dir1sum. Filesize is also ok now, propably just an issue of your viewing gui.
    As you ignored my requests in editing the file and altering a checksum, I just tested it myself using this:


    This is the resulting output:

    Code
    test without altering files
    testfile2 removed
    testfile2 different
    177e997d0a45599c6e25af9f6c12a874  /root/testDestination/testfile2

    As you can see, md5deep is working as I suggested earlier on. It has output if checksums are different, but not, if the file does not exist at all.
    Judging from this, your test scenario is not working. If you copied the files, you can check if they are correct with md5deep. To check, if the file structre is the same you need to do something in addtion.
    This here would work:
    diff -r sourceDir destDir
    It will display all differences in file structure (names only, not looking at data inside the files).

  • Ok, so it is all fine and working as expected. The other command df -hT had an additional space.

    and last one

  • Ok, so it is all fine and working as expected. The other command df -hT had an additional space.

    and last one


    Judging from this, your test scenario is not working. If you copied the files, you can check if they are correct with md5deep. To check, if the file structre is the same you need to do something in addtion.
    This here would work:
    diff -r sourceDir destDir
    It will display all differences in file structure (names only, not looking at data inside the files).

    Thank you very muchj for your time but in my senario nothing helps and does all the comparisons one pass. Each command checks something something else leaves it outside of comparison.
    Not for me

  • Its all working perfectly fine! You just misunderstood something there. You created the file Dir1sums in your /root/ which contains the checksums of your source folder. You compared them to your destination folders and did not get any errors. Now do the same thing with diff -r, if this does not prompt anything, your copy is correct.
    One time you check for file structre, the other time for checksums in identicaly named and placed files.
    You were not able to find dir1sums in your filesystem just because you named it Dir1sums instead. Nothing going wrong, often refered to as an layer 8 problem, but you did not break anything, so its fine.

  • You created the file Dir1sums in your /root/ which contains the checksums of your source folder. You compared them to your destination folders and did not get any errors. Now do the same thing with diff -r, if this does not prompt anything, your copy is correct.

    I didnt put it in there the command had that structured and I just followed If you want type both of the command exactly as it should be to save it somewhere for future use.



    You were not able to find dir1sums in your filesystem just because you named it Dir1sums instead.

    I typed it as Dir1sums also just forgot to mention it.... so problem was not there also

  • The structure is the one, that was already in the old filesystem. I am also pretty sure, if the Dir1sums file would not have been there, that md5deep would have thrown an exception.
    You should restart the whole thing from the beginning.
    1. Use rsync (dont forget -a) and copy the old data onto the new disk.
    2. Use diff -r oldDisk newDisk to check if the files were all copied
    3. Use md5deep to check if all files that got copied were copied correctly
    And thats about it.

  • Ok got it but there is no way to delete 1,2tb and copy everything again.. Its just dont worth the effort and the extra wear off the hdd. Thank you though

  • Hello again


    A tiny bit of knowledge each time. For a few weeks rsync worked probably as I wanted but truth is didnt since yesterday I came across a situation that didnt worked as expected.


    So the issue is.... i have a source folder in the nas and a destination one in an external usb device.
    Using rsync -avuc --progress --stats /mnt/md0/muz/ /mnt/usb/usbshare1/muz copies all contents from source to destination creates all dirs and subdirs as I want. But today I made 3 changes to the source which are
    -deleted a file
    -renamed another one
    -added another one in a different location
    From the above actions only the third worked as expected which is logical.


    The file I renamed in the source had as a result in destination to create a new folder with the renamed name and keep the original one which I didnt want to (since i want source and destination to be identical)


    Last the file i deleted in source kept in destination.


    Bottom line is what should I add in the rsync line to behave the way I want?


    Thank you

    • Offizieller Beitrag

    You should add --delete to the command line.


    Then new files will be added to destination and deleted files will be deleted from the destination.


    However if you use rsync as a backup tool then that is bad. If you by mistake delete a file or a folder, then run rsync, then you loose the file or folder.


    That is why you may not want to use --delete.


    I use the snapshot and hardlink feature of rsync from a script. That way I can have many versioned/timestamped snapshots that looks just like the source did when the snapshot was created, but files that are unchanged are not copied but instead hardlinked from a previous snapshot. And the script even purges old snapshots so that you keep a certain number of monthly, weekly and daily snapshots.


    The snapshots are not perfect, btrfs provides better atomic snapshots. But i take my snapshots at night when there is no activity on the network, so they are pretty OK.


    I have posted my script in this thread: Question: Can rsync be used without using the GUI?


    The plugin rsnapshot does a very similar job. But works on whole shared folders.

  • You should add --delete to the command line.

    I suppose it doesnt matter if its at the end rsync -avuc --progress --stats --delete or somewhere in the middle rsync -avuc --delete --progress --stats



    That is why you may not want to use --delete.

    I already know the danger of using this switch but I am very careful when handling my music database (being there done that for 8 years now and manually in windows and a quadrant file explorer) but i need them to be identical



    The snapshots are not perfect, btrfs provides better atomic snapshots. But i take my snapshots at night when there is no activity on the network, so they are pretty OK.

    I am currently trying out a terramaster nas device with a TOS OS and the hdd's by default are formatted in btrfs. Used the command but gived an opportunity to save snapshots inside the array only and during import it errors a message and afterwards I cant see my sharess for several restarts of the nas.


    Anyway I ll try the delete


    Always there to help!! Thank you

  • Maybe you should consider to keep at least a few older versions. Something like full backup each month, than only diffs and remove everything older than current and last month.
    Dataloss due to mistakes do and will happen at some point.

  • Or the following scenario: your fs fails, you recover but loose some inodes. Just before you recover your rsync starts working and deletes your backup. You should really be using more than just a plain copy.

  • Or the following scenario: your fs fails, you recover but loose some inodes. Just before you recover your rsync starts working and deletes your backup. You should really be using more than just a plain copy.

    Well I have an external backup having older state of data (1-2 weeks) and rsync both internally and externally are manually initialized so ... I am doing the best I can...thank you though

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!