Rsync with NTFS

1st Official Post

    Hello

    I have have been using Rsync with ext4 formatted drives in a Pi4 with OMC and love it.

    Although, I have a problem that is puzzling me how to address


    My mp3 and music files are stored in a 5TB external HD formatted in NTFS.

    I use to have a backup of all folders in different drives. The backup ended eliminated for a different reasons, and I now want to recreate it. The 5TB is about 85% used.

    I Would like to do it in OMV and use rsync to do it, but I would like to keep the possibility of :


    1) Keep the "original " external HD in the NTFS format (it is used in multiple PCs, Macbook, etc and I travel with it)

    2) Map it at my OMV server and shared it without needing to format to ext4

    3) Add another new HD of same size to be the backup destination drive (I am fine if that second drive will be in ext4 or NTFS)

    4) Any file added on the 'Original' drive (NTFS) should mirror on the new "backup drive" using Rsync.


    Final curveball.

    Life brings challenges (in particular on COVID19 world) I have been working remotely during the week, and my home it is a few hours away.

    My OMV server is on a Raspberry Pi4 and it is set in the apartment I am using during the week while working.

    On weekends, I turn everything off and commute to my home to spend the weekend with family.


    I would like to bring the original NTFS external drive with me, and in the eventuality I would add any files on it over WE, I would like when reconnecting to my OMV server, and turning everything up, to HOPEFULLY sync (AKA backup) any new file added to my new ext4 backup with rsync.


    I have been falling in love with Linux and experimenting everything I can with it, but would need some guidance.

    I don't know if what I am asking above is possible.

    Hopefully it is, and I really need some tips and advices on how to do it.

    Any suggestion to this "road warrior lone-wolf" would be greatly appreciated.

    Best,

    Brimo60

    • Official Post

    You need to rethink this.


    Use a EXT4 drive as the original connected to the NAS. Let it stay connected. Use a NTFS drive for backups and portable use. Use rsync to keep the folders on the portable NTFS drive synced with the original. Possibly use two NAS. One remote and one at home. Use the NTFS drive to keep them synced. Check out rdiff-backup.


    Or simply get Spotify...

    Yeah Spotify is cool, and I do use it too.

    Just an old guard ...and still have my beloved old files...

    Let me see if I understood your suggestion.


    Copy all folder/files into the ext4 at OMV (call it "original').

    Use rsync to "sync" from this ext4 HD into the NTFS portable drive (calling it in now NTFS backup)


    Here is what is not clear to me:

    1) Are both HD mapped in my OMV NAS as type local drives?

    2) Or your suggesting is to rsync from ONE ext4 HD mapped in OMV, but using Rsync type "Remote" into the NTFS HD?

    In that case I believe the NTFS drive should be connected to a local PC or Mac in my network, correct?

    Thank you!

    • Official Post

    The NTFS backup drive can either be connected to the NAS or to a client in the network. Then use rsync (or some other suitable sync software) to sync the files in the shared folder on the NAS to the backup folder on the NTFS backup drive. Possibly using a script.


    If you connect to the NAS then you need to mount the NTFS filesystem. But you don't need to share it. So NTFS will work fine. Sync locally to backup. Possibly by logging in to the NAS using SSH. Or automatically at a pre-determined time if the backup NTFS filesystem is connected.


    Then use the backup NTFS portable drive on-the-go. And take it with you.


    You decide how to sync. To/from or bidirectional. You may have to run rsync twice for bidirectional. Or use some other backup/sync tool. I do something similar with ebooks to a tablet. I use the android app FolderSync to sync from the NAS.

    Well I can only suggest to make some test before. Rsync ext with NTFS filesystem is not so recommended. I am doing some test too from ext4 to exfat external USB disk and when you think all is good you find that for example all timestamps have been changed and the next rsync process will think that files have been changed and all files are resynced even if they have not changed.

    • Official Post

    Do some experimenting. Yes, mixing NTFS and ext4 is bad for you and your data. But it is commonly done. You may need to use more relaxed sync settings to accommodate differences between EXT4 and NTFS.


    I would never use NTFS for my backups. But I have done it in the distant past. Worked fine. Mostly...


    Once you have several copies of the files, it becomes important how and where you introduce changes and additions to the files. You need to consider that as well. Partially that can be handled with bidirectional syncs, but bigger changes will quickly mess things up. And you will have to spend time consolidating your data.


    Also you need to consider how fast your data is changing. There is little point in making new copies of old files you already have backed up once (in multiple copies at several locations).

    Thanks Adoby


    The drive is organized to hold my mp3, ACC and FLAC files. It grew a lot with time as should be predictable.

    Originally some folders were files that where located in smaller drives (like 1T or 2T).

    Once the drive was getting full a new HD was added and new folder created.

    Then, I started to create folders to file only for international singers, compilations, etc. - some had different name strings, and it does made easier for using "super tagging" softwares. MP3tag is my preferred one (but Windows only so far)


    The amount of files created problem . Find a software that will manage the metadata database.

    E.g. Itunes cannot handle that amount by far. It seems the design limit it to about 40-45K tracks. I am over that many years ago.


    Have to start to use a SQL or MySQL db on backend to store the metadata. In a PC, Helium Music Manager is and was the best experience

    Then IOS come to my life as well as MacBook. Whole new game of considerations.


    On Mac, Swinsian is pretty much the only SW that works great for that amount of songs.

    Works really well. On Linux I did not experiment too much (yet).


    Updates of metadata - adding or deleting tracks - regardless how good those SW are, it is also time consuming,

    To a point where I started to save new tracks into a "New Music" folder and set the SW to 'watch' couple folders only, and then doing overall update (all folders) once a month overnight.


    I think i see your point about "where you introduce changes and additional files".

    I believe once the "full copy" is created, I will select 2-3 folder (where new files are added) to apply rsync. That based on you advice - by the way , thank you for that.


    Hoping not to turn into a "never ending pests of one more question" , Could you explain better what do you mean for "use more relaxed sync settings to accommodate differences between EXT4 and NTFS"?

    I am sure that can help my testing.


    Thanks again!

    Brimo60

    • Official Post

    EXT4 and NTFS may handle characters differently. UTF8 or UTF16. Also may have different case sensitivity.


    So it may be a good idea to, for instance, use only 7-bit ascii characters in filenames.


    EXT4 and NTFS also have different time resolution. But rsync has some options to handle inexactly matching times.


    It was a while since I mixed NTFS and EXT4 like this, so there may be other issues.


    It is usually faster to connect and copy/sync directly from the NAS. But then you need to be aware of differences and possible problems when copying/syncing between EXT4 and NTFS. A safer and easier way might be to copy/sync from a Windows client instead. Then SMB/CIFS handles a lot of the problems for you. And if most of the files are unchanged, then very little need to be synced so speed is hardly an issue.


    Tip: I use MusicBrainz Picard to manage my music collection. And match with releases, when possible. And simply store everything in a simple folder structure on my NAS. Separate folder trees for Classical/Artists/Various Artists/No Release.


    Tip: To "archive" static data, like completed music albums matching some release and with full perfect metadata, you can simply zip the album and store it at a (preferably more than one) safe location. And skip it in future backups. You can at any time check the integrity of the zip files and verify that they are unchanged. You may even want to check out the par2 package. Using par2 you can add error correction records to your archived files. Either individual zip files or groups of zip files. So if a backup file goes bad and you don't have a good copy elsewhere, you may be able to repair the file(s) using the error correction records. Commercial version of RAR (WinRAR) has similar functions.


    Tip: Most Linux music player can easily handle small (;)) music libraries like yours. I like the old clunky Guaydeque Music Player. It can handle HUGE libraries gracefully. But newer Linux players are prettier. And perhaps (?) just as good at handling large music libraries? Typically Linux music players use SQLite.

    Those are not tip or advices but real treats!

    Thank you for your time and willingness to share your knowledge and experiences with others..

    Your Zip idea is really good and will definitely consider.

    As you can imagine, my collection started many years ago (Naspter times to be precisely). I am from old good "Vinyl" times and music aficionado up my last day.

    Would I start things today it will very unlikely go thru the pains of mp3 tracks.

    As you well said, with Spotify, Apple Music, Amazon and the relatively low cost associated and convenience of accessing your prefered songs, it a no brainer not having this kind of things to worry about.
    Quality changed, options changed and much more.

    Thank you for the tips regarding Linux.

    Once I have more to say I will share my findings!:)

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login