How best to utilise my drives for storage

1st Official Post

    Hi,


    I have been using OMV for about a year but am currently trying to simplify my setup.


    My drives are configured accordingly:

    • 1 x SD 128gb card is used for the OS
    • 1 x SSD 240gb for docker with an appdata, compose and docker folder. I also have an OS backup folder which stores a fsarchive of the OS SD card
    • 1 x HDD 5tb for general media and ntfs storage. My laptop backs up to this and some docker container data is here if media related and high in storage. For example tvheadend recordings and photoprism pictures.
    • 1 x HDD 5tb which is a borgbackup of the SSD and HDD


    My concerns are:

    • My HDD used for backup is only 5tb, I may run out of storage in due course
    • It is difficult to test if the backups are working - I occasionally mount the 5TB borgbackup archive and check I recognise my data
    • The OS backup is difficult to quickly test the FS archive
    • My data is spread across various drives with only one backup
    • I don't have a offsite backup - to add in due course


    I have read the forums and my thoughts are:

    • Using mergefs combine my docker drive and general media drive and then snapraid to my backup HDD - I understand that mergefs doesn't work well with docker and so I don't think this is the best solution. I also understand that in the event of a drive failure I could lose data even with the parity data on my backup HDD?
    • Keeping my existing drives but using rsync to keep an exact copy of all the data on my backup drive. Then I could easily swap the drives over in case of a failure. This would mean I would need a much larger backup drive as I understand rsync doesn't de-duplicate like borgbackup does. In case of a failure I could then buy a new drive and copy the data only for the relevant drive from my backup drive.
    • Keeping my existing setup but just buying a larger more expensive drive when it's full and selling the old one
    • I like the idea of gradually adding storage with mergefs when needed but am concerned about losing data if a drive breaks.
    • How do I test the backups for the option you recommend I use above?


    Would someone please be able to advise the best approach for my needs?


    Thank you

  • chente

    Approved the thread.
    • Official Post

    The first thing I would do is format that NTFS drive to a native linux file system. It is not recommended to use an NTFS file system continuously in OMV, it is only compatible to be able to temporarily connect and copy large amounts of data, but little else. I guess it hasn't given you problems yet since you don't mention it, but it's easy for it to start giving them in the near future.


    Regarding the rest, what I would do is use the two disks that you have in a single volume through mergerfs, this would expand the available storage. And for backups I would buy a larger USB type drive that allows you to manually plug it in for backups. This will provide you with an offline backup, highly recommended for several reasons.

    Regarding the sensitive information that you want to back up regularly, I suppose you have data on the NAS, as well as audiovisual media, I would do incremental automatic local copies only of the documents (I suppose it will take up much less space than the rest) so that when you connect the external backup disk it will include too.

    Thank you very much for your detailed reply.


    I realised I made a mistake it's not NTFS but EXT4 but I use a network share for my windows PC to backup file history. I mistakenly called this NTFS.


    Just to clarify your other point, so would I merge my two HDDS (one currently used for media and one for backup) and keep my docker SSD separate? I would then buy a larger external harddrive to backup the mergedfs drive as an incremental backup?

    • Official Post
    Quote from TobyB

    Just to clarify your other point, so would I merge my two HDDS (one currently used for media and one for backup) and keep my docker SSD separate? I would then buy a larger external harddrive to backup the mergedfs drive as an incremental backup?

    Yeah. That's what I would do. That would solve future capacity expansions. In addition, the backup would not be online on the server, therefore it would be safe from certain risks that online backups have.

    The only downside to this is that the copy is done manually, so there won't be frequent updates. That's why I say that I would also make local backups of the documents.

    This approach allows you to automate local incremental backups of the documents folder. This folder is supposed to contain documents that take up much less space than the folder where you store your movies, for example.

    When you manually backup everything you can also include the files from that document backup. So in that external backup you have a complete copy of all your data.

    Personally I do the same, but instead of a USB disk I use a second server that is usually turned off. I turn it on manually just for backups.

    Thank you that seems great, I think I will go with that solution.


    So my drives will be like the below:

    SD card = OS system

    SDD = docker data

    HDD + HDD = merged HDD used for media and document files

    External HDD - used for offline backups when needed


    Sorry for all the questions. I'm getting there but still have a few more.

    • Where should I incrementally local backup to? Because once I merge my two HDDs, I will lose the existing disk I was using for backups. Or will I need to buy another disk for that as well?
    • Some of my docker containers use databases which I would like to backup too. I suppose I could use a script on my SSD to backup these and then I can just store these where I would store my other local backups ready for my offline drive?
    • Do you use rsync for your offline drive when it's turned on or something else?
    • Official Post
    Quote from TobyB

    Where should I incrementally local backup to? Because once I merge my two HDDs, I will lose the existing disk I was using for backups. Or will I need to buy another disk for that as well?

    You can do it wherever you want, just create a new folder for it. If you have enough space on the SSD it may be a good idea to do it there. Otherwise you can do it on any of the other two disks (or on the mergerfs volume, less advisable because you will have it distributed).

    Quote from TobyB

    Some of my docker containers use databases which I would like to backup too. I suppose I could use a script on my SSD to backup these and then I can just store these where I would store my other local backups ready for my offline drive?

    The same as before. In that folder you would create two subfolders, one for document backups and one for docker backups.

    Quote from TobyB

    Do you use rsync for your offline drive when it's turned on or something else?

    Yes, I use rsync. But since you are going to use a USB disk you can have a look at the openmediavault-usbbackup plugin, I have never used it but you might be interested. The main functionality is that when you connect the disk, the backup is updated and then the disk is automatically unmounted.


    _________________________________________________________________________________


    Since you have the operating system on an SD card I assume that we are talking about a Raspberry. Make sure the drives have enough power by using external power supplies. Do not connect all the drives directly to the Raspberry.

    Thank you that makes sense.


    So the local backup is more if I need to retrieve a previous version of a file rather than protecting if a drive breaks which is where the external offline backup comes in to protect from failure.

    • Official Post
    Quote from TobyB

    So the local backup is more if I need to retrieve a previous version of a file rather than protecting if a drive breaks which is where the external offline backup comes in to protect from failure.

    Exact. They are different protections. In this case it would be better to have the online backup of documents on the SSD and have the documents on the mechanical disks. In this way you will be more protected also if one of the two mechanical discs breaks. It all depends on the available space, how often you make external backups, etc.

    You can still go further. You can set up a second server at a family member's or friend's house that does automatic online backups. I also do that. :) That protects you if someone breaks into your house to steal or if there is a fire or flood, etc.

    You can take this as far as you feel comfortable.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login