Backing up USB Installation

I think a few of the methods you're looking for is in this guide under OS and Data backup. I clone my USB boot drives, per the guide, in an offline process. It's simple and works well. There's also an OMV plugin for backing up the boot drive.

If you're looking for backup, I wouldn't use a mirror. Rsync is better than a mirror.

BTW: Since you're using a flash drive to boot, if you haven't done it already, you really need to install the flashmemory plugin. (Otherwise, the life of your thumbdrive will short.) This is in the guide as well.
____________________________________________________________________

Other forum members use other methods. Perhaps they'll chime in as well.

RAID may help with availability of data. For some types of drive errors the data continues to be available. But RAID provides no protection if you delete a file by mistake or if a program error cause file corruption. To protect against that and any other drive errors you need backups. Without backups you will loose data, sooner or later. RAID is no guarantee against loss of data. By using more drives in RAID you ensure that you will experience more drive problems, not less.

Rsync is a good tool to update a backup copy of a data drive. Rsync can also create a set of versioned snapshots that shares unchanged files. That means that you can have for instance 7 daily, 4 weekly and 4 monthly full backups but they take up only a little more space than one complete backup.

I have now made my script for versioned rsync snapshots with purge available online. Rsnapshot is similar.

https://github.com/WikiBox/snapshot.sh

Quote from shockwave

I already setup my drives as a mirror (raid 1). Is it possible to change it to Rsync without losing all the data? What makes it better?

I'm going to assume that you're using mdadm RAID1. As for splitting a RAID1 mirror, I'll ask @geaves to chime in.

What makes Rsync better? RAID1 will replicate the effects of data loss or corruption, to the second drive, instantly. (As @Adoby would put it, this is Bad^TM ) A RAID mirror is not backup.
Backup is having two separate copies of your data. Rsync can provide backup. You'll have at least two separate copies of data, or more, separated in time. This means you'll have a fall back if something goes wrong and, eventually, something will go wrong. Murphy's Law.
___________________________________________________________

As far as a recommendation on how to proceed, that would depend on your skill level. There are simple ways to implement backup that can be done in the GUI and there are more feature rich ways to do it, but restoration is more complex. It all depends on what you're comfortable with but it's important to note that backup is worthless, if you can't restore it.

I don’t want this to sideline @shockwave’s question. Hopefully it will wind up helping us both.

@crashtest In your Guide at about page 67 you show how to repoint shares to a rsync’ed drive. It works. I’ve done it. Love it.

Two questions:

• How often should you backup?
• You warn to turn off automatic Rsync in the event of a failing or failed data drive. If you set it up for a daily backup at 2a.m., for instance, and your data drive dies one night at 1a.m., does the scheduled Rsync run and if so what does it do to the backup drive?

Is it possible to use the rsnapshot plugin to do a similar thing. As I understand the rsnapshot plugin you have to create a second set of shares on the backup drive corresponding to those shares on the data drive. Is rsnapshot more for recovering that lost file (or shared folder) that went missing (that My fat fingers deleted)?

Is a two-drive setup backup - one for Rsync and one for rsnapshot - the way to cover the bases?

I also have used the guide by @WastlJ on a Rsync of disks on two different machines that works great. Is there a similar way to use rsnapshot remotely?

I guess that is more than two questions.

@Adoby you chime in on this too. I’ve looked at your script. I would love to figure out how to use it. Rsnapshot is my next challenge.

The only real challenge with my script is to figure out the source and destination paths from the point of view from the server that runs the script. I have only used the script when both source and destination are already mounted. Then you just have a cron job to run the script.

I like to run rsync jobs on the destination server. Pull. That way you avoid having rsync writing to a unmounted mount point. If run from the source server that could happen. Push.

Quote from Agricola

Two questions:

1. How often should you backup?

2. You warn to turn off automatic Rsync in the event of a failing or failed data drive. If you set it up for a daily backup at 2a.m., for instance, and your data drive dies one night at 1a.m., does the scheduled Rsync run and if so what does it do to the backup drive?

1. This depends on the user, what they're doing, etc. There's no right answer for everyone.
If I was using Rsync exclusively, I would backup no more than once a week, probably on Sunday or Monday. The thinking is, on Friday or Saturday night when we usually watch a show, I'd be able to spot something going wrong before data is replicated on a schedule.
(If I had a single backup, I would give serious thought to running the job manually.)
In my case, including one 4TB USB drive that's dedicated as a traveling media store, I actually have 3 physical backups and I'm using ZFS on the primary and one backup server.

2. The reason for turning off automated Rsync is as you suggested:
If automated, when something goes wrong with the primary drive, it's possible that part or all of the problem may be replicated to an otherwise clean drive. This is similar to what happens, instantaneously, in a RAID1 mirror.

There are a number of things that can be done to address this problem.
- Setting up SMART stat's, testing, and file system notification E-mails. (An ounce of prevention.)
- Before manually running a backup, take a quick look at shares then backup.
(Or)
- If automated, use a longer backup interval that allows more time to find or notice a problem. A two week interval, or even more, is not unreasonable. I'd rather lose 2 weeks worth of data, versus running a greater chance of corrupting a clean backup.
____________________________________________

There is something to be said for more sophisticated approaches like the Rsnapshot plugin and Adoby's Rsync script, for multiple versions of backup using hard links, and there's no space penalty for backing up everyday. It's a far better approach. But restoring from that kind of backup involves getting on the command line which is difficult for some users. AFIK, it can't be done from the GUI.

Quote from crashtest

But restoring from that kind of backup involves getting on the command line which is difficult for some users. AFIK, it can't be done from the GUI.

@crashtest So Rsnapshot is best for restoring media accidentally deleted. Since it looks doable at my skill level I may still set up an extra drive on my main server just for Rsnapshot of my Media folder. I edit and transcode my Plex files manually and that is where I would most likely accidentally delete the wrong file.

I have one backup on the main server, and two more hc2's with one drive each that also backup the main server ("pull" using Rsync). Right now I have them set up to backup twice a week each. According to your thinking I might just back them off to once a week each; say, Monday, Tuesday, Wednesday in the wee hours. Thanks for the insight.

Quote from Adoby

The only real challenge with my script is to figure out the source and destination paths from the point of view from the server that runs the script. I have only used the script when both source and destination are already mounted. Then you just have a cron job to run the script.

Easy to say I'm afraid, my friend. One day I will be deeply indebted to you when I put all the pieces together and figure it out, but for now I must leave it lay. I do appreciate your input and follow your comments even if I don't understand half of what you are writing.

Quote from Morlan

@Agricola Setting up Rsnapshot is nearly the same as setting up rsync. It requires some time to set it up initially but because it is basically running on rsync it's not much harder.

That's exactly what I was thinking. I just need an extra drive to try it out. I am in the process of moving my main server over from an Odroid HC2 to a NanoPi M4 so now I will have a use for one of the two unused sata connections.

Quote from Morlan

You can also trigger the backup manually if you changed something important in between the intervals.

It sounds like the best plan. I often do that on days when I have more than usual server activity.

Quote from Agricola

@crashtest So Rsnapshot is best for restoring media accidentally deleted. Since it looks doable at my skill level I may still set up an extra drive on my main server just for Rsnapshot of my Media folder.

Is it worth an extra disk solely for Rsnapshot "file version" protection? You'd have to make that decision, but it might be worth it if for nothing else but the extra level of protection - a 2nd backup. I believe in multiple backups.

On the other hand, an adequate explanation and comparison of the two, an Rsync disk-to-disk mirror versus what the Rsnapshot plugin does, takes in a lot of ground.
_______________________________________________

An Rsync'ed disk-to-disk mirror may contain accidentally deleted file(s) as well. When automated, the retention period would depend on the job interval . If you're concerned about delete protection, with automated or manual jobs, removal of the --delete switch will mean that the destination disk will retain files deleted from the source disk, indefinitely. Simply copy deleted files, from the destination disk, back to the source - done.
When using this option, you could add the --delete switch back in from time to time, for house keeping, to clear out old deleted/unwanted files. In the bottom line, with a bit of thought, it can be made to work and it's not complicated.

Rsnapshot is suitable for protecting all files and even previous versions of the same file, because it uses hard-links and scripts in a way that prevents same file duplication while preserving unique files from permanent deletion. (At least until a cleanup script purges older files by date.) This means that every version of a file, no matter how minor the change may be, is preserved in it's original state as of the time of backup. All unique versions are available for restoration, even after the latest version has been deleted from the source filesystem. (Again, this dependent on the retention period.) Rsnapshot is similar to automated ZFS snapshots. Most of my ZFS based network shares have snapshot based file version protection for up to a year. I believe Rsnapshot has similar retention periods.

On the other hand, restoration of Rsnapshot files is another matter altogether. It's not as simple to restore as is to setup, and it's not done in the GUI. Is it better? Yes. Is it complicated to restore? It can be. Numerous decision points may be involved in selecting "what" and "when".

(Unlike Rsync disk-to-disk backup) Rsnapshot deleted file restorations are not a simple copy operation from the destination disk, back to the source. A decision on which backup to pull the file from must be made. Getting the latest version of a deleted file depends on "when" the file was last backed up and that may require a bit of searching.
Further, with Rsync disk-to-disk, restoring shares and other drive data doesn't require a new drive, drive formatting, etc. It's a matter of repointing existing shares from the failed source drive to the backup drive. Restoration of shares is relatively simple and can be done in a matter of minutes, not hours. The Rsnapshot plugin can't do this with anywhere near the speed or simplicity. To replace a failed data drive, multiple operations are involved.

For these reasons, in the guide, the focus on Rsync disk-to-disk was to provide something that's effective, easy for beginners, that can be setup and restored from within the GUI, and includes functional restoration of network shares after a drive failure, in minutes not hours.
(With beginners in mind, keeping all of that in the GUI and off the command line, was a creative exercise. )

Easy is not always best, or vice-versa.

If you don't install the flash plugin correctly, then make sure you have a spare thumb drive with a recent clone of the system drive.

Why do you think you need RAID1 at all? By all means use it if you want to, but it might be a good idea to first figure out why you want to double the risk of a drive failure as well a double the drive cost.

If you need compressed and/or encrypted backups, then duplicati is great. But much more compute intensive and complex.

Quote from Adoby

If you don't install the flash plugin correctly, then make sure you have a spare thumb drive with a recent clone of the system drive.

@shockwave . Adoby has a point here. If the plugin is not installed correctly, flash media wear will increase. I don't know what happened in Technodad's case but setting up /etc/fstab for the flashmemory plugin will enhance it's performance. The plugin's displayed web page shows what is needed, and easy methods for doing these manual edits are in the guide as well. (I've never heard of what seemed to happen to Technodad, but a lot of users have done these edits without a problem. Otherwise, it wouldn't be recommended.)

Quote from shockwave

I have an extra 2tb external hard drive I can setup with r-sync. In that case, should I leave the current two 3tb drives in raid 1?

Provided that the entire data store doesn't exceed the capacity of the drive (2TB) you could do that. Just use the mount point for the RAID1 set as the source drive. But it's a shame to do it this way. mdadm RAID1 is kind of a waste of 1 of your 3TB disks. This is why I was hoping @geaves would chime in with a safe way to break the RAID1 set.

In any case, copying all of your data to the external drive would facilitate a transition out of the RAID1 set. Your data would be backed up in the event that breaking the set went south. (Note: The first sync is going to take a LOT of time - possibly several hours to a day or so. You'd want a complete and confirmed copy. Completion can be confirmed my manually running the job again and seeing no files transferred. At that point, the transfer is complete.)

Quote from crashtest

would chime in with a safe way to break the RAID1 set.

There isn't, none that I know of, you would need to rsync your files to the external, then break apart the array (this can be done in the gui) wipe both drives and start again. To begin, rsync back to one drive and set up your shares as the folder structure will be in place on the drive, then rsync to the second drive. Then set up a scheduled job for rsync from drive 1 to drive 2 in after hours, until it's all reset and running leave the usb alone at least you know you have a backup.

Quote from shockwave

Will there be any issue with my docker programs once I put the info back? Will I need to change anything?

This depends on where your dockers are stored. If they're in the default location, your boot drive, there's no effect.

If your dockers are stored in a shared folder on the RAID1 pair:
- After repointing the appropriate shared folders to the backup drive, I believe the location of your Docker containers will follow the shared folder, just as network shares follow their shared folders.
__________________________________________________________________________________

Quote from shockwave

I can use the following command mentioned on page 64 of the guide: rsync -av --delete /srv/dev-disk-by-label-DATA/ /srv/dev-disk-by-label-RSYNC/

The command line in the guide (and below) is just an example:
rsync -av --delete /srv/dev-disk-by-label-DATA/ /srv/dev-disk-by-label-RSYNC/

Since I don't know what your disk labels are:
Follow the guide to find out what your source and destination mount points are. Read it over carefully and follow the steps.

Note that your RAID1 array will your "source", it goes on the left side of the command line. The destination, the external disk, will be on the right side of the command line. For the first run, until you have two full confirmed copies, I'd recommend leaving out the --delete switch.

Just as an example, without the --delete switch in the line, your command line "might" look something like this:
rsync -av /srv/dev-disk-by-label-RAID1/ /srv/dev-disk-by-label-EXTDRV/
_________________________________________________________________

Quote from shockwave

Can this be done in the plugin or do I need to open a shell?

As the guide specifies, the command line is added to a Scheduled Job.
_________________________________________________________________

Note that my command line will be very different from yours (I'm running ZFS on the source drive), so the following is for an example only.

To determine my mount points, I hit the down arrow on one of the data column headers, selected "columns" in the drop down menu, and checked the Mount Point box.
My source is ZFS1 and my destination is DATA1. DATA1 is a USB external drive.
(Pic edited for clarity.)

__________________________________________________________________________

With the mount point information from the Filesystem screen, set up a Schedule Job with your mount point information in the command line.

Note that my job is not Enabled. I run it manually with the run button.

_____________________________________________________________________________________

When the job is running, files will scroll by similar to the following.

Depending on the speed of your drives and how much data is being transferred, this is going to take a long time. Hours to a day or more. It's best to walk away and let work.
________________________________________________________________________

Things may happen where the job may be interrupted. To be sure it's complete, run the job again. If files start transferring again, that's fine. When the following screen comes up, the job is done.

**But it wouldn't hurt to run the job at least one more time, just to be sure.**
__________________________________________________________________________

The -> guide was written in a "Walk Through" format. The info you need starts on page 63. Read it over and if you're not comfortable with what's there, don't try to set it up. On the other hand, after you read through the setup process, I'll try to answer any questions you may have.

(Remember, if you do try it, leave the --delete switch out of the command line in the first run.)