System -> Backup (RSYNC) documentation needed

Hi All,

Sadly I couldn't find a clear help on the backup available in the System section. My entire system with user data is on single 2TB SSD M.2. (OMV 6.9) I would like to set up once-a-week RSYNC backup onto USB SSD 2TB in such a way that I could easily restore it (sync new files, deleting destination files if they are gone at the source).

System->backup looks like a default simple option that includes RSYNC variant. But, there is no information on what will be copied! Moreover, I understand that if it is a system backup there should be a list of some 10 directories excluded for RSYNC (live server) backup... but there is no default suggestions for OMV. More over, there is no obivous System Restore menu.

Help!

Thank you, Ryecoaaron.

Could you please point me to a documentation page that explains the restore process for OS backup?

I assume:

1. Install fresh copy of OMV

2. Boot off of USB flash of any live Linux

3. RSYNC backed up files (from USB external source) over to the freshly installed system (with deletion of absent at the source, but present at the destination files)?

4. (later restore user data)

Could this work?

Zitat von ryecoaaron

The System -> Backup plugin (openmediavault-backup) is meant to backup the OS only. Since it is recommend to store data on a different disk (in case of failure), the plugin doesn't have options for what to do with data. You can look at the script to see what it excludes. It sounds like you should setup shared folders and use the rsync plugin to do what you want.

There is no restore menu because it isn't possible to do safely from a running system. There are many threads about his on the forum.

BTW, Will the dd version of System->Backup copy everything there is? (in my case the user data are together with the system) And how does dd work? Is it copying drive -> to a file?

Of course, since my destination USB backup drive is SSD, it doesn't make sense to do dd-backup once a week -- it will wear out the SSD faster than syncing with RSYNC of only changed files.

My idea of backup is to attempt a live-system&userdata backup with zero downtime. Moreover, if a crash happens, but the user data is needed immediately locally (even before full system restore) -- with RSYNC it's no problem (although I guess that a dd-image can be mounted and need data can be retrieved also, but a damaged dd image could mean loss of everything, while a dammage of several files on an SSD is much lower risk). Does this plan make sense?

Thank you.

Zitat von ryecoaaron

It makes a byte for byte image of the OS partition.

All of the options in the backup plugin are live and all work. If you worry about a damaged dd image file, then VMs would have a lot of problems (and they don't). I prefer borgbackup since it gives you multiple copies, incremental backups, compression and dedupe, and bitrot detection.

Yes

Thank you so much for a quick and to the point reply.

I prefer files to be saved naturally (without compression)... That is why I'm looking at RSYNC or (now maybe) dd. In a case of dd the restoration process (or even cloning) is a breeze, but the ammount of write-cycles is much higher than RSYNC. With Borg or any other compression -- it's taking extra CPU time and once multiple compressed files with assorted section of data are created, there is probably a lot of work to be done if you want to quickly pool out a 7 days older copy of a file ... unless I'm mistaken and Borg backup has a nice file browser to easily chose a backed up directory and rescue an individual file?

Thanks a lot again.

All the best,

Sasha

Zitat von ryecoaaron

The borgbackup plugin allows you to mount an archive. Then you could pull anything out of the archive just an rsync backup. The borgbackup option in the backup plugin does not have this and you would have to run the archive mount from the command line.

I didn't worry about rescuing one file from the OS backup because typically people need to restore the entire OS. Because you are mixing the OS and data, you will have to do a lot more command line work to recover one file using borgbackup. If you can't use more than one drive for OS and data, I would recommend partitioning the OS drive. Backup the OS partition with the backup plugin reducing the chances you will need to restore one file. Backup the data partition with a backup plugin like rsync, rsnapshot, or borgbackup.

Thanks.

I know that I could create a separate partition for the user data, but at the moment I had no clue how much to dedicate for OS and how much for user data. Besides that, I was not sure what kind of software I would use. Even now I'm not sure if I'd like to add a Kanboard server, a mail server... Each software would need to be configured to store user data separately and predict space required for OS with all of the apps.

But I thought: in Linux everything is a file, everything can be mounted into a folder ... Yes, having something on a separate physical drive is a good thing... but I'm using an Orange Pi 5. This little thing has only ONE fast port for system/user drive -- a single M.2 slot. Since I'm using Nextcloud and have a huge collection of photos to store, I know that I would want the user space to be fast. On the other hand, the OS drive must be fastest... So, it has to be a single slot -- single user and OS drive.

OK, then, I thought, should I partition it as a separate user data and OS? Well... then in a case of complete server reinstall (not a backup restore situation), I would have to remember how I configured user space, then reconfigure every single server software to point to the separate user partition. Too much work.

So, one single fast drive for all data, default software and data locations, and (thus) a very much needed reliable and fast backup to an external USB SSD with a possibility to restore it to a working server OS with the user data.

I know it's a mess, but I wasn't really sure how the separation of user data and OS works in OMV. It is quite simple in a regular Linux -- you have your home directory and that's all the user ever going to have. Here... I was not so sure. And, frankly, still figuring things out.

Thank you.

Zitat von ryecoaaron

16GB for the OS is fine IF you run your apps in docker (kanboard, mail server, nextcloud are all better to be in docker) and store everything docker in the data partition. I have an Orange Pi 5 Plus and have OMV on the SD card but everything data is on the nvme stick.

But functionality of OS depends on so many little files read/write... I know that Linux is a small OS and can probably fit most of it's libraries in RAM... but still, all of the logs and temporary files... I can hardly imagine an SD card survive for longer than a year. Yes, there is a cool option in OMV to reduce number of writes (which prolongs the SD card lifespan)... But I'm nervous about having my server OS on M.2 Samsung NVME (instead of old-good-reliable HDD -- I don't know how long it will survive the crazy number of thumbnails for my photo collection, all of the indexing and such... let along an SD card.

I already had a setup of bare metal and no docker Ubuntu Server + Nextcloud on a Raspberry Pi 4 8GB with OS on a SSD USB M.2 enclousure + USB HDD 2TB having all the user data... But the amount of time I dedicated tuning all of this up was enormous. And now, that it's stuck with some outdated library update and an impossiblity to upgrade to PHP8 ... I don't even want to try anymore.

Here, with OMV I did docker everything (although now relearning not to use a Portainer or a Yacht) -- it's great! I use AIO for Nextcloud and with shivering look at automatically configured dozens of services that I had to do by hand previously. Yes, it's great, but now I have to learn where all the user data is stored... and that is for each container. Even the whole Idea of config+container+storage... where is each thing stored? What do I backup?... It's a mess in my head now. Besides that OMV also has it's configs, plugins...

I've been doing virtual machines since 90s (say hi to Basilisk II), and understend that docker is a lite version of a virtual machine... but with all of the security and a wierd naming of the storage... Ah... I thought, let is store where it wants to store -- I'll back it all up in one shot :))

Yes, it's laziness, but it's also a lack of free time.

I hope to figure out a good backup scheme. Thank you for help.

Zitat von ryecoaaron

The System -> Backup plugin (openmediavault-backup) is meant to backup the OS only. Since it is recommend to store data on a different disk (in case of failure), the plugin doesn't have options for what to do with data. You can look at the script to see what it excludes. It sounds like you should setup shared folders and use the rsync plugin to do what you want.

There is no restore menu because it isn't possible to do safely from a running system. There are many threads about his on the forum.

OK. I think I understand it now. [after reading your previous reply -- probably not]

In my situation, with all the docker files stored at /srv path... I see that /srv path is excluded from System Backup. It is OK to edit the script and allow /srv path to be backed up (using RSYNC) so that with the system all of my user data will be backed up also?

Zitat von ryecoaaron

I have an OMV system running on a cheap usb stick for over five years now with the flashmemory plugin. I have a couple of RPis running for 3+ years on a SD card with the flashmemory plugin. I have killed SD cards in as little as two days without the flashmemory plugin.

I have a Samsung 960 Pro nvme stick that is 7 years old and I have written way more data than your photo collection will on it. It has a high churn of 15-30 VMs. It will outlive spinning rust. At work, I have systems with a raid 5 array of four SSDs that write 2TB per day and the drives are five years old. spinning drives are not more reliable than nvme. In fact, I have never had an nvme stick fail and I have run multiple OMV VMs on mine without the flash memory plugin for months.

But since you can't know when storage is going to fail, that is why you have backup.

That is a selling point for using docker.

You literally tell Docker where to store your data in the docker-compose. Anything else doesn't even need to be backed up. If you put all of your volumes on a data drive in a directory name dockerdata (example) and have each container put their data in the subdirectory with the name of the container, how can you not know where the data is? The compose plugin also handles the backup and restore of the data. You just give it a sharedfolder and it knows where to put things.

The config is in the database which gets backed up with the OS. The plugins use shared folders and put all of their data on the data drives.

Nope. There is no virtualization in docker. It just putting the container processes in their own little jail.

Alles anzeigen

Wow! Thank you!

Let me see if I get this now:

1. If Docker Compose plugin backsup all Docker data, and if this is my main user data (in fact all of my userdata) then this can be considered my user data backup

2. The best option is to have OS on it's own partition so that I could use dd-backup with super easy restore procedure (although in my current situation a System Backup via RSYNC is best since dd will do too much).

So, Compose backup -- for user data, and dd-backup for OS seams to be the most ideal solution for a backup.

If that is right, then the only question left: how often does Compose Backup happens and how does it work? Is it incremental or replacement, is it daily or weekly...?

BTW Yes, that's what I meant by VM Lite -- no need to emulate hardware, only control available resources (RAM, ports, storage). Though, AFAIK to the services within a Docker container -- they feel like their own installation of OS, but in reality it is a sandbox. (Right?)
Anyway, I was pleasantly surprised by how little resources a Docker container uses compared to having a VM. It looks like the whole idea of docker was inspired by Wine.

Thanks a lot. You really saved me a lot of time by correcting my missconceptions.

All the best.

Zitat von ryecoaaron

You set the schedule. It uses rsync.

Nope. Wine is meant to emulate Windows binaries. Docker is an interface to the Linux kernel c-groups.

Oh! That was so counterintuitive, that the choice of the backup destination is selected at the main Compose Settings page, but it isn't activated until the schedule page is used to do so. There must be a link on the main page taking a user to the schedule page...

Thanks a lot.

Last clarifying question, if I may: is it rsync with the delete option or not? And, if System Backup and Copmose backup both pointing to the same backup destination, will they create subdirectories?

PS: What I meant by Wine idea, is that Wine isn't a proper virtual machine -- only libraries are recreated and resources are managed. AFAIK different Docker containers can have different versions of the same libraries to run necesary software (right?). Both Wine and Docker have no virtual CPU/MB/RAM/HDD/NC/AUDIO/VIDEO emulated. It's just that it was impossible to play 3D games in VM, but possible in Wine because of no need of emulation of the hardware.

Zitat von ryecoaaron

It does have the delete option

https://github.com/OpenMediaVa…n/omv-compose-backup#L131

https://github.com/OpenMediaVa…n/omv-compose-backup#L194

I can't make plugins that make sense to everyone. All of the settings for the plugin are on that tab including for back. The Schedule tab is just literally scheduling the backup. It creates cron job(s) and I didn't feel there should be settings in the cron job. There is a wiki about the plugin and it is linked to in the Files tab. It has a section on the Schedule tab. https://wiki.omv-extras.org/do…v6_plugins:docker_compose

No.

Alles anzeigen

Excellent! Thank you!

About interface and ease of use: IMHO the popularity and success of a software greatly depends on simplicity and clarity of the interface. E.g. I was using Webmin for a long time, but majority of it's plugins are simply GUI fields for each config file. Once I learned where to find those configs, I didn't see much advantage in editing them via GUI vs editing them in nano. Eventually, I started using Webmin simply for it's dashboard with system info and for internal filemanager plus the terminal portal.

One more example: today I had my Pfsense HDD data corrupted. There was no sense in trying to restore the OS... so I downloaded and prepared USB image, installed it on a new SSD, loaded ONE config file and Pfsense not only set all the firewall rules, but also pulled all necessary plugins and configured them according to the config file. One single config file for all Pfsense and all plugins... The entire restoration process took me 20 min (including image preparation).

It's simple things that matter.

I really enjoy Compose plugin and really love the idea of no longer requiring of secondary project like Portainer or Yacht to be installed alongside with OMV. Don't take my vision of interface for a harsh critisism... I know it's not easy to come up with a simple and yet useful ideas.

Thanks for all your help.