Help with Easy Backup solution

    • OMV 4.x
    • This is a pretty good article that they pretty much replicate rsnapshot with scripts and rsync. Maybe a whole new plugin using some of these ideas with a restore and remote capabilities since I think this can do a push to remote that rsnapshot can't do.
      omv 4.1.11 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.11
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • ryecoaaron wrote:

      flmaxey wrote:

      It would be great to be able to pick a source drive and a destination drive, a schedule, and it's done. Simple. Restore could be a button, with a "are you sure" caution popup, that restores to a new drive (raid pair, etc.) with the same label or maybe another drive that's wiped and mounted.
      So, you want something that effectively clones one drive to another drive? Not just an rsync? ie creates the partition table and filesystem then rsyncs the data.
      No... Perhaps I should have said, "wiped, formatted and mounted".
      Something like clonezilla would be a duplication of effort and really, for backup and restore purposes, not as easy or flexible as simple Rsync file and folder replication.

      As an example, one couldn't "clone" a RAID5 array (partitions, flags, and all) onto a single drive.
      ((I hesitate to say "can't" in this forum, because someone might step up and provide some abstract method and reason for doing it. :) Partitioning single drives and setting up RAID5 arrays with the partitions comes to mind... 8| ))

      On the other hand, it is possible to Rsync the files and folders of a RAID1 array (or even a RAID5 for that matter) onto a single drive of sufficient size for the content, and restore those files and folders back onto a new or repaired RAID array. An Rsync plugin could also be used to move files and folders, with permissions intact, between drives with different file formats (ZFS to EXT4, then back to a BTRFS formatted setup).

      In any case, the user would responsible for wiping, formatting and mounting a backup or a replacement, whether it's a single drive, some kind of array, a Unionfs drive set, etc.

      It's a plugin that could be used for simple backup and restoration, that's really easy to understand, and provide additional utility as noted above. I could explain the use of it in the New User Guide, for the benefit of all.
      ________________________________________________

      BTW, given other traffic I've noticed on the forum:
      This winter, I'm planning to add content to the New User Guide (addendums?) or maybe write something separate, for intermediate use. I'm planning to go through the various utility plugins, explain their uses and provide some examples. There's a lot of functionality in those plugins that users are missing out on.
      Good backup takes the "drama" out of computing
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
      2nd Data Backup: OMV 3.0.99, R-PI 2B, 16GB boot, 4TB WD USB MyPassport - direct connect (no hub)
    • ryecoaaron wrote:

      This is a pretty good article that they pretty much replicate rsnapshot with scripts and rsync. Maybe a whole new plugin using some of these ideas with a restore and remote capabilities since I think this can do a push to remote that rsnapshot can't do.
      I'll have to read up on this more closely when I get back this coming weekend but,,, that would be so freaking cool! 8o
      I'm guessing your thinking about remote drive sync and restoration, in a plugin!....

      And it didn't occur to me, until I read your post above, that Rsync could be used in combination with Remote Mount, to off load an entire drive to a remote host. After the first sync, it would be efficient, changes only! And with the scripts, even the equivalent of snapshots could be off loaded onto a remote host. It would appear to be local.
      Enter a user and password, as in Remote Mount, connect to the remote host, make source and destination selections, set a schedule, test... I like it. :thumbsup:

      Something like that would be the absolute best of both worlds - simplicity for new users (for local backup and restore) with really advance capabilities for the experienced (remote sync).
      Good backup takes the "drama" out of computing
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
      2nd Data Backup: OMV 3.0.99, R-PI 2B, 16GB boot, 4TB WD USB MyPassport - direct connect (no hub)
    • flmaxey wrote:

      Rsync could be used in combination with Remote Mount,
      I don't want to have to use remotemount. The plugin would either use rsync or rsync+ssh.

      henfri wrote:

      I think rsnapshot can do remote..
      rsnapshot can do a remote pull but not a remote push. The current rsnapshot plugin can't do either.

      henfri wrote:

      I would not try and replicate rsync functions but use and extend them.
      What are you talking about? I want to use rsync (no need to extend) since rsnapshot can't do a remote push.
      omv 4.1.11 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.11
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • ryecoaaron wrote:

      flmaxey wrote:

      Rsync could be used in combination with Remote Mount,
      I don't want to have to use remotemount. The plugin would either use rsync or rsync+ssh.
      You are way more qualified than I am to chose the right approach. (Not to mention being the Dev that codes it. :) )
      But I believe the net concept as I understand what you're purposing, push/pull remote sync capability, is roughly the same.
      Good backup takes the "drama" out of computing
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
      2nd Data Backup: OMV 3.0.99, R-PI 2B, 16GB boot, 4TB WD USB MyPassport - direct connect (no hub)
    • henfri wrote:

      askubuntu.com/questions/35180/…he-server-from-the-client
      What is this link for? It says what I have been saying. You can't do a remote push. You *can* push with rsync and then do an rsnapshot. Some people don't like that because you keep two copies of the data on the destination.
      omv 4.1.11 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.11
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • Hello Aaron,

      I was refering to this:
      Another option is to use a reverse ssh tunnel and then to a "remote" pull from the receiver's side:
      #!/bin/bash -xI
      VAL=${1:-hourly}
      ssh -vNR 1998:localhost:22 example.com &
      ssh example.com "rsnapshot -c pull-from-sender-rsnapshot.conf ${IVAL}"


      where pull-from-sender-rsnapshot.conf might contain a line like:
      backup localhost:1998/etc/ desktop-1This could be invoked by cron on the sender's side, triggering the remote rsnapshot locally.

      (of course it's up to you, but) Why do you think it is easier developing something from scratch (and having to maintain it) rather than extending rsnapshot to be able to do push?

      Greetings,
      Hendrik
    • henfri wrote:

      I was refering to this:
      I didn't see that but it is not the best workaround and harder to setup.

      henfri wrote:

      Why do you think it is easier developing something from scratch
      Because it isn't developing it from scratch. It is using rsync in a normal way. rsnapshot uses rsync. Personally, I don't like rsnapshot's config files. rsync can do everything easily from the command line. The only thing creative is the clean up.

      henfri wrote:

      ather than extending rsnapshot to be able to do push?
      What is the difference between "extending" rsnapshot with ssh and "extending" rsync with a cleanup routine? I really can't figure out why you think every idea of mine is wrong...
      omv 4.1.11 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.11
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • ryecoaaron wrote:

      This is a pretty good article that they pretty much replicate rsnapshot with scripts and rsync. Maybe a whole new plugin using some of these ideas with a restore and remote capabilities since I think this can do a push to remote that rsnapshot can't do.
      After reading the article, it reminds me of the scripts zfs-auto-snapshot uses to create periodic snapshots. zfs-auto-snapshot is a pretty slick setup; scripts that use cron, auto naming-dating, and cleaning up snapshots. But the ZFS scripts are all about the local pool - and it's about the retention of changes, not hard-linked backup. Still, for a group of scripts, the resultant functionality is nothing short of amazing.

      After reading the article; I have a much better understanding of Rsync "hard-links", and how they're created with --link-dest. It makes sense and the article was an intro to bash scripts. Adding SSH to Rsync would certainly give it security and reach, beyond the local network.

      On the other end of it, what did you have in mind for restore? Pick a name-date and a destination?
      A simple plugin for beginners, with a couple buttons (Backup and Restore) with, maybe, source and destination drop downs to chose a share or a drive seems logical.
      Good backup takes the "drama" out of computing
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
      2nd Data Backup: OMV 3.0.99, R-PI 2B, 16GB boot, 4TB WD USB MyPassport - direct connect (no hub)
    • Hello Aaron,

      rsnapshot is 7500 lines of code and this is also building upon rsync.
      Plus there are many tools, tutorials, documentation around that.

      I am sure that you will come up with a good backup solution in a short time if you start from pure rsync. But you create an isolated solution and you will be on your own supporting it.

      If you extend rsnapshot removing the limitation you all these drawbacks do not exist and more people (not just omv-users) will profit from it.

      My experience: creating a solution = x hours. Making it robust = 2 x hours and years of usage. Maintaining it = x/4 per year.

      I am not sure that you take into account more than "one x".

      Greetings,
      Hendrik
    • henfri wrote:

      But you create an isolated solution and you will be on your own supporting it.
      Just like most the plugins I write for OMV because I rarely get any help and they are specifically for OMV.

      henfri wrote:

      If you extend rsnapshot removing the limitation you all these drawbacks do not exist and more people (not just omv-users) will profit from it.
      There already is an rsnapshot plugin. You can use it with remote mount (or sshfs if you are creative). I don't understand what I am supposed to extend.

      henfri wrote:

      I am not sure that you take into account more than "one x".
      Of course not. I have no idea what I am doing. It amazes me all the people who tell me I am doing things wrong but never help.
      omv 4.1.11 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.11
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • henfri wrote:

      I did not mean to offend you...
      Then maybe you might look back at your approach:
      In this thread, the title is Help with Easy Backup solution

      You suggested rsnapshot. Right at the beginning, porkenstien looked at rsnapshot and stated that it is not what he/she is looking for. I understand that point completely. For beginners - rsnapshot is half of a backup and restore solution, with no restoration option that's easy to understand or implement. Looking at it from that perspective, that of a beginner, half a solution = no solution.

      Since it was obvious that porkenstein was looking for "dirt simple", I picked it up from there. At that point, you criticized my approach (which works AND is documented in detail) for simple drive-to-drive backup with Rsync.
      While I shouldn't have had to do it, this is porkenstein's thread after all, I patiently explained to you that beginners aren't impressed with a lot of restoration options that require complex decision points; that simplicity - something that works and is easy to understand - is the imperative. (This goes back to the thread title.)

      Then, it seemed as if you didn't like ryecoarron's overall approach, with another series of posts, for a potential plugin that would benefit all concerned. It's just not constructive. In the bottom line, ryecoarron is right; there are plenty of arm chair critics out there but few who are actually willing to do something.
      Good backup takes the "drama" out of computing
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
      2nd Data Backup: OMV 3.0.99, R-PI 2B, 16GB boot, 4TB WD USB MyPassport - direct connect (no hub)
    • Hello,

      thanks for your feedback.
      It seems like for you true, valuable support is only coding, not advice. In fact, I think that this not right.

      The solution that you provide in your guide (that I really value. It is really good for Beginners) seems dangerous to me. I have spent lots of time on Backups and I think I know what I am saying...
      It is really dangerous to consider a 'delayed mirror' a backup. And I suggest not to suggest this to beginners. Backups must be versioned. If there is a non versioning solution that is easier, does not make this a solution. Maybe both (the versioned and the easy solution) are no solution, but that does not invalidate what I say.

      Apart from that I must say that I do not understand why a restore would be not easy with rsnapshot. Rsnapshot creates folders with a complete copy of the whole folder structure. A restore can be done with 'cp'. (sorry, if I oversaw a post explaining what is not easy about the restore). Maybe for the restore just a 'button' in the webinterface and an explaination in your guide would do? I do not see where the 'inherint' complexity in rsnapshot restores would be that would justify a new development.

      Next Aaron suggested that he wants to develop a new plugin solving all this. I suggested not to start from scratch, for the reason stated above -and others that I explained.

      ryecoaaron wrote:

      There already is an rsnapshot plugin. You can use it with remote mount (or sshfs if you are creative). I don't understand what I am supposed to extend.
      I need nothing. I am happy with what I have.
      But you stated that you see a limitation in rsnapshot not being able to push. I gave a suggestion how this could be overcome and I understood that you prefer starting from plain rsync. I have outlined why I think that this is the second best option, and I have explained the reasons.
      I do not see why this is offending you. I see that for you 'help' is only coding. I am sorry about that. I did try to understand how to write a plugin but although I think I am reasonable capable in programming the efford understanding the plugin architecture is to high for me.
      If you think that my advice is useless, I'll keep my 2ct for myself in future.

      What I will not is stop warning beginners about bad backup strategies though.

      Regards,
      Hendrik
    • henfri wrote:

      What I will not is stop warning beginners about bad backup strategies though.
      Regards,
      Hendrik
      All things are relative - something is better than nothing and two full copies of data are, without doubt, better than one.
      Note that WHS has a similar server backup solution which is, essentially, "back up the server" - no versions. The backup generated by WHS is the state of the server from the last manually triggered backup.

      In any case, most of the dialog in this thread is about addressing and improving on that situation. If you can come up with something better that's appropriate for beginners (no CLI) and be willing to document or code it, I'm all ears.
      Good backup takes the "drama" out of computing
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
      2nd Data Backup: OMV 3.0.99, R-PI 2B, 16GB boot, 4TB WD USB MyPassport - direct connect (no hub)
    • henfri wrote:

      a 'delayed mirror' a backup.
      A realtime mirror is and never will be a backup. It is redundancy.

      henfri wrote:

      But you stated that you see a limitation in rsnapshot not being able to push. I gave a suggestion how this could be overcome and I understood that you prefer starting from plain rsync. I have outlined why I think that this is the second best option, and I have explained the reasons.
      You gave a workaround that is difficult to setup from the web interface.

      henfri wrote:

      I do not see why this is offending you. I see that for you 'help' is only coding. I am sorry about that. I did try to understand how to write a plugin but although I think I am reasonable capable in programming the efford understanding the plugin architecture is to high for me.
      Trust me, it isn't just this one time you criticize my plugin decisions. Even when I argue my reasons, you keep pushing and this has happened more than a few times. I understand trying to "help" but remember that you are dealing with someone volunteering their time and you might not get everything you want (ie I code plugins to do what is good for me).

      henfri wrote:

      If you think that my advice is useless, I'll keep my 2ct for myself in future.
      I never said your advice was useless but it seems like your ignore my responses when they are different than what you want.

      henfri wrote:

      What I will not is stop warning beginners about bad backup strategies though.
      If rsync is a bad backup strategy while rsnapshot is a good strategy (remember I use rsnapshot), then people should listen to other people.

      I am done here. No new plugin will be written and the rsnapshot won't be changed. So, everything should not explode.
      omv 4.1.11 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.11
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • henfri wrote:

      So, the CLI is the only problem about the restore?
      For beginners? Absolutely. While it might be hard to for you to imagine, we'll say you know nothing about backup strategy. But, you're told backup is a good idea and it's no big deal to do the backup itself - just install a plugin, make a few selections and it's done for you.

      The problem crops up when, as a beginner, you're not sure what went wrong but some of your data is corrupted or missing and you want to do a restore. The catch for you is, you have to write a program in assembly language to get your data back. (And really, that's not a fair comparison because you at least have an idea of what coding is.) Forcing a beginner onto a Linux command line would be even more alien to them, than programming in assembly would be for you.

      Effectively speaking, anything that requires constructing a command line from scratch, where a beginner would have to select a command, switches, set sources and destinations, arguments, etc., is the net equivalent of not having backup at all.
      Good backup takes the "drama" out of computing
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
      2nd Data Backup: OMV 3.0.99, R-PI 2B, 16GB boot, 4TB WD USB MyPassport - direct connect (no hub)
    • Users Online 1

      1 Guest