Need some help setting up backup solution. Brand new OMV NAS on Raspberry Pi 3B+, 32Gb microSD Card, 4TB WD RED NAS + 4TB WD RED NAS

    • OMV 4.x
    • Resolved
    • Need some help setting up backup solution. Brand new OMV NAS on Raspberry Pi 3B+, 32Gb microSD Card, 4TB WD RED NAS = 4TB WD RED NAS

      Hello everyone,

      Like many initiated, I had many digital pictures and home videos floating around in many HDDs. Had a Raspberry Pi 3B+ laying around the house unused and decided to setup my own NAS.

      Long story short, I installed OMV4 and have a dual HDD docking station. Started with a single WD NAS RED 4TB HDD and transferred some content. Installed Plex and it is working fine. So far, so good. Pretty straightforward setup.

      Now, I decided to buy a second WD NAS RED 4TB to be an exact replica (Mirror) of the first one.

      Initially I tried to setup a RAID O but found out both disks needed to be initialized (wiped/formatted). This was a no-go since I already have content on my primary drive.

      I've been doing some reading about the differences between RAID and Backup. Decided to ditch RAID and use a backup solution instead. Now I'm scratching the surface with Rsync and RSnapshot. However what I've seen is you need to select and backup folder by folder.... Not exactly what I'm looking for.

      Is there a user friendly (GUI) way to setup a whole disk backup?

      - Select Source Disk
      - Select Target Disk
      - Set how frequently you want it to run automatically
      - How many days you want to keep your snapshots?
      - Click OK and DONE!

      Without recreating the folders of the Source Disk in the Target Disk.

      I have both Rsync and Rsnapshot enabled on my system.

      I'm putting on hold transferring more content to my primary disk until I get the second one running as backup.

      Any help will be greatly appreciated.

      Thanks!
      RaspberryPi 3B
      32 GB microSD Card
      Storage: 4TB WD Red NAS x 2 (One for Media, One for Backup)
      OMV 4.x
      Hey, I'm using Docker Images!
      - Plex
      - Watchtower

      The post was edited 2 times, last by javcsjc ().

    • Need some help setting up backup solution. Brand new OMV NAS on Raspberry Pi 3B+, 32Gb microSD Card, 4TB WD RED NAS + 4TB WD RED NAS

      Odroid HC2 - armbian - Seagate ST4000DM004 - OMV4.x
      Asrock Q1900DC-ITX - 16GB - 2x Seagate ST3000VN000 - Intenso SSD 120GB - OMV4.x
      :!: Backup - Solutions to common problems - OMV setup videos - OMV4 Documentation - user guide :!:
    • Hello,

      recently there was a similar thread asking for a simple solution. The proposed solution was also rsync.
      I strongly recommended not to use that option (pure syncing) because backups need to be versioned.

      A real live example from some days ago:
      My wife cleaned up here Documents folder including the folder "Photobooks". She thought it was a duplicate and deleted it. But it was not.
      She told me about this in the evening - after the weekly backup had run.

      So, if I had the simple rsync solution, the content of here Documents folder would have been mirrored onto the backup drive -without the folder "Photobooks". Everything would have been gone.
      But I do not have this simple solution, but a versioned backup using rsnapshot. This way I could use the backup from last week and restore from that.

      In short: Use rsnapshot and not a pure rsync.

      Greetings,
      Hendrik
    • My intention was not to propose to use rsync instead of rsnapshot. The linked thread just shows how to rsync the complete drive rather than setting up seperate jobs for individual folders. I assume it is working the same for rsnapshot plugin. Have not tested, though.
      Odroid HC2 - armbian - Seagate ST4000DM004 - OMV4.x
      Asrock Q1900DC-ITX - 16GB - 2x Seagate ST3000VN000 - Intenso SSD 120GB - OMV4.x
      :!: Backup - Solutions to common problems - OMV setup videos - OMV4 Documentation - user guide :!:
    • javcsjc wrote:

      Is there a user friendly (GUI) way to setup a whole disk backup?
      - Select Source Disk
      - Select Target Disk
      - Set how frequently you want it to run automatically
      - How many days you want to keep your snapshots in days
      - Click OK and DONE!

      Without recreating the folders of the Source Disk in the Target Disk.

      The features you're describing, above, is "Rsnapshot" in a nutshell. One of the Dev's was looking at designing a new Rsync plugin so that it would work in a manner similar to rsnapshot, but be able to copy entire drives and have the ability to offload data to a remote host (if needed). It fell through.
      __________________

      As a work around:
      This guide has a section that describes how to set up drive-to-drive Rsync jobs without leaving the GUI. (It begins on page 57 in the current version.) It should be noted that an Rsync job doesn't have the depth of choices for restoration when compared to rsnapshot but, (as you've noticed) the rsnapshot plugin does not support drive-to-drive backup.

      While not in the guide yet, if the source drive fails, recovery options are simple. They consisting of redirecting shared folders to the backup drive and, finally, reversing the process to copy files back to a new drive after it's installed. These recovery options can be done from the GUI and will be covered in an updated version of the guide. (It should be available this fall/winter.)
      __________________

      For the reasons outlined in this post, it might be best to either run the job manually or set the automated task to run every two weeks or so. (With plenty of time to discover a source drive problem and stop an automated Rsync event.)

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
    • Thank you, everyone! I will be experimenting with your recommendations.

      Since this is not really urgent, allow me a couple of days or so to test. If I still have any questions, will let you know.

      This morning took a look at USB Backup on OMV web client - Seems it works using the same approach, shared folder by shared folder.

      Set up a test with Rsnapshot using a single Shared Folder - "Pictures". Will let you know what happens later.

      Thanks again!
      RaspberryPi 3B
      32 GB microSD Card
      Storage: 4TB WD Red NAS x 2 (One for Media, One for Backup)
      OMV 4.x
      Hey, I'm using Docker Images!
      - Plex
      - Watchtower
    • henfri wrote:

      And if the problem happens just hours before the automated task?
      The possibilities for potential problems are infinite. I could ask, what if a power supply fails in a way that takes out the source and destination drives? But that question has no direct relevance to the question asked.
      In any case, I believe I was specific enough in the post above, and in referencing this -> post, for javcsjs's to understand the down side of using Rsync.
      ____________________________________

      With that noted; using @javcsjc's exact words, "a user friendly (GUI) way", explain to him how to do a restoration of an rsnapshot backup, from the GUI. (Please be specific.)

      Similarly, he asked specifically about backing his data drive, without setting up each share individually. If you know of a way to backup (and restore) an entire drive with rsnapshot from within the GUI, again, please share the info.

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk

      The post was edited 1 time, last by flmaxey: edits ().

    • After some initial tests, I've decided to go with Full Disk Mirroring/Backup with RSync, as it is explained on page 58 of the Getting Started with Openmedia Vault document.

      My source disk is: /srv/dev-disk-by-label-Media

      My backup target disk is: /srv/dev-disk-by-label-Backup.

      Note: I wiped, formatted & mounted the backup target disk. So, basically it was blank

      Using a scheduled job, under System --> Scheduled Jobs. Using the following command:

      rsync -av --delete /srv/dev-disk-by-label-Media/ /srv/dev-disk-by-label-Backup/

      I have it running every Tuesday at 1 am. I checked the File Systems this morning and both disks have the exact same size. IT WORKS!

      Question:
      • To see the actual files on my backup drive, do I need to recreate all my folders on the Source drive using SMB/CIFS?
      I would like to see them on my Windows machine, same way I see my folders on my Source drive.

      Thanks
      RaspberryPi 3B
      32 GB microSD Card
      Storage: 4TB WD Red NAS x 2 (One for Media, One for Backup)
      OMV 4.x
      Hey, I'm using Docker Images!
      - Plex
      - Watchtower
    • javcsjc wrote:

      To see the actual files on my backup drive, do I need to recreate all my folders on the Source drive using SMB/CIFS?
      Yes, you need to create shared folders for these folders and add them to SMB/CIFS. Or you create the shared folders in the SMB/CIFS view directly.
      Odroid HC2 - armbian - Seagate ST4000DM004 - OMV4.x
      Asrock Q1900DC-ITX - 16GB - 2x Seagate ST3000VN000 - Intenso SSD 120GB - OMV4.x
      :!: Backup - Solutions to common problems - OMV setup videos - OMV4 Documentation - user guide :!:
    • Hello,

      what made you use pure rsync rather than rsnapshot (there is even a plugin for that)? Just the fact that you would need to setup a task per shared folder?
      Did you read the concerns I raised on pure rsync?

      > Similarly, he asked specifically about backing his data drive, without setting up each share individually. If you know of a way to backup

      No. But how hard is it to setup one job per shared folder... I prefer safety over saving maybe 10 minutes.

      > (and restore)
      Is not easier or harder than with pure rsync


      Regards,
      Hendrik
    • Hi henfri,

      I did try rnapshot for 3 days by setting up only one of my folders.

      While it did backup the folder, I didn't like the nomenclature used by rsnapshot. It created a different folder for each day, nothing wrong with that. However, the name of the folder was Day 0, Day 1, Day 2, etc. instead of using the actual dates when the backup was created.

      This is what I saw on my backup drive:
      - Day 0
      - Day 1
      - Day 2

      This is what I would've like to see:
      - Sunday, September 2nd 2018 <Time>
      - Monday, September 3rd 2018 <Time>
      - Tuesday, September 4th 2018 <Time>

      This simple issue + the fact I had to setup each folder was a deterrent for me.

      For now, I'm backing up my Media drive once a week using rsync.

      rsync -av --delete /srv/dev-disk-by-label-Media/ /srv/dev-disk-by-label-Backup/

      I liked the simplicity of it.

      Thanks!
      RaspberryPi 3B
      32 GB microSD Card
      Storage: 4TB WD Red NAS x 2 (One for Media, One for Backup)
      OMV 4.x
      Hey, I'm using Docker Images!
      - Plex
      - Watchtower
    • javcsjc wrote:

      However, the name of the folder was Day 0, Day 1, Day 2, etc. instead of using the actual dates when the backup was created.
      The timestamp on the folder would tell you that info.

      javcsjc wrote:

      This simple issue + the fact I had to setup each folder was a deterrent for me.
      You don't have to setup a job for each folder. Create a shared folder on each drive with the path of /. If you have two drives, all of your shared folders should be subfolders of one of those root path shared folders. Setup one rsnapshot job from one to the other. Done.
      omv 4.1.14 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • Need some help setting up backup solution. Brand new OMV NAS on Raspberry Pi 3B+, 32Gb microSD Card, 4TB WD RED NAS + 4TB WD RED NAS

      Another possibility is to have a 2nd Pi with the backup drive and use something like Syncthing. You can have this on the same lan or somewhere else. This is probably one of the easiest ways to transmit data between two locations - and what I mean by this is that the setup is very simple and the security concerns around Syncthing’s data port are next to zero. Yes, there’s SSH (with keys), rsync and cron, but Syncthing is easier out the box. Separate power supply, separate location etc etc. I have offsite backups in this fashion because if there’s a house fire all I’m taking is my family and animals. I have my data in various other locations. So for an extra ~£40 for a Pi, case and PSU you can have automated versioned offsite backups with a nice GUI to configure it all. :)


      Sent from my iPhone using Tapatalk

      The post was edited 1 time, last by ellnic ().

    • I didn't know backup would bring such strong emotions among team members. Appreciate all your inputs. I have learned a lot from you.

      Following recommendations from ryecoaaron: Did some fine tuning and created shared folders with path of "/" for both disks (Media and Backup).

      One of the problems I found on the GUI was when I was creating the share folder it forces you add a name. I used "root" and ended up with path "root/".

      That was fixed when I entered the path "/" first but system still forced me to add a name. I use name "root" with path "/". Then I tried to do the same for the other disk. Turns out you cannot have two shared folders with the same name - for obvious reasons.

      I quickly fixed that by using the following naming convention:

      Disk Name: Media
      Share Folder Name: Media_Drive
      Path: /

      Disk Name: Backup
      Name: Backup_Drive
      Path: /

      Then set up rsnapshot on the GUI for taking daily backups of my root and everything underneath. It is working fine. Thanks ryecoaaron!

      P.S. rsync for the whole drive works as previously stated. For the time being, I will use this rsnapshot configuration for my backups. Will look into having a second (offsite) backup location.
      RaspberryPi 3B
      32 GB microSD Card
      Storage: 4TB WD Red NAS x 2 (One for Media, One for Backup)
      OMV 4.x
      Hey, I'm using Docker Images!
      - Plex
      - Watchtower
    • javcsjc wrote:

      One of the problems I found on the GUI was when I was creating the share folder it forces you add a name. I used "root" and ended up with path "root/".

      That was fixed when I entered the path "/" first but system still forced me to add a name. I use name "root" with path "/". Then I tried to do the same for the other disk. Turns out you cannot have two shared folders with the same name - for obvious reasons.
      They have to have a name because they bind mounted to /sharedfolders/SHAREDFOLDERNAME. If you have two with the same name, it would obviously cause problems. As for the path filling in root/ after you type the name, that is just the web interface trying to help. You don't have to use that path as you have found.
      omv 4.1.14 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • javcsjc wrote:

      This is what I saw on my backup drive:

      - Day 0
      - Day 1
      - Day 2

      This is what I would've like to see:
      - Sunday, September 2nd 2018 <Time>
      - Monday, September 3rd 2018 <Time>
      - Tuesday, September 4th 2018 <Time

      I understand that and I don't like that as well.
      But: with pure rsync you have Day 0 only. And if you like, you may just ignore all the other days (and weeks and months later) - until you need them. And then, you'll appreciate the others and figure out from when they are -and if it just by going through all until you find your beloved file that is missing in Day 0.
      Then you have only benefits, no drawbacks of rsnapshot.

      > I liked the simplicity of it.
      It's simple, but useless if you ask me. Others think differently than I do.
      But I would really hate to see that I recomment someone to do something I consider unsafe just to hear a year later that he lost all the photos of his newborn and now his wife has left the "family-admin".

      Greetings,
      Hendrik
    • javcsjc wrote:

      I didn't know backup would bring such strong emotions among team members. Appreciate all your inputs. I have learned a lot from you.
      It's not really an emotional reaction, per-say. Everyone in this thread, without doubt, wants you to succeed.
      What can be lacking, among some of the highly experienced forum members, is the ability to look at server issues and other topics from the perspective of a beginner.
      For example, @henfri is 100% correct - rsnapshot is technically far superior to a simple Rsync, one shot, backup. After figuring out "what went wrong", rsnapshot allows one to get back in time, and select a snapshot before the problem took place.
      On the other hand, it's the skill set and decisions involved in isolating the fault AND the restoration itself (which involves getting on the command line), that may make rsnapshot unsuitable for beginners.

      Based on your original question, and in that it's possible to set up an Rsync backup (and restore) process without leaving the GUI, Rsync backup and restoration is both "doable" and useful for beginners.
      (Not to mention that Rsync backup is a much better use of two disks, when compared to RAID1.)
      In the bottom line, it's better to have a backup and restore process that you can do yourself, versus using something that's too complex or not doing backup at all.

      macom wrote:

      javcsjc wrote:

      To see the actual files on my backup drive, do I need to recreate all my folders on the Source drive using SMB/CIFS?
      Yes, you need to create shared folders for these folders and add them to SMB/CIFS. Or you create the shared folders in the SMB/CIFS view directly.
      I suspect that you want to look at the contents of the backup drive, your shares, etc. Take a look a WinSCP and Putty in the Users Guide. WinSCP will let you look at your server's drives in an environment that's familiar to you. Also, you'll need to become familiar with Putty so you can get on the command line. In Linux (really any operating system) you'll need to able to get on the command line eventually.
      _______________________________________________________________________

      In the next couple months, you'll find the processes for interim restoration (repointing shared folders from the source - to the backup drive) and for restoring the source drive, in a updated version of the guide. Also, as outlined by @ellnic, you'll find processes for replicating network shares to a 2nd host (eventually).

      (And I still haven't given up hope that @ryecoaaron will write a new easy to use Rsync plugin, with rsnapshot like functionality, and remote host sync as well. :) )

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk

      The post was edited 2 times, last by flmaxey: edit2 ().