Planning for SnapRAID (and rsync and rsnapshot?)

Hi, now that my OMV NAS and the backup computer are more or less stabilised (i.e. at least I can access them at much better speeds, the HDDs are pooled meaning that I won't accidentally miss out folders during backup and that the backup computer contains a second copy of the files in NAS. Yes, I know, this is not exactly a backup by sysadmin standards...), I now need a bit help planning my next moves.

You see, I will like do the following with their respective reasons:
1. SnapRAID: It seems to me that most people using pooled HDDs pair their systems with this. Also, the idea of having some form of redundancy by just adding a new HDD for every 4 HDD and not needing to rebuild the array is quite appealing
2. rsync: I understand this is a better way to copy files from one computer to another since this is an incremental sync rather than full sync (e.g. FreeFileSync). I do still want to see the files being changed before initiating rsync.
3. rsnapshot: I understand that this is what most people use as part of their backup strategy and is also a more true backup solution (e.g. allowing going back to a week's ago data). I am still quite unsure of how to implement it properly.
4. More space: I can also forsee myself needing more space, will have to consider this too. Each computer has 5TB, I hope to have at least 6TB.

So, I will need some opinions on the following:
1. In which order should I proceed?
2. What hardware changes I will need to get? (e.g. How big should the new HDDs be? Need to consider getting better machines in long run?)

Thanks!
-----------------------------------------

Specs of OMV NAS (mhddfs)
Core 2 Quad Q6600
6 GB DDR2 RAM running at 667 Mhz
Asus P5KR (5 SATA 3.0GB/s ports, 1 eSATA port)
Seagate ST1000DM005 HD103SJ (1TB)
SAMSUNG HD103SJ (1TB)
Hitachi HDS723030ALA640 (3TB)

Specs of Backup Computer (Windows 7 Ultimate, Drive Bender)
Core 2 Quad Q6600
7 GB DDR2 RAM running at 667 Mhz
Dell XPS 420 (5 SATA 3.0GB/s ports)
Seagate ST2000DL004 HD204UI (2TB)
Toshiba DT01ACA300 (3TB)

1. Hmm good point, didn't realise that backup has higher priority than redundancy. And I thought that having SnapRAID on both computers are needed

2. Ok, time for me to set up rsync on both computers. Doing so for the Windows computer is going to be slightly harder I guess (I did hear about some Windows rsync server client program, gonna check that out)

But will rsync tell me what files will be changed/added/deleted before the sync? This is the main reason I liked FreeFileSync but also preferring something a bit snappier

3. I am thinking of doing this for the entire pool for now, how much HDD space will I need? (I am still quite unsure on the HDD requirements for this; not as clear cut as SnapRAID)

Hardware: 4TB HDDs sound nice but I am quite unsure if this will work with SnapRAID in the current setup given that the largest HDD must be the parity HDD (i.e. no data). The 4TB will be the largest in my situation.

Also, should I invest in another two port Intel NIC card for the backup computer since the OMV NAS has a two port Intel NIC card too? My switch (TP-Link TL-SG2008) do support LACP (802.3ad); I wonder if this can help (I have tried teaming the onboard Intel NIC and another Intel PCIe NIC but I have to disable Jumbo Frames and Receive Side-Scaling)

Ah ok, sorry, should have made that clearer

Erm, sorry I still need help here...

Ah sorry, I understand. I will experiment with that and see how it goes.

Quick updates on this:

1. Still putting this on hold due to 4.

2. As mentioned in another thread (Rsync from OMV to Windows's DeltaCopy = Internal Server Error?), I am putting the rsync on hold until the issue can be resolved. For now, FreeFileSync will have to suffice

3. I made a new data partition from the system drive holding onto the OMV OS and starting running rsnapshot to snapshot the entire 5TB worth of actual data into this data partition. This data partition will also be used for VMs as mentioned in another thread (Baremetal OMV or OMV as a VM in ESXi?) (gonna find time to work on this...). So far so good seeing how about 150GB is needed for the initial backup, but eventually I will have to use a bigger HDD just for snapshots.

EDIT: Nope, upon checking the system logs, apparently the HDD does not have enough space for all data. I am retesting this on a much smaller HDD and see how this leads to.

Now the question would be to how to use it effectively against say, cryptolockers? I understand from https://www.veeam.com/blog/how…ckup-vs-cryptolocker.html that as the viruses can jump from the actual data into even backups if one is not careful in setting up the system. Currently, the way I set up is that the HDD holding onto the snapshots is not accessible by SAMBA unlike the actual data, which has to be accessed via SAMBA by my main computer. Also, I checked the ACLs and privileges of the HDD so that it is not accessible by other accounts other than those required for rsnapshot and VirtualBox.

More testing will have to be done to confirm this.

4. As I am down to my last 100 GB of space, I think I will have to prioritise getting more space first. Having a new 4TB HDD seems to be the right way to go forward seeing how this and creative rearrangment of the HDDs will let me have 7TB on each side instead of 6TB. It will be nice if I can solve this and 1 via getting just one new HDD but I think I am stretching this a little too much.

And yes, now I can see why your brain is fried, my bad for having such... strange requests!

Back, from settling other real-world issues and even a PSU issue.

After looking through more (especially after thinking through and experimenting with rsnapshot), I am having some slight changes to my setup:

Setup A:
1. I won't use rsync plugin at all
2. I will keep my main data in the OMV NAS
3. I will use OMV NAS's Remote Share and rsnapshot to do daily snapshots to the Backup Computer

This way, I can have "real" backups in a separate computer (instead of just file replication i.e. only one version of the backup). (Performing Incremental Data Backups to a Second OMV System)
This however means that I will need to find ways to wake the Backup Computer up before the rsnapshot job and shut it down after it is done.
Also, I am encountering error 4000 while trying to mount the SMB share from the Backup Computer to the OMV NAS using the Remote Share plugin, will investigate further (May be related to Remote Shares Plugin - can remote share mounted but can not save ?)
I also understand that this may not be the intended use case with OMV (rsnapshot - questions)

Setup B:
1. I will keep my main data and snapshots in the OMV NAS
2. I will transfer the required HDDs to the OMV NAS, then set up a second mergefs pool (which will not be shared most of the time) with the new HDDs
3. I will then rsnapshot the main data pool to the second pool daily
4. I may rsync the weekly backup to the Backup Computer (if not, FreeFileSync it is)
5. In any case, the Backup Computer will at least just be a spare computer for me to use in the event of my current computer failing

This seems much easier to setup than Setup A since this is mostly just using the rsnapshot plugin and a bit of the UnionFS plugin
However, it seems that this is essentially a Single Point of Failure situation given that all data and backups are in a single machine; if the machine fails somehow, all data will be gone (unlike in Setup A where at least my backups will be safe in another machine in the same situation)
Also, this setup means that I will have to use eSATA or even go back to my previous usage of thumbdrive
Of course, the only way to alleviate this is to get more HDDs for the Backup Computer but I am not sure how fast can I make this happen

In either case, it looks like SnapRAID will not be implementable until I get more HDDs, or even new systems that have more SATA ports

Am I thinking this through correctly or am I missing something?

I tried Setup A and kept getting error 4000 whether trying to mount a NFS share or a SMB share. Seems to me that the Remote Shares plugin does indeed need to be reworked (Remote Shares Plugin - can remote share mounted but can not save)

I will want to give Setup A another shot though, so I think I will change my focus by working on the Windows machine instead using these guides:
http://blog.gingerlime.com/2007/rsnapshot-server-on-windows/
http://jeffskinnerbox.me/posts…-via-rsync-and-rsnapshot/

Let's see how this works out

I have tried Setup A and it somewhat did not work.

On the Backup Computer, I installed cygwin for rsnapshot and then tried to rsnapshot a test folder on my OMV NAS onto the HDDs on the Backup Computer that are combined as one HDD via Drive Bender.

The first snapshot went through successfully but the second one onwards kept giving me messages about permissions and linking.

I solved the permissions messages by getting cygwin to mount all drives with the noacl flag since they are NTFS (except the network drive that is a EX4 HDD pool from the OMV NAS mounted via SMB) but the linking messages stumped me.

Initially, I thought that NTFS was to blame; many sources on the Internet seem to point to that (e.g. http://www.backupcentral.com/p…hard-link-problem-120084/) until I reread rsnapshot's and rsync's man pages to understand them better and also some documentation on the NTFS system, which shows that this is not exactly true.

So, I decided that cygwin may be the one at fault and searched until I hit this page: http://superuser.com/questions…t-hard-links-in-windows-8, which pointed otherwise and hinted to me that I should use Windows to try creating a hardlink to test further.

I did that on the HDD pool created by Drive Bender, the OS drive, a newly formatted drive, a drive created via creating a Spanned Volume from two spare HDDs I happen to have, and a RAID drive created via the onboard RAID controller on the same two spare HDDs.

Hardlinks can be created on all configurations except the HDD pool created by Drive Bender, leading me to this page: http://stackoverflow.com/quest…code-1-incorrect-function, which confirmed that via fsutil, hardlinks are explicitly stated to be supported in all these configurations except for the HDD pool created by Drive Bender. Rsnapshot then is confirmed to be working in those configurations too.

Currently, I am asking Drive Bender's support about this issue and also at the same time, using the spare HDDs mentioned here to test out Setup B's feasibility. So far, this is way easier to setup but we will see if this is what I want.

Either way, looks like another new HDD will be a nice addition

Now shifted to Setup B (i.e. all HDDs in an OMV box), which means this is my current setup:

-----------------------------------------
Specs of OMV NAS
Core 2 Quad Q6600
8 GB DDR2 RAM running at 800 Mhz
Asus P5KR (5 SATA 3.0GB/s ports, 1 eSATA port)

TOSHIBA MK3263GSX (320GB) (OS drive) (eSATA)

Main Data Pool
Seagate ST1000DM005 HD103SJ (1TB) (SATA 3.0GB/s)
SAMSUNG HD103SJ (1TB) (SATA 3.0GB/s)
Hitachi HDS723030ALA640 (3TB) (SATA 3.0GB/s)

Snapshot Pool
Seagate ST2000DL004 HD204UI (2TB) (SATA 3.0GB/s)
Toshiba DT01ACA300 (3TB) (SATA 3.0GB/s)

Specs of Backup Computer (Windows 7 Ultimate | OS X El Capitan)
Core 2 Quad Q6600
5 GB DDR2 RAM running at 667 Mhz
Dell XPS 420 (5 SATA 3.0GB/s ports, 1 eSATA port)

Seagate Barracuda (500GB) (Windows OS drive) (SATA 3.0GB/s)
Seagate Barracuda (500GB) (OS X drive) (SATA 3.0GB/s)
-----------------------------------------

As predicted, doing this shift was easy, but time-consuming since I have to reformat the HDDs of the snapshot pool from NTFS to EXT4, create a mergerfs pool with them, then rsnapshot all data from main data pool to it.

Unfortunately, for Drive Bender, I am told that as the file system is a virtual filesystem, it has its limitations like not supporting hardlinks (but does support folder links though).

Currently, I am setting up the email notification system and S.M.A.R.T monitoring system via their respective plugins; I need to automate some things a bit here.

May I know how do I get email notifications of the results of the S.M.A.R.T tests?