Looking for data backup and redundancy with a Raspberry Pi

  • I'm looking to backup all my photos and videos. I need to have redundancy, so I was thinking of having two hard drives, with the data mirrored on both hard drives. This way, I always have two copys of every file.

    What is the best way to acheive this with a Raspberry Pi 3B and two USB hard drivers?

  • The best way is to buy a powered dual bay case that supports raid1 directly. That way you minimize the cable mess.


    However, even better could be to skip redundancy and instead go for versioned backups.


    If you use raid1 you have NO PROTECTION against silly user errors. Using versioned backups you can , if needed, go back in time and restore files and folders as they were some time in the past.


    I use a bash script to create versioned rsync snapshots. The script also deletes old snapshots. OMV also has support for rsnapshot that does something similar.


    Here is my script: https://github.com/WikiBox/snapshot.sh

    Be smart - be lazy. Clone your rootfs.
    OMV 5: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4

  • I like the look of that, but the Odroid seems hard to get hold of in the UK.


    I dont really want to buy any new hardware, but I could be pushed towards a Pi4 4GB. I've read they are good for home NAS setups with USB hard drives as it has 2x USB3 ports.

  • Why buy an Odroid? Use a RPi!


    The methods I suggests works equally fine on a RPi4. Currently I use my script on my Odroids, on a RPi4 and in my laptop.


    I agree that the RPi4 is great for a NAS. I currently use a 8GB RPi4 with OMV5. And a 4 bay external USB enclosure with 64 TB storage. Works great.


    The suggestion was to buy an external 2 bay USB enclosure.so you get less problems with cables. And connect that to the RPi. Then use a script or rsnapshot in OMV5 to backup files between the drives instead of trying to setup RAID over USB to get some form of redundancy. OMV does not support RAID over USB, for good reasons: It is not reliable.


    While the RPi4 has two USB3 ports, they share the bandwidth of one port.

    Be smart - be lazy. Clone your rootfs.
    OMV 5: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4

    Edited 2 times, last by Adoby ().

  • I currently use a 8GB RPi4 with OMV5. And a 4 bay external USB enclosure with 64 TB storage. Works great.

    Thanks for sharing what you use for your setup. I currently have a helios4 for my operational data and am considering the best (and most cost effective) way to have backups in a separate location.


    Your suggestion of using a Raspberry Pi and an external USB enclosure sounds like a really good option for me to use as a backup server. I had a couple of questions :):

    1. You mentioned that you have a RPi4 8GB. Was there a reason (i.e. experience) that you went 8GB over 4GB? I guess I'm wondering whether 8GB is needed and justifies the additional cost?
    2. There are a lot of external USB enclosures out there. Do you have any recommendations for what to look for? What to avoid? Or even a good brand to go with? There are some very expensive enclosures and I'm keen to have a good quality and robust setup, but the most expensive option is not always needed to achieve this. I probably only need a 2 bay enclosure at the moment. However, I would be open to getting a 4 bay if it wasn't going to break the bank.

    Thanks

  • I got a 8GB RPi4 because I could. I can't say I notice any great performance improvements. Or any at all. If you were running a lot of dockers I assume that you could perhaps notice a difference.


    I may swap out the 8GB RPi4 for a 4GB in the future, if I think I have more use for the 8GB in some other role.


    I don't know exactly what enclosure to recommend.


    Mine isn't optimal. It doesn't support UAS. ICY BOX IB-RD3640SU3. But it supports any size hdds and it is fast enough to saturate GbE during large file transfers.


    I also have a dual bay Startech S352BU33HR enclosure, it supports UAS but not 16TB hdds.


    You want an enclosure that supports UAS and any size hdds. I don't know any enclosure that does both. But I assume some do?

    Be smart - be lazy. Clone your rootfs.
    OMV 5: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4

  • Two bay, 16TB disk limit, and USAP support. I don't know anything about it but was wondering if it would do what you are looking for.

    Easy data backup: In a Scheduled Job: rsync -av --delete /srv/dev-disk-by-label-SOURCE/ /srv/dev-disk-by-label-DESTINATION/ (HT: Getting Started with OMV5)
    OMV 5 (current) - Thinkserver TS140, Nextcloud, Plex, Airsonic, Navidrome, Ubooquity, Digikam, Wetty, & Heimdall - NanoPi M4 (v.1): backups using Rsync and Rsnapshot - Odroid XU4 (Using DietPi): PiHole - hc2, xu4, Pi 3B+, Odroid H2, and VirtualBox: Testing and playing - Mac user converting to Linux, Debian 10 KDE.

  • The Pi 3 can be a great choice for a backup (even an offsite backup) but not as a primary server. USB 2.0 and RAID simply do not mix, as I learned through a lot of headaches.


    The Pi 4 is a far better candidate for RAID due to the USB 3.0 connections.


    But, even then, it's not a particularly elegant solution because plugging in a bunch of hard drives via USB is an unwieldy mess. You can do it, but it's not pretty. I'd love to see some proper enclosures for a Raspberry Pi NAS.

  • he Pi 4 is a far better candidate for RAID due to the USB 3.0 connections.

    Raid with USB is never a good idea. The board or usb3 make no difference.

    But, even then, it's not a particularly elegant solution because plugging in a bunch of hard drives via USB is an unwieldy mess. You can do it, but it's not pretty. I'd love to see some proper enclosures for a Raspberry Pi NAS.

    If you want multiple drives, get a board that has real sata like a nanopim4 with sata hat or helios64.

    omv 5.6.13 usul | 64 bit | 5.11 proxmox kernel | omvextrasorg 5.6.2 | kvm plugin 5.1.6
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • Raid with USB is never a good idea. The board or usb3 make no difference.

    If you want multiple drives, get a board that has real sata like a nanopim4 with sata hat or helios64.

    This advice is given here every time someones asks, but does it really apply in all setups?
    I've build 2 Pi4 based NAS using an external USB3-HW-RAID enclosure with 2 embedded disc drives and HW RAID chip inside the enclose.

    This setup requires just a single USB3 cable.
    I've not found any drawbacks after 9 months of operation.


    Detailing the reason on 'why USB is generally bad for this USB3-HW-RAID enclosure' would be greatly appreciated.

    omv 5.6.16-1 (usul) on RPi4/4GB with Kernel 5.10.63 and WittyPi 3 V2 RTC HAT

    2x 6TB HDD formatted with ext4 in Icy Box IB-RD3662-C31 / hardware supported RAID1

    For Read/Write performance of SMB shares hosted on this hardware see forum here

  • I think you are confused. I doubt anybody says that USB3 to a HW RAID enclosure is a problem? I have a 4 bay HW RAID enclosure connected to a RPi, myself. Currently not using any RAID, but still...


    What can indeed be a problem is using USB for software RAID.


    There are several reasons:


    1. Several drives will share a USB connection, making for poor and/or unpredictable results.

    2. It is likely that there is a mix of lose USB cables, USB hubs, enclosures and power supplies involved. All with one or two connectors that may be bumped by mistake or simply be faulty.

    3. Users wanting to use USB with software RAID often are novice RPi users with little or no experience and knowledge of Linux and RAID. Needing a lot of help and having difficulties following/understanding written instructions. Not because they are stupid, they just lack experience and knowledge. In combination with point 2 this makes software RAID a really bad idea for them and it is extremely difficult to support. It may even be dangerous and cause data loss, because it is easy to setup software RAID using a GUI, but when something bad happens you may quickly need a spare unused HDD and use the command line...

    Be smart - be lazy. Clone your rootfs.
    OMV 5: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4

  • Adoby , which comment do you actually mean when you say "I think you are confused. I doubt anybody says that USB3 to a HW RAID enclosure is a problem?" ryecoaaron's or mine?

    omv 5.6.16-1 (usul) on RPi4/4GB with Kernel 5.10.63 and WittyPi 3 V2 RTC HAT

    2x 6TB HDD formatted with ext4 in Icy Box IB-RD3662-C31 / hardware supported RAID1

    For Read/Write performance of SMB shares hosted on this hardware see forum here

  • This:


    "Detailing the reason on 'why USB is generally bad for this USB3-HW-RAID enclosure' would be greatly appreciated."

    Be smart - be lazy. Clone your rootfs.
    OMV 5: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4

  • well the OMV documentation and hence the quote ' Quote from ryecoaaron

    Raid with USB is never a good idea. The board or usb3 make no difference.'

    is, from my view, lacking the required detail to make an informed decision.


    Hence users that don't know the difference between SW and HW RAID are getting confused.

    To resolve this issue I'd suggest to avoid generic statements like above and be precise that:

    "USB with a HW RAID enclosure is actually an acceptable or even good solution"

    This is the point I try to make.

    omv 5.6.16-1 (usul) on RPi4/4GB with Kernel 5.10.63 and WittyPi 3 V2 RTC HAT

    2x 6TB HDD formatted with ext4 in Icy Box IB-RD3662-C31 / hardware supported RAID1

    For Read/Write performance of SMB shares hosted on this hardware see forum here

  • This advice is given here every time someones asks, but does it really apply in all setups?

    Yes. I will explain this again. If the external enclosure presents a single drive to the RPi, fine but it is still not completely redundant. There is only one power supply. usb is not a server-grade connection and the controller can sometimes put drives to sleep with the server not knowing (bad). The setup I really hate is multiple single usb drives because they can spin up at different times causing the array to not assemble. The drives can also go to sleep at different times causing major array issues.


    I have yet to have an RPi user explain why they need raid (includes usb raid enclosure). Raid is not backup and you can't honestly expect an RPi to be highly available which is the reason you use raid??

    omv 5.6.13 usul | 64 bit | 5.11 proxmox kernel | omvextrasorg 5.6.2 | kvm plugin 5.1.6
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • I have yet to have an RPi user explain why they need raid (includes usb raid enclosure). Raid is not backup and you can't honestly expect an RPi to be highly available which is the reason you use raid??

    Now we are getting to the bottom ;-)


    From my view the goal for this setup is NOT high availability, but only a highly likely (say i.e. 99% or similar) protection from losing data due to failing of a single disc drive via normal aging.

    The benefit is that achieving this high safety level is very cheap, when compared to other high availability approaches. In other words a good balance between risk, available storage space, technical complexity and money spend.

    Does this make sense?

    omv 5.6.16-1 (usul) on RPi4/4GB with Kernel 5.10.63 and WittyPi 3 V2 RTC HAT

    2x 6TB HDD formatted with ext4 in Icy Box IB-RD3662-C31 / hardware supported RAID1

    For Read/Write performance of SMB shares hosted on this hardware see forum here

  • but only a highly likely (say i.e. 99% or similar) protection from losing data due to failing of a single disc drive via normal aging.

    The problem is, that RAID does not offer real protection against this scenario. A failure of a drive is very often a continous process of accumulating errors instead dead-drop of a disk. During this slow failure data might get corrupted. A RAID will not protect you from that, because the corruption will be mirrored to the other disks. What you need is a versioned backup, which can be rolled back to a state where the files are not corrupted.

    Also if you have a faulty HW-RAID controller you might need exactly the same enclosure to get your RAID up again...

  • Does this make sense?

    Nope because rsync can do that same thing with much more reliable setup. I think I can say all RPi users do not need real time syncing between two drives.

    omv 5.6.13 usul | 64 bit | 5.11 proxmox kernel | omvextrasorg 5.6.2 | kvm plugin 5.1.6
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • During this slow failure data might get corrupted. A RAID will not protect you from that, because the corruption will be mirrored to the other disks.

    Well my experience, from a DC with 3000 servers since the last 3 years, what you describe is an extremely rare case in the real world!

    Also in the real world, there is never a 100% reliable solution for each and every failure scenario.

    The maximum commercially affordable solution gives 99.999% reliability.

    Again, we are just looking for protection against 99% of the most common faults at a low cost.

    omv 5.6.16-1 (usul) on RPi4/4GB with Kernel 5.10.63 and WittyPi 3 V2 RTC HAT

    2x 6TB HDD formatted with ext4 in Icy Box IB-RD3662-C31 / hardware supported RAID1

    For Read/Write performance of SMB shares hosted on this hardware see forum here

  • rsync can do that same thing with much more reliable setup.

    I'm not familiar with rsync yet, OMV documentation only mentions '

    RSync

    The server can be configured to act as a client to pull and push data to remote locations'


    Maybe you wanted to elaborate or point to a better example how rsync would achieve the stated goal "protect from single disc failure"?

    omv 5.6.16-1 (usul) on RPi4/4GB with Kernel 5.10.63 and WittyPi 3 V2 RTC HAT

    2x 6TB HDD formatted with ext4 in Icy Box IB-RD3662-C31 / hardware supported RAID1

    For Read/Write performance of SMB shares hosted on this hardware see forum here

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!