Disaster scenario with rclone (hypothetical)

Hi! I'm new to OMV, migrating from TrueNAS and bringing my zfs tanks along.

I've been setting up my "data protection" scheduled tasks today, using Docker rclone image and it got me thinking.

Let's say I have a local storage of very valuable information (e.g. digitalised movies from 1960) on a RAID1 array. A daily cron job makes a backup of that using "docker run -v /mnt/movies:/data ... rclone ..." to a remote location. I think it's not bad, so far, probably what most people would do?

Then while I'm on vacation a power surge (or a cat) kills both drive bays with my RAID1, which OMV informs me about over email. No problem I think, there's a backup. The server reboots, and later that day, runs the rclone command. Since docker doesn't mind that "/mnt/movies" is now empty (or non-existent), it happily erases the content of my remote backup.

Clearly, returning from vacation and trying to enjoy a movie night of "Dr No", I am up for a disappointment.

Did I miss the plot somewhere? Is this something that people on this forum have a remedy for?

While at it, I'm also curious how others set up their rclone tasks. Perhaps some examples, how does one get error notifications, track when it last succeeded, spread them automatically over hours (rather than concentrating all tasks at midnight), finally, take an automated filesystem snapshot before cloning?

Thank you for your answers! Really appreciate fast and educational response. Let me summarise what I understood:

1. For data you really care about, do backups manually. That makes sense but a manual backup cannot be as fast & frequent as RAID1. Single disk failure while actively working on a data set is still very possible. Besides, we would depend on a human in the pipeline, and that's usually the weakest part. I assume one cannot trust a (tired? stressed? sick? simply distracted?) human to always notice they are doing something wrong, including erasing entire backup media.

2. Use versioned backup. Yes! But even having a year of versioned backups means that after a year since Event X they are as good as dead. That is, unless a human notices earlier that something is missing. But we established by now that a human can not be trusted. There are movies I can live a couple years without watching, still want them close by.

These are indeed existential issues of building a backup system. But they are still not moving me forward in solving my specific problem of an empty mount (if not for a power surge... the maybe due to misconfiguration, which is so far much more likely than ransomware, given my current track record). Any ideas for a simple solution to that, while I'm slowly convincing myself to go the 3-disk route suggested by ryecoaaron ?

Alright, thank you for the collective wisdom and ideas. To close on this thread, that's how I incorporated them in my rclone schedule to solve my immediate problem:

1. Set rclone --max-delete to prevent disastrous consequences in one rclone session.

2. Prepend

diff <(md5sum some/important/file) <(echo 'some/important/file 012345abcdef') &&

to rclone command to guard for a specific file modification (unlikely to modify my movie collection very often).

For anyone reading this thread to find the solution: this isn't a mitigation for all possible failure scenarios. As macom says, there's a risk/effort balance you need to find for your peace of mind.