Recovering from a power surge

Today we had a power surge at my house. Usually I don't care because I have an UPS that handles it. My apsupsd was programmed to do a soft shutdown at 80% remaining battery. Unfortunately this time something went bad. The surge was strange, like a power drop, my work computer (which is not connected to the UPS) suddenly restarted, it was like a burst. To my surprise I receive a monitoring alert that my NAS was down, and when I check it was shutdown, which is strange because soft shutdown shouldn't have been triggered in this case.

I started the NAS again and it never booted, it stays in a black screen with the cursor blinking. I removed my RAID1 disks and my SSD which is mounted in a USB case, and tried to recover the partition using my work computer, with no luck at all.

Sadly, my OMV backups are inside the RAID1 (FSarchiver backups) so I mounted the raid1 using systemrescuecd, so I can recover the backup and dump it to the SSD.

I recovered the backup successfully, but then again black screen, it didn't solve the issue at all. I'm now reinstalling OMV from scratch, but really it's puzzling, maybe my SSD has been damaged?.

I will keep the backups and probably try to restore them later, but really does anyone had a similar experience?.

Regards

Reinstalled and it's working now. Hopefully it will never happen again.

Zitat von auanasgheps

I had a similar experience in the past when I was a kid and didn't had a UPS.

Spikes like these can break your entire system and you're probably aware of this, but sometimes I've seen corrupted partitions/mbr during similar events.

This is an event I'm accounting for, that's why I installed rclone and copy my backups to Google Drive every day after they're created.

Here's my own-written procedure about. if you need more in-depth info let me know.

Spoiler anzeigen

• Install rclone apt install rclone
• If you already have a config file, place it in /root/.config/rclone, otherwise run rclone config to run a new config wizard. Docs about supported storage systems can be found here.
• Ideally, logs should go to /var/log/. Also don't forget to create a logrotate config for rclone otherwise the logfile will become huge since rclone writes a lot of data.
• Once configs are all ready to go, create a scheduled rClone job in OMV GUI to actually execute the sync when you want.

Alles anzeigen

The issue is that I do have a UPS. Maybe the min voltage is too low and it didn't kick in, I'm looking to change that now. But clearly the UPS failed to manage this surge properly.

If someone knows how to change the LOTRANS value from an APC UPS I will be grateful.

Ill probably upload at least the last backup to Gsuite, unfortunately my 5 Mbps upload is quite poor and it takes time.

Zitat von auanasgheps

I'm no UPS expert, but is it "healthy"?

Blackouts are an easy task, but spikes from events like thunderstorms or network issues are unpredictable, although an UPS should manage it!

You could use borbackup method instead of dd/fsarchiver. The first archive would be big as normal, subsequent backups only being with changed files.

However I do not recommend this because Borg is not made to backup filesystems but only user files. I tried to restore borgbackups in test VM and did not work well at all.

If you're using dd, switch to fsarchiver: only backs up the actual used space and compresses it. My fsarchiver backup is 1.5GB, down from 3.5GB. It would take 45 mins with your connection to upload it, maybe you can shedule it in the night.

Otherwise run a SMB/NFS share from a USB stick connected to your router and upload it there. Rclone supports everything.

Alles anzeigen

It's healthy yes, maybe it's not up to the task.

About the backup: yeah I'll probably end up plugging a usb once in a while and doing an autobackup when the usb is detected, something like a cold backup.