Recovering from a power surge

  • Today we had a power surge at my house. Usually I don't care because I have an UPS that handles it. My apsupsd was programmed to do a soft shutdown at 80% remaining battery. Unfortunately this time something went bad. The surge was strange, like a power drop, my work computer (which is not connected to the UPS) suddenly restarted, it was like a burst. To my surprise I receive a monitoring alert that my NAS was down, and when I check it was shutdown, which is strange because soft shutdown shouldn't have been triggered in this case.


    I started the NAS again and it never booted, it stays in a black screen with the cursor blinking. I removed my RAID1 disks and my SSD which is mounted in a USB case, and tried to recover the partition using my work computer, with no luck at all.

    Sadly, my OMV backups are inside the RAID1 (FSarchiver backups) so I mounted the raid1 using systemrescuecd, so I can recover the backup and dump it to the SSD.

    I recovered the backup successfully, but then again black screen, it didn't solve the issue at all. I'm now reinstalling OMV from scratch, but really it's puzzling, maybe my SSD has been damaged?.


    I will keep the backups and probably try to restore them later, but really does anyone had a similar experience?.


    Regards

  • I had a similar experience in the past when I was a kid and didn't had a UPS.

    Spikes like these can break your entire system and you're probably aware of this, but sometimes I've seen corrupted partitions/mbr during similar events.


    This is an event I'm accounting for, that's why I installed rclone and copy my backups to Google Drive every day after they're created.


    Here's my own-written procedure about. if you need more in-depth info let me know.


    OMV BUILD - MY NAS KILLER - OMV 5.x + omvextrasorg


    Core i3-8300 - ASRock H370M-ITX/ac - 8GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker), 3x4TB HDD (Data) - Fractal Design Node 304 - Be quiet! Pure Power 11 350W


  • The issue is that I do have a UPS. Maybe the min voltage is too low and it didn't kick in, I'm looking to change that now. But clearly the UPS failed to manage this surge properly.


    If someone knows how to change the LOTRANS value from an APC UPS I will be grateful.


    Ill probably upload at least the last backup to Gsuite, unfortunately my 5 Mbps upload is quite poor and it takes time.

  • But clearly the UPS failed to manage this surge properly.

    I'm no UPS expert, but is it "healthy"?

    Blackouts are an easy task, but spikes from events like thunderstorms or network issues are unpredictable, although an UPS should manage it!


    Ill probably upload at least the last backup to Gsuite, unfortunately my 5 Mbps upload is quite poor and it takes time.

    You could use borbackup method instead of dd/fsarchiver. The first archive would be big as normal, subsequent backups only being with changed files.

    However I do not recommend this because Borg is not made to backup filesystems but only user files. I tried to restore borgbackups in test VM and did not work well at all.


    If you're using dd, switch to fsarchiver: only backs up the actual used space and compresses it. My fsarchiver backup is 1.5GB, down from 3.5GB. It would take 45 mins with your connection to upload it, maybe you can shedule it in the night.


    Otherwise run a SMB/NFS share from a USB stick connected to your router and upload it there. Rclone supports everything.

    OMV BUILD - MY NAS KILLER - OMV 5.x + omvextrasorg


    Core i3-8300 - ASRock H370M-ITX/ac - 8GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker), 3x4TB HDD (Data) - Fractal Design Node 304 - Be quiet! Pure Power 11 350W

  • It's healthy yes, maybe it's not up to the task.


    About the backup: yeah I'll probably end up plugging a usb once in a while and doing an autobackup when the usb is detected, something like a cold backup.

  • About the backup: yeah I'll probably end up plugging a usb once in a while and doing an autobackup when the usb is detected, something like a cold backup.

    I do this every week as part of my 3-2-1 backup strategy for my data.

    OMV BUILD - MY NAS KILLER - OMV 5.x + omvextrasorg


    Core i3-8300 - ASRock H370M-ITX/ac - 8GB RAM - Sandisk Ultra Flair 32GB (OMV), 256GB NVME SSD (Docker), 3x4TB HDD (Data) - Fractal Design Node 304 - Be quiet! Pure Power 11 350W

  • UPS and surge suppressing power strips have devices in them for clamping high frequency, high voltage, short duration spikes to ground. A "varistor" is one type of component that is commonly used. A varistor is a special type of resister that is connected to the incoming line and ground. For this purpose, they have near infinite resistance at low voltage. When voltage goes up to a "break over point", resistance drops to zero and shorts the high voltage spike to ground.

    Unfortunately, these devices have limited current carrying capacity so, in an attempt to save devices from a surge, they may be sacrificed. And it's worth noting that they may fail to do the job - the protection has limits

    There should be some indicator of whether surge suppression is still working or not.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!