Please help - Missing file system after power outage

vazquezjm · 5. Dezember 2022

Hi.

Power went off for a couple of hours and when back, I see one of the filesystems is missing

Output of xmlstarlet sel -t -c "//fstab" /etc/openmediavault/config.xml

Code

<fstab>
    <!--
                        <mntent>
                                <uuid>xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx</uuid>
                                <fsname>xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx|xxxx-xxxx|/dev/xxx</fsname>
                                <dir>/xxx/yyy/zzz</dir>
                                <type>none|ext2|ext3|ext4|xfs|jfs|iso9660|udf|...</type>
                                <opts></opts>
                                <freq>0</freq>
                                <passno>0|1|2</passno>
                                <hidden>0|1</hidden>
                                <usagewarnthreshold>xxx</usagewarnthreshold>
                                <comment>xxx</comment>
                        </mntent>
                        -->
    <mntent>
        <uuid>739592c1-4c0d-4b37-b06c-b5199e8dc5f8</uuid>
        <fsname>/dev/disk/by-uuid/244d688e-20da-4db2-8c5c-bab621879958</fsname>
        <dir>/srv/dev-disk-by-uuid-244d688e-20da-4db2-8c5c-bab621879958</dir>
        <type>btrfs</type>
        <opts>defaults,nofail</opts>
        <freq>0</freq>
        <passno>2</passno>
        <hidden>0</hidden>
        <usagewarnthreshold>85</usagewarnthreshold>
        <comment>disk1</comment>
    </mntent>
    <mntent>
        <uuid>c731323f-ad08-4414-82e1-babd4b8baccd</uuid>
        <fsname>/dev/disk/by-uuid/6a63fabf-9982-4444-a9c3-6d477d2c26f7</fsname>
        <dir>/srv/dev-disk-by-uuid-6a63fabf-9982-4444-a9c3-6d477d2c26f7</dir>
        <type>btrfs</type>
        <opts>defaults,nofail</opts>
        <freq>0</freq>
        <passno>2</passno>
        <hidden>0</hidden>
        <usagewarnthreshold>85</usagewarnthreshold>
        <comment>disk2</comment>
    </mntent>
    <mntent>
        <uuid>0730d707-4449-462f-ba08-c51b511ca628</uuid>
        <fsname>92148cdc-6266-4109-b810-3a2c789ec6bb</fsname>
        <dir>/srv/mergerfs/mainpool</dir>
        <type>fuse.mergerfs</type>
        <opts />
        <freq>0</freq>
        <passno>0</passno>
        <hidden>1</hidden>
        <usagewarnthreshold>0</usagewarnthreshold>
        <comment />
    </mntent>
</fstab>

Alles anzeigen

Any ideas would be very welcomed!

geaves · 6. Dezember 2022

Zitat von vazquezjm

Any ideas would be very welcomed

This might be a corrupted file system due to the power outage, you could try mount -a from the cli, if that doesn't do anything then you may have to research fixing a btrfs file system.

On an EXT4 drive one would run fsck on an unmounted drive to fix file system errors.

vazquezjm · 6. Dezember 2022

More on this

Result of lsblk

Code

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 223.6G  0 disk
├─sda1   8:1    0   512M  0 part /boot/efi
├─sda2   8:2    0 222.1G  0 part /
└─sda3   8:3    0   976M  0 part [SWAP]
sdb      8:16   0   3.6T  0 disk
sdc      8:32   0   3.6T  0 disk
└─sdc1   8:33   0   3.6T  0 part /srv/dev-disk-by-uuid-6a63fabf-9982-4444-a9c3-6d477d2c26f7

Result of fdisk -l /dev/sdb

Code

Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFAX-68J
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Result of fdisk -l /dev/sdc

Code

Disk /dev/sdc: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFAX-68J
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 16D5CD82-FFDB-44A0-BDCA-9780250747DF

Device     Start        End    Sectors  Size Type
/dev/sdc1   2048 7814037134 7814035087  3.6T Linux filesystem

What's the difference between fdisk outputs? Also for lsblk? Is the disk unmounted?

geaves · 6. Dezember 2022

Zitat von vazquezjm

What's the difference between fdisk outputs? Also for lsblk? Is the disk unmounted

The partition is missing on /dev/sdb, this would confirm what I posted in post 2

Soma · 6. Dezember 2022

Zitat von vazquezjm

What's the difference between fdisk outputs? Also for lsblk? Is the disk unmounted?

The difference is that your disk sdb lost it's partition table.
fdisk doesn't recognize it, at least.

If it was unmounted it would show something like this on lsblk:

Code

pi@xxxxxx:~ $ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   3.6T  0 disk
└─sda1   8:1    0   3.6T  0 part /srv/dev-disk-by-label-wolf1
sdb      8:16   0   3.6T  0 disk
└─sdb1   8:17   0   3.6T  0 part

As for fdisk, it would show the same as the other since they're both the same Brand and Model:

Something like this:

Code

pi@xxxxxx:~ $ sudo fdisk -l /dev/sda
Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: 008-2DR166
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: BD573C08-8962-451E-84E7-DE1A5E64E8A9

Device     Start        End    Sectors  Size Type
/dev/sda1   2048 7814037134 7814035087  3.6T Linux filesystem


pi@xxxxxx:~ $ sudo fdisk -l /dev/sdb
Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: 008-2DR166
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 12B73770-E447-45DC-91E4-D02850F36C36

Device     Start        End    Sectors  Size Type
/dev/sdb1   2048 7814037134 7814035087  3.6T Linux filesystem

Alles anzeigen

The sad news is that your disk has lost it's partition table.

This is the same as a blank disk brand new that wasn't even initialized.

Since you gave few to none info, now the question:

Do you have a backup of the DATA that existed there?

vazquezjm · 7. Dezember 2022

Thank you Soma and geaves !

For some reason, I'm not receiving email notifications for forum posts.

I tried a couple of things I don't even remember now. The truth is sdb was unmounted as you said, so I went ahead and mounted it from the UI.

Thankfully, nothing was lost and mergerfs started flawlessly after that.

vazquezjm · 7. Dezember 2022

Zitat von Soma

Do you have a backup of the DATA that existed there?

Since I was able to recover from failure, I have two topics to investigate now:

Backups (I'm using UrBackup for my PCs, which data lives on the Union FS)
How to shutdown the system from a remote host (which has a UPS attached to it)

geaves · 7. Dezember 2022

Zitat von vazquezjm

Backups (I'm using UrBackup for my PCs, which data lives on the Union FS)

Not something I would do, I would direct the backups to a single drive, how have you deployed UrBackup, I'm asking because I did set mine up in docker, then I think it was macom said that he deploys his directly on OMV and it runs as a service. I have done this and I find it more intuitive than the docker version.

Zitat von vazquezjm

How to shutdown the system from a remote host (which has a UPS attached to it)

Pass probably Soma territory that one

Soma · 7. Dezember 2022

Check the guides section.

There's a nut plugin guide from chente

Might give you a solution

vazquezjm · 7. Dezember 2022

Zitat von geaves

Not something I would do, I would direct the backups to a single drive, how have you deployed UrBackup, I'm asking because I did set mine up in docker, then I think it was macom said that he deploys his directly on OMV and it runs as a service. I have done this and I find it more intuitive than the docker version.

I've using UrBackup for more than a year now (Docker) and very used to it. When I migrated to OMV, I kept using the Docker version and no problems whatsoever.

Zitat von Soma

Check the guides section.
There's a nut plugin guide from chente

Might give you a solution

Will try that.

I'm running Home Assistant on a separate host and I automate the shutdown of it based on certain thresholds. Ideally, when the power goes off, I'd shut down OMV gracefully not to have these problems.

Soma · 7. Dezember 2022

If the UPS system is the HA, I can't help you since I don't know how it runs.

If it has nut on it (which should have or similar program) then you need to configure it to have remote monitoring available.

If both instances run on OMV than it's as easy as (with the openmediavault-nut plugin)

Configure the instance with the UPS as Standalone and set the grace period as you see fit (I have on UPS reaches low battery setting).

Enable remote monitoring on it and set a user:password (don't use a USER of the system, this is only for UPS monitoring) on the proper field.

Enable the plugin on the remote system as a Netclient and set all required fields to match the UPS server.

Set the grace period as you see fit.

Don't enable remote monitoring.

When power goes down, system will shutdown in according to the set grace period of each system individually.

This is in the guide I talked about:

[How-to] Use the openmediavault-nut plugin - Guides - openmediavault

vazquezjm · 7. Dezember 2022

Zitat von Soma

If the UPS system is the HA, I can't help you since I don't know how it runs.
If it has nut on it (which should have or similar program) then you need to configure it to have remote monitoring available.

I guess it should work

Soma · 7. Dezember 2022

Zitat von vazquezjm

I guess it should work

Yes, it should.

Then configure the remote monitoring on it.
Maybe the "slave" meaning in there.

You'll need to find that on the HA forums/support or maybe someone who is already using (HA) can give an input on how to set it.

Good luck

vazquezjm · 7. Dezember 2022

Zitat von Soma

Yes, it should.

Confirming it works! Thanks for the tip

Please help - Missing file system after power outage

Jetzt mitmachen!

Tags