Attempt to expand ZFS corrupted metadata and now pool is not accessible. HALP.

goblest · 18. Mai 2017

Ver 3.0.76

I added 3 disks and attempted an expand using the GUI. I received a message stating there was an error and recommend a reboot before any other action. After the reboot, the pool is not available and I cannot re-import it. I see all the disks available but ZFS cannot read a label.

During my troubleshooting, I upgraded the system. This started on 2.x and is now on 3.x. None of the information below has changed between versions.

Here is some data:

Bash

root@omv:~# zdb -l /dev/sdg
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
    version: 5000
    name: 'data-pool'
    state: 0
    txg: 7660266
    pool_guid: 6643558300144223209
    errata: 0
    hostname: 'omv'
    top_guid: 10820414310469970679
    guid: 10820414310469970679
    vdev_children: 5
    vdev_tree:
        type: 'disk'
        id: 2
        guid: 10820414310469970679
        path: '/dev/sdg'
        whole_disk: 0
        metaslab_array: 173
        metaslab_shift: 33
        ashift: 9
        asize: 1099506909184
        is_log: 0
        DTL: 598
        create_txg: 3877085
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
--------------------------------------------
LABEL 2
--------------------------------------------
    version: 5000
    name: 'data-pool'
    state: 0
    txg: 7660266
    pool_guid: 6643558300144223209
    errata: 0
    hostname: 'omv'
    top_guid: 10820414310469970679
    guid: 10820414310469970679
    vdev_children: 5
    vdev_tree:
        type: 'disk'
        id: 2
        guid: 10820414310469970679
        path: '/dev/sdg'
        whole_disk: 0
        metaslab_array: 173
        metaslab_shift: 33
        ashift: 9
        asize: 1099506909184
        is_log: 0
        DTL: 598
        create_txg: 3877085
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
--------------------------------------------
LABEL 3
--------------------------------------------
    version: 5000
    name: 'data-pool'
    state: 0
    txg: 7660266
    pool_guid: 6643558300144223209
    errata: 0
    hostname: 'omv'
    top_guid: 10820414310469970679
    guid: 10820414310469970679
    vdev_children: 5
    vdev_tree:
        type: 'disk'
        id: 2
        guid: 10820414310469970679
        path: '/dev/sdg'
        whole_disk: 0
        metaslab_array: 173
        metaslab_shift: 33
        ashift: 9
        asize: 1099506909184
        is_log: 0
        DTL: 598
        create_txg: 3877085
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data

Alles anzeigen

Bash

root@omv:~# zdb
data-pool:
    version: 5000
    name: 'data-pool'
    state: 0
    txg: 7660266
    pool_guid: 6643558300144223209
    errata: 0
    hostname: 'omv'
    vdev_children: 5
    vdev_tree:
        type: 'root'
        id: 0
        guid: 6643558300144223209
        children[0]:
            type: 'disk'
            id: 0
            guid: 8732353070277557521
            path: '/dev/cciss/c0d1p2'
            whole_disk: 1
            metaslab_array: 38
            metaslab_shift: 33
            ashift: 12
            asize: 878538653696
            is_log: 0
            DTL: 208
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 3377182915721787660
            path: '/dev/cciss/c1d0p2'
            whole_disk: 1
            metaslab_array: 35
            metaslab_shift: 33
            ashift: 12
            asize: 1025320419328
            is_log: 0
            DTL: 209
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 10820414310469970679
            path: '/dev/sdg'
            whole_disk: 0
            metaslab_array: 173
            metaslab_shift: 33
            ashift: 9
            asize: 1099506909184
            is_log: 0
            DTL: 598
            create_txg: 3877085
        children[3]:
            type: 'disk'
            id: 3
            guid: 12104046298170893874
            path: '/dev/sdh'
            whole_disk: 0
            metaslab_array: 324
            metaslab_shift: 33
            ashift: 9
            asize: 1099506909184
            is_log: 0
            DTL: 597
            create_txg: 3877100
        children[4]:
            type: 'disk'
            id: 4
            guid: 3777217097941026287
            path: '/dev/sdi'
            whole_disk: 0
            metaslab_array: 326
            metaslab_shift: 33
            ashift: 9
            asize: 1099506909184
            is_log: 0
            DTL: 596
            create_txg: 3877115
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data

Alles anzeigen

Bash

root@omv:~# zpool import -a
cannot import 'data-pool': one or more devices is currently unavailable
root@omv:~#

Bash

root@omv:~# zpool import -d /dev
   pool: data-pool
     id: 6643558300144223209
  state: FAULTED
 status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: http://zfsonlinux.org/msg/ZFS-8000-72
 config:


        data-pool  FAULTED  corrupted data
          sdg        ONLINE
          sdh        ONLINE
          sdi        ONLINE

Alles anzeigen

Bash

root@omv:~# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/cciss/c0d0p1 during installation
UUID=d888352f-ec5f-4c46-b993-88e5a8d8a622 /               ext4    errors=remount-ro 0       1
# swap was on /dev/cciss/c0d0p5 during installation
UUID=80740f58-f169-486b-80ad-d87fee4b43a6 none            swap    sw              0       0
/dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
tmpfs           /tmp            tmpfs   defaults        0       0
# >>> [openmediavault]
/data-pool/share01/media /export/media none bind,nofail,_netdev 0 0
# <<< [openmediavault]

Alles anzeigen

Any help would be very much appreciated.

geaves · 23. Mai 2017

The information from zpool import points to here I don't have zfs running, but the message 'see' may help fix the problem.

goblest · 23. Mai 2017

Zitat von geaves

The information from zpool import points to here I don't have zfs running, but the message 'see' may help fix the problem.

Thanks. I did go there and it was no help.

geaves · 23. Mai 2017

Zitat von goblest

Thanks. I did go there and it was no help.

So did you run the zpool status -x? I was also reading somewhere that there had been either an update or change to the zfs plugin....is the same plugin compatible with both versions of OMV.....it 'could' also be the upgrade that has in some way caused the error.

goblest · 23. Mai 2017

Zitat von geaves

So did you run the zpool status -x? I was also reading somewhere that there had been either an update or change to the zfs plugin....is the same plugin compatible with both versions of OMV.....it 'could' also be the upgrade that has in some way caused the error.

I can actually wipe it and go back to 2.x if needed but doubt it will help. Any other command involving the pool status will return a message that states the pool is unknown as it is not presented. So there is no "status".

I have a feeling it is lost but keep fighting. ..hate to do all that work again.

ryecoaaron · 23. Mai 2017

Zitat von goblest

I can actually wipe it and go back to 2.x if needed but doubt it will help.

Not really. zfs isn't supported on Debian Wheezy anymore (never really was) so the zfs stuff has been pulled from OMV 2.x. The plugin isn't the problem either. The zfs packages might be.

goblest · 23. Mai 2017

Zitat von ryecoaaron

Not really. zfs isn't supported on Debian Wheezy anymore (never really was) so the zfs stuff has been pulled from OMV 2.x. The plugin isn't the problem either. The zfs packages might be.

Well that's odd....I was on a stable working version when this all started....so...that is weird. The issue is the label on 0, I am sure....just no way to fix. Thanks for trying though.

ryecoaaron · 23. Mai 2017

Zitat von goblest

Well that's odd....I was on a stable working version when this all started....so...that is weird. The issue is the label on 0, I am sure....just no way to fix. Thanks for trying though.

Thanks for being a smart ass. I said supported not working. If you installed the packages when ZoL still had their repo up for wheezy, then you could install it. Now it is gone.

I was just stating not to go back to OMV 2.x because you will have no zfs instead of label on 0 problem and that the plugin didn't cause this. But what do I know...

goblest · 23. Mai 2017

Zitat von ryecoaaron

Thanks for being a smart ass. I said supported not working. If you installed the packages when ZoL still had their repo up for wheezy, then you could install it. Now it is gone.
I was just stating not to go back to OMV 2.x because you will have no zfs instead of label on 0 problem and that the plugin didn't cause this. But what do I know...

No. I understand and wasn't being a smartass.

Jetzt mitmachen!