Problem after rolling-upgrade/failed disks/resync

  • Hi everyone!


    The story is a bit of a long one:
    I ran a raid 5 with 3 disks and one spare (which is now a raid 6). Since space grew to small I decided to upgrade via the good ol' fail-remove-add-resync-repeat-grow method. Takes me a while because of work and uni to get the time but i digress...


    The last disk I added ended up in a different array (I have no idea how i managed that...) and then one of my disks failed. I replaced it, resynced, noticed that the previous disk was in the wrong array, deleted that, added the disk to the right one, resynced, ... somewhere in there a second disk failed and it took a bit of disk juggeling and about 5 resync attempts but now i have a raid 6 with 4 disks.


    Now to the problems part:


    #1:
    While booting I get an error from fsck, that it can't resolve "UID-<Hexstring>" what appears to be the old mountpoint of my raid.
    I found some info on the web how to repair that, but it somehow interfered with my other problems.


    #2:
    My raid now consists of 4 disks and no spares.
    I made a mistake here while adding the last disk... is it possible to somehow convert one disk to a spare or should i just roll with it and fill the last empty hdd-slot in my server?


    #3:
    I can't mount my raid. And i have no idea why and how to fix it.


    So I'm really desperate and any help would be greatly appreciated.


    Some info:




    Code
    root@sarah:~# more /proc/mdsstat
    /proc/mdsstat: No such file or directory
    root@sarah:~# more /proc/mdstat
    Personalities : [raid6] [raid5] [raid4]
    md0 : active (auto-read-only) raid6 sdd[4] sda[3] sdc[5] sdb[1]
          3906765824 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
    
    
    unused devices: <none>




    Code
    root@sarah:~# ls /media
    3552a985-99e8-4bbb-b958-e988a139541a  8C70884E70884146  cdrom  usb  usb0
  • I run a raid 6 with 3 disks and one spare.


    My Initial reaction. "Yup. And my Car runs on three wheels while the fourth is in the trunk, just in case."


    Greetings
    David

    "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"


    Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.

    Upload Logfile via WebGUI/CLI
    #openmediavault on freenode IRC | German & English | GMT+1
    Absolutely no Support via PM!

  • My Initial reaction. "Yup. And my Car runs on three wheels while the fourth is in the trunk, just in case."


    Greetings
    David


    Since i already acknowledged my mistake and now also edited the initial post to reflect that, did you have any additional reactions?

  • It may be your config.xml which is a bit messed up.


    Also: I can't mount the array is no precise error message. Please show a screenshot or post the error message you get.


    I don't know if you can shrink from raid6 to raid5. Never searched for it.


    Greetings
    David

    "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"


    Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.

    Upload Logfile via WebGUI/CLI
    #openmediavault on freenode IRC | German & English | GMT+1
    Absolutely no Support via PM!

  • Thanks David.


    The filesystems are definetly messed up:



    Mounting looks like this:

    Code
    root@sarah:/media# mount -t ext4 /dev/md0 3552a985-99e8-4bbb-b958-e988a139541a/
    mount: wrong fs type, bad option, bad superblock on /dev/md0,
           missing codepage or helper program, or other error
           In some cases useful info is found in syslog - try
           dmesg | tail  or so


    But i must admit that i'm not 100% sure that it was an ext4 fs.


    I never had to manually mount it before.



    Edit:
    Also I will just stick with the raid 6 and a cold spare.

  • Wait for a reaction from @ryecoaaron, he has more expertise with software raids than I have.


    Greetings
    David

    "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"


    Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.

    Upload Logfile via WebGUI/CLI
    #openmediavault on freenode IRC | German & English | GMT+1
    Absolutely no Support via PM!

    • Offizieller Beitrag

    What is the current output of:


    cat /proc/mdstat
    blkid
    cat /etc/fstab

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Code
    root@sarah:/media# cat /proc/mdstat
    Personalities : [raid6] [raid5] [raid4]
    md0 : active (auto-read-only) raid6 sdd[4] sda[3] sdc[5] sdb[1]
          3906765824 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
    
    
    unused devices: <none>


    Code
    root@sarah:/media# blkid
    /dev/sdb: UUID="504814b6-57d2-0712-8ba2-02ee583d3e5c" UUID_SUB="efa802aa-a1d4-3e15-a193-0640f496606f" LABEL="sarah:0" TYPE="linux_raid_member"
    /dev/sde1: UUID="b685be16-6d19-4cdb-bad8-ec244542f1c6" TYPE="ext4"
    /dev/sde5: UUID="40fefd47-da7d-4a34-8733-37992af43472" TYPE="swap"
    /dev/sdd: UUID="504814b6-57d2-0712-8ba2-02ee583d3e5c" UUID_SUB="4338ebe9-64d3-9598-6230-2747532713fd" LABEL="sarah:0" TYPE="linux_raid_member"
    /dev/md0: UUID="504814b6-57d2-0712-8ba2-02ee583d3e5c" UUID_SUB="4733238f-dab5-f4e3-fc15-2cfbf8fc1169" LABEL="sarah:0" TYPE="linux_raid_member"
    /dev/sda: UUID="504814b6-57d2-0712-8ba2-02ee583d3e5c" UUID_SUB="d8297fec-3113-af18-9621-91cadb3cc4da" LABEL="sarah:0" TYPE="linux_raid_member"
    /dev/sdc: UUID="504814b6-57d2-0712-8ba2-02ee583d3e5c" UUID_SUB="f83c93e5-85c6-6b0b-a5f5-dea6c6c6b9d4" LABEL="sarah:0" TYPE="linux_raid_member"


    • Offizieller Beitrag

    Your blkid output shows the raid array itself as a member of the array. It shouldn't be. I would try the following:


    umount /dev/md0
    mdadm --stop /dev/md0
    mdadm --assemble /dev/md127 /dev/sd[abcd] --verbose --force


    Then post the output of: cat /proc/mdstat again

  • Thank you for your time ryecoaaron.


    Code
    root@sarah:/media# cat /proc/mdstat
    Personalities : [raid6] [raid5] [raid4]
    md127 : active (auto-read-only) raid6 sdd[4] sda[3] sdc[5] sdb[1]
          3906765824 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
    
    
    unused devices: <none>


    blkid still lists md127 as linux_raid_member

    • Offizieller Beitrag

    You must have added it to an array or something. So, it has a superblock on it now. I wouldn't zero because you would probably lose everything.


    Strange the array is read only... I would reboot. Then try to mount the filesystem in the web interface. If it doesn't show up, it is probably lost.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Sometime in between the last two or three resyncs it worked for an evening (i came home after a night out and switched on music which i stream from the NAS).
    But in the morning the raid was degraded again.
    I checked the history command and dont't find anything that leads me to believe I added /dev/md0 to an array...


    After the reboot the Filesystems look like the screenshot from before

    • Offizieller Beitrag

    You should be able to delete the missing filesystems with the delete button. Otherwise, you will have to manually edit config.xml.


    To check the array, try manually mounting it.


    mkdir /media/temp
    mount /dev/md127 /media/temp
    You should see files in /media/temp/.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • The delete button is greyed out for both filesystems. .xml it is then...


    Code
    root@sarah:/media# mount /dev/md0 /media/temp
    mount: unknown filesystem type 'linux_raid_member'


    So I guess it's fubar?

    • Offizieller Beitrag

    Post the output of: fdisk -l


    I don't anything else to try that won't risk the data. If you want to try the following, it may save it but I warned you :)


    mdadm --zero-superblock /dev/md0
    fsck.ext4 -f /dev/md0

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!


  • I'll continue to google around a bit. Thanks to you i know what to look at and maybe... just maybe some other idiot managed to accomplish the same thing as i did and just a bit maybier it got solved somehow...

    • Offizieller Beitrag

    The good news is that there isn't a partition in fdisk output on the array. So, zeroing the superblock *shouldn't* cause data loss. My commands should be relatively safe.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!


  • Mhm, not at first try.


    Edit:
    I found a howto to recover a corrupt superblock:
    https://linuxexpresso.wordpres…xt4-superblock-in-ubuntu/
    do you think that would help?


    mke2fs -n /dev/md0
    gives me a bunch of backups.


    Or this:
    http://kevin.deldycke.com/2007…ng-zero-ized-superblocks/

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!