Missing FileSystem after RAID5 problem

Diomet · Jul 21st 2021

Hello,

I have been browsing and trying several solutions on this forum without success, so I need your help...

Prerequisites: OMV4, HP Z420 machines with builtin raid, 3HDDs of 4TB. One HDD failing.

At a certain point the raid fall. In GUI no RAID displayed, missing filessystem and only the 3 HDDs visible in Disks tab.

2 raid appeared in mdstat in cli, one with 2 HDDs and one with the third one. I stopped them both and create the old with:

Code

mdadm --verbose --create /dev/md127 --chunk=512 --level=5 --raid-devices=3 /dev/sda /dev/sdb /dev/sdc

After this point I had the RAID5 back in GUI, and after about 400 minutes it shows as clean but Missing files system because of a different name.

OMV has sent the following message: "

Code

Status failed Service mountpoint_srv_dev-disk-by-label-storage
    Description: status failed (1) -- /srv/dev-disk-by-label-storage is not a mountpoint

I noticed that the name is changed and I tried to rename it but instead of having "storage" in OMV GUI is "office:storage", including the hostname of the server.

I tried to mount it manually but is says that is not ext4 as it should be.

Bellow are the requested info on this kind of problems:

Code

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : active (auto-read-only) raid5 sdc[1] sdb[3] sda[0]
      7813774336 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      bitmap: 0/30 pages [0KB], 65536KB chunk

unused devices: <none>



# blkid
/dev/sdb: UUID="8a1e7d73-19e5-3531-f65a-f4dc1d377bdb" UUID_SUB="4ff6b63d-4587-e513-7f21-f0ea60a0d609" LABEL="office:storage" TYPE="linux_raid_member"
/dev/sdc: UUID="8a1e7d73-19e5-3531-f65a-f4dc1d377bdb" UUID_SUB="8b7e01e0-2e86-3632-5f28-a8f9362ae701" LABEL="office:storage" TYPE="linux_raid_member"
/dev/sdd1: UUID="48A1-D254" TYPE="vfat" PARTUUID="554f2b97-5a95-4484-8f4b-709396ac7666"
/dev/sdd2: UUID="5c724eba-2d04-4cac-bfd2-638f9f55c0b2" TYPE="ext4" PARTUUID="963556cf-f1de-4833-a661-fb87f9b7df75"
/dev/sdd3: UUID="0d39422a-174e-45e7-a9eb-f496146e3652" TYPE="swap" PARTUUID="4e3a30fc-ed2b-4508-8b75-b9d2dfb37c96"
/dev/sdd4: UUID="3247bd9e-62d0-4fe0-a52d-1a71e4a8951d" TYPE="ext4" PARTUUID="397b356a-4c87-423d-8276-a04771664ea5"
/dev/sda: UUID="8a1e7d73-19e5-3531-f65a-f4dc1d377bdb" UUID_SUB="125a8656-c4d0-fa91-303a-fdee59e467db" LABEL="office:storage" TYPE="linux_raid_member"



# fdisk -l | grep "Disk "
Disk /dev/sdb: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Disk /dev/sdc: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Disk /dev/sdd: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Disk identifier: 1B160302-AE4B-4528-86D4-C98E8C2716C3
Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Disk /dev/md127: 7.3 TiB, 8001304920064 bytes, 15627548672 sectors



# cat /etc/mdadm/mdadm.conf

DEVICE partitions

CREATE owner=root group=disk mode=0660 auto=yes

HOMEHOST <system>

ARRAY /dev/md/imsm0 metadata=imsm UUID=4dde16bf:fbb8d79b:ef7599f6:b4beea23
ARRAY /dev/md/Volume0_0 container=/dev/md/imsm0 member=0 UUID=2b8e433f:e5839917:1bc0f870:848843bb

MAILADDR xxx@xxx.xx
MAILFROM root



# mdadm --detail --scan --verbose
ARRAY /dev/md/office:storage level=raid5 num-devices=3 metadata=1.2 name=office:storage UUID=8a1e7d73:19e53531:f65af4dc:1d377bdb
   devices=/dev/sda,/dev/sdb,/dev/sdc

Display More

Thank you and advance!

crashtest · Jul 21st 2021

To be honest, your situation doesn't look good. Maybe geaves might have something for you but, here's to hoping you have backup.

geaves · Jul 21st 2021

Quote from Diomet

mdadm --verbose --create

This not good, recreating an array is an absolute last resort

Quote from Diomet

md127 : active (auto-read-only)

This should correct itself with mdadm --readwrite /dev/md127

Diomet · Jul 21st 2021

Quote from geaves

This not good, recreating an array is an absolute last resort

I found that from another reply of yours on another topic...

Quote from geaves

This should correct itself with mdadm --readwrite /dev/md127

Yes, solved. Thank you!

I still have for the automatically mount:

Code

status failed (1) -- /srv/dev-disk-by-label-storage is not a mountpoint

I tried to do it manually and:

Code

~# mount /dev/md127 /mnt
mount: wrong fs type, bad option, bad superblock on /dev/md127,
missing codepage or helper program, or other error


In some cases useful info is found in syslog - try
dmesg | tail or so.

Nothing in dmesg though.

Also, no luck on efsck.

It's a "bye bye" situation, isn't it?

ananas · Jul 21st 2021

What's the output of blkid after the assembly of the raid ?

Did you try to specify the filesystem type when mounting ? (mount -t ext4 /dev/md127 /mnt)

geaves · Jul 21st 2021

Quote from Diomet

I found that from another reply of yours on another topic.

I can only recall suggesting that once with the caveat that it's a last resort.

Quote from Diomet

mount /dev/md127 /mnt

The mount point for OMV is /srv

Is the name displayed in mdadm.conf? Also as per ananas what's the output of blkid

Quote from Diomet

It's a "bye bye" situation, isn't it

Not yet

Quote from Diomet

I tried to rename it

How?

Diomet · Jul 22nd 2021

Quote from geaves

Is the name displayed in mdadm.conf? Also as per ananas what's the output of blkid

No, it is not.

The name should be "storage" and mounted "by label"

Code

~# blkid
/dev/sdc: UUID="8a1e7d73-19e5-3531-f65a-f4dc1d377bdb" UUID_SUB="8b7e01e0-2e86-3632-5f28-a8f9362ae701" LABEL="office:storage" TYPE="linux_raid_member"
/dev/sdd1: UUID="48A1-D254" TYPE="vfat" PARTUUID="554f2b97-5a95-4484-8f4b-709396ac7666"
/dev/sdd2: UUID="5c724eba-2d04-4cac-bfd2-638f9f55c0b2" TYPE="ext4" PARTUUID="963556cf-f1de-4833-a661-fb87f9b7df75"
/dev/sdd3: UUID="0d39422a-174e-45e7-a9eb-f496146e3652" TYPE="swap" PARTUUID="4e3a30fc-ed2b-4508-8b75-b9d2dfb37c96"
/dev/sdd4: UUID="3247bd9e-62d0-4fe0-a52d-1a71e4a8951d" TYPE="ext4" PARTUUID="397b356a-4c87-423d-8276-a04771664ea5"
/dev/sdb: UUID="8a1e7d73-19e5-3531-f65a-f4dc1d377bdb" UUID_SUB="4ff6b63d-4587-e513-7f21-f0ea60a0d609" LABEL="office:storage" TYPE="linux_raid_member"
/dev/sda: UUID="8a1e7d73-19e5-3531-f65a-f4dc1d377bdb" UUID_SUB="125a8656-c4d0-fa91-303a-fdee59e467db" LABEL="office:storage" TYPE="linux_raid_member"

Quote from geaves

How?

Code

mdadm --assemble /dev/md/md127 --name=storage --update=name /dev/sd[abc]

geaves · Jul 22nd 2021

Quote from Diomet

mdadm --assemble /dev/md/md127 --name=storage --update=name /dev/sd[abc]

That's how I would have done it but without the md so just /dev/md127

Have you tried ananas suggestion but with /srv so mount -t ext4 /dev/md127 /srv

The blkid shows no reference to the raid, have you done an --examine on each drive mdadm --examine /dev/sda

Diomet · Jul 22nd 2021

I have made the suggested changes and the output is:

Code

~# mdadm --assemble /dev/md127 --name=storage --update=name /dev/sd[abc]
mdadm: /dev/md127 has been started with 3 drives.

Code

~# mount -t ext4 /dev/md127 /srv
mount: wrong fs type, bad option, bad superblock on /dev/md127,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

Code

~# mdadm --examine /dev/sda
/dev/sda:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 8a1e7d73:19e53531:f65af4dc:1d377bdb
           Name : office:storage  (local to host office)
  Creation Time : Tue Jul 20 16:49:16 2021
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
     Array Size : 7813774336 (7451.80 GiB 8001.30 GB)
  Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=688 sectors
          State : clean
    Device UUID : 125a8656:c4d0fa91:303afdee:59e467db

Internal Bitmap : 8 sectors from superblock
    Update Time : Thu Jul 22 11:56:30 2021
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 5cc1687b - correct
         Events : 4869

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

Display More

Code

~# mdadm --examine /dev/sdb
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 8a1e7d73:19e53531:f65af4dc:1d377bdb
           Name : office:storage  (local to host office)
  Creation Time : Tue Jul 20 16:49:16 2021
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
     Array Size : 7813774336 (7451.80 GiB 8001.30 GB)
  Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=688 sectors
          State : clean
    Device UUID : 4ff6b63d:4587e513:7f21f0ea:60a0d609

Internal Bitmap : 8 sectors from superblock
    Update Time : Thu Jul 22 11:56:30 2021
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : f03e5e90 - correct
         Events : 4869

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

Display More

Code

~# mdadm --examine /dev/sdc
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 8a1e7d73:19e53531:f65af4dc:1d377bdb
           Name : office:storage  (local to host office)
  Creation Time : Tue Jul 20 16:49:16 2021
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
     Array Size : 7813774336 (7451.80 GiB 8001.30 GB)
  Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=688 sectors
          State : clean
    Device UUID : 8b7e01e0:2e863632:5f28a8f9:362ae701

Internal Bitmap : 8 sectors from superblock
    Update Time : Thu Jul 22 11:56:30 2021
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : b7a2766a - correct
         Events : 4869

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

Display More

geaves · Jul 22nd 2021

That's odd the name change has appeared to work, but the drives still display office:storage, have you checked mdadm --detail --scan --verbose and mdadm --detail /dev/md127

Diomet · Jul 22nd 2021

Now I have in dmesg:

Code

[67255.849584] EXT4-fs (md127): VFS: Can't find ext4 filesystem

And the requested output:

Code

root@office:~# mdadm --detail --scan --verbose
ARRAY /dev/md127 level=raid5 num-devices=3 metadata=1.2 name=office:storage UUID=8a1e7d73:19e53531:f65af4dc:1d377bdb
   devices=/dev/sda,/dev/sdb,/dev/sdc




root@office:~# mdadm --detail /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Tue Jul 20 16:49:16 2021
     Raid Level : raid5
     Array Size : 7813774336 (7451.80 GiB 8001.30 GB)
  Used Dev Size : 3906887168 (3725.90 GiB 4000.65 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Jul 22 11:56:30 2021
          State : clean 
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : office:storage  (local to host office)
           UUID : 8a1e7d73:19e53531:f65af4dc:1d377bdb
         Events : 4869

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       8       32        1      active sync   /dev/sdc
       3       8       16        2      active sync   /dev/sdb

Display More

geaves · Jul 22nd 2021

Quote from Diomet

EXT4-fs (md127): VFS: Can't find ext4 filesystem

that would suggest you've toasted it by using --create as per your first post. You'll have to rebuild, but I would suggest a clean install, it's quicker and a lot less hassle, but I would secure wipe those drives.

ananas · Jul 22nd 2021

Do not wipe yet ...

You could try

https://www.cgsecurity.org/wiki/TestDisk

to see if some data might get "rescued"

Good luck,

Thomas

Diomet · Jul 23rd 2021

I saw that on the HDDs

Code

Used Dev Size : 3906887168 (3725.90 GiB 4000.65 GB)

So, it seems that some data is there.

Quote from ananas

You could try
https://www.cgsecurity.org/wiki/TestDisk
to see if some data might get "rescued"

Thank you, I will try with this.

geaves · Jul 23rd 2021

Quote from Diomet

So, it seems that some data is there.
Thank you, I will try with this.

I'd forgotten about testdisk, another idea 'might be' to try systemrescuecd as this runs without access to omv

Diomet · Jul 23rd 2021

Let's hope for the best

Missing FileSystem after RAID5 problem

Diomet Jul 21st 2021

Participate now!

Tags