Filesystem is missing / RAID disappeard

Magellan · Sep 21st 2023

Hi

I need some help for rebuilding my RAID1.

I ran OMV since many years, since a couple of weeks one HDD is unhealthy (S.M.A.R.T. values). So I bought an replacement disk.

Two days ago I had a very high CPU usage and many warnings according that. So I did a reboot of OMV without any investigations. After the reboot the RAID was disappeared.

What can I do to get the RAID back or rebuild it directly with my replacement disk?

Code

root@centaurus:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : inactive sdc[1](S)
      3906887512 blocks super 1.2
       
unused devices: <none>

Code

root@centaurus:~# mdadm --detail /dev/md127
/dev/md127:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 1
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 1

              Name : magellan:DatenNAS
              UUID : c0a1e9a7:7c39191e:32221cfb:cd271d52
            Events : 114830

    Number   Major   Minor   RaidDevice

       -       8       32        -        /dev/sdc

Display More

That output is weird. I'm very sure, that my RAID was RAID1 with two devices.

Code

root@centaurus:~# blkid
/dev/sdc: UUID="c0a1e9a7-7c39-191e-3222-1cfbcd271d52" UUID_SUB="1111d18a-71a4-f9b8-3c83-29b9734bcd56" LABEL="magellan:DatenNAS" TYPE="linux_raid_member"
/dev/sdb1: UUID="7AA4-2D1C" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="d8798485-2f81-4c7b-9c72-f8430423b800"
/dev/sdb2: UUID="4cc12b30-85e1-459e-8c29-9d5fcaa471bc" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="454b6af2-bd1a-4570-81b1-1ae0f4fb8e98"
/dev/sdb3: UUID="7dffdff3-c63a-48c7-b635-0b1adbd8d8ca" TYPE="swap" PARTUUID="90b31067-837b-432c-9300-d3dfc84c2b43"

I don't know why blkid doesn't schow /dev/sda. In the WebUI it is there:

Code

root@centaurus:~# fdisk -l | grep "Disk "
Disk /dev/sdc: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFRX-68N
Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFRX-68N
Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Samsung SSD 870 
Disk identifier: 0A7E0373-9E5E-40D3-8230-F064D704264F

/dev/sda has some S.M.A.R.T. errors.

geaves · Sep 22nd 2023

Quote from Magellan

/dev/sda has some S.M.A.R.T. errors.

Could be reason why it has dropped from the array and the fact blkid cannot locate it, the array is currently inactive.

ssh into omv and try;

mdadm --stop /dev/md127 then

mdadm --assemble /dev/md127

Magellan · Sep 22nd 2023

Thanks.

I tried so:

Quote

root@centaurus:~# mdadm --stop /dev/md127

mdadm: stopped /dev/md127

Quote

root@centaurus:~# mdadm --assemble /dev/md127

mdadm: /dev/md127 not identified in config file.

That seems not to work.

So I tried it more explicit:

Quote

root@centaurus:~# mdadm --assemble --verbose /dev/md127 /dev/sd[ab]

mdadm: looking for devices for /dev/md127

mdadm: no recogniseable superblock on /dev/sda

mdadm: /dev/sda has no superblock - assembly aborted

What does that mean? Is my /dev/sda broken? Is there any chance to rebuild the RAID with my replacement disk?

mdadm.conf for information:

Quote

root@centaurus:~# cat /etc/mdadm/mdadm.conf

# This file is auto-generated by openmediavault (https://www.openmediavault.org)

# WARNING: Do not edit this file, your changes will get lost.

# mdadm.conf

#

# Please refer to mdadm.conf(5) for information about this file.

#

# by default, scan all partitions (/proc/partitions) for MD superblocks.

# alternatively, specify devices to scan, using wildcards if desired.

# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.

# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is

# used if no RAID devices are configured.

DEVICE partitions

# auto-create devices with Debian standard permissions

CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system

HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts

MAILADDR ---classified---

MAILFROM root

# definitions of existing MD arrays

ARRAY /dev/md/magellan:DatenNAS metadata=1.2 name=magellan:DatenNAS UUID=c0a1e9a7:7c39191e:32221cfb:cd271d52

Display More

geaves · Sep 23rd 2023

Quote from Magellan

1) What does that mean?
2) Is my /dev/sda broken?
3) Is there any chance to rebuild the RAID with my replacement disk?

1) It means that mdadm cannot find/locate the superblock on that drive

2) Don't know, at this moment mdadm does not recognise the drive as being part of an array

3) Probably

Question, why did you run this line -> mdadm --assemble --verbose /dev/md127 /dev/sd[ab] and what is wrong with it, look at the output from your first post, cat proc/mdstat, blkid, and the screenshot of storage -> disks

Magellan · Sep 23rd 2023

Quote from geaves

Question, why did you run this line -> mdadm --assemble --verbose /dev/md127 /dev/sd[ab]

I thougth it would be a good idea to give the disks to use. Maybe a bad idea. And yes, it is wrong! It should be mdadm --assemble --verbose /dev/md127 /dev/sd[ac]

Should I try that?

geaves · Sep 23rd 2023

Quote from Magellan

Should I try that?

Yep, the array should start but the error re /dev/sda should be the same

Magellan · Sep 23rd 2023

Code

root@centaurus:~# mdadm --assemble --verbose /dev/md127 /dev/sd[ac]
mdadm: looking for devices for /dev/md127
mdadm: no recogniseable superblock on /dev/sda
mdadm: /dev/sda has no superblock - assembly aborted

What's next?

geaves · Sep 23rd 2023

mdadm --stop /dev/md127

mdadm --assemble --force --verbose /dev/md127 /dev/sd[ac]

Magellan · Sep 23rd 2023

Code

root@centaurus:~# mdadm --stop /dev/md127
mdadm: error opening /dev/md127: No such file or directory

root@centaurus:~# mdadm --assemble --force --verbose /dev/md127 /dev/sd[ac]
mdadm: looking for devices for /dev/md127
mdadm: no recogniseable superblock on /dev/sda
mdadm: /dev/sda has no superblock - assembly aborted

same output as in post #7.

Any ideas how to go further? I'm quite lost.

geaves · Sep 23rd 2023

Quote from Magellan

I'm quite lost

So am I, post the output of cat /proc/mdstat and blkid

Magellan · Sep 23rd 2023

Code

root@centaurus:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices: <none>

root@centaurus:~# blkid
/dev/sdc: UUID="c0a1e9a7-7c39-191e-3222-1cfbcd271d52" UUID_SUB="1111d18a-71a4-f9b8-3c83-29b9734bcd56" LABEL="magellan:DatenNAS" TYPE="linux_raid_member"
/dev/sdb1: UUID="7AA4-2D1C" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="d8798485-2f81-4c7b-9c72-f8430423b800"
/dev/sdb2: UUID="4cc12b30-85e1-459e-8c29-9d5fcaa471bc" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="454b6af2-bd1a-4570-81b1-1ae0f4fb8e98"
/dev/sdb3: UUID="7dffdff3-c63a-48c7-b635-0b1adbd8d8ca" TYPE="swap" PARTUUID="90b31067-837b-432c-9300-d3dfc84c2b43"

geaves · Sep 23rd 2023

Bummer, that would explain this line -> mdadm: error opening /dev/md127: No such file or directory from #9

At this moment it appears the system/mdadm is not finding any arrays, which means there's nothing that can be done, try a reboot and run the above commands again

Magellan · Sep 24th 2023

OK, I rebooted OMV.

After the reboot the disk with S.M.A.R.T. errors is /dev/sdc.

/dev/md127 is available again:

Code

root@centaurus:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : inactive sda[1](S)
      3906887512 blocks super 1.2

unused devices: <none>

commands from above:

Code

root@centaurus:~# mdadm --stop /dev/md127
mdadm: stopped /dev/md127

Code

root@centaurus:~# mdadm --assemble --force --verbose /dev/md127 /dev/sd[ac]
mdadm: looking for devices for /dev/md127
mdadm: Cannot read superblock on /dev/sdc
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdc has no superblock - assembly aborted

Still no luck with re-creating the RAID.

I think this disk with errors is over. I there any chance to include my new replacement disk into the RAID?

geaves · Sep 24th 2023

Quote from Magellan

I there any chance to include my new replacement disk into the RAID

No until you can actually get the array started you add another disk.

What makes no sense here is the fact that the array will not start with just one drive, which technically it should, at this moment in time for some reason this is not recoverable.

What's the output of blkid

Magellan · Sep 24th 2023

I thought I created an RAID1. But as mdadm —detail /dev/md127 shows (first post), it is a RAID0. Right?

Does it not start with one disk because it is a RAID0?

Output of blkid:

Code

root@centaurus:~# blkid
/dev/sda: UUID="c0a1e9a7-7c39-191e-3222-1cfbcd271d52" UUID_SUB="1111d18a-71a4-f9b8-3c83-29b9734bcd56" LABEL="magellan:DatenNAS" TYPE="linux_raid_member"
/dev/sdb1: UUID="7AA4-2D1C" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="d8798485-2f81-4c7b-9c72-f8430423b800"
/dev/sdb2: UUID="4cc12b30-85e1-459e-8c29-9d5fcaa471bc" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="454b6af2-bd1a-4570-81b1-1ae0f4fb8e98"
/dev/sdb3: UUID="7dffdff3-c63a-48c7-b635-0b1adbd8d8ca" TYPE="swap" PARTUUID="90b31067-837b-432c-9300-d3dfc84c2b43"

geaves · Sep 24th 2023

Quote from Magellan

shows (first post), it is a RAID0. Right

I've seen that before with users with a raid5, what's the output of mdadm --examine /dev/sda if you did create a raid0 then I would not expect to see inactive from cat /proc/mdstat

Magellan · Sep 25th 2023

Quote from geaves

I've seen that before with users with a raid5

OK.

/dev/sda is the healty disk after reboot:

Code

root@centaurus:~# mdadm --examine /dev/sda
/dev/sda:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : c0a1e9a7:7c39191e:32221cfb:cd271d52
           Name : magellan:DatenNAS
  Creation Time : Tue Mar  7 20:53:18 2017
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
     Array Size : 3906887488 (3725.90 GiB 4000.65 GB)
  Used Dev Size : 7813774976 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=48 sectors
          State : clean
    Device UUID : 1111d18a:71a4f9b8:3c8329b9:734bcd56

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Sep 19 19:17:49 2023
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : faf29f44 - correct
         Events : 114830


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

Display More

Examine /dev/sdc:

Code

root@centaurus:~# mdadm --examine /dev/sdc
mdadm: No md superblock detected on /dev/sdc.

Thanks for your help bytheway...!

geaves · Sep 25th 2023

Well that answers some questions, but what makes no sense is why the array aborts when attempting to re assemble, when one drive is ok.

Try this;

mdadm --stop /dev/md127

mdadm --assemble --force --verbose /dev/md127 /dev/sda

cat /proc/mdstat

DO NOT RUN ANYTHING ELSE

Magellan · Sep 25th 2023

Quote from geaves

DO NOT RUN ANYTHING ELSE

Got it.

Code

root@centaurus:~# mdadm --stop /dev/md127
mdadm: error opening /dev/md127: No such file or directory

Code

root@centaurus:~# mdadm --assemble --force --verbose /dev/md127 /dev/sda
mdadm: looking for devices for /dev/md127
mdadm: /dev/sda is identified as a member of /dev/md127, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md127
mdadm: added /dev/sda to /dev/md127 as 1
mdadm: /dev/md127 has been started with 1 drive (out of 2).

That looks good so far...

Code

root@centaurus:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [ra                                         id10]
md127 : active (auto-read-only) raid1 sda[1]
      3906887488 blocks super 1.2 [2/1] [_U]
      bitmap: 0/30 pages [0KB], 65536KB chunk

unused devices: <none>

Does that mean we have a RAID1 again with 1 working disk?

Seems like:

Next?

geaves · Sep 25th 2023

Quote from Magellan

Does that mean we have a RAID1 again with 1 working disk

Fingers crossed

Quote from Magellan

Next?

The array is showing as auto-read-only to correct that run;

mdadm --readwrite /dev/md127 check it again with cat /proc/mdstat

Filesystem is missing / RAID disappeard

Magellan Sep 21st 2023

Participate now!