Raid 5, 3 HDD's, clean degraded

Darcu · 29. Dezember 2021

Hello guys,

setup:

/dev/sda SSD --> OS

/dev/sdb HDD 14TB

/dev/sdc HDD 14TB

/dev/sdd HDD 14TB

sdb, sdc and sdd are/were one RAID5

Raid is /dev/md127

I had to reinstall OMV 5. For reinstall I pulled out all SATA cable of HDD's. After install I put back and configure my system. Now I get messages:

Code

A DegradedArray event had been detected on md device /dev/md127.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md127 : active (auto-read-only) raid5 sda[0] sdb[1]
      27344500736 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
      bitmap: 28/102 pages [112KB], 65536KB chunk

unused devices: <none>

Alles anzeigen

When I do mdadm --detail /dev/md127:

Code

/dev/md127:
           Version : 1.2
     Creation Time : Thu Jul  2 19:46:53 2020
        Raid Level : raid5
        Array Size : 27344500736 (26077.75 GiB 28000.77 GB)
     Used Dev Size : 13672250368 (13038.87 GiB 14000.38 GB)
      Raid Devices : 3
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Wed Dec 29 10:25:29 2021
             State : clean, degraded
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : MediaServer:RAID
              UUID : d1b4a55e:a6b812ae:41feb7af:1e41a9ca
            Events : 51670

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       8       16        1      active sync   /dev/sdb
       -       0        0        2      removed

Alles anzeigen

In the WEBGUI I see all disks. In S.M.A.R.T. all disks have a "good" status.

When I try RAID --> Recovery I see the missing /dev/sdd in the menu, but when I try to recover it:

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; mdadm --manage '/dev/md127' --add /dev/sdd 2>&1' with exit code '1': mdadm: Failed to write metadata to /dev/sdd


Details:

Fehler #0:
OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; mdadm --manage '/dev/md127' --add /dev/sdd 2>&1' with exit code '1': mdadm: Failed to write metadata to /dev/sdd in /usr/share/php/openmediavault/system/process.inc:196
Stack trace:
#0 /usr/share/openmediavault/engined/rpc/raidmgmt.inc(419): OMV\System\Process->execute()
#1 [internal function]: Engined\Rpc\RaidMgmt->add(Array, Array)
#2 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#3 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('add', Array, Array)
#4 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('RaidMgmt', 'add', Array, Array, 1)
#5 {main}

Alles anzeigen

I don't think that the disk is dead, but I don't know how to go on. Any idea what to do?

To buy a new disk is not that cheap

geaves · 29. Dezember 2021

Zitat von Darcu

mdadm --manage '/dev/md127' --add /dev/sdd 2>&1' with exit code '1': mdadm: Failed to write metadata to /dev/sdd

The above is probably due to this:

Zitat von Darcu

md127 : active (auto-read-only) raid5 sda[0] sdb[1]

the array is in a read only state

try mdadm --readwrite /dev/md127 then try adding /dev/sdd

Darcu · 29. Dezember 2021

Hi Geaves,

Code

mdadm: failed to set writable for /dev/md127: Device or resource busy

edit:
With SMB I have access to the raid/shared folder. I can read & write

geaves · 29. Dezember 2021

Hm, stop the array mdadm --stop /dev/md127 then run the readwrite

Darcu · 29. Dezember 2021

Zitat von geaves

Hm, stop the array mdadm --stop /dev/md127 then run the readwrite

Code

mdadm: Cannot get exclusive access to /dev/md127:Perhaps a running process, moun                                   ted filesystem or active volume group?

geaves · 29. Dezember 2021

cat /etc/fstab

Darcu · 29. Dezember 2021

Zitat von Darcu

Code

mdadm: Cannot get exclusive access to /dev/md127:Perhaps a running process, moun                                   ted filesystem or active volume group?

I found this: https://superuser.com/question…mdadm-to-stop-raid5-array

It is written that it may be a problem with SAMBA or users. I had no SAMBA access while using "mdadm --stop /dev/md127". Then I look into my WEBGUI. I have three users. When I go to my shared folder --> ACL

I do not know: What is Group 1000? Is it normal? I can't find it in users or groups.

Should I delete every user to stop the array and set readwrite?

Soma · 29. Dezember 2021

Darcu

Since you don't post the command:

Are you running them as root?

Darcu · 29. Dezember 2021

Zitat von geaves

cat /etc/fstab

Code

 /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda2 during installation
UUID=7951e886-abd2-4a07-b708-e3bab6e33a8f /               ext4    errors=remount                                     -ro 0       1
# /boot/efi was on /dev/sda1 during installation
UUID=4A9E-C1A4  /boot/efi       vfat    umask=0077      0       1
# swap was on /dev/sda3 during installation
UUID=b988c23d-7d64-42d5-9eee-7b63be103b2e none            swap    sw                                                   0       0
# >>> [openmediavault]
/dev/disk/by-uuid/6eb513cb-8151-4ec9-8f71-b3e518b31255          /srv/dev-disk-by                                     -uuid-6eb513cb-8151-4ec9-8f71-b3e518b31255      ext4    defaults,nofail,user_xat                                     tr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
# <<< [openmediavault]

Alles anzeigen

Darcu · 29. Dezember 2021

Zitat von Soma

Darcu
Since you don't post the command:
Are you running them as root?

yes,

logged in as root

e.g.

Code

root@MEDIASERVER:~# mdadm --stop /dev/md127
mdadm: Cannot get exclusive access to /dev/md127:Perhaps a running process, mounted filesystem or active volume group?

geaves · 29. Dezember 2021

Zitat von Soma

Darcu
Since you don't post the command:
Are you running them as root?

That was my next question, but I'm welcome of the input

geaves · 29. Dezember 2021

There's no reference to the array in fstab

geaves · 29. Dezember 2021

so using that link as a reference, which would have been the next step;

umount /dev/md127

mdadm --stop /dev/md127

mdadm --readwrite /dev/md127

then add /dev/sdd back using recover, then mount the file system

Darcu · 29. Dezember 2021

Zitat von geaves

so using that link as a reference, which would have been the next step;

umount /dev/md127
mdadm --stop /dev/md127
mdadm --readwrite /dev/md127

then add /dev/sdd back using recover, then mount the file system

Alles anzeigen

When I try to umount as root I get:

Code

root@MEDIASERVER:~# umount /dev/md127
umount: /srv/dev-disk-by-uuid-6eb513cb-8151-4ec9-8f71-b3e518b31255: target is busy.

When I try to log into SSH not as root (other user with all rights) and try umount /dev/md127 I get

Code

$ umount /dev/md127
umount: /srv/dev-disk-by-uuid-6eb513cb-8151-4ec9-8f71-b3e518b31255: umount faile                                      d: Die Operation ist nicht erlaubt.

Usually when I am logged in via root it is

Code

root@MEDIASERVER:~#

but now it is only $

I think I have to move to any directory, but I am too stupid

geaves · 29. Dezember 2021

Zitat von Darcu

root@MEDIASERVER:~#

logged in as root that is how it should be.

Ok if I go back to your first post and my reply, the reason mdadm cannot write metadata to the drive you're trying to add is because it's in an active (auto-read-only) state, the output from cat /proc/mdstat.

Everything I have got you do so far should have worked, even if SMB and docker are running you should be able to change the 'state' of the array, but in your case it fails, that would suggest that something is accessing the array.

My only thought therefore is to disable smb and click save (all the shares will remain) then run systemctl stop docker this stop docker and any containers, then try mdadm --stop /dev/md127 if the stop fails go back to post 13

Darcu · 29. Dezember 2021

Zitat von geaves

logged in as root that is how it should be.

Ok if I go back to your first post and my reply, the reason mdadm cannot write metadata to the drive you're trying to add is because it's in an active (auto-read-only) state, the output from cat /proc/mdstat.

Everything I have got you do so far should have worked, even if SMB and docker are running you should be able to change the 'state' of the array, but in your case it fails, that would suggest that something is accessing the array.

My only thought therefore is to disable smb and click save (all the shares will remain) then run systemctl stop docker this stop docker and any containers, then try mdadm --stop /dev/md127 if the stop fails go back to post 13

Alles anzeigen

first of all: Thanks for your fast replies!

I disabled SMB

Then i tried to mdadm --stop... and stopped docker. I didn't install docker so far. So couldn't loaded.

Code

root@MEDIASERVER:~# mdadm --stop /dev/md127
mdadm: Cannot get exclusive access to /dev/md127:Perhaps a running process, mounted filesystem or active volume group                                 ?
root@MEDIASERVER:~# systemctl stop docker
Failed to stop docker.service: Unit docker.service not loaded.
root@MEDIASERVER:~# umount /dev/md127
root@MEDIASERVER:~# mdadm --stop /dev/md127
mdadm: stopped /dev/md127
root@MEDIASERVER:~# mdadm --readwrite /dev/md127
mdadm: error opening /dev/md127: No such file or directory
root@MEDIASERVER:~#

umount and stop seem to work, but now it can't find the directory

geaves · 29. Dezember 2021

Output of the following;

blkid

cat /etc/mdadm/mdadm.conf

Darcu · 29. Dezember 2021

Code

root@MEDIASERVER:~# blkid
/dev/sdb: UUID="d1b4a55e-a6b8-12ae-41fe-b7af1e41a9ca" UUID_SUB="1dd37406-fd44-9c3e-b179-316da379a6ca" LABEL="MediaServer:RAID" TYPE="linux_raid_member"
/dev/sdc1: UUID="4A9E-C1A4" TYPE="vfat" PARTUUID="edd760ec-6ae2-49a6-9057-4bac06cfb8fc"
/dev/sdc2: UUID="7951e886-abd2-4a07-b708-e3bab6e33a8f" TYPE="ext4" PARTUUID="0984dc8b-69b0-4b80-aace-860f4418e579"
/dev/sdc3: UUID="b988c23d-7d64-42d5-9eee-7b63be103b2e" TYPE="swap" PARTUUID="b224a268-4fb6-4511-b89c-556bdf4c8d29"
/dev/sda: UUID="d1b4a55e-a6b8-12ae-41fe-b7af1e41a9ca" UUID_SUB="fc59dfda-4bd3-4f3d-4b60-9fe2cf1572dc" LABEL="MediaServer:RAID" TYPE="linux_raid_member"
root@MEDIASERVER:~# cat /etc/mdadm/mdadm.conf
# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR tillsenner@pm.me
MAILFROM root

# definitions of existing MD arrays
ARRAY /dev/md/MediaServer:RAID metadata=1.2 name=MediaServer:RAID UUID=d1b4a55e:a6b812ae:41feb7af:1e41a9ca

Alles anzeigen

geaves · 29. Dezember 2021

That's interesting because it doesn't show /dev/sdd in the output of blkid and it should do.

Ok this will be a first try mdadm --readwrite /dev/md/MediaServer:RAID

Darcu · 29. Dezember 2021

Code

root@MEDIASERVER:~# mdadm --readwrite /dev/md/MediaServer:RAID
mdadm: error opening /dev/md/MediaServer:RAID: No such file or directory

edit:

Raid 5, 3 HDD's, clean degraded

Jetzt mitmachen!

Ähnliche Themen

6x 3TB RAID6 - one disk dying, what to do?

Replacing disk in Raid1 [OMV6] - entry not visible

Mirror RAID clean, degraded

Mirror RAID clean, degraded