degraded -> rebuild -> missing raid/filesystem

Hello experts,

after the read and write speeds of my nas degraded fromr like 90/60 to 60/30 i checked my raid in the web-manager ....

one hdd was missing in the raid config and the status was degraded .... well i tried to reintegrate (rebuild) the "missing" HDD - did not work - an error message occured ...

so i fast wiped the hdd and tried the rebuild again and voila - it worked ..... for some time ...

after about 2 hours i have got the authentication error message - left click to relogin and - Raid infos and filesystem info is empty ....

i am using omv 0.5.48 amd64 krnl 2.6.32.5 - the raid-level is 6 and there are 12 3GB WD red hdds.

i am willing to send some logs - there is a series of error messages in the smart log .... if you need more infos and logs please tell me which ones.

how can i fix that ???????????

cheers - alex

Zitat von "leucophores"

Hi,
please post the output from

cat /proc/mdstat

fdisk -l

and

blkid

Alles anzeigen

okay here they are

cat /proc/mdstat
Personalities :
md127 : inactive sdb[0](S) sdj[9](S) sdk[10](S) sdl[13](S) sdm[12](S) sdi[7](S) sdh[6](S) sdg[8](S) sdf[4](S) sde[3](S) sdd[2](S) sdc[1](S)
35163186720 blocks super 1.2
unused devices: <none>

$ fdisk -l
-dash: fdisk: not found
$ fdisk
-dash: fdisk: not found
$ blkid[/quote]
-dash: blkid[/quote]: not found
$

no idea about the -dash .... : not found

mayby you can guide me - ssh client to use and any specific settings in omv - thx

Zitat von "leucophores"

Hi,
please post the output from

cat /proc/mdstat

fdisk -l

and

blkid

Alles anzeigen

Hallo,

as addendum i am adding the two log-files of fdisl-l and blkid :

Zitat

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2014.06.17 18:04:33 =~=~=~=~=~=~=~=~=~=~=~=
Linux omv 2.6.32-5-amd64 #1 SMP Mon Sep 23 22:14:43 UTC 2013 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Jun 17 18:01:34 2014 from a2.local

root@omv:~# fdisk -l

Disk /dev/sda: 64.1 GB, 64105742336 bytes
255 heads, 63 sectors/track, 7793 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e9e6c

Device Boot Start End Blocks Id System
/dev/sda1 * 1 7472 60017664 83 Linux
/dev/sda2 7473 7794 2582529 5 Extended
/dev/sda5 7473 7794 2582528 82 Linux swap / Solaris

Disk /dev/sdb: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table

Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdc doesn't contain a valid partition table

Disk /dev/sdd: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdd doesn't contain a valid partition table

WARNING: GPT (GUID Partition Table) detected on '/dev/sde'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sde: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sde doesn't contain a valid partition table

WARNING: GPT (GUID Partition Table) detected on '/dev/sdf'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sdf: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdf doesn't contain a valid partition table

Disk /dev/sdg: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdg doesn't contain a valid partition table

WARNING: GPT (GUID Partition Table) detected on '/dev/sdh'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sdh: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdh doesn't contain a valid partition table

WARNING: GPT (GUID Partition Table) detected on '/dev/sdi'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sdi: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdi doesn't contain a valid partition table

Disk /dev/sdj: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdj doesn't contain a valid partition table

Disk /dev/sdk: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdk doesn't contain a valid partition table

Disk /dev/sdl: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdl doesn't contain a valid partition table

Disk /dev/sdm: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdm doesn't contain a valid partition table

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2014.06.17 18:10:59 =~=~=~=~=~=~=~=~=~=~=~=
Linux omv 2.6.32-5-amd64 #1 SMP Mon Sep 23 22:14:43 UTC 2013 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Jun 17 18:08:04 2014 from a2.local

root@omv:~# blkid
/dev/sda1: UUID="b7793e5d-99da-4267-9f9c-68343805eb46" TYPE="ext4"
/dev/sda5: UUID="b71ce9e7-03a3-44b1-8612-94eae10efc5b" TYPE="swap"
/dev/sdb: UUID="8bf82e81-3cb5-f305-bf2a-2c33ee771f55" LABEL="openmediavault:OMV" TYPE="linux_raid_member"
/dev/sdc: UUID="8bf82e81-3cb5-f305-bf2a-2c33ee771f55" LABEL="openmediavault:OMV" TYPE="linux_raid_member"
/dev/sdd: UUID="8bf82e81-3cb5-f305-bf2a-2c33ee771f55" LABEL="openmediavault:OMV" TYPE="linux_raid_member"
/dev/sde: UUID="8bf82e81-3cb5-f305-bf2a-2c33ee771f55" LABEL="openmediavault:OMV" TYPE="linux_raid_member"
/dev/sdf: UUID="8bf82e81-3cb5-f305-bf2a-2c33ee771f55" LABEL="openmediavault:OMV" TYPE="linux_raid_member"
/dev/sdg: UUID="8bf82e81-3cb5-f305-bf2a-2c33ee771f55" LABEL="openmediavault:OMV" TYPE="linux_raid_member"
/dev/sdh: UUID="8bf82e81-3cb5-f305-bf2a-2c33ee771f55" LABEL="openmediavault:OMV" TYPE="linux_raid_member"
/dev/sdi: UUID="8bf82e81-3cb5-f305-bf2a-2c33ee771f55" LABEL="openmediavault:OMV" TYPE="linux_raid_member"
/dev/sdj: UUID="8bf82e81-3cb5-f305-bf2a-2c33ee771f55" LABEL="openmediavault:OMV" TYPE="linux_raid_member"
/dev/sdl: UUID="8bf82e81-3cb5-f305-bf2a-2c33ee771f55" LABEL="openmediavault:OMV" TYPE="linux_raid_member"
/dev/sdk: UUID="8bf82e81-3cb5-f305-bf2a-2c33ee771f55" LABEL="openmediavault:OMV" TYPE="linux_raid_member"
/dev/sdm: UUID="8bf82e81-3cb5-f305-bf2a-2c33ee771f55" LABEL="openmediavault:OMV" TYPE="linux_raid_member"

Alles anzeigen

sorry for too many lines and thanx - alex

Zitat von "ryecoaaron"

You must not be logging in as root? Use putty for client.

To fix array, try:

Code

mdadm --stop /dev/md127
mdadm --assemble /dev/md127 /dev/sd[bcdefghijklm] --verbose --force

hello,

will that code recover the files and the filesystem ???
and shall i fully wipe the "hdd" that caused the trouble in the first place for bad block relocation ???
or shall i just reassemble the array mdadm --assemble /dev/md127 /dev/sd[bcdefghijkl] --verbose --force and then try to recover the original raid 6 ???

thanks in advance - alex

Zitat von "ryecoaaron"

Wiping the drive doesn't remove the mdadm superblock. You may need to add the following statement if you know which drive is bad (replace X with the proper drive letter):

Code

mdadm --stop /dev/md127
mdadm --zero-superblock /dev/sdX
mdadm --assemble /dev/md127 /dev/sd[bcdefghijklm] --verbose --force

This should fix everything back to raid 6 with the filesystem intact after it finishes syncing and then reboot.

hello again,

using the --zero-superblock caused an arror in the next command but at least i have got back my degraded raid and the filesystem --- thank you so much !

i am now wiping the hdd with the omv built in random writes - then i will check the hdd with spinrite 6 and then i will decide if it is an rma case ....

cheers and thank you soooooo much again - alex