DegradedArray Event

davidh2k · 25. August 2014

Zitat von johnyb

197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 19
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 5
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 378

The Drive seems to have some errors. Would prepare myself to replace it if I were you.

Greetings
David

johnyb · 25. August 2014

Also taken a screen shot of the console...
Wprrying thing it nothing is shown beneath the RAID Management tab !!!!

johnyb · 25. August 2014

"The Drive seems to have some errors. Would prepare myself to replace it if I were you."

Hi David,
I'm panicking more that just a little now. So I had a drive failure, waited until the warranty replacement arrived, that started to rebuild ok before I got to this state. Are you suggesting the replacement drive has failed or one of the others? If its one of the others then my data is lost right? I would appreciate you help in ascertaining the next step please.

johnyb · 25. August 2014

Ive catpured the states for the other drives

SDB

Code

root@OMV:~# smartctl -a /dev/sdb
smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, [url]http://smartmontools.sourceforge.net[/url]


=== START OF INFORMATION SECTION ===
Device Model:     WDC WD30EFRX-68AX9N0
Serial Number:    WD-WMC1T2836068
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   9
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Mon Aug 25 16:30:20 2014 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED


General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                 (40680) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 255) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.


SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   179   178   021    Pre-fail  Always       -       6041
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       493
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   086   086   000    Old_age   Always       -       10881
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       23
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       10
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       482
194 Temperature_Celsius     0x0022   122   113   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0


SMART Error Log Version: 1
No Errors Logged


SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Alles anzeigen

johnyb · 25. August 2014

SDD

Code

root@OMV:~# smartctl -a /dev/sdd
smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, [url]http://smartmontools.sourceforge.net[/url]


=== START OF INFORMATION SECTION ===
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WCC4N0219644
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   9
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Mon Aug 25 16:34:44 2014 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED


General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                 (42420) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 255) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x703d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.


SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   253   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   175   175   021    Pre-fail  Always       -       6225
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       10
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       258
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       10
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       70
194 Temperature_Celsius     0x0022   124   107   000    Old_age   Always       -       26
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0


SMART Error Log Version: 1
No Errors Logged


SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Alles anzeigen

johnyb · 25. August 2014

SDE

Code

root@OMV:~# smartctl -a /dev/sde
smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, [url]http://smartmontools.sourceforge.net[/url]


=== START OF INFORMATION SECTION ===
Device Model:     WDC WD30EFRX-68AX9N0
Serial Number:    WD-WMC1T2864163
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   9
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Mon Aug 25 16:35:45 2014 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED


General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                 (38940) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 255) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.


SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   177   176   021    Pre-fail  Always       -       6125
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       493
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   085   085   000    Old_age   Always       -       11029
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       30
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       484
194 Temperature_Celsius     0x0022   122   114   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0


SMART Error Log Version: 1
No Errors Logged


SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Alles anzeigen

johnyb · 25. August 2014

BTW it would seems the replacement drive (new one) is shown as SDD now whereas the drive I took out (as per original report) was SDC... Is this expected / normal ?

WastlJ · 25. August 2014

Assembling MD array mdOMVRaid5... failed (not enough devices) means that there is at leas one HDD less than i takes to boot up the raid.

Looks like you built up a RAID 5 with only 3 drives and used the 4th as a HotSpare. Correct?

BTW. Codeboxes are now scrollable. Please use them instead of spoilers Thanks!

johnyb · 25. August 2014

This is the steps I took, which I hoped was recovering the raid not just adding a hotspare.
So oringinal raid was 4 drives, one died, so replaced and followed steps as above.. I though this was adding the new drive to the RAID (which was working at the time just without the resilience). So if this did as a hotspare, another drive has failed whilst this was happening I've lost all my data right?

This was an email message I recieved whilst it was repairing...

Personalities : [raid6] [raid5] [raid4]md127 : active raid5 sdd[4] sdb[0] sde[3] sdc[1](F) 8790795264 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/2] [U__U] [==================>..] recovery = 93.4% (2738177408/2930265088) finish=335.8min speed=9532K/sec

WastlJ · 26. August 2014

Maybe @ryecoaaron can take a look at this... I already discussed with david yesterday. We both were not sure about this:

Code

State : clean, FAILED
Active Devices : 2
Working Devices : 3
Failed Devices : 1
Spare Devices : 1

Zitat von johnyb

Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd[4](S) sdb[0] sde[3] sdc[1](F)
8790795264 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/2] [U__U]

unused devices: <none>

Looking at the raid it shows....

Version : 1.2
Creation Time : Tue May 28 15:18:10 2013
Raid Level : raid5
Array Size : 8790795264 (8383.56 GiB 9001.77 GB)
Used Dev Size : 2930265088 (2794.52 GiB 3000.59 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Mon Aug 18 13:20:37 2014
State : clean, FAILED
Active Devices : 2
Working Devices : 3
Failed Devices : 1
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 512K

Name : N36L-OMV:OMVRaid5
UUID : 860d598b:36d96569:15b715f8:06ffde88
Events : 417048

Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
1 0 0 1 removed
2 0 0 2 removed
3 8 64 3 active sync /dev/sde

1 8 32 - faulty spare /dev/sdc
4 8 48 - spare /dev/sdd

Alles anzeigen

johnyb · 26. August 2014

Do you think I need to replace the drive SDB (S/N WD-WMC1T2837415) due to those errors?
I'm assuming something to do with the way I added the new replacement drive (WD-WCC4N0219644 replacing faulty WD-WMV1T2427260) it didnt rebuild into the RAID5 (I think you mentioned hot standby) therefore when I had the wickly bad luck of a 2nd drive failing in this state - I've losts all my data right?

What I dont understand is all the drives are showing in physical disks but when I look in RAID management nothing is there, and if I attempt to create a new one (was going to do it just looking) none of the drives show up?

I would really appreciate help here, and I'm so sorry to keep posting - but as you can imagine I want to get the server working again ASAP - preferablly with all my data (but accept this now may have been lost). Thank you....

ryecoaaron · 26. August 2014

I would do the following:

Code

mdadm --stop /dev/md127
mdadm --assemble /dev/md127 /dev/sd[bcde] --verbose --force

johnyb · 26. August 2014

Code

root@OMV:~# mdadm --stop /dev/md127
mdadm: stopped /dev/md127
root@OMV:~# mdadm --assemble /dev/md127 /dev/sd[bcde] --verbose --force
mdadm: looking for devices for /dev/md127
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has no superblock - assembly aborted
root@OMV:~#

BTW - When 1st booting the system, it stops at the point shown as shown below, and I used Cntrl+D to continue (that way the GUI works). Should I try the above commands when prompted the below BEFORE Cntrl+D?

ryecoaaron · 26. August 2014

Hit ctrl-d to continue and try:

Code

mdadm --stop /dev/md127
mdadm --zero-superblock /dev/sdc
dd if=/dev/zero of=/dev/sdc bs=512 count=1000
mdadm --assemble /dev/md127 /dev/sd[bcde] --verbose --force

johnyb · 26. August 2014

mdadm --zero-superblock /dev/sdc
mdadm: Couldn't open /dev/sdc for write - not zeroing

ryecoaaron · 26. August 2014

I would try booting systemrescuecd and try the same commands.

johnyb · 26. August 2014

Where you say use the systemrescuecd, I assume you mean on the GRUB menu select the 2nd entry (recovery mode). I tried this and the same error is presented...

ryecoaaron · 26. August 2014

Nope, I meant systemrescuecd.

johnyb · 26. August 2014

My appologies....

It would seems depending upon the boot method the drive letters changed so I tried the following in an attempt to identify the correct drive letter, which looks like sde, so then tried the superblock command but got the attached results....
mdadm --stop /dev/md127
mdadm --assemble /dev/md127 /dev/sd[bcde] --verbose --force

ryecoaaron · 26. August 2014

Yep, drive letters can change. I assume sdc is now sde. Must be something wrong with that drive.

Jetzt mitmachen!