SMART Device Error Count increasing on all disks in the volume

SaiKo · 30. April 2016

Upgraded my nas.
I had 8*6TB and bought 12 more (in 3 batches).
Also bought a new mobo/cpu/ram.
Planned on using my old controller but testing different nas distro's i found it didn't have a lot of support so i bought a 9211-8i controller and 2 RES2SV240 expanders.
While waiting for parts i kept busy testing distros and testing and burning in hard drives, on my old controller.
I settled on omv and set it up with an initial 4*4TB volume (zfs mirror).
Now the new controller and expander have arrived i put 10 new 6TB drives in a raidz2 and began copying data off my old drives to the new build.
I am now in the midst of copying data full speed over both 1Gbit nic's as i go and check smart statuses.

I notice drives with increased drive error count and Error: ABRT
At first not all 6TB drives had these but as i am checking back and forth i notice all recent errors are at the same time as the respective disk's Power_On_Hours, and now all drives have errors at this current hour (600+).
Only some had errors previously (300+), i'm copying smart info of one of these below, but all the others seem to have this error because of (or at least at the time of) reading the SMART information.
Is this possible?

Copy/pasting stuff from the below smart output in google gives a surprisingly large number of search results for forum posts here, from older threads but still.
Is there something to do with omv that might cause these errors?

Surely not all of these 10 new drives are failing at the same time?
Could errors like these be caused by a controller, or an expander?
Although a few of the disks show errors at 300+ hours, i didn't have the new controller and expanders back then.

Anyone who can shed light on this?
Thank you.

SaiKo · 30. April 2016

Code

smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.2.0-4-amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net


=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD60EFRX-68L0BN1
Firmware Version: 82.00A82
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 9
ATA Standard is: Not recognized. Minor revision code: 0x001f
Local Time is: Sat Apr 30 18:39:27 2016 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED


General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
 was never started.
 Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
 without error or no self-test has ever
 been run.
Total time to complete Offline
data collection: ( 5744) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
 Auto Offline data collection on/off support.
 Suspend Offline collection upon new
 command.
 Offline surface scan supported.
 Self-test supported.
 Conveyance Self-test supported.
 Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
 power-saving mode.
 Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
 General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 255) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303d) SCT Status supported.
 SCT Error Recovery Control supported.
 SCT Feature Control supported.
 SCT Data Table supported.


SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
 1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
 3 Spin_Up_Time POS--K 238 197 021 - 7075
 4 Start_Stop_Count -O--CK 100 100 000 - 42
 5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
 7 Seek_Error_Rate -OSR-K 200 200 000 - 0
 9 Power_On_Hours -O--CK 100 100 000 - 606
 10 Spin_Retry_Count -O--CK 100 253 000 - 0
 11 Calibration_Retry_Count -O--CK 100 253 000 - 0
 12 Power_Cycle_Count -O--CK 100 100 000 - 31
192 Power-Off_Retract_Count -O--CK 200 200 000 - 26
193 Load_Cycle_Count -O--CK 200 200 000 - 125
194 Temperature_Celsius -O---K 125 112 000 - 27
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 100 253 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
 ||||||_ K auto-keep
 |||||__ C event count
 ||||___ R error rate
 |||____ S speed/performance
 ||_____ O updated online
 |______ P prefailure warning


General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
GP/S Log at address 0x00 has 1 sectors [Log Directory]
SMART Log at address 0x01 has 1 sectors [Summary SMART error log]
SMART Log at address 0x02 has 5 sectors [Comprehensive SMART error log]
GP Log at address 0x03 has 6 sectors [Ext. Comprehensive SMART error log]
SMART Log at address 0x06 has 1 sectors [SMART self-test log]
GP Log at address 0x07 has 1 sectors [Extended self-test log]
SMART Log at address 0x09 has 1 sectors [Selective self-test log]
GP Log at address 0x10 has 1 sectors [NCQ Command Error]
GP Log at address 0x11 has 1 sectors [SATA Phy Event Counters]
GP Log at address 0x21 has 1 sectors [Write stream error log]
GP Log at address 0x22 has 1 sectors [Read stream error log]
GP/S Log at address 0x30 has 9 sectors [Reserved]
GP/S Log at address 0x80 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x81 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x82 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x83 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x84 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x85 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x86 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x87 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x88 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x89 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8a has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8b has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8c has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8d has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8e has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8f has 16 sectors [Host vendor specific log]
GP/S Log at address 0x90 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x91 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x92 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x93 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x94 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x95 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x96 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x97 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x98 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x99 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9a has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9b has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9c has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9d has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9e has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9f has 16 sectors [Host vendor specific log]
GP/S Log at address 0xa0 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa1 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa2 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa3 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa4 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa5 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa6 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa7 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa8 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xa9 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xaa has 1 sectors [Device vendor specific log]
GP/S Log at address 0xab has 1 sectors [Device vendor specific log]
GP/S Log at address 0xac has 1 sectors [Device vendor specific log]
GP/S Log at address 0xad has 1 sectors [Device vendor specific log]
GP/S Log at address 0xae has 1 sectors [Device vendor specific log]
GP/S Log at address 0xaf has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb0 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb1 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb2 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb3 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb4 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb5 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb6 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb7 has 54 sectors [Device vendor specific log]
GP/S Log at address 0xbd has 1 sectors [Device vendor specific log]
GP/S Log at address 0xc0 has 1 sectors [Device vendor specific log]
GP Log at address 0xc1 has 93 sectors [Device vendor specific log]
GP/S Log at address 0xe0 has 1 sectors [SCT Command/Status]
GP/S Log at address 0xe1 has 1 sectors [SCT Data Transfer]

Alles anzeigen

SaiKo · 30. April 2016

Code

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 12
 CR = Command Register
 FEATR = Features Register
 COUNT = Count (was: Sector Count) Register
 LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8
 LH = LBA High (was: Cylinder High) Register ] LBA
 LM = LBA Mid (was: Cylinder Low) Register ] Register
 LL = LBA Low (was: Sector Number) Register ]
 DV = Device (was: Device/Head) Register
 DC = Device Control Register
 ER = Error register
 ST = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.


Error 12 [11] occurred at disk power-on lifetime: 606 hours (25 days + 6 hours)
 When the command that caused the error occurred, the device was active or idle.


 After command completion occurred, registers were:
 ER -- ST COUNT LBA_48 LH LM LL DV DC
 -- -- -- == -- == == == -- -- -- -- --
 04 -- 51 00 0b 00 00 00 00 00 00 00 00 Error: ABRT


 Commands leading to the command that caused the error were:
 CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
 -- == -- == -- == == == -- -- -- -- -- --------------- --------------------
 b0 00 d5 00 01 00 00 00 c2 4f e1 00 00 1d+23:02:28.153 SMART READ LOG
 b0 00 d5 00 01 00 00 00 c2 4f e1 00 00 1d+23:02:28.153 SMART READ LOG
 b0 00 d5 00 01 00 00 00 c2 4f e1 00 00 1d+23:02:28.153 SMART READ LOG
 b0 00 d6 00 01 00 00 00 c2 4f e0 00 00 1d+23:02:28.152 SMART WRITE LOG
 b0 00 d6 00 01 00 00 00 c2 4f e0 00 00 1d+23:02:28.151 SMART WRITE LOG


Error 11 [10] occurred at disk power-on lifetime: 606 hours (25 days + 6 hours)
 When the command that caused the error occurred, the device was active or idle.


 After command completion occurred, registers were:
 ER -- ST COUNT LBA_48 LH LM LL DV DC
 -- -- -- == -- == == == -- -- -- -- --
 04 -- 51 00 0b 00 00 00 00 00 00 00 00 Error: ABRT


 Commands leading to the command that caused the error were:
 CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
 -- == -- == -- == == == -- -- -- -- -- --------------- --------------------
 b0 00 d5 00 01 00 00 00 c2 4f e1 00 00 1d+23:02:28.153 SMART READ LOG
 b0 00 d5 00 01 00 00 00 c2 4f e1 00 00 1d+23:02:28.153 SMART READ LOG
 b0 00 d6 00 01 00 00 00 c2 4f e0 00 00 1d+23:02:28.152 SMART WRITE LOG
 b0 00 d6 00 01 00 00 00 c2 4f e0 00 00 1d+23:02:28.151 SMART WRITE LOG
 61 00 20 00 00 00 02 3b f4 cd 88 40 00 1d+23:02:28.151 WRITE FPDMA QUEUED


Error 10 [9] occurred at disk power-on lifetime: 606 hours (25 days + 6 hours)
 When the command that caused the error occurred, the device was active or idle.


 After command completion occurred, registers were:
 ER -- ST COUNT LBA_48 LH LM LL DV DC
 -- -- -- == -- == == == -- -- -- -- --
 04 -- 51 00 0b 00 00 00 00 00 00 00 00 Error: ABRT


 Commands leading to the command that caused the error were:
 CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
 -- == -- == -- == == == -- -- -- -- -- --------------- --------------------
 b0 00 d5 00 01 00 00 00 c2 4f e1 00 00 1d+22:36:12.690 SMART READ LOG
 b0 00 d5 00 01 00 00 00 c2 4f e1 00 00 1d+22:36:12.690 SMART READ LOG
 61 00 f0 00 00 00 02 30 5a 39 78 40 00 1d+22:36:12.689 WRITE FPDMA QUEUED
 b0 00 d5 00 01 00 00 00 c2 4f e0 00 00 1d+22:36:12.689 SMART READ LOG
 b0 00 d6 00 01 00 00 00 c2 4f e0 00 00 1d+22:36:12.687 SMART WRITE LOG


Error 9 [8] occurred at disk power-on lifetime: 324 hours (13 days + 12 hours)
 When the command that caused the error occurred, the device was active or idle.


 After command completion occurred, registers were:
 ER -- ST COUNT LBA_48 LH LM LL DV DC
 -- -- -- == -- == == == -- -- -- -- --
 40 -- 51 00 68 00 00 c5 ef 87 48 40 00 Error: UNC 104 sectors at LBA = 0xc5ef8748 = 3320809288


 Commands leading to the command that caused the error were:
 CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
 -- == -- == -- == == == -- -- -- -- -- --------------- --------------------
 25 00 00 00 68 00 00 c5 ef 87 48 40 00 15:17:36.991 READ DMA EXT
 25 00 00 00 80 00 00 c5 ef 86 c8 40 00 15:17:36.990 READ DMA EXT
 25 00 00 00 70 00 00 c5 ef 86 58 40 00 15:17:36.990 READ DMA EXT
 25 00 00 00 80 00 00 c5 ef 85 d8 40 00 15:17:36.989 READ DMA EXT
 25 00 00 00 20 00 00 c5 ef 85 b8 40 00 15:17:36.989 READ DMA EXT


Error 8 [7] occurred at disk power-on lifetime: 324 hours (13 days + 12 hours)
 When the command that caused the error occurred, the device was active or idle.


 After command completion occurred, registers were:
 ER -- ST COUNT LBA_48 LH LM LL DV DC
 -- -- -- == -- == == == -- -- -- -- --
 40 -- 51 00 20 00 00 b8 22 77 60 40 00 Error: UNC 32 sectors at LBA = 0xb8227760 = 3089266528


 Commands leading to the command that caused the error were:
 CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
 -- == -- == -- == == == -- -- -- -- -- --------------- --------------------
 25 00 00 00 20 00 00 b8 22 77 60 40 00 15:08:01.820 READ DMA EXT
 25 00 00 00 20 00 00 b8 22 33 e0 40 00 15:08:01.820 READ DMA EXT
 35 00 00 00 08 00 01 c6 a7 b5 f0 40 00 15:08:01.819 WRITE DMA EXT
 35 00 00 00 08 00 01 5b 4b 07 a0 40 00 15:08:01.819 WRITE DMA EXT
 35 00 00 00 08 00 02 2d 0d 6f 18 40 00 15:08:01.819 WRITE DMA EXT

Alles anzeigen

SaiKo · 30. April 2016

Code

Error 7 [6] occurred at disk power-on lifetime: 323 hours (13 days + 11 hours)
 When the command that caused the error occurred, the device was active or idle.


 After command completion occurred, registers were:
 ER -- ST COUNT LBA_48 LH LM LL DV DC
 -- -- -- == -- == == == -- -- -- -- --
 40 -- 51 00 08 00 00 0d dd 95 d8 40 00 Error: UNC 8 sectors at LBA = 0x0ddd95d8 = 232625624


 Commands leading to the command that caused the error were:
 CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
 -- == -- == -- == == == -- -- -- -- -- --------------- --------------------
 25 00 00 00 08 00 00 0d dd 95 d8 40 00 14:30:27.904 READ DMA EXT
 25 00 00 00 20 00 00 0d db b2 78 40 00 14:30:27.904 READ DMA EXT
 25 00 00 00 20 00 00 0d db b1 f0 40 00 14:30:11.068 READ DMA EXT
 25 00 00 00 20 00 00 0d db b1 d0 40 00 14:30:11.068 READ DMA EXT
 25 00 00 00 80 00 00 0d db b1 b0 40 00 14:30:11.066 READ DMA EXT


Error 6 [5] occurred at disk power-on lifetime: 309 hours (12 days + 21 hours)
 When the command that caused the error occurred, the device was active or idle.


 After command completion occurred, registers were:
 ER -- ST COUNT LBA_48 LH LM LL DV DC
 -- -- -- == -- == == == -- -- -- -- --
 40 -- 51 00 80 00 00 00 40 00 a0 40 00 Error: UNC 128 sectors at LBA = 0x004000a0 = 4194464


 Commands leading to the command that caused the error were:
 CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
 -- == -- == -- == == == -- -- -- -- -- --------------- --------------------
 25 00 00 00 80 00 00 00 40 00 a0 40 00 00:02:41.171 READ DMA EXT
 35 00 00 00 10 00 02 ba a0 f2 90 40 00 00:02:23.962 WRITE DMA EXT
 35 00 00 00 10 00 02 ba a0 f0 90 40 00 00:02:23.962 WRITE DMA EXT
 35 00 00 00 10 00 00 00 40 02 90 40 00 00:02:23.962 WRITE DMA EXT
 25 00 00 00 10 00 02 ba a0 f2 90 40 00 00:02:23.962 READ DMA EXT


Error 5 [4] occurred at disk power-on lifetime: 309 hours (12 days + 21 hours)
 When the command that caused the error occurred, the device was active or idle.


 After command completion occurred, registers were:
 ER -- ST COUNT LBA_48 LH LM LL DV DC
 -- -- -- == -- == == == -- -- -- -- --
 10 -- 51 00 80 00 01 4c fd da 18 40 00 Error: IDNF 128 sectors at LBA = 0x14cfdda18 = 5586672152


 Commands leading to the command that caused the error were:
 CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
 -- == -- == -- == == == -- -- -- -- -- --------------- --------------------
 35 00 00 00 68 00 01 4c fb d1 b0 40 00 5d+07:05:36.957 WRITE DMA EXT
 35 00 00 00 80 00 01 4c fb d1 30 40 00 5d+07:05:36.949 WRITE DMA EXT
 35 00 00 00 70 00 01 4c fb d0 c0 40 00 5d+07:05:36.948 WRITE DMA EXT
 35 00 00 00 80 00 01 4c fb d0 40 40 00 5d+07:05:36.948 WRITE DMA EXT
 35 00 00 00 68 00 01 4c fb cf d8 40 00 5d+07:05:36.947 WRITE DMA EXT


SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 604 -
# 2 Short offline Completed without error 00% 581 -
# 3 Short offline Completed without error 00% 557 -
# 4 Short offline Completed without error 00% 533 -
# 5 Extended offline Completed without error 00% 517 -
# 6 Short offline Completed without error 00% 504 -
# 7 Short offline Completed without error 00% 353 -
# 8 Extended offline Completed without error 00% 349 -


SMART Selective self-test log data structure revision number 1
 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
 1 0 0 Not_testing
 2 0 0 Not_testing
 3 0 0 Not_testing
 4 0 0 Not_testing
 5 0 0 Not_testing
Selective self-test flags (0x0):
 After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 27 Celsius
Power Cycle Min/Max Temperature: 24/28 Celsius
Lifetime Min/Max Temperature: 17/40 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (123)


Index Estimated Time Temperature Celsius
 124 2016-04-30 10:42 27 ********
 ... ..(476 skipped). .. ********
 123 2016-04-30 18:39 27 ********


SCT Error Recovery Control:
 Read: 70 (7.0 seconds)
 Write: 70 (7.0 seconds)


SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 0 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 1 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000d 2 0 Non-CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 169349 Vendor specific

Alles anzeigen

Jetzt mitmachen!