Hi everyone,
I'm new here and just started to work with OMV because I build me a NAS/VM Server for private usage. I'm an embedded software developer from Germany, so I'm able to understand most technical things, however I'm not a Linux-expert. I just wanted to give you some background and sorry for my bad English.
Ok, so let's start to describe my problem. I brought me a DELL T20 and put the hardware into a new case, brought 8 WD Red with 3TB, a SATA Controller, a SSD and some additional RAM and put everything together. Because I wanted to use the T20 for other services then a file server, too and because just out of curiosity, I installed XenServer 7 on it. Then I installed OMV3 into a VM and passed the SATA controller though to the VM. At first everything seemed to be ok, but after I enabled the HDD Write Cache in OMV, the Kernel started to throw ATA Error messages while coping files onto the server:
dmesg:
[ 240.620141] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[ 240.620164] ata3.00: failed command: FLUSH CACHE EXT
[ 240.620176] ata3.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 15
res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 240.620204] ata3.00: status: { DRDY }
[ 240.620213] ata3: hard resetting link
[ 240.935765] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 240.937342] ata3.00: configured for UDMA/133
[ 240.937344] ata3.00: retrying FLUSH 0xea Emask 0x4
[ 240.937492] ata3: EH complete
[ 623.852140] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[ 623.852545] ata3.00: failed command: FLUSH CACHE EXT
[ 623.852936] ata3.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 23
res 40/00:fe:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
[ 623.853738] ata3.00: status: { DRDY }
[ 623.854148] ata3: hard resetting link
[ 624.167909] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 624.169503] ata3.00: configured for UDMA/133
[ 624.169505] ata3.00: retrying FLUSH 0xea Emask 0x4
[ 624.169652] ata3: EH complete
At first I thought one of my disk were defect, so I permord a extended SMART test on every disk:
smartctl -a /dev/sda:
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.9.0-0.bpo.1-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, http://www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD30EFRX-68EUZN0
Serial Number: WD-WCC4N3NXT1AS
LU WWN Device Id: 5 0014ee 2b6d2dd29
Firmware Version: 82.00A82
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Mar 8 19:40:22 2017 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (39720) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 399) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x703d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 185 183 021 Pre-fail Always - 5708
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 114
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 179
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 99
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 62
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 81
194 Temperature_Celsius 0x0022 120 114 000 Old_age Always - 30
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 112 -
# 2 Extended offline Completed without error 00% 66 -
# 3 Short offline Completed without error 00% 58 -
# 4 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
As far as I can tell, the SMART data seems ok. So I started to switch the SATA cabels to other HDDs, traced back the ATA Channel to the actual device and noted the Serial-Number to determine if one special HDD is causing the problem, but I could not see any similarities between them. I got this errors with 6 HDDs. So for me, and I'm hoping that i'm right, not a HDD is causing the problem. I suspect it is the SATA controller or the fact, that I use PCI-Passthrough.
Anyway the write performance seems to be pretty bad, too:
dd if=/dev/zero of=testfile bs=1G count=1 (with cache enabled):
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 7.03636 s, 153 MB/s
dd if=/dev/zero of=testfile bs=1G count=1 (without cache enabled):
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 12.9104 s, 83.2 MB/s
Here some information about the raid:
mdadm --examine --brief --scan --config=partitions:
ARRAY /dev/md/Raid metadata=1.2 UUID=68a5bac2:9c6e0fdc:e933b308:9fb409f2 name=openmediavault:Raid
mdadm --detail /dev/md127:
/dev/md127:
Version : 1.2
Creation Time : Mon Feb 27 22:17:32 2017
Raid Level : raid6
Array Size : 17580810240 (16766.37 GiB 18002.75 GB)
Used Dev Size : 2930135040 (2794.39 GiB 3000.46 GB)
Raid Devices : 8
Total Devices : 8
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Mar 8 20:00:26 2017
State : clean
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : openmediavault:Raid (local to host openmediavault)
UUID : 68a5bac2:9c6e0fdc:e933b308:9fb409f2
Events : 189916
Number Major Minor RaidDevice State
0 8 48 0 active sync /dev/sdd
1 8 16 1 active sync /dev/sdb
8 8 64 2 active sync /dev/sde
3 8 0 3 active sync /dev/sda
4 8 96 4 active sync /dev/sdg
5 8 32 5 active sync /dev/sdc
6 8 80 6 active sync /dev/sdf
7 8 112 7 active sync /dev/sdh
I assigned 2 V-Cores to the VM and 6GB RAM. I also disabled the idle3-timeout on the WD Red HDDs. Any ideas what could cause this problem? Can you help me? I'm using OMV 3.0.64.
Greetings
m0nKeY