Unable to access RAID volumes

tkripala · 24. Juli 2013

I have been recently experiencing an odd problem but have not been able to pinpoint the cause of the problem and looking for some help. Here is the scenario:

I have 2 RAID1 volumes. One is 3TB and the other is 750GB.
The 3TB volume has a number of directories shared via samba and mounted to my Windows machines.
These volumes are no longer accessible.
When I log into the OMV web page, I am unable to load the Storage pages (physical disks, RAID Management, Filesystem). The pages sit in loading and then errors out.
On the command line, I executed 'cat /proc/mdstat' and it shows the following:

Code

root@homeserver:~# cat /proc/mdstat
Personalities : [raid1]
md126 : active raid1 sda[0] sdb[1]
      2930265424 blocks super 1.2 [2/2] [UU]
      [==>..................]  resync = 11.1% (325492288/2930265424) finish=252026.3min speed=172K/sec


md127 : active raid1 sdd[0] sdc[1]
      732573424 blocks super 1.2 [2/2] [UU]


unused devices: <none>

Alles anzeigen

The first volume has been stuck at 11.1% resync for the last 24 hours.

When I execute fdisk -l, it hangs after /dev/sda.
When I execute blkid, the command hangs.

I'm not sure what else to try to pinpoint the problem.

Any ideas or suggestions?

Thanks!

ryecoaaron · 24. Juli 2013

Your raid is resyncing but very slowly. Sounds like a drive might be going bad?? Looks for errors in dmesg

tkripala · 25. Juli 2013

Thanks for the tip. I think I may indeed have a bad drive. Here is the offending section (I believe)

Code

[    8.204785] mdadm: sending ioctl 1261 to a partition!
[    8.204788] mdadm: sending ioctl 1261 to a partition!
[    8.212869] md: md127 stopped.
[    8.214213] md: bind<sdc>
[    8.214330] md: bind<sdd>
[    8.217324] md: raid1 personality registered for level 1
[    8.217436] raid1: raid set md127 active with 2 out of 2 mirrors
[    8.217453] md127: detected capacity change from 0 to 750155186176
[    8.218351]  md127: unknown partition table
[    8.421465] mdadm: sending ioctl 1261 to a partition!
[    8.421468] mdadm: sending ioctl 1261 to a partition!
[    8.425564] md: md126 stopped.
[    8.426589] md: bind<sdb>
[    8.426727] md: bind<sda>
[    8.427616] raid1: md126 is not clean -- starting background reconstruction
[    8.427619] raid1: raid set md126 active with 2 out of 2 mirrors
[    8.427635] md126: detected capacity change from 0 to 3000591794176
[    8.428494]  md126:
[    8.428521] md: resync of RAID array md126
[    8.428523] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[    8.428524] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[    8.428528] md: using 128k window, over a total of 2930265424 blocks.
[    8.450905]  unknown partition table
[    9.087374] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled
[    9.087783] SGI XFS Quota Management subsystem
[    9.202065] XFS mounting filesystem md126
[    9.318665] Starting XFS recovery on filesystem: md126 (logdev: internal)
[    9.819947] Ending XFS recovery on filesystem: md126 (logdev: internal)
[    9.828509] fuse init (API version 7.13)
[   10.043472] XFS mounting filesystem md127
[   10.247482] Ending clean XFS mount for filesystem: md127
[   10.324582] EXT4-fs (sdf1): mounted filesystem with ordered data mode
[   10.418721] r8169 0000:03:00.0: eth0: link down
[   10.418728] r8169 0000:03:00.0: eth0: link down
[   10.418880] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   14.047774] r8169 0000:03:00.0: eth0: link up
[   14.047943] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   15.808272] Software Watchdog Timer: 0.07 initialized. soft_noboot=0 soft_margin=60 sec (nowayout= 0)
[   16.140109] lp: driver loaded but no devices found
[   16.142841] ppdev: user-space parallel port driver
[ 1896.328942] ata4.00: exception Emask 0x0 SAct 0xffff SErr 0x0 action 0x6 frozen
[ 1896.328992] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.329023] ata4.00: cmd 61/80:00:00:4c:cd/00:00:26:00:00/40 tag 0 ncq 65536 out
[ 1896.329024]          res 40/00:00:08:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
[ 1896.329109] ata4.00: status: { DRDY }
[ 1896.329133] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.329163] ata4.00: cmd 61/00:08:80:4e:cd/01:00:26:00:00/40 tag 1 ncq 131072 out
[ 1896.329165]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.329257] ata4.00: status: { DRDY }
[ 1896.329281] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.329310] ata4.00: cmd 61/00:10:00:47:cd/01:00:26:00:00/40 tag 2 ncq 131072 out
[ 1896.329312]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.329397] ata4.00: status: { DRDY }
[ 1896.329420] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.329450] ata4.00: cmd 61/80:18:00:48:cd/00:00:26:00:00/40 tag 3 ncq 65536 out
[ 1896.329451]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.329536] ata4.00: status: { DRDY }
[ 1896.329559] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.329589] ata4.00: cmd 61/80:20:00:3d:cb/00:00:26:00:00/40 tag 4 ncq 65536 out
[ 1896.329590]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.329675] ata4.00: status: { DRDY }
[ 1896.329698] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.329728] ata4.00: cmd 61/80:28:80:45:cd/01:00:26:00:00/40 tag 5 ncq 196608 out
[ 1896.329729]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.329814] ata4.00: status: { DRDY }
[ 1896.329837] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.329867] ata4.00: cmd 61/00:30:80:4f:cd/01:00:26:00:00/40 tag 6 ncq 131072 out
[ 1896.329868]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.329953] ata4.00: status: { DRDY }
[ 1896.329976] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.330005] ata4.00: cmd 61/80:38:80:4b:cd/00:00:26:00:00/40 tag 7 ncq 65536 out
[ 1896.330007]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.330091] ata4.00: status: { DRDY }
[ 1896.330114] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.330144] ata4.00: cmd 61/80:40:80:50:cd/01:00:26:00:00/40 tag 8 ncq 196608 out
[ 1896.330145]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.330230] ata4.00: status: { DRDY }
[ 1896.330253] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.330283] ata4.00: cmd 61/80:48:00:49:cd/00:00:26:00:00/40 tag 9 ncq 65536 out
[ 1896.330284]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.330369] ata4.00: status: { DRDY }
[ 1896.330392] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.330421] ata4.00: cmd 61/80:50:80:48:cd/00:00:26:00:00/40 tag 10 ncq 65536 out
[ 1896.330423]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.330508] ata4.00: status: { DRDY }
[ 1896.331092] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.331121] ata4.00: cmd 61/80:58:80:a5:cb/00:00:26:00:00/40 tag 11 ncq 65536 out
[ 1896.331123]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.331208] ata4.00: status: { DRDY }
[ 1896.331231] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.331260] ata4.00: cmd 61/00:60:80:4c:cd/02:00:26:00:00/40 tag 12 ncq 262144 out
[ 1896.331262]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.331347] ata4.00: status: { DRDY }
[ 1896.331370] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.331400] ata4.00: cmd 61/80:68:00:4b:cd/00:00:26:00:00/40 tag 13 ncq 65536 out
[ 1896.331401]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.331486] ata4.00: status: { DRDY }
[ 1896.331509] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.331539] ata4.00: cmd 61/80:70:80:4a:cd/00:00:26:00:00/40 tag 14 ncq 65536 out
[ 1896.331540]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.331625] ata4.00: status: { DRDY }
[ 1896.331648] ata4.00: failed command: WRITE FPDMA QUEUED
[ 1896.331678] ata4.00: cmd 61/00:78:80:49:cd/01:00:26:00:00/40 tag 15 ncq 131072 out
[ 1896.331679]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 1896.331764] ata4.00: status: { DRDY }
[ 1896.331790] ata4: hard resetting link
[ 1906.317836] ata4: softreset failed (device not ready)
[ 1906.317867] ata4: hard resetting link
[ 1916.306740] ata4: softreset failed (device not ready)
[ 1916.306771] ata4: hard resetting link
[ 1926.854799] ata4: link is slow to respond, please be patient (ready=0)
[ 1951.289871] ata4: softreset failed (device not ready)
[ 1951.289903] ata4: limiting SATA link speed to 1.5 Gbps
[ 1951.289906] ata4: hard resetting link
[ 2037.615447] INFO: task md126_resync:1001 blocked for more than 120 seconds.
[ 2037.615481] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2037.615525] md126_resync  D f64733c0     0  1001      2 0x00000000
[ 2037.615531]  f6160cc0 00000046 c138aba0 f64733c0 00000001 c141f120 c141f120 c141a6ac
[ 2037.615538]  f6160e7c c3408120 00000000 00000000 00000003 f64733cc f64733c8 00000000
[ 2037.615543]  c34036ac f6160e7c 0005fad5 00000000 00000000 00000000 00000000 00000000
[ 2037.615549] Call Trace:
[ 2037.615560]  [<f852e422>] ? raise_barrier+0x103/0x130 [raid1]
[ 2037.615568]  [<c102d772>] ? default_wake_function+0x0/0x8
[ 2037.615572]  [<f852ee6d>] ? sync_request+0x1da/0x5e7 [raid1]
[ 2037.615578]  [<c1024b3c>] ? __wake_up+0x29/0x39
[ 2037.615584]  [<f8504c36>] ? is_mddev_idle+0x83/0xd1 [md_mod]
[ 2037.615590]  [<f850549c>] ? md_do_sync+0x818/0xcb1 [md_mod]
[ 2037.615597]  [<c1044536>] ? autoremove_wake_function+0x0/0x2d
[ 2037.615603]  [<f8505f8d>] ? md_thread+0x0/0xf8 [md_mod]
[ 2037.615608]  [<f850606e>] ? md_thread+0xe1/0xf8 [md_mod]
[ 2037.615612]  [<c1024ab3>] ? complete+0x28/0x36
[ 2037.615617]  [<f8505f8d>] ? md_thread+0x0/0xf8 [md_mod]
[ 2037.615621]  [<c1044304>] ? kthread+0x61/0x66
[ 2037.615625]  [<c10442a3>] ? kthread+0x0/0x66
[ 2037.615630]  [<c1003d47>] ? kernel_thread_helper+0x7/0x10
[ 2037.615643] INFO: task smbd:2001 blocked for more than 120 seconds.

Alles anzeigen

If indeed that is the case, I should be able to mark the drive as a failed drive using mdadm, right?

Just want to confirm before I do anything.

Thanks!

ryecoaaron · 25. Juli 2013

yes. You can mark it as failed.

tkripala · 26. Juli 2013

So, in the course of troubleshooting to identify the likely failed volume, either my CPU or my motherboard got fried and the system is completely dead. I'm keeping my fingers crossed that I can simply replace my motherboard and recover my system.

Sigh!

Jetzt mitmachen!