RAID 6 gone, physical drives visible

Hey good news, mdadm seems to have enough informations to start the rebuild. But I see a faulty spare, was there one before?

I would let it rebuild without further action, then make a backup of all your data and start from scratch with a fresh installation of the whole box. And check all the hardware for other problems, such a thunderstorm can do a lot of not so funny things. That's why I have a UPS for all my boxes.

Even if the raid is rebuilding you should see your shared folders from a windows box. Do you?

Hmm....looks like we have breaked the champagne too early...
Is the raid visible in the web-ui? What is the result of blkid? Any reasons in the logs why mdadm has stopped the rebuild?

@ahab666

For the last time and I give up now, the disk which started with all the problems should be zero-wiped or you remove the super-block, with your knowledge I suggest to use either the WD tool or DBAN. I am not going to repeat myself for the third time, just read back.

Once you have wiped the disk clean and checked that all is fine, add it back and then start again.
The raid will simple fail with that corrupt disk, if that disk and only THAT disk is zero-wiped, the raid will start to rebuild.
The danger is now that 7 & 8 are in one set and you will break the raid, so only wipe 7 and start again.

Best of luck.

If these two disks are a part of the raid don't forget to set them as failed and remove them before you wipe.
mdadm --manage /dev/md0 --fail /dev/sdi
mdadm --manage /dev/md0 --remove /dev/sdi
and so on...otherwise the whole raid can break.

Afterwards try to re-add them again.

@ahab666

Alex, how are things going?

@ahab666

Oh dear, why couldn't you just wait till the whole raid was rebuild.
I have no clue yet, try to stop samba and any other service like ftp rsync, stop all of them..

Then try a reboot.

Then DBAN the other disk and don't do anything else till the raid is rebuild 100%

@ahab666

Any news?

@ahab666

DBAN as follow, M = method, quick erase, V =verify ALL passes, F10 start, but before you do this make a screen-shot of the SMART write also down the re-allocated sectors. After one run you check SMART again is the re-allocated increased, then you have a bad disc, most disc can do reliable a few re-allocations, but then your gambling!!!

Is the failed disc still under warranty? get it RMA, I wouldn't risk to put it back in the raid then.

@ahab666

Alex, the first thing you always do when you build a raid, you label the disk and write down the serial and label number in a notepad file or spreadsheet whatever you prefer. You also check the disk SMART details and make a note of it or a screen-shot. At the moment with BLKID command you can identify the disks, so it should not be that difficult to work out which physical disk is. SMART in OMV also gives such information, no need for any other tools in OMV, you have all you need. Another option is mdadm --detail /dev/mdx [x=127 or whatever it is] with cat /proc/mdstat you can get more info about your mdx number. Regards the log files you can use in Window$ the Notepad++ editor.

I suggest before you do anything else, get your most important data of the raid!!!

Then zero-wipe the new disk to make sure it goes in without any problems don't take any risk!

Once you have your most important data you can DBAN every disk if you like, you have 12 so taking one out a time and check/replace should not be any problem. If you bought all 12 in one order the problem might be you will run in time in what I call a batch issue, all disk same age, might die at same time!
Make sure you monitor your raid and make sure you have enabled the warnings by email, and check yourself the SMART status in raid, the only reliable way is you check this yourself. Don't forget new disks can die fast as well, like humans some get very old and some die sadly very young!

Now the most important part is that you didn't tell what was wrong with the disk, did it die completely, a lot of bad sectors, still in warranty?
Anyway you better get to the bottom if it was down to the surge pike or just having old disks [who to blame] or anything else. But like has been suggested get at least an UPS or better in combination with a surge protector in your consumer unit, cost as little as £60 here in the UK. We have this one: http://www.dehn-usa.com/pdbRes…mmer-pdf/31307/952070.pdf But it might be different in your country.

Time for a coffee.

@ahab666

Try these.

hdparm -i /dev/sdx

smartctl -a /dev/sdx | more

lsblk

lsblk -o name,kname,label,uuid,state

x is drive letter.
| more let you scroll through each page.
lsblk will show the mdx
Look on the disk label it should show the serial...

Edit: do you have WD drives?

You better check in the SMART what the load cycle count is, if it is very high you need to patch your drives with wdidle3
This applies to many of their products in all colour ranges, I myself have found: red, green, blue and black drives.
There are rumours that this is killing drives in a raid, but you better check each drive if it has.

A good tool is the UBCD Ultimate boot CD, it has this little tool on the CD-ROM, boot from it and check each disk.