Trouble with growing a RAID5 of SAS drives

Jopa · 1. September 2016

Hi all,

guess this will become a report rather than a question
First my history and intention:
After my proprietary 2-disk RAID1 NAS got to its limits, I looked around for a more flexible solution and found OMV. As it is debian linux based, and I'm familiar with debian linux so far, it looked well for a trial.
But now, as I'm running it, I got to have some trouble with it now and then.
Most of it obviously have to do with my decision for SAS drives. E.g. SMART shows "Status" greyed out, and only Extended Information availably, and when trying to add another drive to my running RAID5, the "grow" from OMV menu threw an error, and added the new drive as a spare. By googling, I found out, that
mdadm --grow --raid-devices=4 /dev/md0could to tell system of the new disk in RAID, and actually OMV web GUI showed RAID management's options "Grow" and "Remove" greyded out after this. As "Detail" showed 4th drive not as spare no more, I started "Recover", which is still running. (10% at the moment.) Hopefully I'll find space of 4th drive added, when Recover finishes.
To answer eventual questions in advance:
1. No, my RAID does not run out of 32bit/16TB limit. I'm just wanting to add a 4th 3TB drive to a RAID5.
2. OMV is running on a SATA drive, the RAID5 is on SAS drives.
3. I had no trouble with setting up a 3-SAS-drive RAID5. I just want to grow it.
4. My LSI-SAS HBA is definitively not the problem. It can work upto 8 drives with at least 4TB each. And the Linux driver works the same.

I'll come back, after recover finished. Nevertheless I'm interested in any experience with OMV and SAS drives.

Best regards,
Johannes

ryecoaaron · 2. September 2016

SAS drives shouldn't cause any problem. What kind of hardware are using? server, motherboard,model of LSI HBA, etc.

Jopa · 2. September 2016

Ok. Yes, and sorry, I did not tel...
MoBo is an intel s1200btl. CPU an i3-3240, RAM is 4GB ECC.
HBA is a LSISAS2008 or 2108 with IT-BIOS, i.e. pure HBA w/o hardware RAID support.
Actually this worked well with initial 3-drives-RAID5, I only run into problems when trying to a drive 4.

ryecoaaron · 2. September 2016

I don't see any problems with what you are doing or the hardware. I have three LSI HBAs in my server and a SAS LTO drive. No problems with it. I have also run OMV on an HP server with 8 SAS drives for a short time with no problems. So, it is hard to say. I would try one or more of the following:

- Update hba firmware
- Zero the 4th drive with dd if=/dev/zero of=/dev/sdX bs=512 count=100000
- Update the kernel to the backports kernel (there is button to do that in omv-extras).
- Start with 4 drive array
- Boot into systemrescuecd and try to grow the array (it has newer mdadm libaries).

Probably something I might think of later.

Jopa · 2. September 2016

Recover finished, while I was at work - and OMV is showing 8.1 TB for the RAID now. I.e. the 4th drive was added.

Now I just wonder, why OMV was not able to do that job by its own means. Why did I need to go down to debian command line and run mdadm by hand?

Btw, re your suggestions:

1. I spent some time to make the originally RAID (IR) controller run as a HBA (IT) with latest firmware.
2. I formatted the 4th drive twice, before I started this thread.
3. I always thought, adding just another drive of the same brand and series to a running software RAID5 should not depend on the kernel. And as results show, it actually did not depend on kernel.
4. Why? See 3.
5. Again: Why? See 3. and my starting post.

Best regards,
Johannes

ryecoaaron · 3. September 2016

mdadm has lots of problems and I am used to using the command line. So, it is more comfortable for me to give advice that way.

1 - ok
2 - You shouldn't format a raid array member. It should be zero'd.
3 - The kernel does matter. It has newer versions of mdadm modules that may fix previous problems.
4 - Why not? Then you don't have to grow it.
5 - systemrescuecd has newer mdadm libraries and a new kernel (see #3).

Just giving you advice from my experience. If you think they are wrong, try something else or file a bug report.

Jopa · 8. September 2016

Sorry, but you are not seeming to be really experienced in using SAS drives, are you?

Re 2: Of course I needed to format the drive, that I wanted to add to the RAID. Or were you able to add a drive formatted with 520B sectors to a RAID based on 512B sectors yet? Second format I just run to be safe, that first did not fail accidentally.

Re 4: It is kind of funny for me, to get recommended to trow all data away. Of course this may be the "easiest" way. Provided, I had a clean sytem and must not care about existing data. Only I needed to care about existing data.

BTW, as I posted, recover after OMV 3 default mdadm did include the 4th drive. So why should I still go to backports or use systemrescuecd? I mean, I just want to use OMV and not spend my time trying out this or that or yet something else.

Best regards,
Johannes

ryecoaaron · 9. September 2016

Zitat von Jopa

Sorry, but you are not seeming to be really experienced in using SAS drives, are you?

I have a stack of eight of 2.5" 146GB 10k rpm SAS drives on my desk at work and I maintain hundreds of them in servers at work. Does that count?

Zitat von Jopa

Of course I needed to format the drive, that I wanted to add to the RAID. Or were you able to add a drive formatted with 520B sectors to a RAID based on 512B sectors yet?

No, you don't want to format the drive. You want to zero it. You format the array not the individual drive. OMV doesn't use partitions on the drive. So, 512b sectors doesn't apply. Trust me. I very experienced with raid and sas drives...

Zitat von Jopa

It is kind of funny for me, to get recommended to trow all data away. Of course this may be the "easiest" way. Provided, I had a clean sytem and must not care about existing data. Only I needed to care about existing data.

"throwing the data away" wasn't my first choice. It was just an option. And really, you shouldn't lose any data. You should have a backup since raid is not a backup.

Zitat von Jopa

BTW, as I posted, recover after OMV 3 default mdadm did include the 4th drive. So why should I still go to backports or use systemrescuecd?

As I said in the previous post, the backport kernel and systemrescuecd have newer versions of mdadm and other drivers which help fix the array.

That is all I can offer. I don't have these issues on any of the servers I own or maintain with similar setups. Sorry if it isn't good enough...

Jopa · 11. September 2016

Sorry, and no: 146GB 10k rpm SAS drives are not comparable with drives of >2TB formatted with 520B sectors at all. Or to say: I actually needed an extra system running Cent-OS and use sg-format to get my disks formatted with 520B sectors reformatted to 512B per sector to get them usable with with any debian. Yes, I actually tried out not only OMV...
Btw: Linux world at all still not seems to be clear about supporting bigger SAS drives. No problem with drives of 147 or 300GB each, but if one goes beyond 2TB, common reaction is kind of "Uh, really want this?" rather than "Come on and get it", which I'd prefere.

ryecoaaron · 11. September 2016

Zitat von Jopa

146GB 10k rpm SAS drives are not comparable with drives of >2TB formatted

That is just what is on my desk since you said I don't understand SAS drives. We have SANs with 100s of TB of storage. Just a hint but they aren't using 146GB drives... I have a system that has a raid 10 array of WD 4tb SAS drives as well. Those drives use 4k sectors when used individual. I always though 520b sector size was a stupid NetApp idea (which we have those at work too) not a requirement of the drive. Most commercial SANs run Linux and plenty of them use 2TB+ SAS drives. So, this isn't a Linux problem.

Jetzt mitmachen!