Extending/Growing encrypted RAID5 on OMV 3.0.59

Hello OMV-Forum,

I have setup a fully encrypted OMV installation using this OMV wiki as well as this OMV thread.

Used version is OMV 3.0.59 (Erasmus).

Basically the RAID5 setup is (as I understand)

ext4 on /dev/mapper/abc

LUKS on /dev/mdxxx creates /dev/mapper/abc

RAID (mdadm) creates /dev/mdxxx

Currently the RAID5 is comprised of 4x 4TB HDDs (/, /boot, swap, etc. are stored on a separate SLC-USB-Stick).
I want to extend the RAID5 with another 2x 4TB HDDs, so it will be a total of 6x 4TB in the end.

Which would be the steps to be taken to add the two new drives to the RAID ?

I am no expert on this, but I guess I would need to
- add the HDDs to the RAID (mdadm)
- no changes should be necessary for the LUKS since it operates on the /dev/mdxxx ?
- extend the ext4 to the full size of the RAID

Could somebody tell me which steps to take (either in the webif or in the CLI) ?
Also should I add both HDDs at once or add one and then when its finished add the second one the same way ?

Thank you very much in advance!

raisOr

Hello doron,

thanks for the input!

I was not aware that I also had to resize the LUKS, after some research I came up with the following cookbook so far.

Unfortunately currently I do not have a testsystem on which I can try this, in the other OMV NAS that I have all bays are already filled with disks

Update 20170312: I was able to successfully grow my RAID5 using the following cookbook

1. Umount filesystem of the RAID5
umount /media/<xyz>

2. close the crypto device
cryptsetup luksClose <crypt device> (taken from /dev/mapper/<crypt device>)

3. Add new drives to RAID5 (before: 4x4TB, after:6x4TB, new drives: /dev/sdbX and /dev/sdbY)
mdadm --add /dev/mdX /dev/sdbX /dev/sdbY
mdadm --grow --raid-devices=6 --backup-file=/root/grow_mdX.bak /dev/mdX

Reshape of the RAID5 with the 2 new drives took around 26hrs to complete. Progress can be checked using cat /proc/mdstat

4. open the crypto device
cryptsetup luksOpen -d /path/to/keyfile /dev/mdX <crypt device>

5. resize the crypto device
cryptsetup resize <crypt device>

6. check and resize the ext4
e2fsck -fn /dev/mapper/<crypt device>
resize2fs -p /dev/mapper/<crypt device>

Note:
The size of the grown RAID5 would be ~19TB, therefore resize2fs gave the following error on my machine:
resize2fs: Size of device /dev/mapper/<crypt device> too big to be expressed in 32 bit susing a blocksize of 4096.

It seems like new ext4 filesystems automatically get created using the -O 64bit option, therefore allowing them to have maximzum size of1024 PiB.
If the ext4 has been created in "32bit" (which was the case on my OMV machine), then the maximum size of the filesystem is limited to 16TiB.

This topic has been adressed in e2fsprogs Version 1.43, unfortunately it seems this version has not made it yet into the jessie repository.
When running dpkg --list | grep e2fsprogs it would report e2fsprogs 1.42.12-2+b1 on my OMV machine.

In order to adress this, you have to build the e2fsprogs Version 1.43 from its sources and then resize the filesystem by running the following commands with the new e2fsprogs version

6.1 download and build e2fsprogs Version 1.43

apt-get install git
cd /usr/local/src
git clone git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git
cd e2fsprogs
./configure
make

6.2 resize the filesystem using e2fsprogs Version 1.43

cd /usr/local/src/e2fsprogs/
./resize/resize2fs
Check whether resize2fs reports version "1.43.5-WIP" (or similar newer version)

./resize/resize2fs -b /dev/mapper/<crypt device>
This will convert the "32bit ext4" to a "64bit ext4". It took approx. 10mins to complete.

./resize/resize2fs -p /dev/mapper/<crypt device>
This will resize the ext4 to the size of the RAID5. It also took around 10min to complete.

./e2fsck/e2fsck -fn /dev/mapper/<crypt device
Check the filesystem for any errors.

7. mount the ext4 filesystem
mount /dev/mapper/<crypt device> /media/<xyz> 

Note:
It seems like systemd was unlocking and automounting my RAID5 during the procedure from time to time. I did not investigate further into this, but I checked after each step that the filesystem was unmounted and the luks device was closed. When it was unlocked/mounted I would just run the command in 1. and 2.
Luckily it did not break anything when I grew my RAID, but maybe somebody has some input in how to prevent systemd (or whatever service was causing this behaviour) from unlocking/mounting the filesystem.

Regards,
raisOr

Hello doron,

following your advice, I have built a Virtual Machine of the OMV setup that I have.
I could then sucesssfully test the cookbook and extend the RAID5 in the VM.

When growing my RAID on the actual server, everything behaved like in the VM, except that I ran into some problems with a limitation in the ext4 (for details see post #3 in this thread, under section 6.).

I could successfully fix that using the steps described in 6.1 and 6.2 and have now successfully extended my RAID5 with the 2 new drives.

I have updated the post #3 so it now documents the steps that I have taken to successfully extend my RAID5.

Special Thanks @doron for not only supporting me with the initial encrypted setup a while back, but also for again giving me great input in this case.