Lucid's Fiber Channel Guide

lucid · 16. Januar 2014

UPDATED:3/26/2014

I CHANGED MANY THINGS WITH THE LAST UPDATE OF THIS GUIDE
- I removed the openiscsi portion of the guide, I still have some of the packages installed. I have to try this on a clean system to be sure if they are needed or not.
- I added the additional steps you need to add SCST's iSCSI.
- Startup uses init.d instead of rc.local

SCST does everything openiscsi does and more. SCST can also be used to present iSCSI as well as FC. So it's a very handy package on a storage array. SCST is supposed to be faster as well. For more info see:
http://scst.sourceforge.net/comparison.html

Donh has been awesome in pointing things out to me. I set things up and never went back to try it again, so he has been my "beta tester". --==THANK YOU Donh!==-- I hope you like the new changes I've made!

For this I used a qLogic 2460 card. Some are branded DELL or HP, they should all be the same. This one is a single port card, but there are 2 and 4 port cards as well. With the multi-port cards, you will have multiple WWN's.

You will see some DD commands to make blank image files or add to them. You can make them whatever size you need, you dont have to use the examples.

Also, in ESXi I was able to get my FC to fail-over to iSCSI.

Update OMV + debian

Add non-free contrib to sources.

Code

nano /etc/apt/sources.list

Code

deb http://debian.uchicago.edu/debian/ squeeze main non-free contrib
deb-src http://debian.uchicago.edu/debian/ squeeze main non-free contrib


deb http://security.debian.org/ squeeze/updates main contrib non-free
deb-src http://security.debian.org/ squeeze/updates main contrib non-free


# squeeze-updates, previously known as 'volatile'
deb http://debian.uchicago.edu/debian/ squeeze-updates main contrib non-free
deb-src http://debian.uchicago.edu/debian/ squeeze-updates main contrib non-free

Alles anzeigen

Code

apt-get update 
apt-get upgrade

** Configure your drives or RAID in the GUI now **
The following commands check rebuild speed setting, then increase raid rebuild speed setting. In a production system, set these to something that won't bog you down during a rebuild, but during the initial build this will help.

Code

sysctl dev.raid.speed_limit_min
sysctl dev.raid.speed_limit_max
sysctl -w dev.raid.speed_limit_max=20000000

SSD, FS, TCP tweaks
For me, I used a small 40gb intel SSD for OS. Not the fastest, but perfect for OMV.

SSD fstab options: discard,noatime,nodiratime
discard enables TRIM.
noatime, nodirtime disables inode time updates to reduce writes. (optional)

Code

nano /etc/fstab

Example:

Code

# / was on /dev/sda1 during installation
UUID=xxxxxxx / ext4 discard,noatime,nodiratime errors=remount-ro 0       1

A handy disk tool:
Next install IOtop. "iotop –only" is your friend. This shows what is using your drives and how fast.

Code

apt-get install iotop
iotop --only

Add to bottom of sysctl.conf, TCP speed bump:

Code

nano /etc/sysctl.conf

Code

net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_syncookies = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.core.netdev_max_backlog = 2500

Apt-get install lotsofstuff
Prereqs for later on and a few tools

Code

apt-get install fakeroot kernel-wedge build-essential makedumpfile kernel-package libncurses5 libncurses5-dev gcc libncurses5-dev linux-headers-$(uname -r) lsscsi patch subversion
apt-get build-dep --no-install-recommends linux-image-$(uname -r)
apt-get install iscsitarget-dkms  iscsitarget-source iscsitarget iscsitarget-module-dkms lsscsi less openmediavault-iscsitarget dnsutils pciutils firmware-qlogic

Check for qlogic firmware

Code

ls /lib/firmware/ | grep ql*

Sync to SCST trunk with subversion, this should make a folder called "trunk"

Code

cd /root
svn co https://svn.code.sf.net/p/scst/svn/trunk

Unload orig-driver, then blacklist it

Code

rmmod qla2xxx
echo blacklist qla2xxx > /etc/modprobe.d/blacklist-qla2xxx.conf

Start build

Install linux source code, extract it

Code

cd /usr/src/
apt-get install linux-source-2.6
tar xjf linux-source-2.6.32.tar.bz2

Link extracted source to linux folder (for ease)

Code

ln -s /usr/src/linux-source-2.6.32 linux
cd /usr/src/linux

Scst patches to kernel:

Code

patch -p1 < /root/trunk/scst/kernel/scst_exec_req_fifo-2.6.32.patch
patch -p1 < /root/trunk/iscsi-scst/kernel/patches/put_page_callback-2.6.32.patch

Get current config for kernel, copy it to working kernel source for build

Code

cp -vi /boot/config-`uname -r` .config

Rename old driver

Code

mv /usr/src/linux/drivers/scsi/qla2xxx /usr/src/linux/drivers/scsi/qla2xxx_orig

Link new driver to source build folder

Code

ln -s /root/trunk/qla2x00t /usr/src/linux/drivers/scsi/qla2xxx

export is to speed up build (use # of cores +1)

Code

cd /usr/src/linux
export CONCURRENCY_LEVEL="5"
make-kpkg clean
make menuconfig

In make menuconfig, make sure both are enabled:
Device Drivers> SCSI device support> SCSI low level drivers> [*]Qlogic 2xxx target mode support
Networking support> Networking options> [*]TCP/IP zero-copy transfer completion notification

Compile and make .deb's. Then you can install these later in another system

Code

fakeroot make-kpkg --initrd --append-to-version=-scst kernel-image kernel-headers
cd /usr/src/
ls

Install newly compiled kernel

Code

dpkg -i linux-image-2.6.32-scst_2.6.32-scst-10.00.Custom_amd64.deb
dpkg -i linux-headers-2.6.32-scst_2.6.32-scst-10.00.Custom_amd64.deb

Update ram images

Code

update-initramfs -u
reboot

Select new Kernel on boot!

Change GRUB_DEFAULT=0 to the entry on the boot menu, I used GRUB_DEFAULT=2 (The third one down). Then update grub.

Code

nano /etc/default/grub
update-grub

Reboot to make sure it works

Code

reboot

Build SCST + Kernel Modules
This installs EVERYTHING for SCST. Not a big deal, but you can be more specific. Then we check that the modules are present for the current working kernel.

Code

cd /root/trunk/
make 2release
make all
make install
ls -l /lib/modules/`uname -r`/extra

Setup SCST service. Ignore warning for now.

Code

apt-get install sysv-rc-conf
insserv scst
update-rc.d scst defaults

Run "sysv-rc-conf" and find "scst", put an X with the space bar under 2 3 4 5. Press "q" to quit.

Code

sysv-rc-conf

SCST Config

Get FC WWN: (target, i.e. SAN)

Code

cat /sys/class/fc_host/host*/port_name

Create SCST config

Code

nano /etc/scst.conf

Example config, use the right UUID's, WWN's. The iSCSI target has a group assigned for security. You can do the same for fiber, but it's not needed for an A to B setup. You will also notice that I have disk2 for the "TARGET_DRIVER qla2x00t" and for "TARGET_DRIVER iscsi". When you use this with ESXi, you get failover if the fiber fails. There are also mechanisms to password protect iSCSI, see the docs for SCST

vdisk_fileio attributes
blocksize 512, specifies block size, must be power of 2 and >= 512 bytes. Default is 512. (dont change with a formatted lun)
zero_copy 1, if set, device uses zero copy access to the page cache
rotational 1, if set disk is mechanical, not ssd
nv_cache 1, enables write through cache
filename X, location of file to write
removable 1, if set device is removeable (cdrom)
read_only 1, if set would make LUN read only.
thin_provisioned 1, if set device is thin disk

Code

HANDLER vdisk_fileio {
        DEVICE disk1 {
                filename /media/--==UUID==--/FC/disk1
                nv_cache 1
        }
        DEVICE disk2 {
                filename /media/--==UUID==--/FC/disk2
                nv_cache 1
        }
        DEVICE disk3 {
                filename /media/--==UUID==--/FC/disk3
                nv_cache 1
        }
}


TARGET_DRIVER qla2x00t {
enabled 1
        TARGET 00:00:00:00:00:00:00:00 {
                enabled 1
                LUN 0 disk1
                LUN 1 disk2
        }
}


TARGET_DRIVER iscsi {
enabled 1
        TARGET iqn.0000-00.lucid:iscsi.lun0 {
                enabled 1
                rel_tgt_id 1
        GROUP ESXi {
                LUN 0 disk2
                LUN 1 disk3
                INITIATOR iqn.0000-00.com.vmware:esxi-00000000
                }
        }
}

Alles anzeigen

Make Drive Image For Target
It is possible to use /dev/sda or /dev/md, and give the target the whole drive/array. See SCST manual. You could also make an image and mount that with a loopback and present it that way, just to give you a taste of what you can do. Read the SCST docs for more. I prefer working with images.

Make a place for the images

Code

mkdir /media/--UUID--/FC
cd /media/--UUID--/FC

Example DD commands.

Make 512Mb

Code

dd if=/dev/zero of=/disk1 bs=1024k count=512

Make 50Gb

Code

dd if=/dev/zero of=/disk1 bs=1G count=50

Add 1Gb to disk1 (with zeros)

Code

dd if=/dev/zero bs=1024k count=1024 >> /disk1

Make 512mb Sparse (thin) disk. (not reccomended, for speed)

Code

dd if=/dev/zero of=disk1 bs=1 count=0 seek=512M

Increase size to 1Gb (sparse)

Code

dd if=/dev/zero of=file.img bs=1 count=0 seek=1G

This should start scst

Code

/etc/init.d/scst start

This should restart scst

Code

/etc/init.d/scst restart

If you add a disk or resize one, update with this:

Code

scstadmin -resync_dev disk1

Reboot and make sure it comes back up as intended

Code

reboot

Done!!

donh · 31. März 2014

Thanks, seems to be working. Now to make proxmox see it.

PS: Removed some of the old stuff to clean this up and make it easier to read.

Jetzt mitmachen!