Lucid's Fiber Channel Guide

    • Lucid's Fiber Channel Guide

      UPDATED:3/26/2014

      I CHANGED MANY THINGS WITH THE LAST UPDATE OF THIS GUIDE

      - I removed the openiscsi portion of the guide, I still have some of the packages installed. I have to try this on a clean system to be sure if they are needed or not.
      - I added the additional steps you need to add SCST's iSCSI.
      - Startup uses init.d instead of rc.local

      SCST does everything openiscsi does and more. SCST can also be used to present iSCSI as well as FC. So it's a very handy package on a storage array. SCST is supposed to be faster as well. For more info see:
      scst.sourceforge.net/comparison.html

      Donh has been awesome in pointing things out to me. I set things up and never went back to try it again, so he has been my "beta tester". :D --==THANK YOU Donh!==-- I hope you like the new changes I've made!

      For this I used a qLogic 2460 card. Some are branded DELL or HP, they should all be the same. This one is a single port card, but there are 2 and 4 port cards as well. With the multi-port cards, you will have multiple WWN's.

      You will see some DD commands to make blank image files or add to them. You can make them whatever size you need, you dont have to use the examples.

      Also, in ESXi I was able to get my FC to fail-over to iSCSI. :D



      Update OMV + debian

      Add non-free contrib to sources.

      Source Code

      1. nano /etc/apt/sources.list


      Source Code

      1. deb http://debian.uchicago.edu/debian/ squeeze main non-free contrib
      2. deb-src http://debian.uchicago.edu/debian/ squeeze main non-free contrib
      3. deb http://security.debian.org/ squeeze/updates main contrib non-free
      4. deb-src http://security.debian.org/ squeeze/updates main contrib non-free
      5. # squeeze-updates, previously known as 'volatile'
      6. deb http://debian.uchicago.edu/debian/ squeeze-updates main contrib non-free
      7. deb-src http://debian.uchicago.edu/debian/ squeeze-updates main contrib non-free


      Source Code

      1. apt-get update
      2. apt-get upgrade


      ** Configure your drives or RAID in the GUI now **
      The following commands check rebuild speed setting, then increase raid rebuild speed setting. In a production system, set these to something that won't bog you down during a rebuild, but during the initial build this will help.

      Source Code

      1. sysctl dev.raid.speed_limit_min
      2. sysctl dev.raid.speed_limit_max
      3. sysctl -w dev.raid.speed_limit_max=20000000


      SSD, FS, TCP tweaks
      For me, I used a small 40gb intel SSD for OS. Not the fastest, but perfect for OMV.

      SSD fstab options: discard,noatime,nodiratime
      discard enables TRIM.
      noatime, nodirtime disables inode time updates to reduce writes. (optional)

      Source Code

      1. nano /etc/fstab


      Example:

      Source Code

      1. # / was on /dev/sda1 during installation
      2. UUID=xxxxxxx / ext4 discard,noatime,nodiratime errors=remount-ro 0 1


      A handy disk tool:
      Next install IOtop. "iotop –only" is your friend. This shows what is using your drives and how fast.

      Source Code

      1. apt-get install iotop
      2. iotop --only


      Add to bottom of sysctl.conf, TCP speed bump:

      Source Code

      1. nano /etc/sysctl.conf


      Source Code

      1. net.ipv4.tcp_window_scaling = 1
      2. net.ipv4.tcp_syncookies = 1
      3. net.core.rmem_max = 16777216
      4. net.core.wmem_max = 16777216
      5. net.ipv4.tcp_rmem = 4096 87380 16777216
      6. net.ipv4.tcp_wmem = 4096 65536 16777216
      7. net.ipv4.tcp_no_metrics_save = 1
      8. net.ipv4.tcp_moderate_rcvbuf = 1
      9. net.core.netdev_max_backlog = 2500


      Apt-get install lotsofstuff
      Prereqs for later on and a few tools

      Source Code

      1. apt-get install fakeroot kernel-wedge build-essential makedumpfile kernel-package libncurses5 libncurses5-dev gcc libncurses5-dev linux-headers-$(uname -r) lsscsi patch subversion
      2. apt-get build-dep --no-install-recommends linux-image-$(uname -r)
      3. apt-get install iscsitarget-dkms iscsitarget-source iscsitarget iscsitarget-module-dkms lsscsi less openmediavault-iscsitarget dnsutils pciutils firmware-qlogic


      Check for qlogic firmware

      Source Code

      1. ls /lib/firmware/ | grep ql*


      Sync to SCST trunk with subversion, this should make a folder called "trunk"

      Source Code

      1. cd /root
      2. svn co https://svn.code.sf.net/p/scst/svn/trunk


      Unload orig-driver, then blacklist it

      Source Code

      1. rmmod qla2xxx
      2. echo blacklist qla2xxx > /etc/modprobe.d/blacklist-qla2xxx.conf



      Start build

      Install linux source code, extract it

      Source Code

      1. cd /usr/src/
      2. apt-get install linux-source-2.6
      3. tar xjf linux-source-2.6.32.tar.bz2


      Link extracted source to linux folder (for ease)

      Source Code

      1. ln -s /usr/src/linux-source-2.6.32 linux
      2. cd /usr/src/linux


      Scst patches to kernel:

      Source Code

      1. patch -p1 < /root/trunk/scst/kernel/scst_exec_req_fifo-2.6.32.patch
      2. patch -p1 < /root/trunk/iscsi-scst/kernel/patches/put_page_callback-2.6.32.patch


      Get current config for kernel, copy it to working kernel source for build

      Source Code

      1. cp -vi /boot/config-`uname -r` .config


      Rename old driver

      Source Code

      1. mv /usr/src/linux/drivers/scsi/qla2xxx /usr/src/linux/drivers/scsi/qla2xxx_orig


      Link new driver to source build folder

      Source Code

      1. ln -s /root/trunk/qla2x00t /usr/src/linux/drivers/scsi/qla2xxx


      export is to speed up build (use # of cores +1)

      Source Code

      1. cd /usr/src/linux
      2. export CONCURRENCY_LEVEL="5"
      3. make-kpkg clean
      4. make menuconfig


      In make menuconfig, make sure both are enabled:
      Device Drivers> SCSI device support> SCSI low level drivers> [*]Qlogic 2xxx target mode support
      Networking support> Networking options> [*]TCP/IP zero-copy transfer completion notification

      Compile and make .deb's. Then you can install these later in another system

      Source Code

      1. fakeroot make-kpkg --initrd --append-to-version=-scst kernel-image kernel-headers
      2. cd /usr/src/
      3. ls


      Install newly compiled kernel

      Source Code

      1. dpkg -i linux-image-2.6.32-scst_2.6.32-scst-10.00.Custom_amd64.deb
      2. dpkg -i linux-headers-2.6.32-scst_2.6.32-scst-10.00.Custom_amd64.deb


      Update ram images

      Source Code

      1. update-initramfs -u
      2. reboot

      Select new Kernel on boot!

      Change GRUB_DEFAULT=0 to the entry on the boot menu, I used GRUB_DEFAULT=2 (The third one down). Then update grub.

      Source Code

      1. nano /etc/default/grub
      2. update-grub


      Reboot to make sure it works

      Source Code

      1. reboot


      Build SCST + Kernel Modules
      This installs EVERYTHING for SCST. Not a big deal, but you can be more specific. Then we check that the modules are present for the current working kernel.

      Source Code

      1. cd /root/trunk/
      2. make 2release
      3. make all
      4. make install
      5. ls -l /lib/modules/`uname -r`/extra


      Setup SCST service. Ignore warning for now.

      Source Code

      1. apt-get install sysv-rc-conf
      2. insserv scst
      3. update-rc.d scst defaults


      Run "sysv-rc-conf" and find "scst", put an X with the space bar under 2 3 4 5. Press "q" to quit.

      Source Code

      1. sysv-rc-conf


      SCST Config

      Get FC WWN: (target, i.e. SAN)

      Source Code

      1. cat /sys/class/fc_host/host*/port_name


      Create SCST config

      Source Code

      1. nano /etc/scst.conf


      Example config, use the right UUID's, WWN's. The iSCSI target has a group assigned for security. You can do the same for fiber, but it's not needed for an A to B setup. You will also notice that I have disk2 for the "TARGET_DRIVER qla2x00t" and for "TARGET_DRIVER iscsi". When you use this with ESXi, you get failover if the fiber fails. :D There are also mechanisms to password protect iSCSI, see the docs for SCST

      vdisk_fileio attributes
      blocksize 512, specifies block size, must be power of 2 and >= 512 bytes. Default is 512. (dont change with a formatted lun)
      zero_copy 1, if set, device uses zero copy access to the page cache
      rotational 1, if set disk is mechanical, not ssd
      nv_cache 1, enables write through cache
      filename X, location of file to write
      removable 1, if set device is removeable (cdrom)
      read_only 1, if set would make LUN read only.
      thin_provisioned 1, if set device is thin disk

      Source Code

      1. HANDLER vdisk_fileio {
      2. DEVICE disk1 {
      3. filename /media/--==UUID==--/FC/disk1
      4. nv_cache 1
      5. }
      6. DEVICE disk2 {
      7. filename /media/--==UUID==--/FC/disk2
      8. nv_cache 1
      9. }
      10. DEVICE disk3 {
      11. filename /media/--==UUID==--/FC/disk3
      12. nv_cache 1
      13. }
      14. }
      15. TARGET_DRIVER qla2x00t {
      16. enabled 1
      17. TARGET 00:00:00:00:00:00:00:00 {
      18. enabled 1
      19. LUN 0 disk1
      20. LUN 1 disk2
      21. }
      22. }
      23. TARGET_DRIVER iscsi {
      24. enabled 1
      25. TARGET iqn.0000-00.lucid:iscsi.lun0 {
      26. enabled 1
      27. rel_tgt_id 1
      28. GROUP ESXi {
      29. LUN 0 disk2
      30. LUN 1 disk3
      31. INITIATOR iqn.0000-00.com.vmware:esxi-00000000
      32. }
      33. }
      34. }
      Display All


      Make Drive Image For Target
      It is possible to use /dev/sda or /dev/md, and give the target the whole drive/array. See SCST manual. You could also make an image and mount that with a loopback and present it that way, just to give you a taste of what you can do. Read the SCST docs for more. I prefer working with images.

      Make a place for the images

      Source Code

      1. mkdir /media/--UUID--/FC
      2. cd /media/--UUID--/FC


      Example DD commands.

      Make 512Mb

      Source Code

      1. dd if=/dev/zero of=/disk1 bs=1024k count=512


      Make 50Gb

      Source Code

      1. dd if=/dev/zero of=/disk1 bs=1G count=50


      Add 1Gb to disk1 (with zeros)

      Source Code

      1. dd if=/dev/zero bs=1024k count=1024 >> /disk1


      Make 512mb Sparse (thin) disk. (not reccomended, for speed)

      Source Code

      1. dd if=/dev/zero of=disk1 bs=1 count=0 seek=512M


      Increase size to 1Gb (sparse)

      Source Code

      1. dd if=/dev/zero of=file.img bs=1 count=0 seek=1G


      This should start scst

      Source Code

      1. /etc/init.d/scst start


      This should restart scst

      Source Code

      1. /etc/init.d/scst restart


      If you add a disk or resize one, update with this:

      Source Code

      1. scstadmin -resync_dev disk1


      Reboot and make sure it comes back up as intended

      Source Code

      1. reboot


      Done!!

      The post was edited 14 times, last by lucid ().