Fast RAID, slow iSCSI

    • OMV 1.0
    • Fast RAID, slow iSCSI

      Hi All

      I'm building a junkyard system. I've got a 3ghz core2 and 9 x 250gb HDDs

      I'm targetting a high iOPS, and reasonable sequential iSCSI device. (Actually the target application deliberately does not use disk cache (licensed feature..), so i'm trying to slip something underneath it)

      Using RAID5 or RAID4, I am getting great local performance, better than RAID10.

      The issue occurs on 4 and 5, but I will stick with RAID 4 though, because it makes this issue simpler..

      Random reads are 300+iops and reads and writes are 200+MB/s (no 'write bottleneck' issue with raid 4's single parity device)

      Win7 connected to the RAID via CIFS is great - i.e uploading a file to the share runs at full speed.

      Watching iostat, I can see that when writing, all devices are written to essentially equally, and no parity reads occur on the devices

      Now, iSCSI.

      Created an iSCSI to /dev/md0, connected to it using win7, and format with defaults.

      Writes are <30mb/s. iostat shows the parity drive is being written at 14M/s, the other 8 drives only getting 3M/s

      Im presuming this is some kind of read-modify-write issue.

      I can replicate it somewhat by using dd if=/dev/zero of=/dev/md0 with small block sizes, but speeds are not as bad

      I checked the NTFS partition alignment - it starts at 1024K (1meg), which is supposed to cover most RAID alignment issues..

      I tried different NTFS block sizes.

      I tried md0 with fileio in ietd.conf
      I also tried a file on ext4 on md0, using fileio
      I tried write-back on the iSCSI target

      A few things worked with varying trade offs-
      - Some good results by creating the RAID 4 with a 4K chunk size (ive forgotten what the side effect was)
      - /dev/loop0 to the md0 device allowed high sequential writes, but killed iops ( even locally, and iop test against loop0 was only 80)


      There must be something the iSCSI is doing, i.e what is the block size that ietd is using to write to disk?

      Next thing i will try is creating a network share, create a VHD and use win7's attach VHD feature


      Any ideas what is going on here?
    • I've experimetned using raid ten to eliminate parity issues.

      I tried even things like far and near(default) layout, and 3 disk mirror sets (f3, n3)

      I got some really high raw iops and throughput on the device

      But once I put an EXT4 filesystem on and then benchmarked, file write and copy results were a whole different story. The array would even just pause all disk activity every few seconds, for no apparent reason. EXT4 works fine on a default raid10 (with 4M chunks..)

      So anyway, I've settled on iSCSI, direct to md0, using fileio hack into the openmediavaul xml, default raid10 layout with 9 disks and 4M chunks.

      iSCSI is only getting 50mb writes, but reads are good, and fileio gives me cached reads (important because the application im using tries to avoid cache but it can't control the linux box!)

      Cheers.