RAID6 benchmarking

    Hello everyone :)


    I have finally got around to building my "final" NAS, comprised of 6x WD red 4TB drives in raid6, using a LSI 9211-8i controller card (dumb).


    I have built the storage partition using RAID6 across all 6 disks, giving 14.5Tb of storage. when the raid was initialised, it took around 12 hours.


    OMV is at version 1.6 and is untweaked. the server has 8gb ram and currently a single socket quad core xeon 5500 running at 2.26ghz


    Code
    root@OMV:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdd[0] sdf[5] sdb[4] sdc[3] sda[2] sde[1]
      15627548672 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/6] [UUUUUU]


unused devices: <none>
root@OMV:~#


    DD testing reads

    Code
    root@OMV:~# dd if=/dev/md0 of=tempfile bs=1MB count=50240
50240+0 records in
50240+0 records out
50240000000 bytes (50 GB) copied, 653.376 s, 76.9 MB/s


    CPU usage whilst running the above

    Code
    32143 root      20   0 10856 1428  580 D [b] 25.9 [/b] 0.0   0:47.56 dd
   53 root      20   0     0    0    0 S [b]  6.5[/b]  0.0   0:25.37 kswapd0
 1059 root      20   0     0    0    0 D[b]   6.5[/b]  0.0   0:20.54 flush-8:96



    DD testing writes

    Code
    root@OMV:~# dd of=/dev/md0 if=tempfile bs=1MB count=50240
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 47.7949 s, 214 MB/s




    the drives are all 4k and according to my interpretation of fdisk -l they are operating in 4k mode. sdg is the boot SSD



    everything looks ok to me? i havent found that much info from people running RAID6, so hopefully this post will help shed a little light on expected performance levels.

    Writing is fine, speeds won't be as fast as with raid5. however the read speed seems to be too low.. Did you check hdparm on the array?


    Greetings
    David

    "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"


    Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.

    Upload Logfile via WebGUI/CLI
    #openmediavault on freenode IRC | German & English | GMT+1
    Absolutely no Support via PM!

    im not sure what i should be checking david? do you know the arguments i should provide to hdparm?
    somethign funny is going on.
    using my windows 7 machine, i am connected to a SMB share on the nas. small files of a few hundred mb copy very fast. however files of a few gig such as movies initially start off at 100+mb and then either stall completely or copy at 2mb/sec.


    ive been out all day so not had much change to look at it.

    got it, hdparm -t /dev/md0 :)


    Code
    root@OMV:~# hdparm -t /dev/md0
/dev/md0:
 Timing buffered disk reads: 1652 MB in  3.00 seconds = 550.14 MB/sec


    and bypassing cache

    Code
    root@OMV:~# hdparm -t --direct /dev/md0


/dev/md0:
 Timing O_DIRECT disk reads: 1152 MB in  3.00 seconds = 383.64 MB/sec


    ive just picked up an idrac 6 card for the server so ill give it a reboot and install the card.

    An iDrac 6 card, on which Dell do you run your NAS, should be a gen11 box like a R510 or T410?

    Homebox: Bitfenix Prodigy Case, ASUS E45M1-I DELUXE ITX, 8GB RAM, 5x 4TB HGST Raid-5 Data, 1x 320GB 2,5" WD Bootdrive via eSATA from the backside
    Companybox 1: Standard Midi-Tower, Intel S3420 MoBo, Xeon 3450 CPU, 16GB RAM, 5x 2TB Seagate Data, 1x 80GB Samsung Bootdrive - testing for iSCSI to ESXi-Hosts
    Companybox 2: 19" Rackservercase 4HE, Intel S975XBX2 MoBo, C2D@2200MHz, 8GB RAM, HP P212 Raidcontroller, 4x 1TB WD Raid-0 Data, 80GB Samsung Bootdrive, Intel 1000Pro DualPort (Bonded in a VLAN) - Temp-NFS-storage for ESXi-Hosts

    Somehow. ;)
    Yes they are great servers, reliable and powerful. I have some T410 and T420 among my weaponry at work, but to be honest: Even if a T410 consumes a little less than 200 watt what is not that much for a real server system I wouldn't have one at home and pay the power bill. :)

    Homebox: Bitfenix Prodigy Case, ASUS E45M1-I DELUXE ITX, 8GB RAM, 5x 4TB HGST Raid-5 Data, 1x 320GB 2,5" WD Bootdrive via eSATA from the backside
    Companybox 1: Standard Midi-Tower, Intel S3420 MoBo, Xeon 3450 CPU, 16GB RAM, 5x 2TB Seagate Data, 1x 80GB Samsung Bootdrive - testing for iSCSI to ESXi-Hosts
    Companybox 2: 19" Rackservercase 4HE, Intel S975XBX2 MoBo, C2D@2200MHz, 8GB RAM, HP P212 Raidcontroller, 4x 1TB WD Raid-0 Data, 80GB Samsung Bootdrive, Intel 1000Pro DualPort (Bonded in a VLAN) - Temp-NFS-storage for ESXi-Hosts

    At least not in germany/the eu...


    Greetings
    David

    "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"


    Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.

    Upload Logfile via WebGUI/CLI
    #openmediavault on freenode IRC | German & English | GMT+1
    Absolutely no Support via PM!

    ive got the server plugged into a mains power meter and its currently pulling 147 watts whilst writing to the RAID6 at 103mb/sec (gigabit ethernet maxed out). when i built the server, i removed one of the cores processors and underclocked the cpu, set all the bios settings for maximum power saving and whatever else i could do. with the server idle it uses 117 watts. as the server is only used to stream films downstairs and the occasional bit of lan filesharing i should really look into setting up WOL for it.


    now that version 1.0 of OMV supports the onboard broadcom NIC's i can remove the quad port intel card i have too :)


    anyway, back on to RAID matters. following the reboot, copying files across onto a SMB share seems to be OK. im not sure if it was a problem with my pc or the server. i would like to set up NFS from my windows 7 pc but i just cant get it to work (which is slightly embarrassing as i am a networking/infrastructure specialist!). NFS works just fine from my openelec media pc downstairs, so i am more than convinced the problem is with windows being "special"

    some cool power stats ive just pulled off the enterprise idrac i put in this morning. the formatting is a little messed up...


    Code
    Power Monitoring
System Board System Level140W | 478 BTU/hr


    following up the strange SMB performance, where i could copy small files without problem, but anything over a couple of gig would result in the upload stalling - i have installed the backports kernel, rebooted and done some testing.


    SMB uploads/downloads to the server are working fine...at the minute!


    edit: SMB uploads broke again. so i unplugged the cheapy but silent netgear 8port 1gig switch i was using and put the HP 2910 gig switch back into action. currently transferred 250gig at 110mb/sec over it :)


    toptip: dont use cheap and nasty hardware :)

    Zitat von davidh2k

    Writing is fine, speeds won't be as fast as with raid5. however the read speed seems to be too low.. Did you check hdparm on the array?


    Greetings
    David


    just bumping this again... i have tested read speeds of the array using hdparm and they are mighty fast...


    but when testing with dd read speeds are quite poor.


    Code
    root@OMV:~# dd if=/dev/md0 of=tempfile bs=1MB count=10240
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 147.281 s, 69.5 MB/s
root@OMV:~#


    any ideas why this is so?


    i can read from the nas across a gig network at about 88MB/sec and upload at 100MB/sec roughly (this is using SMB/CIFS share). i would have thought reads would be at least as fast as writes, given the dual parity overhead on write. but this isnt the case.


    aside from the odd speeds, the nas is working perfectly :)

    To be honest I can't tell why dd is slower than hdparm. That the network is slower on write could be caused by a faulty network driver.


    Greetings
    David

    "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"


    Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.

    Upload Logfile via WebGUI/CLI
    #openmediavault on freenode IRC | German & English | GMT+1
    Absolutely no Support via PM!

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!

Benutzerkonto erstellen Anmelden