My build : 25+ TB RAID6 /LVM / 10 GbE

1. offizieller Beitrag

    Hello all, here is my build :

    • CPU : intel Q6600
    • Mobo : ASUS P5E
    • RAM : 8 Gb GSkill
    • Power : Corsair 750 Modular
    • System drive : 1x OCZ SSD 120 Gb
    • Data drive(s) : 16x Seagate Barracuda 2Tb 7200rpm in a MDRAID RAID6
    • HBA : intel SASUC8I + intel Expander RES2SV240
    • Management Network : integrated Marvell NIC
    • Storage network : Dual 10GbE SFP+ intel NIC (x520-da2)
    • OS : OMV4 (previously OMV3)
    • Case : Norco RPC-4224

    System is used to present LVM volumes via NFS to a VMware ESXi 6.5 cluster.


    Previous limitations (resolved by recent actions) :
    Network bottleneck on storage side (2x intel 1Gbit/s NIC) replaced by dual SFP+ 10GbE NIC


    Current limitations/problems :
    BIOS loses settings after power loss (motherboard bug, battery is ok)
    intel SASUC8I is known to be unstable when using SMARTD. SMARTD currently disabled.
    intel SASUC8I doesn't support drives over 2Tb.
    Can't seem to go over 40-50% of 10GbE bandwisdth. Could be due to a PCIe lanes limitation


    Next upgrade :
    Replace SASUC8I with IBM/LSI M1015


    Thoughts :
    Back in the days I used Openfiler + iSCSI + ESX 4. I then discovered that except for SCST, IET doesn't support SCSI-3 reservations and therefore is no-go for more recent versions of ESXi. I now stick to NFS without issues.
    Maybe I'll change my mobo+CPU someday (more power, less consumption)


    Conclusion : Very happy of OMV !! :D

    Sure, I have 1600 VA UPS.


    When i say "power loss", i mean switching off the power supply completly. In that particular case mobo loses settings.
    If PS is still plugged in (although machine is shut down) settings remain.


    I checked/changed BIOS battery and it doesn't help, so I guess mobo is somehow faulty.

    Here is the upgrade i did to my build last month:


    Hypervisor :
    - VMware ESXi 6.7
    - Ryzen 5 2600
    - 16Gb Gskill DDR4 3200
    - MSI X470 Gaming Plus (onboard NIC supported by custom VIB module)
    - Norco RPC-4224
    - WD SSD Green 120 Gb


    OMV (as a VM) :
    - Debian 9 stable
    - kernel backports
    - 6 vCPU
    - 8 Gb vRAM
    - IBM M1015 target mode (passthrough)
    - intel RES2SV240 SAS expander
    - intel 2x 10GbE NIC (passthrough)



    Migration went like a charm, only needed to reconfigure network settings, mount fs and share again (base + nfs)


    New system consumes about 30-40% less power and is about twice faster in IO (move from SATA2 to SATA3 controller i assume, since all disks were SATA3). CPU is mostly idle and stock fan remains low speed.


    VMware gives extra opportunities (cold snapshots, disaster recovery options, consolidation...), without any loss of performance, only loss of hardware monitoring (lmsensors).


    Very happy so far.

    • Offizieller Beitrag
    Zitat von jollyrogr

    IBM M1015 = LSI 9210 = SATA2

    Nope. IBM M1015 = LSI 9220 (which is the OEM version of the 9211) = 6gbps or SATA3. They both use SAS2008 controller. I have a 9211 but @flmaxey should be able to confirm since he has an M1015 flashed with 9211 firmware.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    • Offizieller Beitrag

    I flashed the Dell Perc H200, but the IBM card is a SATA3 6GB item.
    Since the group of HBA's that can be flashed to IT mode are all based on the LSI SAS2008 chip (which is SATA3), I can't see where LSI SAS2008 SATA3 based firmware would downgrade a SATA3 6GB controller to SATA2.
    Performance Test - M1015.






    https://www.servethehome.com/i…art-1-started-lsi-92208i/

    • Offizieller Beitrag

    I had a SATA2-3GBS HBA (an Adaptec 5405Z). Since it had a bit of age on it, I looked into the speed of current drives versus the bandwidth of this controller. I found that consumer drives that are rated for SATA3 - 6GBS can only maintain that transfer rate for a few millisec's. After that very short burst, a sata 2 controller can easily keep up with drives spinning at 7200rpm or less. For consumer drives, advertising them as "SATA3-6GBS" is a sales gimmick. It takes SSD's to get real benefit from SATA3-6GBS. (Which is why most HBA benchmarking is done with SSD's.)


    I swapped out the SATA2 Adapter HBA, for the flashed Perc H200, for transparent SMART stat's pass through. Otherwise, 3GBS (SATA2) would have been fine.

    • Offizieller Beitrag
    Zitat von flmaxey

    If I had @ryecoaaron 's money, I would!

    Funny. I only have one nvme drive 8)

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Zitat von flmaxey

    I had a SATA2-3GBS HBA (an Adaptec 5405Z). Since it had a bit of age on it, I looked into the speed of current drives versus the bandwidth of this controller. I found that consumer drives that are rated for SATA3 - 6GBS can only maintain that transfer rate for a few millisec's. After that very short burst, a sata 2 controller can easily keep up with drives spinning at 7200rpm or less. For consumer drives, advertising them as "SATA3-6GBS" is a sales gimmick. It takes SSD's to get real benefit from SATA3-6GBS. (Which is why most HBA benchmarking is done with SSD's.)


    I swapped out the SATA2 Adapter HBA, for the flashed Perc H200, for transparent SMART stat's pass through. Otherwise, 3GBS (SATA2) would have been fine.

    Yeah, my Adaptec 5805Z also kept up with the disks just fine. I too, swapped out for an LSI SAS2008 controller based card (Supermicro AOC-USAS2-L8I), so I could have IT mode pass through and access the SMART data. But, I also noticed that idle power went down almost 10 watts. The Adaptec (at least my Adaptec) was a big power hog.

    • Offizieller Beitrag
    Zitat von ryecoaaron

    Funny. I only have one nvme drive 8)

    That's exactly one more than I have. :)



    Zitat von Markess

    The Adaptec (at least my Adaptec) was a big power hog.

    Yeah, I noticed my Adaptec was really hot, even when idle. (I was giving thought to retro fitting it with a small GPU fan.) Along with high heat comes low power efficiency.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!

Benutzerkonto erstellen Anmelden