My build : 25+ TB RAID6 /LVM / 10 GbE

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • My build : 25+ TB RAID6 /LVM / 10 GbE

      Hello all, here is my build :
      • CPU : intel Q6600
      • Mobo : ASUS P5E
      • RAM : 8 Gb GSkill
      • Power : Corsair 750 Modular
      • System drive : 1x OCZ SSD 120 Gb
      • Data drive(s) : 16x Seagate Barracuda 2Tb 7200rpm in a MDRAID RAID6
      • HBA : intel SASUC8I + intel Expander RES2SV240
      • Management Network : integrated Marvell NIC
      • Storage network : Dual 10GbE SFP+ intel NIC (x520-da2)
      • OS : OMV4 (previously OMV3)
      • Case : Norco RPC-4224
      System is used to present LVM volumes via NFS to a VMware ESXi 6.5 cluster.

      Previous limitations (resolved by recent actions) :
      Network bottleneck on storage side (2x intel 1Gbit/s NIC) replaced by dual SFP+ 10GbE NIC

      Current limitations/problems :
      BIOS loses settings after power loss (motherboard bug, battery is ok)
      intel SASUC8I is known to be unstable when using SMARTD. SMARTD currently disabled.
      intel SASUC8I doesn't support drives over 2Tb.
      Can't seem to go over 40-50% of 10GbE bandwisdth. Could be due to a PCIe lanes limitation

      Next upgrade :
      Replace SASUC8I with IBM/LSI M1015

      Thoughts :
      Back in the days I used Openfiler + iSCSI + ESX 4. I then discovered that except for SCST, IET doesn't support SCSI-3 reservations and therefore is no-go for more recent versions of ESXi. I now stick to NFS without issues.
      Maybe I'll change my mobo+CPU someday (more power, less consumption)

      Conclusion : Very happy of OMV !! :D
    • Here is the upgrade i did to my build last month:

      Hypervisor :
      - VMware ESXi 6.7
      - Ryzen 5 2600
      - 16Gb Gskill DDR4 3200
      - MSI X470 Gaming Plus (onboard NIC supported by custom VIB module)
      - Norco RPC-4224
      - WD SSD Green 120 Gb

      OMV (as a VM) :
      - Debian 9 stable
      - kernel backports
      - 6 vCPU
      - 8 Gb vRAM
      - IBM M1015 target mode (passthrough)
      - intel RES2SV240 SAS expander
      - intel 2x 10GbE NIC (passthrough)


      Migration went like a charm, only needed to reconfigure network settings, mount fs and share again (base + nfs)

      New system consumes about 30-40% less power and is about twice faster in IO (move from SATA2 to SATA3 controller i assume, since all disks were SATA3). CPU is mostly idle and stock fan remains low speed.

      VMware gives extra opportunities (cold snapshots, disaster recovery options, consolidation...), without any loss of performance, only loss of hardware monitoring (lmsensors).

      Very happy so far.
    • jollyrogr wrote:

      IBM M1015 = LSI 9210 = SATA2
      Nope. IBM M1015 = LSI 9220 (which is the OEM version of the 9211) = 6gbps or SATA3. They both use SAS2008 controller. I have a 9211 but @flmaxey should be able to confirm since he has an M1015 flashed with 9211 firmware.
      omv 4.1.17 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • I flashed the Dell Perc H200, but the IBM card is a SATA3 6GB item.
      Since the group of HBA's that can be flashed to IT mode are all based on the LSI SAS2008 chip (which is SATA3), I can't see where LSI SAS2008 SATA3 based firmware would downgrade a SATA3 6GB controller to SATA2.
      Performance Test - M1015.





      servethehome.com/ibm-m1015-part-1-started-lsi-92208i/

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
    • I had a SATA2-3GBS HBA (an Adaptec 5405Z). Since it had a bit of age on it, I looked into the speed of current drives versus the bandwidth of this controller. I found that consumer drives that are rated for SATA3 - 6GBS can only maintain that transfer rate for a few millisec's. After that very short burst, a sata 2 controller can easily keep up with drives spinning at 7200rpm or less. For consumer drives, advertising them as "SATA3-6GBS" is a sales gimmick. It takes SSD's to get real benefit from SATA3-6GBS. (Which is why most HBA benchmarking is done with SSD's.)

      I swapped out the SATA2 Adapter HBA, for the flashed Perc H200, for transparent SMART stat's pass through. Otherwise, 3GBS (SATA2) would have been fine.

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
    • jollyrogr wrote:

      Time to get with the program and upgrade to all NVME!
      If I had @ryecoaaron 's money, I would! :)

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk
    • flmaxey wrote:

      If I had @ryecoaaron 's money, I would!
      Funny. I only have one nvme drive 8)
      omv 4.1.17 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • flmaxey wrote:

      I had a SATA2-3GBS HBA (an Adaptec 5405Z). Since it had a bit of age on it, I looked into the speed of current drives versus the bandwidth of this controller. I found that consumer drives that are rated for SATA3 - 6GBS can only maintain that transfer rate for a few millisec's. After that very short burst, a sata 2 controller can easily keep up with drives spinning at 7200rpm or less. For consumer drives, advertising them as "SATA3-6GBS" is a sales gimmick. It takes SSD's to get real benefit from SATA3-6GBS. (Which is why most HBA benchmarking is done with SSD's.)

      I swapped out the SATA2 Adapter HBA, for the flashed Perc H200, for transparent SMART stat's pass through. Otherwise, 3GBS (SATA2) would have been fine.
      Yeah, my Adaptec 5805Z also kept up with the disks just fine. I too, swapped out for an LSI SAS2008 controller based card (Supermicro AOC-USAS2-L8I), so I could have IT mode pass through and access the SMART data. But, I also noticed that idle power went down almost 10 watts. The Adaptec (at least my Adaptec) was a big power hog.
      Primary: OMV 4.x, Asrock Industrial IMB-181-L, Pentium G3220T, 16GB, HP 10GbE
      Backup: OMV 4.x, Supermicro X9SIL, Xeon 1220L, 8GB ECC, Mellanox 10GbE
      Learning & Exploring: OMV on Proxmox, Asus M5A97 LE R2.0, Opteron 3320EE, 12GB ECC, Intel 1GbE
    • ryecoaaron wrote:

      flmaxey wrote:

      If I had @ryecoaaron 's money, I would!
      Funny. I only have one nvme drive 8)
      That's exactly one more than I have. :)


      Markess wrote:

      The Adaptec (at least my Adaptec) was a big power hog.
      Yeah, I noticed my Adaptec was really hot, even when idle. (I was giving thought to retro fitting it with a small GPU fan.) Along with high heat comes low power efficiency.

      Video Guides :!: New User Guide :!: Docker Guides :!: Pi-hole in Docker
      Good backup takes the "drama" out of computing.
      ____________________________________
      Primary: OMV 3.0.99, ThinkServer TS140, 12GB ECC, 32GB USB boot, 4TB+4TB zmirror, 3TB client backup.
      OMV 4.1.13, Intel Server SC5650HCBRP, 32GB ECC, 16GB USB boot, UnionFS+SNAPRAID
      Backup: OMV 4.1.9, Acer RC-111, 4GB, 32GB USB boot, 3TB+3TB zmirror, 4TB Rsync'ed disk