200 TB NAS storage with OMV

    • 200 TB NAS storage with OMV

      We have recently setup a NAS with 80 TB of space, with maximum capacity of 200 TB on easily available Off-the-shelf hardware.

      Hardware we used:

      1. Xeon Quadcore processor
      2. Intel motherboard
      3. 8GB DDR3 RAM
      4. 4U 36bay hot swap chasis with redundant power supply and Expander Backplane & Rail Kit.
      5. LSI SAS 4port Raid controller
      6. 120 GB SSD drive for OS.
      7. WD NAS disks(4 TB).

      Detail off Our Setup:-

      A. We used 120 GB SSD hard disk for OS.

      B. In intial phase we have setup 24 disk of 4TB each with total capacity 24 x 4 (TB) = 96TB.
      We have setup two hardware Raid 5 volumes with 11 Disk each and two global hotspare disks.
      Each raid volume has around ~40 TB of space, hence 80TB in total currently.

      C. We left the 12 chasis bays unused for future expansion.

      D. Now the best part:- 1. The LSI raid card can support 6TB drives.
      So in future we plan to replace 4TB disks with 6TB disks or we still have the bandwidth to expand 12 x 6 TB= 72 TB due to 12 unused bays.
      Reason for using 4TB disks is that 4TB disks are comparatively cost effecting compared to 6TB disks.

      E. If we used all 36 Disks of 6TB than we could achive total capacity of 36 x 6TB = 216 TB.

      F. LSI Raid card we have has support of maximum of 128 disk of 6TB. Just imagine 128 x 6 TB = 768 TB at one place.

      G. No vendor lock-in for any hardware and software, no license fees and free software, no limited hardware support for only three years. Every component like motherboard, raid card, chasis back panel etc. is replacable and easily available in the market.

      I. The whole setup was so seamless and smooth, and server was ready for deployment within two days. Previously we had setup OMV for 10 TB NAS, two years ago.

      Our heartful thanks to the
      a. OMV developers(specially Mr. Volker) for creating such a product.
      b. Our Hardware vendor Mr. Amit Kalra (email ID:- amit@buycomps.com, website:- exlsystems.com/).


      Thanks & Regards

      Amit Bondwal
      Email: amit@udyansh.com
      website:- udyansh.com

      "We solve business problems by deploying the best and most suitable technology. We engage with your business, meet your team, understand your processes. Plan the best solution. We develop, engage and deploy. We are proud to say that we deals in open source"
      Images
      • IMG_1249.jpg

        152.05 kB, 1,024×768, viewed 3,045 times
      • IMG_1251.jpg

        122.36 kB, 1,024×768, viewed 1,786 times
      • IMG_1254.jpg

        236.79 kB, 1,024×768, viewed 1,641 times
      • IMG_1250.jpg

        187.05 kB, 1,024×768, viewed 1,791 times
    • Ouuch....
      I would like know the cost?
      Open Media Vault 2.2.6 (Stone burner) in Prod
      Open Media Vault 3.0.32 (Erasmus) in Test

      openmedivault Docker Container
      https://github.com/prbond/openmedivault-dockerfile

      Dev :
      openmediavault-fail2ban 1.1.5 for OMV2.X
      openmediavault-fail2ban 1.3.0 for OMV3.X
      https://github.com/prbond/openmediavault-fail2ban
      https://github.com/OpenMediaVault-Plugin-Developers/openmediavault-fail2ban
    • Gosh,

      Which firm needs to store up to 200Tb ??? Are you working in the movies? The video game industry?
      I used to mount storage for SAP business storage or even backup systems but it never reached the 10Tb limit (or even maybe at the end of the year) !!

      Cheers
      - ASROCK FM2A88X-ITX+ (SATAIII (6Gb/s) x6 (for the DATA), mSATA x1 (for the OS) ==> maybe the only afordable motherboard with 6xSATAIII ports)
      - AMD A6 7400K 3.5GHz (overpowerful for NAS but the cheapest for this motherboard)
      - Corsair 2Go DDR3 1333MHz C9 (x2)
      - INTEL GIGABIT CT DESKTOP ADAPTER SINGLE PORT RJ45 PCIE (to avoid backports problems with network controller)
      - COOLER MASTER G450M (80+ bronze)
      - WD Red 2To 64Mo 3.5" SATAIII (6Gb/s) (x4 for the DATA, 2 SATA ports left for future use)
      - 32 Go SSD mSATA KingSpec Half-Size Solid State (for the OS which allows 6 SATA HDDs)
      - Fractal Design Node 304 black (HDD 3.5" x6)
      - RAID 5 xfs
      - OMV 1.12 (Kralizec) - 3.16.0-0.bpo.4-amd64
    • @pr_bond :- the hardware cost without hard disks is around Rs. 300000/- (Indian Currency in Rupee) and each 4TB WD NAS disk cost Rs. 13500/-, each 6Tb disk cost is Rs. 19500/-, and as per today's rate 1 US Dollar equals to 62.28 Indian Rupee.
      So total cost with 24 disk is :-- 13500 x 24 = 324000 + 300000 = Rs. 624000/-


      @tiste:- Our client already had two 10 TB storage and they are almost full, everyone have separate requirements.

      Thanks & Regards

      Amit Bondwal
      Email: amit@udyansh.com
      website:- udyansh.com

      "We solve business problems by deploying the best and most suitable technology. We engage with your business, meet your team, understand your processes. Plan the best solution. We develop, engage and deploy. We are proud to say that we deals in open source"
    • the hardware cost without hard disks is around : 4,228.61 € (euro) / 4,814.07 $
      each 4TB WD NAS disk cost : 190.320 € (euro) / 216.620 $
      each 6Tb disk cost is : 274.837 € (euro) / 312.734 $
      Total : 8,791.11 € (euro) / 10,008.13 $

      You have a lot of money !!!!
      Open Media Vault 2.2.6 (Stone burner) in Prod
      Open Media Vault 3.0.32 (Erasmus) in Test

      openmedivault Docker Container
      https://github.com/prbond/openmedivault-dockerfile

      Dev :
      openmediavault-fail2ban 1.1.5 for OMV2.X
      openmediavault-fail2ban 1.3.0 for OMV3.X
      https://github.com/prbond/openmediavault-fail2ban
      https://github.com/OpenMediaVault-Plugin-Developers/openmediavault-fail2ban
    • pr_bond wrote:

      the hardware cost without hard disks is around : 4,228.61 € (euro) / 4,814.07 $
      each 4TB WD NAS disk cost : 190.320 € (euro) / 216.620 $
      each 6Tb disk cost is : 274.837 € (euro) / 312.734 $
      Total : 8,791.11 € (euro) / 10,008.13 $

      You have a lot of money !!!!


      it's a business selling a NAS system to another business that needs a lot of storage.... 10K$ is nothing...
    • Wow that's sick...

      "OpenMediaVault is primarily designed to be used in home environments or small home offices, ...

      [IMG:http://i.imgur.com/7McSfST.jpg]
      OpenMediaVault 1.12 on Debian Wheezy 7.8 64bit | 3.16 Backport Kernel | OMV-Extras.org 1.10
      ASRock Q1900-ITX | 1x4GB Corsair Vengeance SODIMM DDR3 | 1xWD Red 3TB | 1xApacer 1,8" SATA1 SSD 32GB as system drive | be quiet! Pure Power L8 350W | SilverStone Milo ML04 Black
    • I am always surpised (but it's understandable) when firms are ready to pay 10k for hardware (and often much more) but don't want to pay a licence for software so they use home and free environnements at their own risks...
      - ASROCK FM2A88X-ITX+ (SATAIII (6Gb/s) x6 (for the DATA), mSATA x1 (for the OS) ==> maybe the only afordable motherboard with 6xSATAIII ports)
      - AMD A6 7400K 3.5GHz (overpowerful for NAS but the cheapest for this motherboard)
      - Corsair 2Go DDR3 1333MHz C9 (x2)
      - INTEL GIGABIT CT DESKTOP ADAPTER SINGLE PORT RJ45 PCIE (to avoid backports problems with network controller)
      - COOLER MASTER G450M (80+ bronze)
      - WD Red 2To 64Mo 3.5" SATAIII (6Gb/s) (x4 for the DATA, 2 SATA ports left for future use)
      - 32 Go SSD mSATA KingSpec Half-Size Solid State (for the OS which allows 6 SATA HDDs)
      - Fractal Design Node 304 black (HDD 3.5" x6)
      - RAID 5 xfs
      - OMV 1.12 (Kralizec) - 3.16.0-0.bpo.4-amd64
    • @tiste:- Every free thing is not bad or risky, same thing is with linux also. If you compare bugs resolving time between linux and windows and no. of contributers in both, you will find linux resolve it much faster and community behind it is very big. Hardware cost is same for both, then why to pay license fee for software, if one can solve your requirements, and to operate both you needs skilled people.
      And about risk on closed source software, you become dependent on one company while in open source, you will hire other freelancer to solve it.


      Thanks & Regards

      Amit Bondwal
      Email: amit@udyansh.com
      website:- udyansh.com

      "We solve business problems by deploying the best and most suitable technology. We engage with your business, meet your team, understand your processes. Plan the best solution. We develop, engage and deploy. We are proud to say that we deals in open source"
    • @amitbondwal : -I have two questions for you.
      Do you consider when desingn 200TB NAS other O.S. like FreeNAs or Nas4Free?
      and same question for Filesystem, Do you consider to use ZFS instead RAID (You can use ZFS on OMV if you need)?

      My interest is to know if you do some test with ZFS or XFS for example for Disk lost & time to reconstruct array of Disk.
      OMV 4.1.11 x64 on a HP T510, 16GB CF as Boot Disk & 32GB SSD 2,5" disk for Data, 4 GB RAM, CPU VIA EDEN X2 U4200 is x64 at 1GHz

      Post: HPT510 SlimNAS ; HOWTO Install Pi-Hole ; HOWTO install MLDonkey ; HOHTO Install ZFS-Plugin ; OMV_OldGUI ; ShellinaBOX ;
      Dockers: MLDonkey ; PiHole ; weTTY
      Videos: @TechnoDadLife
    • @raulfg3: We had used freenas in past, but the main reason behind using OMV is that, it's works on Debian based OSes and we have more expertise in Debian than Unix (Free NAS is unix based) and we had used OMV before.

      As per my knowledge to use ZFS in linux you have to do a lot of tweaks (may be it could be easier now, but I did not use it on linux), so we did not go with ZFS . We used available Filesystem XFS and ZFS is not available in OMV.

      We did not do any test with ZFS because it is not available in OMV. We had a very nice experience with XFS file system and now it has become default filesystem in Redhat7, centos7 etc.

      Four month back, two asterisk server at one of our client was having high loads from some time due to high IO waits on call recordings, they were heavy files, It was running on ext4 filesystem, after doing a lot of tweaks but performance did not improve much so we test XFS file system on one server and load of server drastically reduced, later we put other server on XFS filesystem for recording partition.

      So we go with available options in OMV, our expertise, our experience and use XFS file system.

      Thanks & Regards

      Amit Bondwal
      Email: amit@udyansh.com
      website:- udyansh.com

      "We solve business problems by deploying the best and most suitable technology. We engage with your business, meet your team, understand your processes. Plan the best solution. We develop, engage and deploy. We are proud to say that we deals in open source"
    • thanks a lot.

      If you want to test ZFS I write a short guide to start: [HOWTO] Instal ZFS-Plugin & use ZFS on OMV

      My point is that ZFS is more roboust and scalable for 200TB that XFS , and time to recostruct array (scrub in ZFS) is considerable small.

      PD: Time to create by first time a big array of disk is inmediate in ZFS and really long in 20TB RAID5 like you use, so please test it on some test PC to know if meets your requeriments.
      OMV 4.1.11 x64 on a HP T510, 16GB CF as Boot Disk & 32GB SSD 2,5" disk for Data, 4 GB RAM, CPU VIA EDEN X2 U4200 is x64 at 1GHz

      Post: HPT510 SlimNAS ; HOWTO Install Pi-Hole ; HOWTO install MLDonkey ; HOHTO Install ZFS-Plugin ; OMV_OldGUI ; ShellinaBOX ;
      Dockers: MLDonkey ; PiHole ; weTTY
      Videos: @TechnoDadLife
    • @raulfg3: thanks for information, agree with you about ZFS, next time, we will keep in mind to test a zfs filesystem. We used hardware raid, I did not check whole raid building time but it take almost 3 hours to 20% of raid array building on ~40TB raid.

      @ryecoaaron:- thanks for information.

      Thanks & Regards

      Amit Bondwal
      Email: amit@udyansh.com
      website:- udyansh.com

      "We solve business problems by deploying the best and most suitable technology. We engage with your business, meet your team, understand your processes. Plan the best solution. We develop, engage and deploy. We are proud to say that we deals in open source"
    • That is quite lovely. I've just started with a new employer that's dealing with the constraints of 2 XIV's, one in and one out of warranty. One of the reasons I took the position is because the current director is very forward thinking (former sysadmin himself) and more interested in leveraging skillsets and ideas of current staff as opposed to simply honoring tradition and paying MS and IBM or associated costs from lame 3rd party support contractors because that's what previous directors had done.
    • @blindguy:- It is Chenbro 4U 36 bay Hot swap Chassis. I think MegaRAID Storage Manager(MSM) software for the LSI card, because we used bios firmware to configure hardware raid, which was Megaraid storage software.



      Thanks & Regards

      Amit Bondwal
      Email: amit@udyansh.com
      website:- udyansh.com

      "We solve business problems by deploying the best and most suitable technology. We engage with your business, meet your team, understand your processes. Plan the best solution. We develop, engage and deploy. We are proud to say that we deals in open source"