RAID on VM

  • Hi,


    I am new in this so maybe I will ask a stupid question, but searching the forum I didn't find the answer.


    I have installed OMV3 on a VMWare ESXi 6.5 and on this server I have 2 x 1T HDD that are not in RAID
    Now I want to assign some HDD for OMV3, but don't know if it makes sense to make a RAID in OMV.


    If I add 4 x 100GB virtual disk from the same physical to OMV and create a RAID 10 for example will it help me? From my logic makes no sense if the physical disk will crash on 4 virtual disk will crash. If there any reason for me to create a RAID on OMV side or not?


    On Server side I don't want to create a RAID, as I will use ESXi just for testing some VM and the OMV I will install just for share and NextCloud, and the data that will be stored on NextCloud I want to back up them to some other NAS that I have at home.


    P.S. If you know a thread about sync from OMV to some other NAS please let me know :)


    Thanks.

    • Offizieller Beitrag

    Nope, there is no point in creating a raid array if all of the vmdks are on the same drive.


    There is no way to create a raid array on ESXi. You have to use a hardware raid controller.

    omv 7.0-32 sandworm | 64 bit | 6.5 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.9 | compose 7.0.9 | cputemp 7.0 | mergerfs 7.0.3


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Nope, there is no point in creating a raid array if all of the vmdks are on the same drive.


    There is no way to create a raid array on ESXi. You have to use a hardware raid controller.

    Thanks.


    Yes I know that I will have to use the hardware raid controller from the HP Microserver Gen8.


    Because is more for testing the ESXi and the OMV3 will be used for NextCloud and for some share folders, I think it makes no sense at all to create a RAID and lose the 1TB HDD.


    I will have just to find a solution to move data automatically to backup from OMV to my other NAS solution that is a Zyxel 320.

  • Don` t know. Think there is no hotplug on the gen8 i think.


    Gesendet von meinem SM-G935F mit Tapatalk

    HP Proliant Gen8 with ESXi6.5


    Storage - Raid5 with 4 x 3TB WDRed - System on microSD / Datastore on Samsung 850Pro - 250GB

  • Don` t know. Think there is no hotplug on the gen8 i think.

    No, there's no hotplug. But as with every other hardware out there there's always the possibility that hardware will fail.


    RAID even in its most basic implementation needs regular testing (scrubs are one thing, simulating a failed drive another -- without doing both regularly one can not trust in promises people associate with RAID. BTW: simulating a failing drive would also be needed to see how a specific RAID implementation can cope with this situation but that's of course hard to test).


    Playing device pass-thru means adding another layer of complexity to an already complex setup known from the forums from 'My RAID array has gone. Why?' every few days. In other words: without testing this sounds like a great recipe for future data loss to me :)

  • @tkaiser


    I agree with you 100% in all! ;)
    Let me backup the raid5 to an external drive and then i will try to simulate (or real) a disk fail.



    the possibility to cover everything with one box is of course very tempting ... but I want at least two things:


    keep it simple as possible
    und secure!




    before I had openmediavault natively on the hp n40l and virtualbox for the vm´s with a very low performance ...

    HP Proliant Gen8 with ESXi6.5


    Storage - Raid5 with 4 x 3TB WDRed - System on microSD / Datastore on Samsung 850Pro - 250GB

  • Let me backup the raid5 to an external drive and then i will try to simulate (or real) a disk fail.

    Very good idea since I remember a few really horrible situations at customers with failed passed through storage devices that failed in a bizarre way differently than expected that the VM had to be shutdown since logging to death (years ago though)


    before I had openmediavault natively on the hp n40l and virtualbox for the vm´s with a very low performance ...

    If the VMs were on a RAID5 before and now are on a SSD you gave the answer why already. Easy to test with iozone and '-i 2 -I' (this tests for random IO using direct IO which pretty much simulates what happens in virtualized environments).


    Any RAID5 with just a few HDDs will show here very low numbers, almost any SSD very high numbers and to get something similar with HDDs you would need to adjust your RAID topology, eg. RAID10 (I personally wouldn't use any more but rely on clustered zmirrors with all devices in a single pool. Even with 7.2k HDDs random IO as high as with a consumer SSD as long as you throw in enough disks -- sorry for the OT, already preparing a customer call on storage expansion ;) )

  • Zitat von tkaiser

    If the VMs were on a RAID5 before and now are on a SSD you gave the answer why already. Easy to test with iozone and '-i 2 -I' (this tests for random IO using direct IO which pretty much simulates what happens in virtualized environments).

    OMV3 and the datastore for the 4 virtualbox vm´s was stored on den samsung 850 pro.. but with no seperate partition for the "datastore". Maybe ext4 is not so fine for that..


    please could you explain how to test with ".. iozone and '-i 2 -I' .. " ? :)


    Maybe i go back to Gen8 with ESXi6.5 and the SSD for the VM´s and the N40L with OMV3 for the Data or i replace the N40L with a Qnap and enable iSCSi for the VM´s. :)


    Good luck with your storage expansion! ;)

    HP Proliant Gen8 with ESXi6.5


    Storage - Raid5 with 4 x 3TB WDRed - System on microSD / Datastore on Samsung 850Pro - 250GB

  • OMV3 and the datastore for the 4 virtualbox vm´s was stored on den samsung 850 pro.. but with no seperate partition for the "datastore". Maybe ext4 is not so fine for that..

    I overlooked totally that you were running on a N40L before so most probably it's dog slow Turion vs. Gen8 Intel CPU on your new MicroServer (even the smallest Celeron should be at least twice as fast as the old Turion).


    Wrt iozone it's on a somewhat recent Debian or Ubuntu an 'apt install iozone3' and then I always recommend to chdir to your storage and do


    Code
    iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
    iozone -e -I -a -s ${DRAM*3}M -r 16384k -i 0 -i 1 -i 2

    ${DRAM*3} has to be replaced with the amount of physical/assigned memory your machine has mutiplied with 3. Just to check the other numbers since in some circumstances the -I (direct IO) parameter gets ignored and the data does not come from or goes to disk but instead filesystem buffers (write benchmarks with btrfs are an example). If that's the case you get a huge mismatch between benchmarks with 100MB filesize and the '3 times the available memory' filesize since in the latter case buffers are of not much use any more.


    The first iozone line tests through IO requests with varying blocksizes, this is especially interesting when your workload makes use of these IO patterns ('cold storage' does not but a datastore for VMs does definitely). Especially with RAID the results with small blocksizes are interesting since due to the chosen chunk size every 'overwrite' (data change) that is smaller than the chunk size gets VERY expensive with direct IO (you can check the chunk size with mdadm -E and your first RAID member, eg. /dev/sda).


    Anyway: If you've only 'cold data' on the array there's no need to care that much, this only gets interesting if your storage is subject to (especially random) writes with small amounts of data (eg databases or those datastores for VMs).


    We made an experiement half a year ago with an outdated/abandoned macOS server VM on a RAID6 based datastore. A macOS update to latest version took +1 hour. Then reverting back to latest snapshot, moved the datastore over to a zmirror that was optimized for random IO (all disks in a single zpool) and the whole procedure again in less than 10 minutes. You might experience the same when you grab a QNAP and use it as iSCSI target for your VMs. RAID5 vs RAID10 might make a huge 'snappiness' difference :)

  • Nice one!


    yesterday was the day! ;)
    Backup of 6,4TB data to my qnap finished.
    i have simple removed the disk and guess what! :)


    the hypervisor has stopped the openmediavault vm instantly.
    The gen8 doesn´t support hotplug so i have to reboot the whole system.
    Back online the openmediavault vm starts with one missing disk but clean raid5 and health filesystem.
    i added the disk with
    mdadm --manage /dev/md127 --add /dev/sdd
    and now the raid5 is recovering ... should take a while ..


    summary i would say ... it works after a disk failure ... but it is not perfect because the ESX stoped the vm and only a reboot of the whole system brings it back to normal.


    at first appearance i would say .. Performace is better then before (native OMV).


    SMB write speed 112MB/sec
    After raid recover i will make the tests with iozone! ;)


    thx

    HP Proliant Gen8 with ESXi6.5


    Storage - Raid5 with 4 x 3TB WDRed - System on microSD / Datastore on Samsung 850Pro - 250GB

  • it works after a disk failure ... but it is not perfect because the ESX stoped the vm and only a reboot of the whole system brings it back to normal.

    Well, since wasting disks to implement the R in RAID is only about providing availability I would say this simply sucks and is pretty insufficient since the specific implementation thwarts the whole technology :)


    I had the same already 18 years ago when I learned why hardware RAID controllers are so dangerous. Moved a whole internal 8 disk RAID (RAID5 back then since we couldn't do better with one configured hotspare) to another host. RAID controller an insanely expensive thing from AMI (IIRC MegaRAID) sitting in an own 5.25" drive bay and refusing to boot the new server. Since 'hotspare missing' (8 drive bays, one internally damaged). So all the money spent on 'business contuinity' was simply wasted since the RAID controller could not be convinced from forgetting about the 'additional availability guarantee' and we had only two alternatives: exchanging the hotspare with an array member to boot with a degraded array (since I tested a rebuild and measured array performance before that was not an option since +36 hours crappy performance, full restore from tapes would have lasted even longer in case something went wrong) so we moved everything back into the old server (5 in the morning). Lesson learned...


    Given that your 'RAID with passthrough drives' adds another level of complexity I would think twice about keeping such a setup. On the other hand those 'My array has gone, what happened?' threads are always interesting to read ;)


    But you did test which is already a lot different to the usual 'RAID drama' happening here in the forums all the time :)

    SMB write speed 112MB/sec

    It's 2017, every cheap ARM toy with Gigabit Ethernet can do this. That's not a number any more :)

  • Zitat von tkaiser

    Well, since wasting disks to implement the R in RAID is only about providing availability I would say this simply sucks and is pretty insufficient since the specific implementation thwarts the whole technology


    in many things the implementation is flawed but, without raid i had lost data, right! ;)


    the mdadm softraid exist since ... lets say 10 years ... maybe longer. From an old Dell server to a miniITX board with ipfire as OS ... from 500GB Disk up to 1TB and now 3TB. ext3 to ext4 and so on
    Everthing works fine the last years and i never ever lost any bit of data.


    in my castle there are many many things working 24/7 so therefore it is very important to find the right balance between energy consumption, stability and speed.
    every day at 2 am omv-vm send wol to my qnap and when its up rsync makes the rest. ;)


    in worstcase i stop automatic shutdown of the qnap. bamm! :)



    Zitat von tkaiser

    I had the same already 18 years ago when I learned why hardware RAID controllers are so dangerous. Moved a whole internal 8 disk RAID (RAID5 back then since we couldn't do better with one configured hotspare) to another host. RAID controller an insanely expensive thing from AMI (IIRC MegaRAID) sitting in an own 5.25" drive bay and refusing to boot the new server. Since 'hotspare missing' (8 drive bays, one internally damaged). So all the money spent on 'business contuinity' was simply wasted since the RAID controller could not be convinced from forgetting about the 'additional availability guarantee' and we had only two alternatives: exchanging the hotspare with an array member to boot with a degraded array (since I tested a rebuild and measured array performance before that was not an option since +36 hours crappy performance, full restore from tapes would have lasted even longer in case something went wrong) so we moved everything back into the old server (5 in the morning). Lesson learned...


    i spend a lot of time in fixing problems on our dell servers and megaraid illness so i know what you mean! ;)






    Zitat von tkaiser

    It's 2017, every cheap ARM toy with Gigabit Ethernet can do this. That's not a number any more


    yes, i know! :)
    but for me was only important to reach the borders of my 1gb network and have a reason to upgrade the whole over the next years to 10gb :D


    maybe I should take a closer look at zfs .. :P

    HP Proliant Gen8 with ESXi6.5


    Storage - Raid5 with 4 x 3TB WDRed - System on microSD / Datastore on Samsung 850Pro - 250GB

  • Hello Rowie,


    You said

    I've almost the same setup as you...but I'm failing to create RDMs.


    My config :
    - HP Prolinant Gen8
    - ESXi 6.5 update1 (HP customize) on Internal USB
    - 1 SDD (for VMs) connected to ODD port- sata 5.
    - 1 HDD 640 GB (for tests) - later, I'm going to connect WD RED
    - 1 datastore in ESXI linked to the SSD where I have 1 VM (OpenMediaVault)
    Now, Each time I'm following the link you gave above to attach the HDD directly, I got a strange message "Incompatible device backing specified for device ‘x’".


    Questions :
    - how did you configure your sata controller in the BIOS ? (Legacy, AHCI or Bi120)
    - is there a specific config to make in ESXi (like creating datastore first for the HDD) ?


    Thanks,
    Didier

  • Hi Didier,


    - sata controller = ahci
    - dit you set the pointers with vmkvstools ? (https://kb.vmware.com/selfserv…playKC&externalId=1017530)
    - is your gen8 up to date?
    - du you have activated a valid esx license?
    - and last but not least .. please make automatic backups! ;)


    If one if the discs failed .. you have to reboot the gen8 because there is no hotpluf feature.
    Hope you doing well.


    Best regards,
    Ronald


    Ps: please excuse the delay ... i am in vienna on a security symposium ... writing on my mobilphone .. ;)




    Gesendet von meinem SM-G935F mit Tapatalk

    HP Proliant Gen8 with ESXi6.5


    Storage - Raid5 with 4 x 3TB WDRed - System on microSD / Datastore on Samsung 850Pro - 250GB

  • Hi,


    Many thanks for your answer and no issue for the delay, please enjoy your security symposium.
    I'm not at home for the moment. I'll check tonight again.


    Otherwise, my HP Gen 8 is new (received 1 week ago) and with standard equipement (4GB of mem, Celeron cpu and Bi120 sata controller).
    Changed sata Bios settings to AHCI.
    Bios is J06 (2015) (I think this is the last).
    Vmware ESXi 6.5 update 1 (HP customized) and activated (free licence).


    I did indeed created the RDM pointers with vmkfstools and that was ok (no errors) and a file was correctly created on /vmfs/volumes/DatastoreVM/HDDtest.vmdk
    This "DatastoreVM" is the datastore created on the SSD.
    Issue was when I tried to add the vmdk file to the VM...adding a new "existing HDD" was ok and then when trying to save the modificarion, got that strange error.


    I'll keep you posted.
    Thanks,
    Didier

  • Hello,


    Finally got it...was very simple in fact.
    Issue was that I had created a datastore also on the HDD I wanted to map.


    So to summary,
    - datastore creation on the SSD (port sata 5) to store the VM and to store also the RDM pointer files
    - in ssh, list all drives with "ls -l /vmfs/devices/disks" and choose the one you want to attach
    - run the following command :


    Code
    vmkfstools -z /vmfs/devices/disks/t10.ATA_____WDC_WD2000JS2D55NCB1__________________________WD2DWCANL1692385 /vmfs/volumes/DatastoreVM/HDD200.vmdk

    - edit VM, add another scsi controller, save
    - edit VM, add existing hard disk, choose the vdmk created (HD200.vdmk) and choose the other scsi controller just added, save


    That's it.

  • Fine, nice work!


    Gesendet von meinem SM-G935F mit Tapatalk

    HP Proliant Gen8 with ESXi6.5


    Storage - Raid5 with 4 x 3TB WDRed - System on microSD / Datastore on Samsung 850Pro - 250GB

  • Well hello everyone,


    It looks like this is the right place to pose my questions, therefore let me begin first with the server config.


    I have a HP ProLiant Gen8 with ESXi5.5 (genuine licence) installed on a 4GB SD HC card (booting/working fine)
    Sadly still the Intel G1610T processor (will eventually upgrade to Xeon E3-1265L v2), but managed to upgrade to 16GB ECC RAM though.
    Also I have an OCZ 120GB SSD installed on the SATA5 port (ODD with the silly floppy to SATA power adapter) with all my VMs out of which one is OMV 3.0.86 (16GB vmdk - enough? to much?)
    On the "real storage" side, in bays 1 and 2 I have 2x 3TB WD RED, and bays 3 and 4 I have 2x 1TB WD (one RED one GREEN)
    The first pair I intend to use only for OMV, whilst the second pair will serve my training program in my virtual lab and has no implication to OMV.
    Now, these 2x3TB red drives I took them out of a Zyzel NSA325 where they have been working under RAID1
    Although I have a backup of the data that these 2 drives contain, I have did not format these drives yet, as I'm still learning if I should use the on-board HP RAID controller Bi120 or serve the drives raw (RDM) directly to OMV, as per the safest/fastest solution out of the two.
    Strangely enough, after adding these 2 drives to OMV VM, is seems that the RAID profile configured by my Zyxel NAS had been learned/accepted by OMV after the first reboot, which is a very nice touch if question 3 is positive!
    I can't remember exactly but I think Zyxel NSA325 runs some custom ver. of Linux (don't ask me which, as Linus isn't my strongest point).



    No let me return to the questions:


    1. Knowing my config. would you advise to use RAID Bi120 controller and serve 1 logical drive to OMV or just raw disk mount to OMV and create a RAID1 at VM level? I would appreciate some personal points as of why you advise one or the other, please.


    2. I have all my "precious" data saved on a WD My Book Duo (although only one drive loaded) USB3 device, which unfortunately is not supported by ESXi from what I have been told. So if I will end up restoring my data, it will be via LAN I guess, but could rSynch serve my purpose then?


    3. I wonder if there is a way to make the data available too, not just the RAID config. that was "inherited" from my old Zyxel NAS, considering they both run on Linux (hence I haven't formatted/reconf. the drives yet)?


    I'm sure I will have more questions as things evolve, but let's make a starting point please.


    Your assistance is much appreciated.
    Many thanks,
    Brian

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!