Mobo and other HW suggestions wanted for new OMV box with ECC

    I know we've already covered these boards... But: http://www.asrock.com/mb/Intel/X99%20WS/


    I think it would be a breach of trade description if this board doesn't support ECC as it clearly states at the top "Server grade X99 for making the most out of DDR4 ECC and RDIMMs"


    'Making the most' would surely mean using the ECC ability.


    The description also states:


    Microsoft® Windows® Server 2012 R2 64-bit / Server 2012 64-bit / Server 2008 R2 64-bit support


    Xeon® E5-1600/2600 v3 Processor Family for the LGA 2011-3 Socket, up to 18 Cores and 160W


    Supports Quad Channel DDR4 3200+(OC) memory with max. capacity up to 128GB
    Supports ECC*, RDIMM Memory (*ECC is supported with Intel® Xeon® processors)


    Intel® Dual Gigabit LAN with Teaming Function (I217LM + I210AT)


    10 SATA3, 1 eSATA, 1 Ultra M.2 (PCIe Gen3 x4 & SATA3)


    These are all indicating that it is designed to be used a server board or at the very least, a workstation, which I guess is indicated in the model name.


    I have sent ASRock a message to ask them to confirm if this board simply takes and will boot with ECC memory or if the feature is actually enabled. They're pretty fast at replying so I should have an answer soon.


    Funnily enough, if I didn't hate Asus, they actually do an X99 board with IPMI: https://www.asus.com/Commercia…s_Workstations/X99WSIPMI/


    Which is listed under servers and workstations. So I'm hoping that the X99 architecture can support ECC.

    • Offizieller Beitrag

    Doesn't the line - Supports ECC*, RDIMM Memory (*ECC is supported with Intel® Xeon®processors) - mean it will use it? I've never seen a board that will let you use ecc but not enable it.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.2 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Well, this is what I am hoping. If the X99s do indeed support ECC, then it opens up a ton of boards with a square ILM.


    What I will probably do is place an order for the board from Amazon, who have it for a very reasonable £254.99. Amazon have arguably one of the best customer service policies so in the event that it does not have ECC or there are any other problems, I can have it collected.


    The ASRock X99 WS is one of the higher end boards that they have with ECC, but the extreme 4, 5 and 6 also appear to as well and are sub £200.


    This also looks nice and has 10G Ethernet: http://www.asrock.com/mb/Intel/X99%20WS-E10G/


    They have quite a nice range of X99s, not all say ECC: http://www.asrock.com/mb/index.asp?s=2011-3


    Wow.. This one has 18 SATA ports.... http://www.asrock.com/mb/Intel/X99%20Extreme11/


    But it's £745 and way overkill.


    :P

    This board is, for lack of a better word, gorgeous. The thing looks amazing for a start and luckily the connectors are mostly in the same places. The UEFI is a joy to use and very smooth. For anyone interested in this board, it's EATX but only measures ~10.5 inches across so doesn't cover the grommets in this V71 chassis. I'll be benchmarking and testing soon, but just wanted an opinion on the GPU.. The X99s don't come with integrated graphics so just to get up and running I dropped an Nvidia 9500gt in that I had spare. I think this card pulls about 50w at maximum - bearing in mind it will never be maxed.


    Do I need to think about getting a smaller generic 32MB VGA card before I hook all the drives up?


    The PSU calc website mentioned earlier in this thread recommends a minimum of ~362watt with the Nvidia, assuming no caps have aged - but ideally recommends about 412watt.


    Is this cutting it too close with a 400w PSU?

    • Offizieller Beitrag

    nVidia always tells you to use too large of a power supply with their cards. This mainly happens because they don't know how large each rail is. I have a q6600 with two 980gtx cards (much, much bigger and faster than the 9500gt) pulling a large load on both cards (folding@home) and it is only 450 watts on the kill-a-watt. So, with a 400 w PSU, I think you will fine unless you have 20 drives writing at the same time while running all those cores at full load.

    Well I don't know what magic ASRock have woven into this board, but I am getting lower CPU temps by about 5c and higher pass mark and geekbench scores :P losing IPMI doesn't seem so bad all of a sudden.


    The Nvidia card I have in here is running the hottest at 37c, all else is below that, CPU is about 26c - and this is with fans on silent mode in the UEFI. It's very quiet. I could easily sit this next to a TV and it probably wouldn't affect viewing.


    I'm still considering changing the Nvidia card purely from an overkill and ambient temp point of view, any recommendations on a crappy low powered generic card?

    • Offizieller Beitrag

    Any power consumption readings? :)


    Do you need a video card? Some systems will boot without them. If it has a serial port, you wouldn't even need a video card to login locally.

    Power readings:


    Nvidia in, no Seagates yet, Debian 7 OS, Corsair H105, all 3 of the V71 chassis fans on high and LEDs on:


    POST: spikes to 97w then down to 89w
    Booting: 82-83w
    Xfce login screen: 51w
    Logging in: 62w
    Sat on Xfce desktop: 50w
    Geekbenching: 150-186w


    Guess I don't need to worry about the Nvidia. :P I can't imagine the Seagate drives and a little transcoding will push this up too much.


    I'd probably keep a card in, not sure if it will boot without it. At least there is the option to remove it if I need to.

    • Offizieller Beitrag

    Very nice :) I can't imagine going over the 200w mark even with the drives in.

    Btw, I haven't forgotten about a write up thread with benchmarks etc. Have had a busy few days. Hoping I might get something done this weekend but might be next week. Very tempted to order a cheapo decibel meter so I can add that to it too. This case is unbelievably quiet considering what's in it and how cool it keeps it. It's all very well me saying it's quiet, but that is partially subjective.


    Edit: also, I haven't come across any board niggles or bugs yet. Totally impressed with this board. :)

    This is been a really interesting thread that I'd been following closely, but lost track of for a while. While its too late now, I thought I'd mention in case it helps others in the future:


    Zitat von ellnic

    ....
    I'm not sure if this is the CPU or a bug related to the 2687W, because it was tested prior to me getting it and there weren't any weird issues like this. In any case, it looks like I might have just upgraded to 14 cores ;)...


    I had a past experience with Supermicro where using a specific processor from a listed "compatible" family caused the sensors to report incorrectly. In my case using a specific model dual core Celeron in my X7 board (and I tried more than one) resulted in questionable temps and fan speeds of "0" on all fans. BIOS reported each processor and all its particulars properly, but sensors didn't function correctly. If I popped a different speed Celeron from the same family, or a Core 2 Duo in, any Core 2, even ones NOT on the compatability list, everything functioned properly after a BIOS reset. Supermicro suggested I take it up with my local distributor.


    Zitat von ryecoaaron

    Doesn't the line - Supports ECC*, RDIMM Memory (*ECC is supported with Intel® Xeon®processors) - mean it will use it? I've never seen a board that will let you use ecc but not enable it.


    While its not the case here, I'm currently working with an MSI motherboard (AM1I) that listed ECC memory on the tested list, but doesn't support ECC. I thought it odd to see ECC listed on such a low end consumer board, so put some in and did a test install of Ubuntu. dmidecode reported properly but EDAC modules wouldn't load, telling me that "BIOS Blocked module from loading". MSI confirmed that ECC memory will work, but ECC function won't. Which is...strange.


    Thanks to everyone for such an interesting read

    @Markess Sounds like your Supermicro symptoms are the same as mine. My Supermicro board is in the process of being RMA'd but I had to fight to get it returned. The guy I was dealing with said that Supermicro boards are for replacement or repair, never a return and refund - but they are making an exception. I'm not sure this aligns with UK consumer rights and Mr. PayPal but hey ho. What this experience has taught me is:


    1. Supermicro are not a great company to deal with, this might be because I am a consumer, but then most on this forum are and I want to warn others. I constantly felt like I was chasing them. They didn't send emails and updates when promised, they seemed almost bothered when I called, and overall I didn't feel like my ~£280 investment was a good one.


    2. If you compare the build quality of the Supermicro board with the ASRock, there is no competition in my eyes. The ASRock wins hands down. It's appears to be better made, it's UEFI is smoother, it performs better. As stated previously, this board is flawless. I haven't noticed anything even so much as annoying with it yet. For example, with the Supermicro board in the EFI, there were a number of issues including the boot device selection drop downs that were so long they rendered off the view of the screen and wouldn't scroll. Rendering boot devices 1-6 (approx) useless unless you could see what you wanted in the drop down when you clicked on it, which was about 2/3 hidden!


    3. Buy your stuff from a large retailer. I had to search for a small reseller that had this board, because Supermicro boards are generally sold in bulk via B2B only. This caused some complications when the RMA was issued, for example, it was less than 28 days in my possession but the retailer went on date of order, and they initially refused to collect it and accept the RMA because of this. Conveniently forgetting that I reported the problem to Supermicro 5 days after receipt and it was only because I was constantly waiting on them to email me something or check something for me that it took so long. Finally they agreed, but I don't think they're too pleased about it. If I hadn't tested 2 CPUs and those in other boards, they probably would have tried blaming me. Should have stuck with Amazon/eBuyer/etc to begin with :)


    Was the AM1I board you tested for ECC an AMD board? From what I found when looking into verifying ECC seemed to indicate that the EDAC modules only loaded for AMD CPUs.... I hope this is the case anyway, because I see no mention of anything EDAC related in the kernel log and lsmod | grep EDAC gives no result.


    Glad you're enjoying the thread by the way :)

    This is probably because I do not have the recommended amount of drives for a RAIDZ2, however when playing around with the Ashift value this morning I noticed somehing. Usually, to get the most space but at the loss of some performance, you would use Ashift 9. To max out on performance, but lose some space, you would use Ashift 12.


    Default OMV Plugin Ashift (box unticked) - 20.3TB usable space

    Code
    /mnt/Tank # time dd if=/dev/zero of=ashift.bin count=20000 bs=1M conv=fdatasync && sync  
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB) copied, 30.6986 s, 683 MB/s
dd if=/dev/zero of=ashift.bin count=20000 bs=1M conv=fdatasync  0.02s user 10.40s system 33% cpu 30.701 total


    Ashift 12 - 20.3TB usable space

    Code
    /mnt/Tank # time dd if=/dev/zero of=ashift12.bin count=20000 bs=1M conv=fdatasync && sync
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB) copied, 30.6864 s, 683 MB/s
dd if=/dev/zero of=ashift12.bin count=20000 bs=1M conv=fdatasync  0.02s user 10.87s system 35% cpu 30.689 total


    So it looks like the default is 12.


    Now look what happens when I set the Ashift to 9...


    Ashift 9 - 21.4TB usable space

    Code
    /tank # time dd if=/dev/zero of=ashift9.bin count=20000 bs=1M conv=fdatasync && sync
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB) copied, 32.8503 s, 638 MB/s
dd if=/dev/zero of=ashift9.bin count=20000 bs=1M conv=fdatasync  0.01s user 10.39s system 31% cpu 32.853 total


    To me, that certainly looks like pretty much the same write performance and slightly less CPU - AND 1.1TB extra in the process. I think I'll be sticking with this.


    Here is a screen shot of the config:



    Just a note: The default plugin behavior is to use Device Alias by path - I set this to by ID, then if the assignments change on a reboot, the tank will still mount.

    Zitat von ellnic

    1. Supermicro are not a great company to deal with, this might be because I am a consumer, but then most on this forum are and I want to warn others. I constantly felt like I was chasing them. They didn't send emails and updates when promised, they seemed almost bothered when I called, and overall I didn't feel like my ~£280 investment was a good one.


    I've been using FreeNAS/NAS4Free, where Supermicro is a preferred brand due to its generally solid support by the underlying BSD OS, including enterprise oriented features like IPMI, multiple HBAs, etc. If a Linux, or even Windows, based OS can meet a users needs. however, there's a lot more supported hardware options to choose from and the argument for paying the Supermicro price premium is much less compelling. Even if you need ECC and IPMI, there's a lot more hardware to choose from with Linux.


    I've had great personal experience with ASrock. I chose an ASRock H97 ITX board for my desktop and it just works flawlessly. I think (just my opinion) that Supermicro tends to be conservative with BIOS/UEFI to maintain compatibility with OSs that bring hardware support on more slowly, like BSD. So they tend to be more clunky.


    Zitat von ellnic

    Was the AM1I board you tested for ECC an AMD board? From what I found when looking into verifying ECC seemed to indicate that the EDAC modules only loaded for AMD CPUs.... I hope this is the case anyway, because I see no mention of anything EDAC related in the kernel log and lsmod | grep EDAC gives no result.


    Yes, its AMD. A local retailer here had a one day combo sale for motherboard and APU for US $39 (down to $25 if I bother to send in the rebate form). That's less than a Raspberry Pi 2 here, so I figured I'd give it a go for a little testing. So I was surprised to find ECC on the compatibility list for such a low spec/inexpensive item. I'm not sure why MSI would go to the expense of putting the extra traces in on the motherboard, and then block their use in BIOS. Perhaps the same PCB is also used in POS, signage, or other industrial setups that need ECC?


    To maintain the "Spousal Approval Factor" I keep most of the hardware in my modest home office, or cabinets off the bedroom or in the family room, all of which tend to get HOT in our Central California summers...and springs...and fall. So heat management has become my main priority over throughput or number crunching ability. Lately, I've decided that FreeNAS/BSD/ZFS, which doesn't handle low power states and drive spindown gracefully, just isn't the best fit for my particular needs. My modest Supermicro X7 based FreeNAS box with 8GB or RAM and a simple 2 disk mirror draws 65watts at idle. So I'm back lurking the forums here and trying OMV.


    So far, the AMD set-up on OMV is drawing only 10 watts at the plug with drives spun down. Once I have that configured well, I'll try disabling the GPU core and setting up suspend/WOL. Ought to be able to shave a watt or two off with that. I'd rather be testing monumental throughput and number crunching like you, but getting the heat signature tamped down addresses my needs better right now.


    Excellent work by the way!

    ASRock X99 WS. If you don't mind not having IPMI and integrated graphics it's a cracking board with a great UEFI and it's covered in blue heat sinks so it looks great. The only thing that I would mention about it, but it's hardly a complaint, is that two of the 10 SATA ports are shared with the eSATA and mSATA. So if both the eSATA and mSATA are in use, the remaining number of usable SATA ports is 8 - but I'm running 9 drives off the SATA ports (8 in the array and 1 SSD for the OS) leaving the eSATA free so it suits me fine. The build quality and stella UEFI are immediately obvious and it benchmarks slightly better (and cost less). Most of their X99 boards seem to support ECC which has been confirmed by email from them, check them out. :)


    I've had a hellishly busy week but I'm hoping once I get everything set up over the next few days (in the process of sorting out Emby at the moment) that I will be able to do that write up thread I've been promising.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!

Benutzerkonto erstellen Anmelden