Building OMV automatically for a bunch of different ARM dev boards

    Quote from nasnas

    I thought ROCK64 or other kernel update can solve my issue

    'chances are USB-3 bulk-streams are broken on the USB controller on the odroidxu4' (and same might apply to RK3328 as well, I will suggest to Rockchip engineers buying such a Seagate thing too and if it's there the same and they get it fixed in their kernel it might take another few months until ODROID micro community stops rejecting reality)

    Quote from nasnas

    Can you recommend a 2.5" disk enclosure which works with Linux UAS driver?

    If we're talking about that 5TB Seagate then it gets complicated since this is a 15mm disk. Besides that I use many JMS567 and one ASM1153 2.5" enclosures without problems (funnily all from same vendor since inexpensive, they list chipsets and I buy based on this only).


    Well, the only problems occur when trying to let the attached disk be powered by the host. I've some combinations that

    • work all the time (eg. a 750GB HDD used as backup disk)
    • some that only work on my Mac (eg. a Samsung EVO 750 works bus-powered on my MacBook known to provide as much as 1100mA on the USB3 ports but not on any SBC)
    • some disks only work bus-powered in the JMS567 enclosures but not in the ASM1153 (so maybe ASM1153 needs more power or the firmware there has a different 'power on' strategy -- at least I've two disks that run in underpowering problems only in the ASM1153 enclosure)
    • and then I've now 2 disks which work fine with Mass Storage on XU4 but run in underpowering problems when allowing to use UAS... they both work fine on an Orange Pi PC or Plus 2E but that's not comparable since there we're limited to USB2/Hi-Speed.

    I don't really understand this since Hardkernel recommends using a 5V/6A PSU with the original Cloudshell instead of the standard 5V/4A. If increasing available current by 2A should be of any help it would mean that there are no current limiters on the USB3 ports (which is somewhat surprising, usually it's 900mA with USB3 or 1100mA on more recent Macs, most Orange Pis for example have 2 1500mA current limiters, one for the upright USB port and one for the two horizontal). At least my ODROID-XU4 doesn't change its inability to power some external disks reliably regardless of the standard 5V/A PSU or my bench PSU capable of providing much much more.


    BTW: Of course there's also a potential advantage by choosing ready to run external USB disks. The vendor might chose a combination of hardware and software that should work reliably with just 800mA (to leave some safety headroom on USB3 ports). The 'branded' firmware of these devices might be adjusted in a way that it allows disks with a rather high spin-up peak consumption to reliably work in such an enclosure since the firmware does some magic to allow for slower spin-up up or there's an extra capacitor to compensate for peak consumption and the firmware therefor delays disk spinup by 0.3 seconds or something like that).


    When using a separate disk enclosure you'll have to take care of this yourself which means for me: only buying enclosures that can also be powered externally after running into the above issues over time:

    Quote from nasnas

    Is it worth to try usb-storage quirks=0x0bc2:2322:t (US_FL_NO_ATA_1X) option with my Linux PC which runs Kernel 4.4?

    A bit off-topic since in this thread we try to focus on OMV on ARM devices (and typical caveats OMV/Samba/Linux will be blamed for that are totally unrelated).


    It seems your x86 Linux has this enclosure already UAS blacklisted so I doubt the quirk will do anything.


    If I would do a x86 Linux distro I would patch the kernel in a way to UAS blacklist everything with Seagate's vendor ID for the simple reason that most probably all these devices suffer from the same problems (all using ASM1153 with same broken firmware -- also responsible for 'eating' the disk's last sector for example) but Seagate constantly produces new variants with different device IDs. And some PCs have somehow broken USB3 host controllers (eg. Fresco Logic FL1009 has broken bulk-stream support) while others not so you get obscure bug reports you'll waste a lot of time with and once you figure out 'yet another broken Seagate variant' in 2 months the same story starts over again since now Seagate sells a new disk using 0x0bc2:5678 this time (still relying on the ASM1153 with broken firmware). So simply blacklist 0x0bc2:* and you're done.

    It is really hard to find a reliable seller who sells a real Linux UAS compatible enclosure on ebay/amazon.
    Time to forget the perfect UAS compatibility in the ARM & x86 Linux world for a while.
    I can live without UAS since I still have around 90MB/sec Samba performance with XU4. Let's see how it can be improved with Rock64.


    Anyway, my Seagate 5TB has worked very stably with XU4 for a couple of weeks without extra power cable.
    When I looked into the XU4 schematics, they used 2Amp load switch on the USB 3.0 power rail. So it is very reasonable.
    If I ran a very heavy load like transcoding on all 8-cores at 2Ghz, the peak current could be over 3Amp. So I must use a 6Amp power supply because the HDD in-rush current could be over 1Amp.
    Another Toshiba 2TB external HDD and the problematic Seagate 5TB could work together thanks to the 6Amp PSU.


    Do you know the maximum clock of the Rock64 CPU and the spec of the load switch on the USB 3.0 port?
    If it is too much out of topic, I can wait until I receive it in August or September. :)

    Quote from nasnas

    Anyway, my Seagate 5TB has worked very stably with XU4 for a couple of weeks without extra power cable.

    Good to know but that's what I get when attaching the aforementioned 7.2k 2.5" disk with UAS enabled in an ASM1153 enclosure and powered by XU4:


    To explain what's happening: '[ 1503.633731] bash (1771): drop_caches: 3' is the start of the benchmark (I always flush caches prior to any benchmark since I don't like numbers without meaning).


    Then at 1503.960555 when iozone generates sequential loads the small bug that most probably needs a quirk for Exynos USB3 host controller appears in the log.


    And around 1536.494331 iozone has finished with sequential writing/reading and starts with random writes, now under-powering occurs and dmesg output reports a lot of troubles (naive people will consider this being 'UAS is broken' while it's just the additional HDD heads moving around creating high load/consumption and then things start to fail). Same test with an externally powered enclosure will be fine. But I have none with ASM1153 and will not buy one anytime soon.


    I always see people testing with hdparm or dd (most of the times using absolutely useless parameters). Both tools are a joke since they do only sequential writes/reads. This is boring since this is no stress for any HDD (platters rotate, heads move only very slowly). You need random IO keeping the heads busy (iozone -i 2) to trigger powering problems.


    To make things worse on ODROID XU4 we have all the time the internal USB hub in between that even disappears from the bus in such situations! See the last lines in dmesg output above: 'reset high-speed USB device number 2 using xhci-hcd' and then 'New USB device found, idVendor=05e3, idProduct=0610' (that's the USB3 part of the internal Genesys Logic hub). Sorry but 'reliable storage' is about avoiding unnecessary complexity like this!


    (full dmesg output as reference. I shut ODROID-XU4 down afterwards and will not power it on that soon again)

    And some RK3328 updates. Today we did some testing with ROCK64 again and even if there's still a lot of room for improvements it already looks really good: https://forum.armbian.com/inde…findComment&comment=35323 (compare with the LanTest numbers for ODROID-XU4 a few posts above and you get the idea what you can expect in real-world scenarios already)


    Also some storage performance updates without and with a good USB3 hub in between: https://forum.armbian.com/inde…findComment&comment=35316


    Soon there will be another RK3328 SBC called 'Le Fly' most probably also using Gigabit Ethernet and priced competitively: https://libre.computer/products/boards/roc-rk3328-cc/ (no contents yet)

    I tried the the first iozone command several times. There was no error at all. So the USB cable comes with the Seagate 5TB HDD box seems to have very low resistance.


    Anyway, RK3328 development progress is very impressive. :o
    I hope somebody can show me the LanTest numbers with an HDD instead of the ideal SSD.

    Quote from nasnas

    I hope somebody can show me the LanTest numbers with an HDD instead of the ideal SSD.

    There you go: this is ROCK64 with the Hitachi 2.5" (7200 rpm) disk from a few days ago that refused to work on XU4 with UAS. Works now flawlessly: picture and dmesg output -- I tried it before with a JMS567 enclosure, a hub in between and different cables. The disk in the ASM1153 enclosure was connected at 372.998527 and since then no problems any more)

    We have still issues so these numbers are preliminary anyway and also the disk is pretty old and not that fast. I just repeated the standard iozone test on the fastest partition (I also used for the NAS test) and post the XU4 / Mass Storage numbers again for a comparison (as expected UAS scores better but we're also using 2 different devices here so that test is not that scientific anyway):


    Quote from nasnas

    Anyway, RK3328 development progress is very impressive. :o

    Well, yes. Both we as community make nice progresses and Rockchip engineers themselve diagnose issues and send funny reports made with an USB3 analyzer back to us. Looks then like this for example:



    BTW: I elaborated a little bit about USB3 cable/connector issues since I ran in exactly this few hours ago: https://forum.armbian.com/inde…findComment&comment=35333

    Silly me! I forgot that I'm currently not testing my OMV/Armbian builds but ayufan's OMV images for ROCK64. For whatever reasons rock64_fix_irqs.sh does not get automatically executed at startup so after manually executing it performance increases (instruction the CPU cores to take care of individual subsystems, especially getting IRQ processing for Ethernet and USB3 away from cpu0 where both get in conflict otherwise):


    Just by tweaking some settings we get 9 MB/s more in write direction and even 15 MB/s in read direction. Settings matter! :)

    Quote from tkaiser

    rock64_fix_irqs.sh

    And now iozone numbers also improved (but this must be due to switching from ondemand governor to performance, IRQ affinitiy can't be the reason):

    Code
    random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    15353    17885    18069    17338      675     1227
          102400      16    49141    56370    52564    52499     2661     4941
          102400     512   102425   104195   105165   106311    44485    56117
          102400    1024   103314   102459   104835   107574    63781    64271
          102400   16384   100834   102401   101345   103759   101399    99320

    Final update: these are the results of RK3328 with appropriate settings and a rather slow 2.5" HDD:

    Settings are fine, consumptions is as low as possible. The above numbers were generated with ondemand settings (cpufreq low when not needed and only increasing when necessary). No heatsink needed, idle temperatures around 50°C, full NAS load 70°C (ambient temperature currently 30°C)


    Great! Appreciate your valuable tests.
    My XU4 CPU temperature is near 75°C with similar NAS performance.


    BTW, your HDD seems to be much faster than mine. My 5TB is 5400rpm while yours is 7200rpm.
    Stupid Seagate also broke the UAS compatibility against Linux.
    I have to invest more time to search a Linux friendly enclosure this weekend before Rock64 arrives here.

    Quote from nasnas

    BTW, your HDD seems to be much faster than mine. My 5TB is 5400rpm while yours is 7200rpm.

    Well, your HDD should be a lot faster since 5 platters (10 heads) rotating with just 5400 rpm result in higher bandwidth. Anyway I used XU4 one last time, latest OMV image (upgraded to 3.0.81, kernel 4.9.33), same disk/partition, same enclosure (picture and debug output). This time it worked (ASM1153 enclosure powered by XU4) but performance is shit:


    Code
    Host powered in ASM1153 enclosure                             random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    11859    12746    16141    15766      624     1103
          102400      16    38879    40961    43359    43955     2595     4624
          102400     512    89819    88433    94175    95327    38726    52457
          102400    1024    89186    89756    93010    96308    52115    58614
          102400   16384    78021    80624    83628    85159    85717    85184


    Same disk, same partition (the fastest), same enclosure, same powering scheme (SBC provides power via USB3 port), different performance.


    What happened?

    • Why does it work at all since all my previous attempts with this combination failed? Since I adjusted input voltage to 5.2V (bench PSU) instead of 5.0V. It's just another example for Ohm's law 'surprisingly' being valid everywhere. People babble about amperage ratings while they're suffering from under-voltage in reality. And if they report stuff like this in ODROID forum they're told to do kernel updates
    • During the sequential write tests the disk made ugly noises (clicking). We're talking about underpowering here. Obviously the disk controller's firmware implements a strategy to deal with this: Reduce consumption by slowing down (no disconnection happened, see the end of debug log)

    As usual inappropriate powering is reason N°1 for SBC troubles. Someone not beeing such an electronics noob than me might want to have a look what's happening here (measuring voltage-drops on a connected disk enclosure while generating load -- that's what Ohm's law is about --> you only run in trouble once consumption increases and you have to measure on the right spot to detect this!).


    I won't further waste my time with this device since unfortunately on XU4 we deal with another issue: USB3 receptacle crappiness. I tested through maybe 10 USB3 receptacles recently with different cables and in XU4 they fit less tight. That's a huge issue if you look at how shitty USB3-A is:


    Unlike the USB2/Hi-Speed data lines where it doesn't really matter how tightly the cable fits for the USB3 SuperSpeed data lines to operate properly you must fully insert the cable. Since the SuperSpeed data pins are the outer ones even slightly touching/bending the cable might result in sporadic loss of connection (then you get either xhci or uas errors in dmesg output -- these messages don't talk about software issues but just drivers reporting an underlying hardware problem: a shitty connector loosing contact)


    On ROCK64 cables fit more tightly but of course the same can happen there (see my report from yesterday). USB3-A is crap and until we get USB-C everywhere (made by engineers thinking about average users out there and what happens outside of their labs) you need to pay a lot of attention to cable/contact issues.


    In other words: Now that we know how well RK3328's USB3 implementation works I really hope a hardware manufacturer soon will eliminate cable/contact problems by simply adding a JMS578 (or JMS567 or ASM1153) on the PCB combined with the usual 7/15 pin SATA/power connector. Dual powering input (5V and 12V via separate barrel jacks), step-down converters on the board, 12V on pins 13-15 of the SATA power connector, PCB traces thick enough to avoid voltage drops.


    This way people can use a 5V/3A PSU to reliably power board and a connected 2.5" disk while by using a 12V/2.5A PSU they can also power a 3.5" disk. Relying on ROCK64 price list: 1GB variant for $40, 2GB for $50 and 4GB for $60. Problem solved.

    And a final settings comparison between ODROID-XU4 (big.LITTLE design combining 4 slow A7 cores with 4 fast A15) and ROCK64 (4 A53 cores): these OMV images rely on basic 'Armbian tuning' (IRQ affinity, IO scheduler settings for rotating disks and so on) but also contain further tweaks. We optimize system performance in 3 locations: /etc/init.d/armhwinfo, /etc/rc.local and /etc/cron.d/make_nas_processes_faster (the first two executed on startup, the latter executed every minute)


    Similar settings on both boards (configuring Receive Packet Steering (RPS) in the same way and tweaking ondemand cpufreq governor to be responsive to IO loads)

    Different settings but doing the same (sending USB3 IRQs to cpu2 and Ethernet IRQs to cpu3 -- these are little cores on ODROID-XU4):


    Code
    echo 4 >/proc/irq/$(awk -F":" "/xhci/ {print \$1}" </proc/interrupts | sed 's/\ //g')/smp_affinity
echo 8 >/proc/irq/$(awk -F":" "/eth0/ {print \$1}" </proc/interrupts | sed 's/\ //g')/smp_affinity
echo 4 >/proc/irq/$(awk -F":" "/usb3/ {print \$1}" </proc/interrupts | sed 's/\ //g')/smp_affinity
echo 8 >/proc/irq/$(awk -F":" "/usb5/ {print \$1}" </proc/interrupts | sed 's/\ //g')/smp_affinity


    And the cronjob being executed every minute to enhance IO performance of NAS daemons is also different since on ODROID-XU4 we move these daemons to the A15 cores:



    Code
    for i in `pgrep "ftpd|nfsiod|smbd|afpd|cnid"` ; do ionice -c1 -p $i ; done
for i in `pgrep "ftpd|nfsiod|smbd|afpd|cnid"` ; do ionice -c1 -p $i ; taskset -c -p 4-7 $i ; done


    These OMV images contain also other improvements (eg. better Samba settings) but the above is essentially 20-30 MB/s more for free just by looking at what's really happening and tweak settings accordingly. To analyze what's going on and to improve performance all you need is an open mind, not being trapped in vendor micro communities and then 'htop', 'iostat 5' and 'armbianmonitor -m' in 3 different terminals while looking at /proc/interrupts and cpufreq 'time_in_state' statistics (on big.LITTLE implementations these are two files of course in a different location, do a 'find /sys -name time_in_state' to get the idea).


    'sudo armbianmonitor -m' on some platforms lacks detailed CPU utilization reports (see example) so that's what 'iostat 5' is for (especially %iowait and %sys vs. %user is interesting there).


    In case anyone runs into trouble with these OMV images always provide output from 'sudo armbianmonitor -u' (see example and there especially at the end).


    And obviously you can't fix hardware problems (underpowering, SD card crappiness or cable/contact issues) with software. Same with hardware limitations. For example soon a lot of current Raspberry Pi users will head over to something called Banana Pi M2 Berry (at least in Germany and Europe sales will start soon). After they'll find out that compatibility to Raspberries is not that great some might think about a 'NAS only' use case just to realize that there will be neither a performant OMV image for this device nor a way to improve performance. Since even with tweaks like the above the SATA implementation there is crippled and especially writes to such a Berry will always be slow as hell.

    "Changing to 5.2V from 5.0V" seems to be very important key fact for many other ARM SBC as an HDD-NAS solution.
    I hope we can see a new SBC which contains a good powerful SATA port and stable HDD power supply circuit soon as you suggested.

    Quote from nasnas

    "Changing to 5.2V from 5.0V" seems to be very important key fact for many other ARM SBC as an HDD-NAS solution.

    Well, the problem is that the average user is neither aware of under-voltage being such a problem nor Ohm's law (which makes this hard to diagnose since only if consumption increases due to higher activity the problem will be triggered).


    Here's a perfect example combining two hardware products of the same vendor together and running into issues: https://forum.odroid.com/viewtopic.php?f=146&t=27548


    Both affected users combine ODROID-XU4 with the original Cloudshell (enclosure + USB-to-SATA brige + display, everything including 2.5" disk powered by XU4's USB3 port).


    User @reza reported that issues started after 'upgrading my lan cable to get 1Gb/s speed on lan I get hdd offline when using heavy copy and paste'. When switching from Fast Ethernet to Gigabit Ethernet the overall board consumption increases a lot since before we were talking about NAS processes maxing out at ~11 MB/s and later this can be ten times more which also increases power demands of connected disk and host (there are a lot of USB IRQs to be processed, the Gigabit Ethernet PHY usually needs a few more 100mW, the disk has also a lot more to do compared to 'almost idle' with Fast Ethernet.


    On ODROID XU4 it's even worse as on other Gigabit Ethernet equipped SBC since everything is USB here. And the used RTL8153 chip needs a lot more power compared to other SBC that combine an internal GbE capable MAC with the usual external RTL8211 PHY connected through RGMII. So even if your PSU is able to provide 6A the real consumption increases a lot and if there's high resistance somewhere between PSU and disk or disk controller then voltage there might drop below 4.7V or even 4.6V and then disks start to behave strange (varies, so check the datasheet! HDDs usually need 5V for the motors while most SSDs transform the available voltage to 3.3V or below for controller and flash cells so they are more robust in this regard especially since consumption is lower anyway).


    @crazyquark, the 2nd user affected with the same Cloudshell/XU4 combination is pretty sure that his issues can't be power related: 'I have a 6A PSU so it's not a power issue'. Well, if you run in under-voltage it doesn't matter whether your PSU provides 4A, 6A or 100A. Since as soon as SBC+disk start to consume more the voltage drops below acceptable levels. But of course in addition to or alternatively you can also run in under-current problems so the recommendation to use a 5V/6A PSU is not wrong. But the 6A won't help with under-voltage and the 5V should be 5.25V anyway.


    All their reports contain the same message worth a search in ODROID forum: 'reset SuperSpeed USB device number 3 using xhci-hcd' site:odroid.com


    And of course that' just a simple under-voltage issue Cloudshell users suffered already two years ago from.


    Quote from nasnas

    I hope we can see a new SBC which contains a good powerful SATA port and stable HDD power supply circuit soon as you suggested.

    I ordered one few days ago: ESPRESSObin. Can be fed with 12V so an internal step-down converter provides stable 5.0V all the time. Uses reliable cabling and protocols (SATA and later I might try to use NVMe with a mPCIe to M.2 adapter too), fast storage and network implementation with low CPU load since made for NAS. No USB3-A receptacle crappiness, no under-powering drama.

    Quote from tkaiser

    No USB3-A receptacle crappiness, no under-powering drama.

    Just to elaborate on that: above we have a few issues that could be nailed down to under-powering (symptom when UAS was disabled: 'reset SuperSpeed USB device number 3 using xhci-hcd').


    Here user @Kosmatik after being told to 'Disable UAS! It's evil!' runs not in under-powering troubles but USB3 cable crappiness this time: https://forum.odroid.com/viewt…t=26016&start=200#p191201


    Exactly same symptom, the log reads 'reset SuperSpeed USB device number 3 using xhci-hcd' (it's a simple cable issue leading to connection losses and the try to re-negotiate the connection later). This makes it of course a little bit hard to diagnose stuff since once you solved your under-voltage problem a simple USB3 cable not fitting exactly can later show the very same symptoms as the power problems before.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login