Building OMV automatically for a bunch of different ARM dev boards

Another update with regard to the 'cheap and energy efficient single drive NAS' journey. Xalius, another dev, tested yesterday with ROCK64 and a rather performant SSD (Samsung 850 Pro with 512GB, this drive is bottlenecked by SATA 3.0 that means on a SATA port it will exceed 520 MB/s with sequential workloads even with rather small block sizes, also random IO performance is that high that numbers aren't different than sequential numbers). ROCK64's companion was a JMS578 disk enclosure, UAS capable of course.

We're talking about +380MB/s on average with UAS (full log here: http://pastebin.ubuntu.com/25064848/). This is close to the maximum what we can get with USB3 SuperSpeed anyway (400MB/s or to quote the specs: "At a 5 Gbps signaling rate with 8b/10b encoding, the raw throughput is 500 MBps. When link flow control, packet framing, and protocol overhead are considered, it is realistic for 400 MBps or more to be delivered to an application.")

Please remember: This is a $25 single board computer featuring an ARM SoC made for TV boxes (Rockchip RK3328). The CPU cores are somewhat underpowered compared to any recent x86 box but USB3 performance is simply impressive here and IMO we don't need any more storage tests wrt RK3328 USB3 host performance (well, most probably prefixing benchmark tests with 'ionice -c1 ' will further improve the 380 MB/s above but that's just tuning numbers).

A prerequisit for such impressive USB performance is 'USB Attached SCSI' (UAS) of course. It's way more efficient than the older USB Mass Storage protocol. And this does not only result in better performance but also less CPU utilization. To get an idea I simply tested with and without UAS (see [1] how to disable this on ROCK64). I used the usual iozone test but only with 2 blocksizes: 4KB and 16MB. Performance differs a lot (not that important if you want to use HDDs instead):

random    random
                    write  rewrite    read    reread    read     write
          UAS 4K    18129    23421    27535    23642    15982    17115
 Mass Storage 4K    13410    14787    16924    16926    11617    10639
         UAS 16M   217132   219776   354657   354838   353728   183282
Mass Storage 16M   149317   163379   180475   180800   180795   116282

It's obvious that Mass Storage performance is much worse compared to UAS (especially with the huge blocksize -- since my test SSD is cheap crap normally the sequential write difference would also be much higher). But as already said: not that important if it's about using spinning rust (HDDs). So let's look at CPU utilization (I let an 'iostat 5' run in another terminal).

First CPU utilization for 4K block size:

4K UAS:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
write      0.09    0.00   17.85    4.25    0.00   77.82
rewrite    0.10    0.00    9.43    4.86    0.00   85.62
read       0.10    0.00    7.60    5.68    0.00   86.62
reread     0.67    0.00   14.42   10.03    0.00   74.88
rand read  0.10    0.00    1.05   14.97    0.00   83.87
rand write 0.00    0.00    1.11   10.09    0.00   88.80


4K Mass Storage:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
write      0.00    0.00   29.81    7.43    0.00   62.75
rewrite    0.00    0.00   21.16    9.95    0.00   68.89
read       0.29    0.00   22.65   10.84    0.00   66.22
reread     0.97    0.00   20.66   10.96    0.00   67.41
rand read  0.00    0.00   17.60   12.08    0.00   70.32
rand write 0.00    0.00   20.10    9.60    0.00   70.30

I'm not entirely sure but AFAIK we should not rely on %iowait (and therefore %idle too) but can only have a look at %system. Then CPU utilization with Mass Storage is multiple times higher than with UAS on average and if we look at random IO then it's even 15 times higher). So if it's about random IO with small record/block sizes then UAS is way more efficient wrt CPU utilization (which is important since we should keep in mind that RK3328 is a quad core CPU and %25 utilization means one single CPU core is entirely busy and might bottleneck storage performance already).

With 16 MB block size it looks a lot better for Mass Storage:

16M UAS:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
write      0.00    0.00    5.97   20.00    0.00   74.03
rewrite    0.00    0.00    3.15   24.19    0.00   72.66
read       0.45    0.00   10.05   16.74    0.00   72.76
reread     0.63    0.00    9.95   16.65    0.00   72.76
rand read  0.53    0.00    8.28   17.99    0.00   73.20
rand write 0.00    0.00    3.60   22.09    0.00   74.32


16M Mass Storage:
write      0.00    0.00   11.20   23.73    0.00   65.07
rewrite    0.00    0.00   10.67   24.53    0.00   64.80
read       0.27    0.00   10.63   23.68    0.00   65.42
reread     0.27    0.00    9.76   13.38    0.00   76.58
rand read  0.36    0.00    9.85   13.37    0.00   76.42
rand write 0.00    0.00    5.08   20.41    0.00   74.50

The larger the block size gets the less UAS has an advantage wrt lower CPU utilization. Which is understandable based on how UAS works (see link above).

[1] How to set USB quirks (eg. for external Seagtate disks) or disable UAS on ROCK64 OS images:

Let's take disabling UAS as a somewhat stupid example (you don't want to disable UAS since it's great!). It's necessary to create a file called eg. /etc/modprobe.d/disable-uas.conf (filename doesn't really matter) with the following contents:

options usb-storage quirks=0x174c:0x55aa:u

(USB vendor and product ID have to match of course, check 'lsusb' output). And then you need to execute 'update-initramfs -u' since otherwise your settings won't be applied (initrd)

Zitat von ryecoaaron

I probably know the answer to this question but it would be nice if all of the OMV arm images were created the same way. I'm guessing the build process does not work for the RPi2/3?

Here you go: OMV_3_0_79_RaspberryPi_2_3_4.9.35.7z

Most important info: this is not the result of a totally automated build but manually throwing three things together and fixing some stuff also manually. At least this image has never been booted so it's 'as clean as possible' (userland generated from scratch few weeks ago by debootstrapping Debian Jessie from official repos). I used an Armbian OMV userland (armhf) together with latest official RPi kernel (4.9.35) and https://github.com/bamarni/pi64

There's 0% Raspbian involved, it's a true ARMv7 userland and also prepared to run ARMv8 code (a 64-bit mainline kernel 4.11.2 is included but not active, once you switch to this kernel you could always do an 'dpkg --add-architecture arm64' later to boost some stuff extremely on RPi 3 but since all Raspberries are such a horribly slow choice for a NAS I didn't investigate further).

Since it's based on Armbian almost all the Armbian tools work too eg. CLI monitoring or uploading debug info:

macbookpro-tk:~ tk$ ssh root@192.168.83.145
root@192.168.83.145's password: 
 ____                 _                            ____  _   _____ 
|  _ \ __ _ ___ _ __ | |__   ___ _ __ _ __ _   _  |  _ \(_) |___ / 
| |_) / _` / __| '_ \| '_ \ / _ \ '__| '__| | | | | |_) | |   |_ \ 
|  _ < (_| \__ \ |_) | |_) |  __/ |  | |  | |_| | |  __/| |  ___) |
|_| \_\__,_|___/ .__/|_.__/ \___|_|  |_|   \__, | |_|   |_| |____/ 
               |_|                         |___/                   


Welcome to ARMBIAN 5.27 experimental Debian GNU/Linux 8 (jessie) 4.9.35-v7+   
System load:   0.79 0.27 0.10  	Up time:       0 min		
Memory usage:  7 % of 970MB  	IP:            192.168.83.145
CPU temp:      46°C           	
Usage of /:    59% of 2.6G   	


[ General system configuration: armbian-config ]
Last login: Wed Jul 12 10:32:21 2017 from macbookpro-tk.fritz.box


root@raspberrypi:~# armbianmonitor -u
/var/log/armhwinfo.log has been uploaded to http://sprunge.us/edGJ
Please post the URL in the Armbian forum where you've been asked for.


root@raspberrypi:~# armbianmonitor -m
Stop monitoring using [ctrl]-[c]
Time        CPU    load %cpu %sys %usr %nice %io %irq   CPU
09:58:52:  600MHz  0.16  10%   2%   5%   0%   2%   0% 48.3°C
09:58:57:  600MHz  0.15  10%   2%   5%   0%   2%   0% 48.3°C^C

As usual with all my OMV builds a working Ethernet/Internet connection is needed on first boot (OMV installation will finish itself and then reboot automatically within 2-5 minutes).

What's missing

• filling the ext4 partition with zeros to shrink archive size (at the moment the archive is 700MB in size but might be just 300-350MB after doing this)
• adding the relevant Raspbian repository that contains firmware and kernel updates and also some useful tools like vcgencmd to talk to the main OS running on the VideoCore IV
• Adding the needed 'apt install' commands to /etc/init.d/firstrun to get RPi kernel/firmware/tools automatically updated

If someone familiar with this Raspberry stuff provides information with regard to this, I'll add this and re-publish the image afterwards. Otherwise not.

I've just a quick&dirty boot test on an RPi 3 but since I tested the used rootfs extensively on a much more capable device (Clearfog Pro using ARMADA 388 SoC, the same engine as Helios4 ) there shouldn't be that much surprises.

Zitat von tkaiser

If someone familiar with this Raspberry stuff provides information with regard to this, I'll add this and re-publish the image afterwards. Otherwise not.

I looked through https://github.com/debian-pi/raspbian-ua-netinst/issues/435 just to be ensured that I won't further waste time on this (since Raspberries are such an unbelievable bad choice for any NAS). If someone tests through and provides both the necessary repo for /etc/apt/sources.list.d/ and a single 'apt install' command line that works for sure in an unattended fashion I'll add this though.

Silly me wasted his time with Raspberries -- unbelievable. New image uploaded which should fix all of the above issues.

Debug output from Raspberry Pi 3 with a crappy 4 GB SD card. Debug output from Raspberry Pi 2 with a good Samsung 64 GB SD card.

Kernel+firmware will be updated automatically now (I added the needed repo and the 'apt install' call to firstrun routine). So now it looks like this:

root@raspberrypi:~# apt-cache show raspberrypi-bootloader
Package: raspberrypi-bootloader
Source: raspberrypi-firmware
Version: 1.20170703-1
Architecture: armhf
Maintainer: Serge Schneider <serge@raspberrypi.org>
Installed-Size: 12259
Recommends: raspberrypi-kernel
Homepage: https://github.com/raspberrypi/firmware
Priority: extra
Section: admin
Filename: pool/main/r/raspberrypi-firmware/raspberrypi-bootloader_1.20170703-1_armhf.deb
Size: 3552060
SHA256: 4ac26e71e888203129a05b6291e055d0e7eaeab0859481cfb9cfb83b7ee7909d
SHA1: 6e7044523fbaef01b97fb2736d9c7fd29d5f40dd
MD5sum: 2d6b8b53290c964b2a140eedc52780f1
Description: Raspberry Pi bootloader
 This package contains the Raspberry Pi firmware and bootloader.
Description-md5: f3fa59829d377b6fc81f91a4f5aa78da

Only downside: Since now RPi kernel + firmware stuff will be installed on first boot if you use a crappy SD card this can take ages (with my shitty 4GB noname SD card above it took 12 minutes, with a good Samsung EVO+ the whole procedure was done in less than a minute)

New image at same location: OMV_3_0_79_RaspberryPi_2_3_4.9.35.7z (now 'only' 370 MB in size, that's due to all the proprietary 'firmware' crap those Raspberries need).

Settings/performance might be better compared to the older Raspbian based OMV images (now it's ARMv7 userland with some optimizations) but of course I won't do any performance tests since Raspberries with their single USB2 port and shared Ethernet/storage bandwidth are the wrong choice for a NAS anyway.

Again: this OMV image is not based on boring ARMv6 Raspbian at all, we're only fetching kernel+'firmware' updates from https://archive.raspberrypi.org/debian/ -- the rootfs is a clean armhf Debian with Armbian tweaks applied and there's an 64-bit mainline kernel + modules in place so adventurous users can try to release the handbrake and switch from ARMv7 software to a true 64-bit ARMv8 experience. Please don't ask how to do this since it's useless with OMV use case anyway and it needs some skills to benefit from.

Zitat von jata1

thanks for spending the time/effort on doing this for a raspberry. I have one and I think it works great with OMV but agree that it is slow - especially network throughput

The problem is that with every other SBC around you can simply add a cheap RTL8153 USB Gigabit Ethernet dongle (7 bucks on Aliexpress) to increase overall performance a lot (30 MB/s) but not with those Raspberries since there's only one USB2 connection to the outside. I measured this long ago and as soon as real disk accesses are involved performance drops dramatically: see the LanTest screenshots at the bottom of post #2 https://forum.armbian.com/inde…findComment&comment=28825

Anyway: final image, this time I added also the RPi Foundation's libraspberrypi-bin package that provides vcgencmd, raspivid and other commands that deal with the proprietary VideoCore stuff. Seems to work. So I added also /usr/sbin/raspimon which then looks like this:

root@raspberrypi:~# raspimon 
To stop simply press [ctrl]-[c]


82.2'C 1200/ 947 MHz 1110000000000000010 1.3250V
82.7'C 1200/ 933 MHz 1110000000000000010 1.3250V
82.7'C 1200/ 931 MHz 1110000000000000010 1.3250V
82.7'C 1200/ 918 MHz 1110000000000000010 1.3250V
82.2'C 1200/ 935 MHz 1110000000000000010 1.3250V
79.9'C 1200/1163 MHz 1110000000000000000 1.3250V
75.8'C 1200 MHz 1110000000000000000 1.3250V

(this was with a heavy load on RPi 3 with a pillow on it to trigger throttling. The RPi kernel is cheating on us reporting the CPU cores would run at 1200 MHz while in reality it's throttling down to 900 MHz already -- that's why we need vcgencmd here --> to be able to talk to the main OS controlling the hardware, the so called 'firmware'). It also shows you when you suffer from under-voltage, always worth a test while running a quick NAS benchmark.

I also added a very simple health logging in /var/log/raspihealth.log. On every clean shutdown there will be a single line logged containing status information whether under-voltage, frequency capping and throttling happened or not since the last boot. If you see there entries that are not just a single 0 then you're in trouble and should run raspimon to see what's going on (under-voltage with Rasperries is a huge problem due to the crappy Micro USB DC-IN connector).

Quick explanation of these weird 19 digits displayed by raspimon and maybe appearing in /var/log/raspihealth.log

The three digits on the right are about actual situation while those three on the left are flags that are set if one of the three different problems occured since last boot and now (official description differs but is wrong in my opinion after testing through all three issues extensively)

1110000000000000010
|||             |||_ under-voltage
|||             ||_ currently throttled
|||             |_ arm frequency capped
|||_ under-voltage has occurred since last reboot
||_ throttling has occurred since last reboot
|_ arm frequency capped has occurred since last reboot

OMV_3_0_79_RaspberryPi_2_3_4.9.35.7z (tested on RPi 2 and 3, no 'apt upgrade' hassles like with the Raspbian based version)

MD5 hashes:

• OMV_3_0_79_RaspberryPi_2_3_4.9.35.7z = 8b4599cac9afc17cc60b357851fa60b4
• OMV_3_0_79_RaspberryPi_2_3_4.9.35.img = 36b7cdabdf3e33337243f1680509162f

@ryecoaaron In case you pick this image up after some testing it's necessary to explain in the release notes that this is not an Armbian image (no support in Armbian forum for any issue!) but just based on an Armbian userland. Most problems people will face with this image are as usual either related to SD card crappiness (use Etcher!) or powering problems. And since average users reject reality it's useless to argue with them. But that's why I recommend to only recommend Etcher any more and why there's now raspimon and /var/log/raspihealth.log included.

If users report problems they now have tools to check at least most common problem sources (checking for SD card corruption is 'armbianmonitor -c $HOME' and then afterwards doing an 'sudo armbianmonitor -u' and providing the output of both -- this is valid for all these Armbian based OMV builds then)

Zitat von scuub

Played with the 2nd version last night. Everything works perfectly. I m gonna burn this one now.

Please re-download if not already done. The version from few hours ago is better since contains debug/health stuff too.

In the meantime I also did a quick benchmark comparison to the former official Erasmus image using same kernel but ARMv6/Rasbian userland. With 128K blocksize it looked like this few months ago:

And now it looks like this with identical test setup (Samsung EVO 840 SSD in JMS567 disk enclosure -- same results with the 3rd partition of the Samsung EVO Plus 64GB SD card though, so the SSD doesn't play any role here since everything is bottlenecked by RPi's horriby low IO/network bandwidth anyway): Results are more or less the same with some minor improvements wrt handling of small/many files:

I also tested again with an attached RTL8153 Gigabit Ethernet dongle. First with 128K blocksize (old results ARMv6/Rasbian userland, new results with ARMv7/Armbian userland) just to realize that maybe there's something wrong with benchmarking (that's pretty common, at least I throw away more results than I publish). So I tested also with 1M blocksize and then it looks like this:

Since both SSD and Ethernet are behind the same USB bus and have to share bandwidth this is the best possible (since every single byte has to pass the USB2 connection twice). In other words: you can't really benefit from an external Gigabit Ethernet dongle, Rasperries are always the worst NAS choice ever. Even an ultra cheap NanoPi NEO 2 (OMV available from manufacturer implementing all tweaks we developed here the last months!) will outperform any Raspberry easily.

And when we look at SBC like ROCK64 that start at $25 and show excellent NAS performance there's really no reason to misuse a Raspberry for OMV purposes

Zitat von scuub

I logged by in the webinterface now and even after the update the error pops up when clicking the tabs.

Never seen such stuff and I installed and upgraded +10 times within the last 24 hours.

Maybe our 'environment' differs?

• I use a 5V/2A PSU tested and known to provide 4.9V at 2A real consumption
• Cable between PSU and Raspberry Pi is rated AWG20 (low resistance)
• OMV lives on a genuine Samsung EVO+ SD card
• I burned the image with the only reliable tool: https://etcher.io

Further readings:

• https://forum.armbian.com/inde…other-people-experiences/
• https://tech.scargill.net/a-question-of-lifespan/
• http://www.cnx-software.com/20…iability-and-tips-tricks/

TL;DR: On Single Board Computers if software starts to behave weird it's 99% of the times related to SD card crappiness or power problems (especially on devices that use the shitty Micro USB connector). It's not worth analyzing any of these problems unless SD card and powering problems are tested and solved

How do the first five lines of output from 'sudo raspimon' look like?

Zitat von scuub

It s definately just the Network tab that shows the error. After closing the error everything works in the tab.

Can't reproduce. I burned the image again, tried out clicking around in the Network tab, then did all upgrades, tried again, rebooted, tried again. No errors.

Have you checked MD5 hashes of the image you burned (I downloaded just in case and checked hashes for both archive and image -- they do match so at least nothing bad happened when uploading the image)

Zitat von jata1

I have 2-3 minor issues and hoped you could help me resolve them.

Not really since I already put the Raspberries back in the drawer and deleted the image locally (to be honest: Raspberry Pi as OMV host is insane, this thing is way too slow).

Please think about resolving this stuff on your own and then posting a short mini tutorial with your findings below New approach for Raspberry Pi OMV images

1) worked for me (but needed a reboot)

2) no idea about log entries, maybe you need to blacklist modules? Isn't /var/log handled by folder2ram (see 'df -h' output please)? If so something spamming syslog is no problem anyway

3) /etc/cron.d/make_nas_processes_faster should be self-explanatory? It's explained on the last page of this thread: Building OMV automatically for a bunch of different ARM dev boards

4) No idea but it should work the usual way (enabling a module here and a DT overlay there)

Zitat von jata1

CRON job that also runs every minute - not sure what this is or why it needs to run every minute

Simple reason: I decided to go with /etc/cron.d/make_nas_processes_faster instead of 'doing it correctly' (using the established Linux ways to set/inherit IO scheduling and priority classes) since I hate stuff that breaks later.

If these OMV images are shipped with adjusted start scripts or other tweaks it might happen that future upstream package updates revert these tweaks. I don't like this so it's a cron job executed every minute (this is necessary since new forked client processes always have a new and unique PID). 2 disadvantages: It's not 'the right way' (I don't care since it works and survives future 'apt upgrade' calls) and freshly forked NAS daemons might remain in an 'unaccelerated' state up to 59 seconds.

On appropriate OMV devices (read as: not Raspberries but minimum SBC with at least Gigabit Ethernet and USB3, fast eMMC or SATA) using ionice is good for significant higher IOPS and up to 7 MB/s (for free!). Of course on Raspberries this won't make any difference since they're all bottlenecked by shared network/storage bandwidth (never tested though).

So if you're on a Raspberry or prefer clean logs over performance there's nothing wrong to delete /etc/cron.d/make_nas_processes_faster

Zitat von jata1

4. make_nas_processes_faster cron job runs every minute:
once a minute is too much for me and fills the log

Huh?

1) folder2ram (flashmemory plugin) handles /var/log so there's no reason to fear anything that happens wrt log files. They are kept in RAM and are flushed to disk very rarely so you don't need to fear 'SD card wear out'.

2) rsyslog allows filtering out unwanted messages: http://www.rsyslog.com/discarding-unwanted-messages/ -- why so much hassles configuring here and there if you're after 'clean logs'? You can get this in a single location and this is the official way. Discard everything you consider useless.

3) make_nas_processes_faster executing only every hour makes this task more or less useless anyway. Each time you establish a new client SMB connection a new smbd process will be forked. This process should get a better IO scheduler priority class. If you execute this only once per hour then this client process might get accelerated maybe only after 59 minutes. But as already said: Raspberries are bottlenecked by their slow IO implementation anyway so you can simply disable the cron job if you fear log entries (or filter them out, see 2) above)