Building OMV automatically for a bunch of different ARM dev boards

Important: the final result of this full thread (highly optimized OMV images for ARM boards) can be found here: https://sourceforge.net/projec…s/Other%20armhf%20images/

Apologies for probably asking in the wrong place but I didn't find a developer sub forum here.

Purpose of this post is asking for support to get OMV built in a fully automated way for ~50 different single board computers. I know that for many SBC already OMV OS images exist but either hardware is not sufficient (eg. Raspberries with their slow Fast Ethernet and limited USB bandwidth) or the quality of the images is questionable (someone used the OS image he found somewhere on the net, installed OMV inside, didn't care about kernel/settings and pushed it out... so the result is neither fast nor trustworthy).

We (Armbian team) thought let's change this since we implemented a scripted build system to create Debian/Ubuntu based OS images fully automated from scratch currently supporting +50 devices: https://www.armbian.com/download/

Getting OMV built automagically wasn't that hard: adjusting kernel configs to meet OMV requirements as first step and adding a customization routine that gets called at the end of image creation and adding OMV unattended to freshly created OS images. Still WiP latest attempt looks like this (executed on a x64 build host in a chrooted ARM environment): https://github.com/igorpecovni…mage.sh.template#L31-L126

My problems currently with implementing this are

• I wonder how I can influence/prevent overwriting of /etc/defaults/cpufrequtils -- OMV's default settings are counterproductive on armhf and arm64 since most kernels we're dealing with here perform badly with 'conservative' cpufreq governor
• some of those small ARM boards with fast Gigabit Ethernet but only slow storage (USB2 or SDIO 2.0) would benefit a lot from compress=lzo as btrfs mount option (and of course noatime,nodiratime). I found ways to specify that globally but the problem here is that a lot of ARM devices are supported by more than one kernel flavour (eg. a smelly 3.4 kernel with multimedia features and latest and greatest mainline kernel). I would consider using btrfs these days with something lower than 4.4 dangerous, so the question is: are btrfs options accessible or can btrfs be 'hidden' when old kernels are used (I would assume no, so we have to fix this in the build system and check kernel variant at build time)

I'll stop now and wait for feedback since maybe this is the wrong place to discuss this anyway? Progress can be monitored here: https://forum.armbian.com/inde…ges-for-sbc-with-armbian/

Kudos for all your great work. Tried really hard to destroy my various OMV builds the last two days but to no avail. Even mean CLI modifications got absorbed by OMV and performance is also not bad on those small gems (though a little tuning on ARM boards could improve performance significantly)

Thanks for the prompt response and invitation to dev forum!

Wrt kernel configs: https://github.com/igorpecovni…a0a05c37393d379c104ba475a

When people on your forum report 'using Armbian' that doesn't tell much since Armbian is a build system debootstrapping somewhat optimized Debian or Ubuntu images. So in case they're using Ubuntu they run in troubles with OMV (due to PHP 7), if they use Jessie they're fine. The focus of this attempt is 'Jessie only' for exactly that reason: avoiding dependency hell.

That's also the reason why Armbian's desktop images switched from Jessie to Ubuntu Xenial since a lot of the required dependencies there were easier to met. So currently at least with all Allwinner boards it's a broken situation since users wanting Desktop stuff (HW accelerated video decoding and such stuff) get only the Ubuntu Armbian flavour while OMV will only run (flawlessly) with the Jessie variant. Once Stretch is released, OMV confirmed to work well on Stretch and Armbian also evaluated/resolved the Desktop dependencies this will be resolved eventually.

Will now play a bit around and become a bit more familiar with OMV so thanks for the pointers where to look for (will use your ODROID-C2 image for comparisons from now on). But it'll take some time until I get back since other work has higher priority in the meantime :\

Disclaimer: won't look into this for at least the next three weeks. But thought I provide something to test and a status update now. Yesterday I added some minor tweaks to the OMV installer routine to match what I found on the ODROID-C2 image more closely.

One remaining issue is resizing a 2nd partition on the SD card to the maximum (2nd since we're not using a separate boot partition) but this is by intention since I prefer sane solutions over hacks. Needs some patience and details can be followed here: https://github.com/igorpecovnik/lib/issues/650

You'll find here for the next few weeks an OMV image for ODROID-C2 automatically created from scratch yesterday: http://kaiser-edv.de/tmp/NumpU8/

Armbian's build system built u-boot and kernel, then debootstrapped Jessie freshly, applied the Armbian tweaks to the rootfs and added OMV as final step before closing the image. No one was ever logged into this image, it will finish installation on first boot and reboots 1 time automatically (needs 2 minutes maximum).

It would be great if people more experienced than me regarding OMV could give it a try and leave feedback here (as already said, I will check that later).

Areas of testing:

• general behaviour: what's wrong, what looks wrong, what's missing?
• nand-sata-install -- this is Armbian's default mechanism to move the rootfs away from SD card to NAND, eMMC, USB or SATA disks (successfully tested with OMV by me with SATA on a Clearfog Pro, USB on a Pine64 and eMMC on an Orange Pi Plus 2E). Idea behind is simple: burn the image on a slow/small SD card, boot it once, call nand-sata-install and then the only thing remaining on SD card is the bootloader while rootfs is on a fast connected disk or eMMC (please note that I've not the slightest idea about bootloader behaviour on all the ODROIDs -- the 'move to eMMC' feature is maybe an Allwinner board feature, NAND is Allwinner only for sure). On ODROID-C2 the best way to test this would be an USB attached disk manually partitioned with one 4GB partition and a 2nd using the remaining space.

In any case the output of 'sudo armbianmonitor -u' running in a shell is required if problems are to be reported (Armbian collects system info on every boot and this call tries to upload this together with latest dmesg output to an online pasteboard service) and a problem description an OMV noob like me is able to understand

@ryecoaaron

Just as a quick comparison your build after all updates/upgrades applied and the result of the fully automated Armbian build below:

root@openmediavault:~# uname -a
Linux openmediavault 3.14.29-51 #1 SMP PREEMPT Thu Mar 31 22:37:02 BRT 2016 aarch64 GNU/Linux
root@openmediavault:~# free
             total       used       free     shared    buffers     cached
Mem:       1760956     504668    1256288      28924      14744     333968
-/+ buffers/cache:     155956    1605000
Swap:            0          0          0

And this is the automated Armbian build after deactivating display to gain more available DRAM (as per https://github.com/igorpecovni…board_details/odroidc2.md )

root@odroidc2:~# uname -a
Linux odroidc2 3.14.79-odroidc2 #2 SMP PREEMPT Wed Apr 5 20:33:28 CEST 2017 aarch64 GNU/Linux
root@odroidc2:~# free
             total       used       free     shared    buffers     cached
Mem:       2022212     355848    1666364      25944      14288     200244
-/+ buffers/cache:     141316    1880896
Swap:       131068          0     131068

Kernel is more recent since Armbian maintains repo with packages built from up-to-date kernel sources to provide security fixes ASAP.

Almost forgot: SSH password is also 'openmediavault' with these builds. And this is how the output of 'armbianmonitor -u' looks like: http://sprunge.us/WTfV

Quote from ryecoaaron

I've always avoided installing OMV on nand because it could make the board useless when the nand reached its write lifetime. Granted, using the flashmemory plugin and disabling monitoring would help this.

Oh, I don't think this is an issue since Armbian uses a 10 minutes commit interval for writes to disk and /var/log gets written only every hour:

/dev/mmcblk0p1 / ext4 rw,noatime,nodiratime,errors=remount-ro,commit=600 0 0
/dev/mmcblk0p1 /var/log.hdd ext4 rw,noatime,nodiratime,errors=remount-ro,commit=600 0 0
log2ram /var/log tmpfs rw,nosuid,nodev,noexec,relatime,size=51200k,mode=755 0 0

Please see also http://tech.scargill.net/a-question-of-lifespan/

Quote from ryecoaaron

As you can tell by the sourceforge download numbers, it wasn't a high priority to keep the image up to date.

Hmm... well, Armbian team tries to provide security relevant fixes as fast as possible. Eg. 'Dirty COW' was fixed within 24 hours after it came to our attention, then unattended-upgrades are available so every user of a more recent Armbian image had the fix installed within the next 24 hours (apt upgrade includes kernel updates) and so only a reboot was needed to save our users.

But that's not important, just another by-product of generating an OMV image with Armbian's build system.

Looking forward to your feedback! Thanks!

Quote from ryecoaaron

Your image enables flashmemory in config.xml but that doesn't truly enable it. The service needs to be enabled

I thought naivly that would already be done by https://github.com/igorpecovni…ize-image.sh.template#L98? I see, the systemd unit file is missing. Will look into that later.

Thanks for the list of directories to take care of. I just used iotop to watch a bit what happens and obviously this setting for rrdcached needs some attention: '-j /var/lib/rrdcached/journal/' -- removing it greatly reduces 'Actual DISK WRITE' and I don't see an advantage configuring rrdcached to ensure writing a journal to disk (real storage) to redirect that to RAM later. Better remove -j at all when /var is in flash since it's useless anyway when redirected to a tmpfs?

Seems like a good idea to setup some basic block device layer monitoring when looking into this ('egrep "pgpgout|pgpgin" </proc/vmstat' in a loop or something like that to get frequency and amount of actual writes to disk).

And regarding kernel it seems that was a misunderstanding. Kernel is in an Armbian repo and will get updated automatically (so no need to download new images if a kernel update happens):

ii  linux-image-odroidc2           5.27                      arm64        Linux kernel, version 3.14.79-odroidc2

I was only surprised to find your OMV installallation still being at 3.14.29 after I applied all updates in OMV. But maybe I simply missed something.

Small update: 'nand-sata-install' needs NO testing currently but a fix we try to implement the next days/weeks: https://forum.armbian.com/inde…findComment&comment=29041

nand-sata-install should now work properly after being updated with latest version (has to be done now using wget/curl, with next major Armbian update in a few weeks it will also be available through 'apt upgrade'):https://forum.armbian.com/inde…all-script/#comment-29117

Will then look into folder2ram next (still needs some time since I wait for a NanoPi NEO Plus 2 dev sample to arrive and will then do performance tests/optimizations with NanoPi NEO 256MB, NEO Plus 2 512 MB and Clearfog Pro)

I let a new C2 image build with latest OMV fixes: https://github.com/igorpecovni…22b1893053a3e240f0aded585

Image(s) available at http://kaiser-edv.de/tmp/NumpU8/

Looks good to me (full log here: http://sprunge.us/ZMXS)

root@odroidc2:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p1  4.0G  1.5G  2.5G  37% /
udev             10M     0   10M   0% /dev
tmpfs           344M  6.2M  338M   2% /run
tmpfs           859M     0  859M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           859M     0  859M   0% /sys/fs/cgroup
tmpfs           859M     0  859M   0% /tmp
folder2ram      859M  1.7M  858M   1% /var/log
folder2ram      859M     0  859M   0% /var/tmp
folder2ram      859M  332K  859M   1% /var/lib/openmediavault/rrd
folder2ram      859M  760K  859M   1% /var/spool
folder2ram      859M   12M  848M   2% /var/lib/rrdcached
folder2ram      859M  8.0K  859M   1% /var/lib/monit
folder2ram      859M     0  859M   0% /var/lib/php5
tmpfs           172M     0  172M   0% /run/user/0


root@odroidc2:~# armbianmonitor -d mmcblk0p1
Sun Apr  9 14:37:58 CEST 2017      11/17 pages written after 5 sec
Sun Apr  9 14:37:59 CEST 2017       6/0 pages written after 1 sec

I added a monitoring variant to watch writes to block devices so if OMV has been transferred using nand-sata-install to an USB HDD for example one would call 'armbianmonitor -d sda1' instead to watch how frequent writes happen.

Next task is building an XU4 image with mainline kernel of course making use of UAS. I just need some confirmation about IRQ affinity (since I don't have an XU4 I can not test): http://forum.odroid.com/viewtopic.php?f=146&t=26016#p186035

Quote from ryecoaaron

I can test

Great, here you go: http://kaiser-edv.de/tmp/NumpU8/

Completely untested, creation log (skipping build log of both u-boot and kernel) here: https://pastebin.com/cs7vXpd5

Wrt UASP only a few USB-to-SATA bridges are capable, please see http://linux-sunxi.org/USB/UAS…ipsets_in_disk_enclosures for some.

Regarding performance: In case 'ondemand' should be used I would recommend setting the following also (adding to /etc/rc.local in this case):

echo ondemand >/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 
sleep 0.1 
echo 1 >/sys/devices/system/cpu/cpufreq/ondemand/io_is_busy 
echo 25 >/sys/devices/system/cpu/cpufreq/ondemand/up_threshold 
echo 10 >/sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor

Then IRQ distribution might not be ideal and needs some testing/tweaking.

And then I found it always helpful to allow afpd child processes (those that handle client sessions) higher prioritized IO handling. On my TimeMachine ARM boards crond runs this every minute (should work with smbd too):

for i in $(pgrep -P $(pgrep -u root afpd)) ; do ionice -c1 -p $i ; done

I just ordered an XU4 -- should be here within 2 days (but 'unfortunately' Xunlong also sent out a bunch of new Orange Pi dev samples that will arrive at the same time). After reading through http://magazine.odroid.com/wp-…agazine-201702.pdf#page=8 I understood that there are some issues with missing HMP and mainline kernel now that need some attention.

Final goal is to define CPU affinity of interrupt handling and daemons (if the A7 cores are fast enough move all IRQs to them and let the A15 cores do the 'real work'). This will take some time since I don't want to fool myself solely looking on misleading iperf3 or Windows Explorer numbers but try to optimize storage layer first (plenty of UASP capable disk enclosures around) and then look at combined network/storage performance using Helios LanTest.

Idea is then to check $BOARD=odroidxu4 in build script and add to the final OS image an 'optimize-omv-on-xu4-performance.service' unit like cpuset.service from https://github.com/mad-ady/odroid-xu4-optimizations

No idea yet whether it really needs cgroups, maybe a simple pgrep/taskset loop can do the job too. I think I'll start with assigning USB3 IRQs and /sys/class/net/eth0/queues/rx-0/rps_cpus to cpu1-3 and try to assign all relevant OMV daemons to cpu4-7. Then looking later into increasing /proc/sys/net/core/rps_sock_flow_entries and /sys/class/net/eth0/queues/rx-0/rps_flow_cnt and the usual net.core.*/net.ipv* tweaks going into /etc/sysctl.d -- but my goal is not to produce nicest looking benchmarks but good real-world performance (having also low latency in mind)

TL;DR: No testing needed at the moment, I'll be able to look into it soon myself

Testing is done. I compared performance on both XU4 kernels (which implies 'UASP vs. Mass Storage Protocol' too): https://forum.armbian.com/inde…findComment&comment=29304

And now there's something to test too: http://kaiser-edv.de/tmp/NumpU8/

Still unresolved is creating the 2nd partition on the installation media but everything else should be fine-tuned and ready for some (performance) tests.

BTW: When testing with an old 2.5" HDD I pulled out of a MacBook in an UASP capable enclosure (JMS567) I noticed syslog being flooded with xhci errors. So maybe UAS on XU4 is not ready for the masses now (but I didn't do intensive tests to verify that, maybe it was just a cabling problem)

In the meantime I started to upload a few more OS images based on OMV 3.0.70 and Armbian 5.27: kaiser-edv.de/tmp/NumpU8/

There are still some issues with some boards (the only with 'finished tweaking' is the one for XU4) but this needs some time and fixes will then later be available through 'apt upgrade' since part of 'board support package'.

Wrt Cloudshell2 unfortunately everything as expected: http://forum.odroid.com/viewtopic.php?f=146&t=26016#p186466

And just to document the fixes and for some background info: Armbian project started partially since vendor OS images performed so badly when SBC were used as servers/NAS. So part of the continual tweaking we did over the years was improving NAS performance (eg. identifying IRQ balancing as broken/bad, defining IRQ/CPU affinity, optimize cpufreq scaling settings and so on).

With the OMV tweaks that are now applied it's 'just' the following also:

• process names that match the pattern "ftpd|nfsiod|smbd|afpd|cnid" become better IO priority (and on XU4 they are sent to the big cores only)
• on XU4 the USB3-Ethernet IRQs are processed by cpu3 (little) only
• on XU4 /sys/class/net/eth0/queues/rx-0/rps_cpus is set to cpus 0-2 (little) only

I'm currently thinking about tweaking that a little more on the quad core boards but that needs testing (and fixes come as updates later anyway)

What's still missing is some tuning for smb.conf's global section. What is the appropriate way to tweak this on a 'per board' or 'per family' (ARM boards) basis?

Quote from ryecoaaron

There may be some general tweaks that help but I would think they would be same across most boards

I was talking about the stuff Hardkernel is now using testing with their Cloudshell2:

socket options = TCP_NODELAY IPTOS_LOWDELAY
min receivefile size = 16384
use sendfile = yes
write cache size = 524288
getwd cache = yes

Similar settings with some explanations can be found here for example:

socket options = IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY
min receivefile size = 2048
use sendfile = true
write cache size = 1024000
getwd cache = yes
aio read size = 2048
aio write size = 2048
read raw = yes
write raw = yes
oplocks = yes
max xmit = 32768
dead time = 15
large readwrite = yes

Is it possible with OMV to tweak smb.conf's global section without risking to loose the changes with an update? If so I would start a bit to test around (will take some time since I have no 'native' Windows around anywhere, just some virtualized servers)

My real Samba use case is preparing the transition from AFP (Netatalk) to SMB (Samba) but that requires a Samba version recent enough and some special settings in smb.conf's global section. I'll postpone discussing this until Stretch is released (Samba 4.5.8 looks sufficient)