Which energy efficient ARM platform to choose?

  • Btrfs doesn't do snapshots on its own. I would believe you're talking about OpenSuse's snapper instead?
    ...



    And experiences from 6 years ago are pretty much worthless (the work on btrfs started a decade ago!).

    I agree, my reservations for Btrfs are mostly based an FUD. Not the best way to make such decisions, but I can’t dismiss them straightaway either. This is part of the reason why I joined the conversation here and was hoping to get some more insight/arguments for or against certain filesystems and on which hardware they can be used.


    Making a case for ZFS is pretty straight forward due to the overwhelming amount of articles that appraise it as the best filesystem around (see e.g. this or this article).


    And yes, you are correct, the issues I had were related to OpenSuse's snapper utility, if I remember correctly, I wasn't really running out of disk space, it was more related to the meta data and the system thus reporting that I was running out of disk space and thus cannot mount the partition. In the meantime, I switched my main production systems from openSUSE to CentOS/Fedora and back to old-school file systems. I started reading up on the btrbk utility, which seems to be a rather good approach towards that problem.

  • I agree, my reservations for Btrfs are mostly based an FUD. Not the best way to make such decisions, but I can’t dismiss them straightaway either

    We're all affected by this and I would guess the older the more ;)


    douglas-adams.jpg


    If things go wrong it's kind of a normal reaction to blame stuff that is new to you or unknown or not fully understood yet. Then instead of a failing/weak PSU dropping disks out of ZFS pools OMV's proxmox kernel will be blamed, people take reports of btrfs filesystems failing on ReadyNAS boxes (just the symptom of data corruption at the mdraid/lvm layer below) as a proof that btrfs' own raid implementation is not ready (while it's not even used here in reality), the same OMV users happily living with mdraid's write hole putting their data at risk with a degraded RAID5/6 whine they can not use btrfs' raid56 mode since 'write hole' and so on...


    And what's also a real problem wrt awareness is users loving to shoot the messenger. In situations with faulty hardware traditional filesystems often do not report any error and you get that's something wrong only by accident: if unreliable hardware not just silently corrupted data but destroyed filesystem structures or them realizing that a lot of data is corrupted/lost only when it's way too late and also the corruption already spread into all backups (if existent).


    With those modern 'checksumming' filesystems like btrfs and ZFS you're notified almost immediately that something's wrong. But users then blame the filesystem instead of realizing that there's something wrong with their storage hardware below. Users seem to be happy suffering from silent data corruption but hate it to realize that their hardware sucks.


    ZFS and btrfs sit in exactly the same boat here but since ZFS is more used on server grade hardware (relying on hardware components that do not suck™) while btrfs is used also on commodity hardware reports of bizarre btrfs failures are quite normal and happen more often. In the end FreeNAS guys when inventing and spreading the myth that ZFS would need at least 8GB ECC DRAM did something good since users who feel bound to this requirement buy more likely hardware that does not suck™.

  • if you do snapshots then choose a tool that automates everything for you and deletes unneeded snapshots based on a retention policy

    Just want to mention, that snapper is doing this. You can configure snapper to create snapshots on a time basis (every hour or what ever). Additionally it will create a snapshot before and after any apt something. Cleanup is also done based on configurable criteria. If you do not want to have time based snapshots you can disable.

  • Just want to mention, that snapper is doing this

    Yep. But snapper only does snapshots and as such only prevents from logical failures (like accidentally deleting/overwriting data or when used on the rootfs after failing software upgrades allowing to revert to last known working state -- this was the problem @dkxls ran into years ago on OpenSuse and the challenges are outlined here).


    If you want to use snapshots also for 'real' backup protecting from physical failures then with those modern filesystems other tools like btrbk or znapzend are way more interesting since those do all the necessary snapshot handling embedded in an automated way to transfer data to another disk or even better another location (e.g. a different SBC in another room, building or town).


    Physical separation of the backups is crucial, see lesson 3 here: http://www.taobackup.com/

  • I also have to mention that I recently did a rollback with snapper on the HC2. It worked in the sense that the system was up and running as expected, but snapper was not configured any more. Probably because arm devices are not using grub which seems to be needed for a propper rollback.


    I will have a look at btrbk. Thanks

  • I also have to mention that I recently did a rollback with snapper on the HC2

    Yep, on the OMV images for ARM (except RPi and two or three other boards where I had bootloader troubles with btrfs) rollbacks are possible since we ship with btrfs for the rootfs now for a long time.


    But I'm almost always talking about btrfs for data shares and here it really gets interesting with OMV5 in the future since not 'everything outdated as hell' like today (Debian 10 relies on kernel 4.19 on x86 and includes btrfs-progs 4.20.1-2 and btrbk 0.27). One of the 'btrfs problems' is that filesystem support mostly depends on kernel version which is also the reason I remained relatively silent wrt btrfs in the past since an awful lot of OMV users love to skip updates, and a stock OMV3 installation on x86 might still run with a kernel 3.16 and btrfs-tools 3.17 which is a bit risky with some advanced btrfs features.

  • Alright, having considered my options for a small, energy efficient NAS, I believe the Helios4 is the best bet at the moment. I listed my requirements/wish-list in a previous post, and ended up with essentially these three options to consider:

    • Helios4

      • Marvell ARMADA 388
      • ARMv7 32-bit (Dual-core ARM Cortex-A9)
      • 2 GB ECC RAM
      • 4x SATA-III
    • MACCHIATObin

      • Marvell ARMADA 8040
      • ARMv8-A 64-bit (Quad core Cortex-A72)
      • 4-16 GB (DDR4 DIMM slot with optional ECC)
      • 3x SATA-III
    • A Rockchip RK3399 based SBC (e.g. the ROCKPro64 with a PCIe SATA controller or the NanoPi M4 with the SATA HAT)

      • Rockchip RK3399
      • ARMv8-A 64-bit (Dual-core ARM Cortex-A72 + Quad-core ARM Cortex-A53)
      • 2 or 4 GB non-ECC RAM
      • Up to 4x SATA-III (depending on the choice of the SBC and controller)


    The reason for going eventually with the Helios4 is that it's the only board that comes with 2GB ECC memory and an integrated SATA-III controller right away, is reasonably priced, and the board is actually designed to be used in NAS systems. The MACCHIATObin is also attractive due to the powerful 64-bit CPU and large amount of memory, but is significantly more expensive and it's unclear if it can be purchased with ECC memory right away (buying ECC separately and replacing the non-ECC one would make it obviously even more expensive). Also, it seems the MACCHIATObin targets a slightly different user group as it actually has additional 10 GBit and 2.5 GBit network controllers. I ruled out the Rockchip RK3399 based boards due to the non-ECC memory, based on the discussion above.


    Lucky that the 3rd Helios4 campaign is still on for another day. ^^


    Thanks @tkaiser for the fruitful discussion. Kudos to the guys at kobol.io ( @gprovost ) for putting the Helios4 together.

  • I am trying to research and take in all this information, yet I feel just as lost as when I started. I think I may be looking too much into it and getting caught up in particulars that don't matter for my needs.


    I want to ensure personal documents and photos are kept safe. This seems to lead to a minimum of a two disk system with btrfs (and I already back up to backblaze). Movies & TV are somewhat curated, so would only need a couple of terabytes of storage for the foreseeable future. Radarr/Sonarr to be used, they will then be streamed to a Shield.


    One thing the seems to be constant in these threads is to avoid USB where possible, or at least USB with the A type connector? An idle power consumption of under ~10w would be preferable.


    As @dkxls informative summary of @tkaiser excellent posts shows - the Helios4 seems to fit most of the criteria - though it just seems a little bit overkill and expensive for what I really need? The HC2 would be great with the exception of wanting to use btrfs to backup to a second disk and the fact that it uses a USB host.


    As far as I understand it - no such device currently fits... How bad is it, really, to have a HC2 with USB for storage? Or is the Nano pi m4 with SATA Hat a better option than the HC2?


    Apologies for the confusing text, but I am just as confused about what I need.


    TL;DR:
    -Low power consumption
    -Fairly low storage capacity required (~3-4 TB)
    -Reliable backup of personal documents, media not required but would be 'nice'

  • 2x HC2

  • If you want full bitrot protection on one HC2, with automatic bitrot correction, you could use a single big HDD and use half as redundant duplicate storage for btrfs. Btrfs can then use this duplicate data to automatically and transparently fix any bitrot errors. And then, as suggested above, use another HC2 in the same way for backups in several versions.


    I haven't tried this setup, but it should be possible to configure using btrfs and it should work fine and give extremely secure storage for your data. Especially if you have the two HC2 in different parts of the house or even, if you have fast internet, at totally different sites.


    If you do setup something like this, please document it and post it here! Would be very interesting for some minor part of my own data. Not a lot, just the stuff I care especially for. Scanned family photos and so on. Less than a TB. I might create shared folders with btrfs duplicate storage, on a separate partition on my IronWolves, just for extra secure automatically bitrot protected storage, on two HC2s.

    Be smart - be lazy. Clone your rootfs. This help is Grateful™.
    OMV 4: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4

  • you could use a single big HDD and use half as redundant duplicate storage for btrfs. ... use another HC2 in the same way for backups in several versions

    IMO this doesn't make that much sense. While you theoretically can make a btrfs raid1 out of 2 partitions on the same HDD what if this HDD dies? Also for 1 TB of usable data you need 4TB on your HDDs.


    I would better use btrfs on both OMV instances (full disks without partitioning) and then let btrbk create snapshots on both and send them efficiently via btrfs send/receive to the 2nd OMV instance. Btrfs by default will duplicate metadata on spinning rust (so there's always two copies of filesystem metadata) and you can run periodically btrfs scrub start on both hosts. Once there's data corruption on either OMV host you need to copy data over from the other instance.


    BTW: what do you think of https://forum.khadas.com/t/vim…-nas-server-variant/4287/

  • IOnce there's data corruption on either OMV host you need to copy data over from the other instance.

    I don't like the manual task of copying over data. The combined automated bitrot detection AND silent/automatic correction is what makes btrfs interesting for me.


    Without that I might just as well use ext4, syncs, hardlink and checksums. I am already testing/writing software to do the ext4/sync/hardlink/checksum thing, but that only runs like a backup job. This is for a slowly changing (growing) large long-lived file repository. Typical home nas usage.


    Sure 4x storage is a waste. But I don't have a lot of files that are really important. No matter what I do I need a backup. So really it is only 2x extra storage for automatic bitrot protection. The cost of 3TB to keep my most important 1TB safe and sound could be OK. 36TB to keep 12TB various stuff safe and sound, not so much...


    What would be nice is some extra software on top of btrfs that automatically fixes bitrot from a remote backup.

    Be smart - be lazy. Clone your rootfs. This help is Grateful™.
    OMV 4: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4

  • A two SATA SBC NAS with a slot for nvme ssd would be great. But for me to buy one it would also need more RAM than 4GB and really good software support. Otherwise I might just as well keep using my HC2s like now.


    16-32 GB RAM, or even more, would be nice for running plenty of dockers and having plenty of disk caches as well. Actually, with enough extra RAM I would skip the nvme ssd. Perhaps slots for extra RAM?

    Be smart - be lazy. Clone your rootfs. This help is Grateful™.
    OMV 4: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4

  • Perhaps slots for extra RAM?

    Affordable general purpose ARM SoCs lack

    • ability to address more than a few GB RAM (it's either 3GB or 4GB with the currently available SoCs)
    • DIMM support
    • (ECC memory support)

    Also these features make everything a lot more expensive and as such you can then also choose an x86 design like ODROID H2 for example. But something like this (with 32GB RAM and the additional acrylic enclosure for 2 x 3.5" HDD) is three times more expensive than what I'm interested in. It needs to be as inexpensive as a HC2 but without the little USB hassles and with the ability to add 1-3 additional 3.5" disks to the 'head unit'.

  • Thanks for the replies. This is kind of where it all just comes unstuck for me. Again, apologies for lacking the appropriate knowledge.


    If you want full bitrot protection on one HC2, with automatic bitrot correction, you could use a single big HDD and use half as redundant duplicate storage for btrfs. Btrfs can then use this duplicate data to automatically and transparently fix any bitrot errors. And then, as suggested above, use another HC2 in the same way for backups in several versions.

    This does sound like a good idea with the exception of tkaiser's point: what if the disk fails? But I love the simplicity of the idea and the automatic bitrot detection/fixing. I would diffidently consider this option,

    I would better use btrfs on both OMV instances (full disks without partitioning) and then let btrbk create snapshots on both and send them efficiently via btrfs send/receive to the 2nd OMV instance.

    Is this possible to do with two HC2s, or does your suggestion presume a single HC2 with two USB disks (based on your previous posts regarding USB I think i know that answer...)? Do they need to be physically connected together, or would it be possible to have them in different locations?



    Thanks again.


  • Is this possible to do with two HC2s, or does your suggestion presume a single HC2 with two USB disks (based on your previous posts regarding USB I think i know that answer...)? Do they need to be physically connected together, or would it be possible to have them in different locations?

    This is possible with two HC2, and them being in different locations is prerequisite for this being called some sort of backup. You should run automated scrubs anyway if you're a btrfs user and as such problems will be written to syslog (in case of files with checksum errors they will be written with their full path to syslog).


    And you can always check output of btrfs device stats $fs to get an idea if something is wrong. If sometimes in the future there will be data corruption you'll then copy over the data from the other HC2.

  • This does sound like a good idea with the exception of tkaiser's point: what if the disk fails?

    Then you can still access your files from the backup server, while you revive or replace the failed drive. You can even reconfigure the backup HC2 to fully replace the failed server, since you have full hardware redundancy.


    No difference from a drive failure on a single drive server without redundancy. If it is really important data you most likely have secondary offline backups as well.


    But with two separate disks, with btrfs' version of RAID1, you would indeed get better protection from a single disk failure. But that is not possible with HC2s, unfortunately.


    And if one of the two disks failed, you still would have to get a new disk, only not quite as urgently. And with two disks, instead of one, you are about twice as likely to experience a disk failure. And two drives use more power, and generate more heat, than one.

    Be smart - be lazy. Clone your rootfs. This help is Grateful™.
    OMV 4: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4

  • If you want full bitrot protection on one HC2, with automatic bitrot correction, you could use a single big HDD and use half as redundant duplicate storage for btrfs.

    Quick test with mkfs.btrfs -f -d dup -m dup (the 'dup' profile should be suitable to get the duplicated chunks be written to locations far far away from each other -- at least that's the design goal behind 'dup' which is default for metadata to minimize harm if a disk starts to die).


    Olimex Lime A10 (native SATA):

    Code
    mkfs.btrfs -f -d single -m dup /dev/sda1 random random
    kB reclen write rewrite read reread read write
    1024000 1024 54720 59136 61272 46079 32372 56500
    1024000 16384 78330 80932 58478 80669 77251 79362
    mkfs.btrfs -f -d dup -m dup /dev/sda1 random random
    kB reclen write rewrite read reread read write
    1024000 1024 22234 26812 59756 58773 27719 26923
    1024000 16384 26651 23958 78806 79219 75714 23699

    As expected a massive decrease in write performance due to constant head movements. Now testing with USB3-to-SATA bridge (where benchmarking btrfs makes no sense since for whatever reasons the -I flag in iozone -e -I -a -s 1000M -r 1024k -r 16384k -i 0 -i 1 -i 2gets ignored. The disk is an older 2.5" notebook HDD not able to provide more than 90 MB/s sequentially.


    RockPro64 (USB3 via ASM1153):


    Code
    mkfs.btrfs -f -d single -m dup /dev/sda1 random random
    kB reclen write rewrite read reread read write
    1024000 1024 215514 240554 192410 192691 190798 231142
    1024000 16384 401241 366650 381807 375545 373801 390407
    mkfs.btrfs -f -d dup -m dup /dev/sda1 random random
    kB reclen write rewrite read reread read write
    1024000 1024 167942 148305 209686 209880 209528 159986
    1024000 16384 177979 185367 396280 381482 381011 180694

    If you're going this route I would love to see some 'write barrier' related tests with this scenario on an USB3 SBC like ODROID HC2. While running intensive write loads (iozone -e -I -a -s 1000M -r 4k -i 0 -i 1) repeatedly cutting power.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!