Headless, automated backup of the system SD-card on RPi

  • Hi,
    I would like to regularly, automatically backup my system SD-card on OMV 4.1.15 on RPi2. Clonezilla does not exist for ARM, omv-backup is a pain to recover in case of error, so I wrote my own. I'm planning to use the following commands in a weekly cron job. Will this do the job?


  • omv-backup is a pain to recover in case of error

    Why? If you use the dd option, you would only have to dd two files back to the sd card.

    omv 5.6.13 usul | 64 bit | 5.11 proxmox kernel | omvextrasorg 5.6.2 | kvm plugin 5.1.6
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • In case of system failure I would not have a system to dd with, only my Windows Laptop with Etcher.

    Your windows laptop can probably boot Linux.

    Can you elaborate a bit on what I'm supposed to to to use omv-backup?

    Write the mbr image that the plugin creates to the sd card to create the boot sector and partition table and then write the os image to the partition.

    omv 5.6.13 usul | 64 bit | 5.11 proxmox kernel | omvextrasorg 5.6.2 | kvm plugin 5.1.6
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • omv-backup creates the following files:

    Code
    -rw-rw----+ 1 root users 751 Dec 14 16:21 backup-omv-14-Dec-2018_16-21-15.blkid
    -rw-rw----+ 1 root users 2865857738 Dec 14 17:12 backup-omv-14-Dec-2018_16-21-15.dd.gz
    -rw-rw----+ 1 root users 443 Dec 14 16:21 backup-omv-14-Dec-2018_16-21-15.fdisk
    -rw-rw----+ 1 root users 446 Dec 14 16:21 backup-omv-14-Dec-2018_16-21-15.grub
    -rw-rw----+ 1 root users 512 Dec 14 16:21 backup-omv-14-Dec-2018_16-21-15.grubparts
    -rw-rw----+ 1 root users 1563 Dec 14 16:21 backup-omv-14-Dec-2018_16-21-15.packages

    What do I have to do to restore them back to an SD card?

  • Only with the amd64

    Yeah, there it will work by following the instructions and installing/updating GRUB will do the trick. But by looking at the situation with SBC and each and every ARM SoC family implementing a very own boot procedure I doubt the omv-backup plugin provides the prerequisite for a successful restore.


    We need to differentiate between RPi (basing not on an ARM but a VideoCore SoC) and all the other SBC where SPL (secondary program loader) and u-boot need to be written somewhere at the beginning of the boot media (not being part of any partition or filesystem structure)

  • Code
    dd if=/dev/mmcblk0 of=imgfile.img bs=1M

    The problem with this attempt (or any use of dd or other tools running below the filesystem layer) is that the created image will most probably contain a lot of junk due to the way filesystems implement deletion: only marking contents as being deleted now while still containing all previous data.


    Therefore I would at least once execute the following in your running OMV install:


    Code
    dd if=/dev/zero of=/zeroes bs=1M || (sync ; rm /zeroes)

    This will zero out all unused space and a resulting compressed image will be way smaller afterwards. You should see this with omv-backup as well. To get a compressed image ready for Etcher I simply would run pxz -9 on the created image. This will take ages on a slow RPi so you might compare with default settings (-3 or simply suppressing the compression level) since most probably only a little bit of further compression is achievable by using anything above -3.

  • Thanks for the feedback. So I guess, I'm not all wrong doing the backup my way :-)
    Good advice with zeroing the unused space - although that is taken care of by pishrink.sh afterwards if I'm not mistaken.
    Compression on the fly is out of the question with the slow RPi anyways - this will increase the backup time by a couple of hours (depending on the size of the SD card) and max out CPU load during that time.

  • So I guess, I'm not all wrong doing the backup my way

    Well, if I understood the restore procedure from @ryecoaaron correctly then the plugin creates an dd image only of the rootfs partition. If so this wouldn't be enough for the RPi since the main operating system (ThreadX) lives on the FAT32 boot partition. And without ThreadX no Linux/OMV will boot.

  • correctly then the plugin creates an dd image only of the rootfs partition. If so this wouldn't be enough for the RPi since the main operating system (ThreadX) lives on the FAT32 boot partition. And without ThreadX no Linux/OMV will boot.

    Correct, it doesn't backup everything on installs that have more than one "OS" partition. This was done because too many people have a data partition on the OS drive. But you probably could write the RPi install image to an SD card and then restore the OS partition (haven't tried).

    omv 5.6.13 usul | 64 bit | 5.11 proxmox kernel | omvextrasorg 5.6.2 | kvm plugin 5.1.6
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

    Edited once, last by ryecoaaron ().

  • But you probably could write the RPi install image to an SD card and then restore the OS partition (haven't tried).

    While this should work there is still a problem since the ThreadX version on the installation image is outdated in the meantime and won't get magically updated after an OS restore (since it's part of raspberry-firmware package, apt/dpkg stuff lives on the OS partition and almost always RPi Trading guys update both ThreadX and their kernel packages at the same time due to some -- mostly undocumented -- dependencies).


    Same problem with all the ARM based installations since u-boot/SPL on the installation images might be outdated and Armbian/ayufan images package u-boot in a similar way as on the RPi.


    BTW: I'm fine with this procedure (since I don't believe in 'live cloning' below the filesystem due to my job, and am always doing offline clones of the SD cards) but users should be aware that the omv-backup plugin might not work as expected with all the SBC.

  • Well, if I understood the restore procedure from @ryecoaaron correctly then the plugin creates an dd image only of the rootfs partition. If so this wouldn't be enough for the RPi since the main operating system (ThreadX) lives on the FAT32 boot partition. And without ThreadX no Linux/OMV will boot.


    In omv-backup settings, it will let me configure a root device. If I set this to /dev/mmcblk0, omv-backup will still only dd the contents of /dev/mmcblk0p2 (the rootfs). Is this a bug or wanted behavior?


    screenshot.png

  • omv-backup will still only dd the contents of /dev/mmcblk0p2 (the rootfs). Is this a bug or wanted behavior?

    If I understood @ryecoaaron correctly then intended behavior due to potential existence of data partitions on the device.


    Desired behavior on the various SBC would be also backup of partition table, 1st partition (usually /boot) and if it's not Raspberry Pi then also all data between start of device and partition table since u-boot/SPL sits there. This way combined with an adapted pishrink script really small device backups that are able to be processed with Etcher seem doable.


    On the other hand it's live cloning below the filesystem layer so prone to transferring an inconsistent state (not that likely on the RPi since here the default ext4 commit of 5 seconds is used on our image for compatibility reasons with RPi firmware/kernel updates that might try to tinker with /etc/fstab. On all the other SBC with a 10 minute commit interval chances of cloning the OS partition inconsistently are much higher.

  • Well, almost 10 open Samba tdb databases. If they're open contents might be inconsistent. Usually tdbbackup is needed to get them in a consistent state but maybe Samba on its own runs this regularly (tdpbackup can also be used to verify integrity of these databases at startup and then grab the latest backed up version in case of broken databases): https://wiki.samba.org/index.p…g_the_samba_backup_script


    Anyway, since it's known that without special precautions like snapshots and applications being aware of snapshots happening (see VSS for example) live cloning (especially below the filesystem layer) can always result in a mess I stay with my offline cloning attempts :)

  • It does not have to be offline - only the services with write access to important files can be stopped upfront.
    Or even better: Have a special systemctl target that takes care of that and go back to graphical.target after dd is finished.
    To the files listed above:
    - /var/cache/minidlna/files.db is in a ram folder anyways
    - fail2ban.sqlite3 could be imho placed in a ram folder as well


    - as for the smb databases, I'm not sure if they can be placed in a ram folder, bc I don't know if they need to persist across boots (probably not)


    - the dhcp-lease should be in a ram folder as well


    So from my point of view, no issue with live cloning here :-)

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!