Revive botched apt upgrade?

DHGE · 30. Januar 2018

Hello I just did an upgrade of my ODROID hc1:

Bash

root@hc1:/home/user#apt upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  armbian-config armbian-firmware armbian-tools-jessie hostapd linux-dtb-next-odroidxu4 linux-jessie-root-next-odroidxu4 linux-u-boot-odroidxu4-next sunxi-tools
8 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 3,306 kB of archives.
After this operation, 137 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://apt.armbian.com/ jessie/main armbian-config all 5.38 [30.0 kB]
...
Get:8 http://apt.armbian.com/ jessie/jessie-utils sunxi-tools armhf 1.4.2-2~armbian5.38+1 [36.2 kB]                                                                                                                      
Fetched 3,306 kB in 12s (273 kB/s)                                                                                                                                                                                       
(Reading database ... 48425 files and directories currently installed.)
Preparing to unpack .../armbian-config_5.38_all.deb ...
...
Processing triggers for initramfs-tools (0.120+deb8u3) ...
update-initramfs: Generating /boot/initrd.img-4.9.61-odroidxu4
cryptsetup: WARNING: failed to detect canonical device of /dev/mmcblk1p2
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
W: mdadm: no arrays defined in configuration file.
update-initramfs: Converting to u-boot format
Setting up armbian-config (5.38) ...
Setting up armbian-firmware (5.38) ...
Setting up armbian-tools-jessie (5.38) ...
Setting up hostapd (2:2.6-4~armbian5.38+1) ...
Setting up linux-dtb-next-odroidxu4 (5.38) ...
Setting up linux-jessie-root-next-odroidxu4 (5.38) ...
Setting up linux-u-boot-odroidxu4-next (5.38) ...
Updating u-boot on /dev/mmcblk1
Setting up sunxi-tools (1.4.2-2~armbian5.38+1) ...Shell-Script
Quellcode bearbeiten


    root@hc1:/home/user#apt upgrade
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Calculating upgrade... Done
    The following packages will be upgraded:
      armbian-config armbian-firmware armbian-tools-jessie hostapd linux-dtb-next-odroidxu4 linux-jessie-root-next-odroidxu4 linux-u-boot-odroidxu4-next sunxi-tools
    8 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Need to get 3,306 kB of archives.
    After this operation, 137 kB of additional disk space will be used.
    Do you want to continue? [Y/n] 
    Get:1 http://apt.armbian.com/ jessie/main armbian-config all 5.38 [30.0 kB]
    ...
    Get:8 http://apt.armbian.com/ jessie/jessie-utils sunxi-tools armhf 1.4.2-2~armbian5.38+1 [36.2 kB]                                                                                                                      
    Fetched 3,306 kB in 12s (273 kB/s)                                                                                                                                                                                       
    (Reading database ... 48425 files and directories currently installed.)
    Preparing to unpack .../armbian-config_5.38_all.deb ...
    ...
    Processing triggers for initramfs-tools (0.120+deb8u3) ...
    update-initramfs: Generating /boot/initrd.img-4.9.61-odroidxu4
    cryptsetup: WARNING: failed to detect canonical device of /dev/mmcblk1p2
    W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
    W: mdadm: no arrays defined in configuration file.
    update-initramfs: Converting to u-boot format
    Setting up armbian-config (5.38) ...
    Setting up armbian-firmware (5.38) ...
    Setting up armbian-tools-jessie (5.38) ...
    Setting up hostapd (2:2.6-4~armbian5.38+1) ...
    Setting up linux-dtb-next-odroidxu4 (5.38) ...
    Setting up linux-jessie-root-next-odroidxu4 (5.38) ...
    Setting up linux-u-boot-odroidxu4-next (5.38) ...
    Updating u-boot on /dev/mmcblk1
    Setting up sunxi-tools (1.4.2-2~armbian5.38+1) ...
    Processing triggers for initramfs-tools (0.120+deb8u3) ...
    update-initramfs: Generating /boot/initrd.img-4.9.61-odroidxu4
    cryptsetup: WARNING: failed to detect canonical device of /dev/mmcblk1p2
    W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
    W: mdadm: no arrays defined in configuration file.
    update-initramfs: Converting to u-boot format

    root@hc1:/home/user# ls /boot
    armbian_first_run.txt.template  boot-desktop.png  boot.ini.out             dtb                    dtb.old                      lost+found                   uInitrd                   vmlinuz-4.9.61-odroidxu4
    boot.bmp                        boot.ini          config-4.9.61-odroidxu4  dtb-4.14.15-odroidxu4  initrd.img-4.9.61-odroidxu4  System.map-4.9.61-odroidxu4  uInitrd-4.9.61-odroidxu4  zImage

    root@hc1:/home/user# reboot
Processing triggers for initramfs-tools (0.120+deb8u3) ...
update-initramfs: Generating /boot/initrd.img-4.9.61-odroidxu4
cryptsetup: WARNING: failed to detect canonical device of /dev/mmcblk1p2
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
W: mdadm: no arrays defined in configuration file.
update-initramfs: Converting to u-boot format


root@hc1:/home/user# ls /boot
armbian_first_run.txt.template  boot-desktop.png  boot.ini.out             dtb                    dtb.old                      lost+found                   uInitrd                   vmlinuz-4.9.61-odroidxu4
boot.bmp                        boot.ini          config-4.9.61-odroidxu4  dtb-4.14.15-odroidxu4  initrd.img-4.9.61-odroidxu4  System.map-4.9.61-odroidxu4  uInitrd-4.9.61-odroidxu4  zImage


root@hc1:/home/user# reboot

Alles anzeigen

After reboot I can not reach the machine any more via browser or ssh. The machine does not connect to my router any more.
LEDs look normal.
Power cycling did not help.

This happened to me once before when updating OMV itself. At that time I did a time consuming reinstallation via a new image loaded from Sourgeforge.
Note that kernel updates went smooth on this setup before and the cryptsetup error is to be expected since I have a LUKS-encrypted data storage (decrypted after boot via the web interface).

Is there anything I could do to avoid a new installation?

e.g. pulling the SD-card and fixing something via chroot

tkaiser · 30. Januar 2018

https://github.com/armbian/bui…93#commitcomment-27203859

You might want to open a thread in Armbian forum. I won't -- currently given up on the project since it moves in the wrong direction...

tkaiser · 30. Januar 2018

Zitat von DHGE

After reboot I can not reach the machine any more via browser or ssh. The machine does not connect to my router any more.

That's a similar (or most probably the same) symptom of the kernel not able to load an appropriate .dtb. Now with the USB node missing the internal RTL8153 Ethernet adapter is behind you end up with a board without network. Attaching any common USB Ethernet dongle to the USB2 port might help to recover from this problem and then most probably a quick fix would be to revert back to the 4.9 DT:

Code

apt install linux-dtb-next-odroidxu4=5.35

Maybe you need to purge the affected package first, maybe it's a better idea to downgrade all affected packages down to 5.35.

But please open up a thread over at Armbian forum. It simply sucks that after every single minor upgrade boards are bricked and this needs to change (but I'm just too tired to repeat the same things over and over again)

DHGE · 30. Januar 2018

Zitat von tkaiser

You might want to open a thread in Armbian forum.

Thanks for the update!

HHmmmm - when my time gets eaten then I will choose the way.

I think I will install Debian Stretch and then OMV4 and learn something on this endeavor.
https://wiki.debian.org/InstallingDebianOn/OdroidHC1

chroot won't be of much use on my Skylake notebook ...

tkaiser · 30. Januar 2018

Zitat von DHGE

I think I will install Debian Stretch and then OMV4 and learn something on this endeavor.

Well, the secret behind 'plain Debian' being more stable than Armbian on SBC is that they simply don't touch the stuff that can brick boards that often or at all (bootloader and kernel stuff) while Armbian for whatever obscure reasons permanently tries to update this without ensuring that proper testing happens.

It should also be noted that upstream Debian uses these kernel packages: https://packages.debian.org/se…eywords=linux-image-armmp (which might lack important stuff compared to Hardkernel's repos, eg. something that will break UAS on HC1 for sure and is long fixed in Armbian on every platform) and I also doubt that any of the various performance fixes are present there (SMP/IRQ affinity fixes, IO scheduler, appropriate kernel config and so on). Please don't be surprised if NAS performance will drop noticeable.

Making some noise over in Armbian forum to influence especially the 'project owner' seems like a better idea to me.

DHGE · 30. Januar 2018

Zitat von tkaiser

It should also be noted that upstream Debian ...

Making some noise over in Armbian forum to influence especially the 'project owner' seems like a better idea to me.

Thanks - point taken!
Had good results doing this for education purposes on a Raspi2.

Before I vent on Armbian forum's turf I will try your USB-net approach - no way since I only have wireless USB-adapters

tkaiser · 30. Januar 2018

Zitat von DHGE

I will try your USB-net approach - no way since I only have wireless USB-adapters

Maybe time to buy an RTL8153 just for the drawer (in DE eg. this one or with some patience on Aliexpress for half the price). Has been some sort of a lifesaver for me multiple times (an el cheapo USB2 dongle would do it as well but I like that RTL8153 is the most performant one around and driver support on almost any platform is fine since RealTek tried to get their drivers included in all major OS)

DHGE · 30. Januar 2018

I think I got caught between a armbian repository update 5.38 -> 5.39
~~When I did the upgrade not all packages where in the repository~~ or they botched the 5.38 version with the wrong dtb.

Manually (dissecting the deb-file) I installed linux-dtb-next-odroidxu4_5.39_armhf
After an apt upgrade (brave or?):

Bash

user@hc1:~$ ls -l /boot
total 18540
-rw-r--r-- 1 root root    1624 Nov 23 11:39 armbian_first_run.txt.template
-rw-r--r-- 1 root root   38518 Nov 23 11:39 boot.bmp
-rw-r--r-- 1 root root    4882 Nov 23 11:39 boot-desktop.png
-rw-r--r-- 1 root root   12405 Nov 23 11:58 boot.ini
-rw-r--r-- 1 root root       0 Dec  7 14:01 boot.ini.out
-rw-r--r-- 1 root root  141316 Nov 23 11:36 config-4.9.61-odroidxu4
lrwxrwxrwx 1 root root      20 Jan 30  2018 dtb -> dtb-4.9.61-odroidxu4
drwxr-xr-x 2 root root    3072 Jan 30 10:05 dtb-4.9.61-odroidxu4
-rw-r--r-- 1 root root 5155946 Jan 30 09:59 initrd.img-4.9.61-odroidxu4
drwx------ 2 root root   12288 Nov 23 11:58 lost+found
-rw-r--r-- 1 root root 2389804 Nov 23 11:36 System.map-4.9.61-odroidxu4
lrwxrwxrwx 1 root root      24 Jan 30 09:59 uInitrd -> uInitrd-4.9.61-odroidxu4
-rw-r--r-- 1 root root 5156010 Jan 30 09:59 uInitrd-4.9.61-odroidxu4
-rwxr-xr-x 1 root root 6063608 Nov 23 11:36 vmlinuz-4.9.61-odroidxu4
lrwxrwxrwx 1 root root      24 Nov 23 11:38 zImage -> vmlinuz-4.9.61-odroidxu4

Alles anzeigen

So all should be well again now for the other updaters with an ODROID hc1

tkaiser · 30. Januar 2018

Zitat von DHGE

I think I got caught between a armbian repository update 5.38 -> 5.39

Nope, this was just Igor throwing the old packages with higher version number into the repository again to prevent more XU4, HC1 and HC2 users ending up with boards without network after upgrade/reboot. As a reaction to your observations reported over at Github...

DHGE · 30. Januar 2018

Maybe Armbian is a one man show ATM ...

Jetzt mitmachen!