From dc7162e06b01b3f6d1243a9f28affd52fc68872a Mon Sep 17 00:00:00 2001 From: Otavio Salvador Date: Wed, 29 Jul 2026 18:10:12 -0300 Subject: [PATCH] updatehub-rollback-guard: Add guard for updates which never validate Rolling a bad update back relies on the bootloader counting boot attempts, and that counter only advances when the board actually reboots: in bootcount_env.c both bootcount_load and bootcount_store are no-ops while upgrade_available is 0, and a healthy image clears that flag through updatehub-active-validated. A boot which fails services but leaves the init system running is not a hang, so nothing reboots it. The init system stays up and keeps petting the hardware watchdog, the counter never reaches its limit, and the broken update sits in the active slot indefinitely -- unreachable, when the same failure also takes the network down. Add a package closing that gap: a timer started at boot which, once UPDATEHUB_VALIDATION_TIMEOUT has elapsed without the image having validated itself, reboots so the bootloader counts the attempt and the boot script can switch back to the previous slot. It deliberately does not write bootcount or updatehub_active, as incrementing is U-Boot's job and the slot decision belongs to the boot script; causing the reboot is enough. It honours BOOTCOUNT_ENV from /etc/default/updatehub-active, so a machine keeping the counter outside the U-Boot environment is left alone rather than policed against state the backend never wrote. The timer sets DefaultDependencies=no and is wanted by emergency.target and rescue.target as well as timers.target. Both are needed: the default dependencies would order it after sysinit.target, which a boot broken early enough never reaches, and those two targets isolate without pulling timers.target in. Missing either leaves the timer inert in precisely the boot it exists to rescue. Enable it from updatehub-runtime.bbclass alongside the active/inactive backend rather than requiring every machine to opt in. Two conditions gate it, both of which would otherwise leave the package installed but silently doing nothing: the u-boot backend, since the boot counter state is read from the U-Boot environment, and systemd in DISTRO_FEATURES, since the grace period is a timer. Expressing a grace period under sysvinit needs a different mechanism, so no attempt is made to enable it there and those images keep their previous behaviour. Tested on an Allwinner A40i board (kirkstone-based downstream tree). An update whose boot failed to mount the data partition dropped to the emergency shell and left the board unreachable for thirty minutes, recoverable only from the serial console. With the guard the same failure rebooted itself once the grace period expired, U-Boot counted the attempt and reverted to the previous slot, and the board was back on the network about five minutes after the failed boot with no console intervention. A healthy update validates within seconds and is left untouched. Also verified that a sysvinit DISTRO_FEATURES leaves the package out of UPDATEHUB_RUNTIME_PACKAGES and parses without errors. Backport of master commit 68daa34bfbcec978862594eca8265977c133d1b8 (PR #122). Signed-off-by: Otavio Salvador --- classes/updatehub-image.bbclass | 8 +++ classes/updatehub-runtime.bbclass | 6 ++ .../updatehub/updatehub-rollback-guard.bb | 55 +++++++++++++++++++ .../updatehub-rollback-guard | 49 +++++++++++++++++ .../updatehub-rollback-guard.service | 9 +++ .../updatehub-rollback-guard.timer | 16 ++++++ 6 files changed, 143 insertions(+) create mode 100644 recipes-core/updatehub/updatehub-rollback-guard.bb create mode 100644 recipes-core/updatehub/updatehub-rollback-guard/updatehub-rollback-guard create mode 100644 recipes-core/updatehub/updatehub-rollback-guard/updatehub-rollback-guard.service create mode 100644 recipes-core/updatehub/updatehub-rollback-guard/updatehub-rollback-guard.timer diff --git a/classes/updatehub-image.bbclass b/classes/updatehub-image.bbclass index 60f3421..3336247 100644 --- a/classes/updatehub-image.bbclass +++ b/classes/updatehub-image.bbclass @@ -71,6 +71,14 @@ # The active and inactive image schema requires a backend to identify and choose the image to be # used for next boot. It supports: 'u-boot', 'grub' or 'grub-efi'. # +# UPDATEHUB_VALIDATION_TIMEOUT +# +# How long a freshly installed image has to validate itself before updatehub-rollback-guard +# reboots it so the bootloader can roll back. Defaults to '5min', accepts any systemd time span. +# Enabled automatically with the 'u-boot' active/inactive backend when 'systemd' is in +# DISTRO_FEATURES; there is no sysvinit equivalent. Keep it comfortably above the time the agent +# needs to reach the validation callback on the slowest supported hardware. +# # UPDATEHUB_INSTALL_MODE # # There are multiple installation modes supported. This is usually machine dependent as it depends diff --git a/classes/updatehub-runtime.bbclass b/classes/updatehub-runtime.bbclass index f079dbf..4ea4746 100644 --- a/classes/updatehub-runtime.bbclass +++ b/classes/updatehub-runtime.bbclass @@ -167,6 +167,12 @@ python () { raise bb.parse.SkipRecipe("'%s' in UPDATEHUB_ACTIVE_INACTIVE_BACKEND is not a valid active/inactive backend. Valid active/inactive backends are: %s" % (active_inactive_backend, ' '.join(valid_active_inactive_backends))) elif active_inactive_backend: d.appendVar('UPDATEHUB_RUNTIME_PACKAGES', ' updatehub-active-inactive-backend-%s' % active_inactive_backend) + + # Guards against updates that never validate (see UPDATEHUB_VALIDATION_TIMEOUT). + # Requires the u-boot backend (reads its boot counter) and systemd (it's a timer). + if active_inactive_backend == 'u-boot' and \ + bb.utils.contains('DISTRO_FEATURES', 'systemd', True, False, d): + d.appendVar('UPDATEHUB_RUNTIME_PACKAGES', ' updatehub-rollback-guard') } def sanitise_version(ver): diff --git a/recipes-core/updatehub/updatehub-rollback-guard.bb b/recipes-core/updatehub/updatehub-rollback-guard.bb new file mode 100644 index 0000000..d345a60 --- /dev/null +++ b/recipes-core/updatehub/updatehub-rollback-guard.bb @@ -0,0 +1,55 @@ +# Copyright 2026 (C) O.S. Systems Software LTDA. + +SUMMARY = "Roll back an update which fails to validate itself" +DESCRIPTION = "Reboots the system when a freshly installed image does not \ +validate itself within UPDATEHUB_VALIDATION_TIMEOUT, so U-Boot can count the \ +boot attempt and roll back to the previously working image." +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" + +SRC_URI = " \ + file://${BPN} \ + file://${BPN}.service \ + file://${BPN}.timer \ +" + +S = "${WORKDIR}" + +UPDATEHUB_VALIDATION_TIMEOUT ?= "5min" + +# The timeout is baked into the installed files, so two machines configuring it +# differently must not share a package. +PACKAGE_ARCH = "${MACHINE_ARCH}" + +# Nothing is built; do not stage a cross toolchain for three text files. +INHIBIT_DEFAULT_DEPS = "1" + +# The grace period is a systemd timer. Fail loudly rather than install units +# which no init system will ever run. +REQUIRED_DISTRO_FEATURES = "systemd" + +inherit features_check systemd + +do_configure[noexec] = "1" +do_compile[noexec] = "1" + +# The timer is what gets enabled; it pulls in the service when it elapses. +SYSTEMD_SERVICE:${PN} = "${BPN}.timer" + +do_install() { + install -Dm 0755 ${WORKDIR}/${BPN} ${D}${bindir}/${BPN} + install -Dm 0644 ${WORKDIR}/${BPN}.service ${D}${systemd_system_unitdir}/${BPN}.service + install -Dm 0644 ${WORKDIR}/${BPN}.timer ${D}${systemd_system_unitdir}/${BPN}.timer + + sed -i -e 's,@VALIDATION_TIMEOUT@,${UPDATEHUB_VALIDATION_TIMEOUT},g' \ + -e 's,@BINDIR@,${bindir},g' \ + ${D}${bindir}/${BPN} \ + ${D}${systemd_system_unitdir}/${BPN}.service \ + ${D}${systemd_system_unitdir}/${BPN}.timer +} + +# systemd.bbclass doesn't follow the timer's Unit= to package the .service. +FILES:${PN} += "${systemd_system_unitdir}/${BPN}.service" + +# fw_printenv, to read the boot counter state from the U-Boot environment. +RDEPENDS:${PN} += "u-boot-fw-utils" diff --git a/recipes-core/updatehub/updatehub-rollback-guard/updatehub-rollback-guard b/recipes-core/updatehub/updatehub-rollback-guard/updatehub-rollback-guard new file mode 100644 index 0000000..0a5ea44 --- /dev/null +++ b/recipes-core/updatehub/updatehub-rollback-guard/updatehub-rollback-guard @@ -0,0 +1,49 @@ +#!/bin/sh +# -*- shell-script -*- +# +# Copyright 2026 (C) O.S. Systems Software LTDA. +# +# Force a reboot when a freshly installed image fails to validate itself. +# +# U-Boot only counts boot attempts (bootcount) while upgrade_available=1; a +# healthy boot clears that flag via updatehub-active-validated. A boot that +# fails services but leaves systemd running is not a hang -- systemd keeps +# petting the watchdog -- so bootcount never advances and a broken image can +# sit in the active slot forever. This script reboots once the timeout below +# elapses so U-Boot counts the attempt and can roll back to the other slot. +# +# No 'set -e': runs in an already-broken boot, so a failing diagnostic must +# never stop the script short of the reboot. + +TIMEOUT="@VALIDATION_TIMEOUT@" + +# BOOTCOUNT_ENV=0 means the counter isn't in the U-Boot environment: nothing to police. +BOOTCOUNT_ENV=1 +if [ -f /etc/default/updatehub-active ]; then + . /etc/default/updatehub-active +fi +[ "$BOOTCOUNT_ENV" = "1" ] || exit 0 + +# Log to kmsg too: a boot broken enough to need this often has no working journal. +log() { + logger -t updatehub "updatehub-rollback-guard: $*" 2>/dev/null + echo "updatehub-rollback-guard: $*" > /dev/kmsg 2>/dev/null +} + +# Single call: a second fw_printenv re-reads storage, which may be what's broken. +{ read -r upgrade_available; read -r bootcount; } </dev/null) +EOF + +# Not a probationary boot. +if [ "$upgrade_available" != "1" ]; then + exit 0 +fi + +log "image did not validate itself within $TIMEOUT (bootcount=${bootcount:-?}); rebooting so U-Boot counts the attempt and can roll back" + +# Not touching bootcount/updatehub_active: that's U-Boot's and the boot script's job. +# sysrq is the real fallback, for when PID 1 can't be reached. +systemctl --no-block reboot || echo b > /proc/sysrq-trigger + +exit 0 diff --git a/recipes-core/updatehub/updatehub-rollback-guard/updatehub-rollback-guard.service b/recipes-core/updatehub/updatehub-rollback-guard/updatehub-rollback-guard.service new file mode 100644 index 0000000..d247522 --- /dev/null +++ b/recipes-core/updatehub/updatehub-rollback-guard/updatehub-rollback-guard.service @@ -0,0 +1,9 @@ +[Unit] +Description=Roll back an update that failed to validate itself +# No deps: must run even when the data partition/agent/targets pulling them in have failed. +DefaultDependencies=no +IgnoreOnIsolate=yes + +[Service] +Type=oneshot +ExecStart=@BINDIR@/updatehub-rollback-guard diff --git a/recipes-core/updatehub/updatehub-rollback-guard/updatehub-rollback-guard.timer b/recipes-core/updatehub/updatehub-rollback-guard/updatehub-rollback-guard.timer new file mode 100644 index 0000000..d8af180 --- /dev/null +++ b/recipes-core/updatehub/updatehub-rollback-guard/updatehub-rollback-guard.timer @@ -0,0 +1,16 @@ +[Unit] +Description=Grace period before rolling back an unvalidated update +# Default deps order this after sysinit.target, which a broken-enough boot never reaches. +DefaultDependencies=no +IgnoreOnIsolate=yes +Conflicts=shutdown.target +Before=shutdown.target + +[Timer] +# From boot, not from timer start, so a slow/partial boot can't extend the grace period. +OnBootSec=@VALIDATION_TIMEOUT@ +Unit=updatehub-rollback-guard.service + +[Install] +# emergency/rescue.target isolate and don't pull in timers.target, so list them explicitly. +WantedBy=timers.target emergency.target rescue.target