From 8b7f6da7416bd6ef600f2229d8cc383e7c02c9c9 Mon Sep 17 00:00:00 2001 From: Igor Pecovnik Date: Sun, 20 Sep 2026 06:13:43 +0200 Subject: [PATCH] download-external: mirror only the newest version of each package The aptly path downloads every version an upstream index still advertises, then deletes all but the newest. packages.microsoft.com keeps ~350 builds of "code" that pass GLOB: Packages filtered: 355 -> 53. Download queue: 53 items (7.92 GiB) ... [/root/.aptly/public/pool/main/c/code] kept: code_1.138.0-1789458676_arm64.deb total size is 219,966,878 7.92 GiB fetched, 210 MiB kept, per release and per arch -- and "code" alone runs 18 times a night (6 releases x 3 arches) against a single upstream "stable" suite. The step already knows which version it wants; it just works it out after paying for the rest. Ask the index first. pin_newest() reads the same Packages file aptly is about to read, takes the newest version of each package name with dpkg --compare-versions, and pins the mirror filter to those exact versions, ANDed with the existing GLOB. One extra request, no new trust: it is the same index, fetched over the same transport. It engages only where it pays. Sources whose index already holds one version per name -- every rpi-*, zfs-*, google-chrome, gh, min, fastfetch, the Armbian repo itself -- fail the "entries > names" check and take exactly today's path. A source with more than 500 distinct names is left alone too, rather than building a filter with thousands of clauses; that is what keeps apt.armbian.com (2267 names) out of it. Failure handling ---------------- Three things can go wrong, and each now has one answer: skip the slot. A skipped source is one stale nightly; the alternatives are a wrong repository or a red matrix. - The pin matches nothing. A filter that matches nothing used to publish an empty repository in silence -- the one way this change could go wrong, via a GLOB whose upper bound excludes the newest version of everything. The mirror's package count is now read after the update and warn_skip fires if the pin emptied it. An unreadable count changes nothing. (An earlier draft redid the mirror unpinned instead; that cost three times the characters, and skipping is the safer of the two.) - The index is unreadable. pin_newest had three failure paths returning 1 and the caller could not tell them apart, so all three fell through to an unpinned mirror. For "no duplicate versions" that is correct and harmless. For a transient 404 on the Packages file it is not: the pin silently turns off and the mirror fetches the whole back catalogue. Once the $Version floors come out of os/external/*.conf -- which this change is what makes possible -- that means "code" mirroring all 358 of its versions, roughly 52 GiB per release per arch against 7.9 GiB with the floor. So that path gets its own status and skips. The 500-name path still falls through: it is bounded by construction, and mozillateam, the largest source, has 289. - aptly cannot create the mirror. mk_mirror was guarded at the first call site but not at the retry that recreates after a failed update, and a command in an if-branch body is subject to errexit, so a failed recreate killed the slot instead of reaching warn_skip. The guard now lives inside mk_mirror and covers both call sites. Fitting it under the expression cap ----------------------------------- The first version of this failed to parse: (Line: 626, Col: 14): Exceeded max expression length 21000 GitHub caps a run: block at 21000 characters once expressions are counted. That step was already at 19614 on main -- 1386 of headroom -- and the newest-version pin took it to 25504. Most of the space came back from the mirror-create call, which was written out four times in two shapes (with and without components). $COMPONENTS unquoted expands to nothing when empty, so one mk_mirror() covers both. The rest came from comments and from eight debug echoes that restate a value printed again nearby -- FILTER_ARGS printed a value the pin then overwrites, "Original KEY" duplicates the "Config loaded" line, and "Checking if mirror exists..." has both of its outcomes logged one line later. ORIG_FILTER_ARGS went too: it was written, never read, left over from the unpinned redo above. Final size 20819, so 181 of headroom. This step is close enough to the cap that it wants splitting before much else is added to it. Measured against the live indexes of all 47 aptly sources, with each source's real GLOB applied: edge amd64 8886M -> 184M 97.9% code amd64 8750M -> 227M 97.4% code arm64 8111M -> 210M 97.4% code armhf 7069M -> 150M 97.9% zulu amd64 1204M -> 172M 85.8% zulu arm64 1195M -> 170M 85.8% chromium-* ~280M -> ~260M ~9% everything else unchanged per run: 44.0 GiB -> 10.5 GiB downloaded (76.1% avoidable) Verified: no source is emptied by the pinned filter, checked across every source and architecture; pin_newest returns the same filters against the live indexes (code -> 3 clauses, zulu -> 276, apt.armbian.com at 2267 names declines, a bad DIST declines); all three pin_newest return paths and both mk_mirror call sites leave the slot via warn_skip with exit 0 under set -euo pipefail; bash -n and YAML parse clean; and a manual run of the caller against this branch completed successfully -- https://github.com/armbian/armbian.github.io/actions/runs/35467743134 Signed-off-by: Igor Pecovnik --- .../infrastructure-download-external.yml | 106 ++++++++++++------ 1 file changed, 71 insertions(+), 35 deletions(-) diff --git a/.github/workflows/infrastructure-download-external.yml b/.github/workflows/infrastructure-download-external.yml index b0d0f49713..9fded24525 100644 --- a/.github/workflows/infrastructure-download-external.yml +++ b/.github/workflows/infrastructure-download-external.yml @@ -764,8 +764,6 @@ jobs: mkdir -p "$APTLY_ROOT" printf '{ "rootDir": "%s" }\n' "$APTLY_ROOT" > "$APTLY_CFG" - echo "::debug::APTLY_ROOT=$APTLY_ROOT" - echo "::debug::APTLY_CFG=$APTLY_CFG" echo "::debug::Config file contents:" cat "$APTLY_CFG" >&2 @@ -789,7 +787,6 @@ jobs: MIRROR="${{ matrix.name }}-${{ matrix.release }}-${{ matrix.arch }}" echo "::debug::MIRROR_NAME=$MIRROR" - echo "::debug::Original KEY='${KEY:-}'" # KEY may be: # - "unstable contrib non-free" (suite + components) @@ -798,9 +795,6 @@ jobs: read -r DIST REST <<<"${KEY:-}" COMPONENTS="$REST" - echo "::debug::DIST='$DIST'" - echo "::debug::REST='$REST'" - echo "::debug::COMPONENTS='$COMPONENTS'" # Special/flat cases: do not pass components case "${KEY:-}" in @@ -823,12 +817,58 @@ jobs: URL="${URL/http:\/\//https:\/\/}" echo "::debug::URL='$URL'" - echo "::debug::FILTER_ARGS='${FILTER_ARGS[*]}'" echo "::debug::ADDITIONAL_FILTER='$ADDITIONAL_FILTER'" echo "::debug::ARCH='${{ matrix.arch }}'" + # Some upstreams advertise every historical build: ~50 versions of + # "code" pass GLOB, 7.9 GiB, of which the prune below keeps 210 MiB. + # aptly queues from the filtered index, so pin the filter to the newest + # version of each name first -- same index aptly is about to read. + # Engages only where the index actually holds duplicates. + PINNED_FILTER="" + pin_newest() { + local comp stem body name ver entries=0 + local -A newest=() + local -a comps=() + if [[ -n "$COMPONENTS" ]]; then + read -r -a comps <<< "$COMPONENTS" + else + comps=("") + fi + for comp in "${comps[@]}"; do + if [[ -n "$comp" ]]; then + stem="${URL%/}/dists/${DIST}/${comp}/binary-${{ matrix.arch }}/Packages" + else + stem="${URL%/}/Packages" + fi + body="" + body="$(curl -fsSL --max-time 60 "${stem}.gz" 2>/dev/null | gzip -dc 2>/dev/null)" || true + [[ -n "$body" ]] || body="$(curl -fsSL --max-time 60 "${stem}.xz" 2>/dev/null | xz -dc 2>/dev/null)" || true + [[ -n "$body" ]] || body="$(curl -fsSL --max-time 60 "${stem}" 2>/dev/null)" || true + # 2 = index unreadable; caller must not run unpinned + [[ -n "$body" ]] || return 2 + while read -r name ver; do + [[ -n "$name" && -n "$ver" ]] || continue + entries=$((entries + 1)) + if [[ -z "${newest[$name]:-}" ]] || dpkg --compare-versions "$ver" gt "${newest[$name]}"; then + newest["$name"]="$ver" + fi + done < <(awk '/^Package: /{p=$2} /^Version: /{if (p != "") {print p, $2; p=""}}' <<< "$body") + done + # nothing to gain when the index already holds one version per name, + # and a filter naming hundreds of packages is not worth building + (( entries > ${#newest[@]} )) || return 1 + (( ${#newest[@]} > 0 && ${#newest[@]} <= 500 )) || return 1 + local q="" + for name in "${!newest[@]}"; do + q+="${q:+ | }${name} (= ${newest[$name]})" + done + PINNED_FILTER="$q" + echo "::notice::pinning ${#newest[@]} package(s) to newest of ${entries} versions" + } + pin_newest || { [[ $? == 2 ]] && warn_skip "index unreadable: $URL $DIST"; true; } + # Drop mirror if it already exists from previous run - echo "::debug::Checking if mirror exists..." if aptly -config="$APTLY_CFG" mirror show "$MIRROR" &>/dev/null; then echo "::notice::Dropping existing mirror: $MIRROR" aptly -config="$APTLY_CFG" mirror drop "$MIRROR" || true @@ -836,56 +876,52 @@ jobs: echo "::debug::Mirror does not exist yet" fi - # Create mirror (distribution + optional components). - # `|| warn_skip` catches a missing/dead Release file at - # the URL — common cause of "broken source" — and skips - # the slot instead of letting `set -euo pipefail` fail - # the whole job. - echo "::debug::Creating mirror..." - if [[ -n "$COMPONENTS" ]]; then - echo "::debug::aptly -config="$APTLY_CFG" -ignore-signatures ${FILTER_ARGS[*]} ${ADDITIONAL_FILTER} -architectures="${{ matrix.arch }}" mirror create "$MIRROR" "$URL" "$DIST" $COMPONENTS" + # $COMPONENTS unquoted on purpose: empty means "no components". + mk_mirror() { # shellcheck disable=SC2086 - aptly -config="$APTLY_CFG" -ignore-signatures "${FILTER_ARGS[@]}" $ADDITIONAL_FILTER -architectures="${{ matrix.arch }}" mirror create "$MIRROR" "$URL" "$DIST" $COMPONENTS \ + aptly -config="$APTLY_CFG" -ignore-signatures "${FILTER_ARGS[@]}" $ADDITIONAL_FILTER \ + -architectures="${{ matrix.arch }}" mirror create "$MIRROR" "$URL" "$DIST" $COMPONENTS \ || warn_skip "aptly mirror create failed (URL='$URL' DIST='$DIST' COMPONENTS='$COMPONENTS')" - else - echo "::debug::aptly -config="$APTLY_CFG" -ignore-signatures ${FILTER_ARGS[*]} ${ADDITIONAL_FILTER} -architectures="${{ matrix.arch }}" mirror create "$MIRROR" "$URL" "$DIST"" - # shellcheck disable=SC2086 - aptly -config="$APTLY_CFG" -ignore-signatures "${FILTER_ARGS[@]}" $ADDITIONAL_FILTER -architectures="${{ matrix.arch }}" mirror create "$MIRROR" "$URL" "$DIST" \ - || warn_skip "aptly mirror create failed (URL='$URL' DIST='$DIST')" + } + + # Swap in the pinned filter. + PINNED_APPLIED=0 + if [[ -n "$PINNED_FILTER" ]]; then + [[ -n "${GLOB:-}" ]] \ + && FILTER_ARGS=(-filter="( ${GLOB} ), ( ${PINNED_FILTER} )") \ + || FILTER_ARGS=(-filter="${PINNED_FILTER}") + PINNED_APPLIED=1 fi - echo "::debug::Mirror created successfully" + + echo "::debug::Creating mirror..." + mk_mirror # Update mirror with retry logic for EOF errors - echo "::debug::Updating mirror..." MAX_RETRIES=3 RETRY_COUNT=0 UPDATE_SUCCESS=false - while [[ $RETRY_COUNT -lt $MAX_RETRIES && "$UPDATE_SUCCESS" == "false" ]]; do if aptly -config="$APTLY_CFG" -max-tries=20 -ignore-signatures mirror update "$MIRROR"; then - echo "::debug::Mirror updated successfully" UPDATE_SUCCESS=true else RETRY_COUNT=$((RETRY_COUNT + 1)) if [[ $RETRY_COUNT -lt $MAX_RETRIES ]]; then echo "::warning::Mirror update failed (attempt $RETRY_COUNT/$MAX_RETRIES), retrying..." sleep 2 - # Recreate mirror if it got corrupted - echo "::debug::Recreating mirror after failure..." aptly -config="$APTLY_CFG" mirror drop "$MIRROR" || true - if [[ -n "$COMPONENTS" ]]; then - # shellcheck disable=SC2086 - aptly -config="$APTLY_CFG" -ignore-signatures "${FILTER_ARGS[@]}" $ADDITIONAL_FILTER -architectures="${{ matrix.arch }}" mirror create "$MIRROR" "$URL" "$DIST" $COMPONENTS - else - # shellcheck disable=SC2086 - aptly -config="$APTLY_CFG" -ignore-signatures "${FILTER_ARGS[@]}" $ADDITIONAL_FILTER -architectures="${{ matrix.arch }}" mirror create "$MIRROR" "$URL" "$DIST" - fi + mk_mirror else warn_skip "aptly mirror update failed after $MAX_RETRIES attempts (URL='$URL' DIST='$DIST')" fi fi done + # A pinned filter matching nothing would publish an empty repository. + if [[ "$PINNED_APPLIED" == "1" ]] && [[ "$(aptly -config="$APTLY_CFG" mirror show "$MIRROR" \ + 2>/dev/null | awk -F': *' '/^Number of packages/{print $2;exit}')" == "0" ]]; then + warn_skip "newest-version pin matched no packages (URL='$URL' DIST='$DIST')" + fi + # Snapshot. Failure here is rare (local aptly state op, # not a remote fetch) but `set -e` would still kill the # job, so route through warn_skip for consistency.