Skip to content

fix(notify): keep bump PRs open on update, retry failed repos with backoff - #129

Open
ssestak wants to merge 3 commits into
mainfrom
housekeep/notify-consumer-repos-retry
Open

ssestak wants to merge 3 commits into
mainfrom
housekeep/notify-consumer-repos-retry

Conversation

@ssestak

@ssestak ssestak commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes two problems in the consumer-repo notification script observed during the 2.6.0 rollout.

1. Updating an existing bump PR auto-closed it. The update path force-reset the PR branch to the default branch HEAD before committing the bumped files. For a moment the PR had zero commits and GitHub auto-closed it, while the script still reported UPDATED. In the last two runs 12 of 13 "updated" PRs ended up closed with a stale title, and the next run then opened duplicate PRs.

  • Existing PRs are now brought up to date by merging the default branch into the PR branch (POST /repos/{repo}/merges), so the PR never loses its commits.
  • On a merge conflict the branch is deleted and recreated and a fresh PR is opened.
  • The script verifies the PR is still OPEN before reporting UPDATED.

2. No resilience to GitHub outages. During a GitHub incident (degraded Git operations) 22 repos failed in a row because Contents API writes were rejected, and the run had no way to recover.

  • Failed repos are retried in passes with exponential backoff: up to 10 passes, starting at 2 s and doubling up to 5 min.
  • Only failed repos are retried; files already bumped on the branch are not rewritten, so partial progress is kept.
  • Tuning is read from repository variables NOTIFY_RETRY_MAX_ATTEMPTS, NOTIFY_RETRY_BASE_DELAY, NOTIFY_RETRY_MAX_DELAY (already set to the defaults), so it can be changed without a merge.

Also

  • gh errors are no longer discarded; the last stderr line is printed next to each FAIL so logs show the actual API response.
  • A failed gh pr create was reported as CREATED because stderr was captured into the URL variable.
  • New NOTIFY_REPOS env var and repos input on workflow_dispatch to process an explicit list of repos, e.g. to re-run only the ones that failed.
  • Script restructured into functions so the per-repo flow can be re-run and unit tested.

Test plan

  • New BATS suite (13 tests) with a gh CLI mock: backoff schedule, create path, leftover-branch reuse, merge-based update without force-reset, conflict → recreate, closed PR → FAIL, no rewrite of already-bumped files, retry until success, give up after max attempts, only failed repos retried, self repo skipped, dry run. Full bats -r . passes (140 tests).
  • actionlint on the workflow.
  • --dry-run against two real consumer repos (one create, one update) produced the expected plan.
  • After merge: re-run the workflow via workflow_dispatch with the repos input for the repos that failed in the 2.6.0 run.

🤖 Generated with Claude Code

https://claude.ai/code/session_015HbhiukLQv71hkruK31fyT

… backoff

The update path force-reset the PR branch to the default branch HEAD before
committing the bumped files. That left the PR with zero commits for a moment
and GitHub auto-closed it, while the script still reported UPDATED. Existing
PRs are now brought up to date by merging the default branch into the PR
branch; on a merge conflict the branch is recreated and a fresh PR opened.
The script also verifies the PR is still open before reporting success.

Repos that fail (typically during a GitHub outage, as writes via the Contents
API are the first to go) are now retried in passes with exponential backoff:
up to 10 passes, starting at 2 s and doubling up to 5 min, configurable via
NOTIFY_RETRY_* env vars. Only failed repos are retried and files already
bumped on the branch are not rewritten, so partial progress is kept.

Other fixes along the way:
- gh errors are no longer discarded; the last stderr line is printed next to
  each FAIL so logs show the actual API response
- a failed `gh pr create` was reported as CREATED because stderr was captured
  into the URL variable
- NOTIFY_REPOS env var / `repos` workflow_dispatch input to process an explicit
  list of repos, e.g. to re-run only the ones that failed

Adds a gh CLI mock and BATS tests covering the create/update/retry paths.

Claude-Session: https://claude.ai/code/session_015HbhiukLQv71hkruK31fyT
The action-refs BATS suite rewrites every ref to @main in its teardown; those
changes slipped into the previous commit. No functional change.

Claude-Session: https://claude.ai/code/session_015HbhiukLQv71hkruK31fyT
NOTIFY_RETRY_MAX_ATTEMPTS, NOTIFY_RETRY_BASE_DELAY and NOTIFY_RETRY_MAX_DELAY
are read from repository variables so they can be changed without a merge.
Script defaults apply when the variables are not set.

Claude-Session: https://claude.ai/code/session_015HbhiukLQv71hkruK31fyT

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant