[kelos-self-update] Prompt Tuning: reviewers can approve a superseded PR head

[kelos-bot]

🤖 Kelos Self-Update Agent @gjkim42

Latest verdict

• Status: VALID
• Checked at: 2026-09-21T06:04:27Z
• Evidence: Revalidated default branch a18a124: all four regular/API reviewer prompts still omit live-head verification before analysis/publication and a required reviewed SHA; workers, PR-responder, and squash still lack a final-push review handoff. PR #1496 retains a workflow-only approval after a bot force-push replaced the branch with runtime-only changes; the commits share a parent and have no changed-file overlap. PR #1747 retains a Claude approval last updated at 2026-09-13T23:47:57Z, before its final force-push at 23:51:58Z. Rechecked review/comment/force-push timelines for Recover empty Codex Sessions after pod recreation #1496, Suspend idle Sessions automatically #1569, Preserve workspace repository usernames during authentication #1602, and Fix session conversation alignment with long headers #1747, all root spawner prompts, the latest 20 generated merged PRs (all image updates), the latest 20 generated issues, and current open issues/PRs for overlap. Open PR #1249 adds reviewer checklist guidance, not commit binding. No superseding fix or clearly stronger in-scope candidate found; retain the proposal and acceptance criteria.

Area: Prompt Tuning

Summary

Kelos reviewer prompts can publish or retain an APPROVE sticky comment for a commit that is no longer the pull request head. The comments do not identify the reviewed commit, and the agents neither verify that their checkout matches the live PR head before analysis nor recheck the head immediately before publishing.

The integrity gap continues after publication. Branch-mutating worker, PR-responder, and squash flows do not define a final-head review handoff, so a valid verdict can remain visible after those agents push a different diff.

This is the highest-impact prompt-tuning candidate after reviewing the root self-development spawner prompts, the 20 most recent merged PRs labeled generated-by-kelos, the 20 most recent generated issues, and current open generated issues for overlap.

Current prompt gap

• kelos-reviewer.yaml and kelos-claude-reviewer.yaml fetch main and review origin/main...HEAD; neither reads headRefOid, verifies the checkout against it, or rechecks it before publication.
• kelos-api-reviewer.yaml and kelos-claude-api-reviewer.yaml have the same gap on their PR paths. Their issue-review paths do not need commit binding.
• The regular sticky template and API sticky template, plus their Claude counterparts, contain a verdict and scope but no reviewed SHA.
• kelos-workers.yaml and kelos-pr-responder.yaml provide only a generic URL handoff. kelos-squash-commits.yaml force-pushes and reports completion without refreshing reviewer verdicts for the rewritten head.

A reviewer-side guard is required even if branch-mutating prompts add a handoff, because a maintainer or concurrent agent can change the head while one review run is already analyzing it.

Evidence from agent activity

The head changed between a review request and publication

On PR #1602, a maintainer posted /kelos review at 07:55:00 UTC on 2026-08-08. The PR was force-pushed to final head f448565 at 07:59:52 UTC. The sticky approval, originally created on August 6, was updated at 08:02:06 UTC on August 8, before merge. Its body identifies no reviewed SHA. These timestamps establish that the branch changed between the request and publication, but do not establish which commit was analyzed; this is evidence of missing traceability, not proof of an outdated review.

A regular verdict remained stale after later pushes

On PR #1569, the regular reviewer published its last sticky verdict at 10:29:40 UTC on 2026-08-08. The branch was force-pushed at 10:29:58 and again at 10:40:18 to final head 0906431. Only the API-specific reviewer ran afterward, although its prompt deliberately leaves general correctness to the regular reviewer. The PR merged without a current regular verdict.

A surviving approval described a different diff

On PR #1496, the reviewer published an APPROVE describing a serial Session StatefulSet restart change. The branch was then force-pushed and merged without another regular review. The reviewed commit 8e56ed8 changed only .github/workflows/deploy-dev.yaml; the final head 51b1344 changed only internal/sessionruntime/codex.go and internal/sessionruntime/server_test.go. The surviving approval therefore described a diff with zero file overlap with what merged.

Proposed minimal prompt changes

1. In all four reviewer prompts, read the PR's live headRefOid before analysis, fetch and check out that exact commit, and verify git rev-parse HEAD matches it.
2. Immediately before creating or updating the sticky comment, read headRefOid again. If it changed, do not publish the old verdict: review the new complete diff or publish a clearly non-approving stale-run status that records both SHAs and requests a fresh run.
3. Add **Reviewed commit**: <full SHA> to every PR sticky-review template. Keep issue-only API review comments unchanged.
4. Make reviewer handoff the terminal branch-mutation step in kelos-workers, kelos-pr-responder, and kelos-squash-commits. After the final push, request the regular reviewer and the API reviewer when the diff has API surface; rerun any Claude reviewer whose sticky marker is already present. If another commit or force-push is necessary, repeat verification, CI, and every applicable reviewer handoff for the new final head. Do not request another review when the run makes no branch change.

Related but not duplicate

• Closed issue #752 proposed push-triggered polling-source retriggers. These reviewers are command-triggered webhook spawners, and a single in-flight review still needs commit binding.
• Closed issue #962 proposed webhook event deduplication and debouncing, not verdict-to-commit integrity.
• Open issues #1089 and #1203 concern routing reviewer commands to PRs rather than plain issues; they do not address a PR head changing during or after review.
• Targeted issue and PR searches for headRefOid, reviewed commits, stale reviewer heads, and superseded heads found no duplicate beyond this slot. No open PR references [kelos-self-update] Prompt Tuning: reviewers can approve a superseded PR head #1412 or implements the proposed binding.

Acceptance criteria

• Every Codex and Claude regular/API PR reviewer verifies that its checkout matches the live PR head before analysis and includes the exact full reviewed SHA in its sticky comment.
• Every reviewer rechecks the live PR head immediately before publication. A detected head change cannot leave an APPROVE or REQUEST CHANGES verdict presented as current; the agent reviews the new head or publishes a non-approving stale-run status.
• kelos-workers, kelos-pr-responder, and kelos-squash-commits request the appropriate dedicated review for their final pushed head, refresh every reviewer type that had already published a sticky verdict, and do not mutate the branch after that handoff without repeating it.
• Focused self-development manifest tests pin the commit-binding instructions and reviewed-commit field across all four PR reviewer variants, plus the final-head re-review handoff in each branch-mutating prompt.
• Issue-only API review behavior remains outside this commit-binding change.