Skip to content

fix(ci): retire queued runs when PR returns to Draft - #2537

Draft
seonghobae wants to merge 14 commits into
seonghobae/coverage-incomplete-approval-20260930from
fix/draft-queue-retirement
Draft

seonghobae wants to merge 14 commits into
seonghobae/coverage-incomplete-approval-20260930from
fix/draft-queue-retirement

Conversation

@seonghobae

@seonghobae seonghobae commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Incident

A fresh organization-wide sample found that CodeQL PR, SAST Semgrep, Security Scan, and Python Security runs could remain queued after the pull request was converted back to Draft. The workflows had per-PR cancel-in-progress: true groups and ready_for_review triggers, but no converted_to_draft trigger, so the Draft transition could not enter the same concurrency group and retire the stale Ready run.

RED

Added test_converted_to_draft_retires_queued_run_without_runner first.

4 failed in 0.37s

All four failures were the intended missing-event assertion.

GREEN

  • subscribe the four workflows to converted_to_draft;
  • require every entry job to observe github.event.pull_request.draft != true;
  • preserve existing per-PR concurrency, permissions, scanner thresholds, and Ready-head behavior;
  • record the incident and rejected alternatives in the doctoring note and changelog fragment.

Local contract evidence:

4 passed in 0.07s
14 passed in 0.46s

Safety

The replacement Draft event reaches workflow-level concurrency and cancels the older same-PR run, then skips before runner admission. No required result is synthesized, no failed check is rerun or changed to success, and no model/provider path is introduced. No force push, destructive rebase, bypass, or merge was performed.

Acceptance

Ready is review admission only. Hosted exact-head checks and independent review remain mandatory before merge. After ordinary protected merge, verify one live Draft transition retires the older queued run for all four workflows without assigning a runner.

Stack repair (2026-09-30)

Retargeted from main@37b10243cec3d160ecc9c1be75c71428b160a703 to canonical security prerequisite #2531 (codex/security-baseline-final-20260930@d1aa3659fca527a6c7330151f3ab4df3d7578391). The semantic source tree remains unchanged from f1823128a51d74590eec524e183a09fcbdd47f0f. A single empty fast-forward commit bf8acdc6855e854692b4bf940b97490400a9b33d refreshed Checks on the new base; no force update, approval, or merge was performed. Prior Python Security/Trivy failures were the inherited vulnerable PyJWT/PyO3 base and are not current-head GREEN evidence. Fresh mergeability and hosted Checks are required on the new stack.

Current owner-stack integration (2026-10-01)

  • exact head: 76aa57533b1332af2d3976e32437dba70fafe47a
  • canonical prerequisite: .github#2536@18c886cbb8c41f5f9c2bcae99640b97d53a4806e
  • ancestry: ordinary two-parent integration preserves prior fix(ci): retire queued runs when PR returns to Draft #2537 head bf8acdc6855e854692b4bf940b97490400a9b33d and adopts the exact security/coverage owner head as the second parent
  • effective prerequisite-relative delta: the original seven queue-retirement paths only; 11 ahead / 0 behind
  • RCA: exact run 36710180872, job 109887107430, failed on PyO3 GHSA-36hh-v3qg-5jq4 and GHSA-chgr-c6px-7xpp; the canonical prerequisite already repairs and contracts those dependencies
  • workflow-consumer suite: 672 passed; warnings-fatal repository suite: 5,259 passed, 5 optional skips, 40 subtests
  • stale contract repair: the SAST single-consumer oracle now requires the full closed-and-Draft admission guard
  • publication: ordinary single-parent fast-forward from cf8fa367a590c97a440e434731d469909920e4bc, tree 536bc45136c2c9ad406fac0dd54ec5caeb2f2ab2, force=false
  • lifecycle: Ready only for fresh exact-head hosted Checks and independent review; ordinary merge remains HOLD

No predecessor result transfers. No force update, manual rerun, synthetic status, bypass, merge, or release was used.

Concurrent exact-head refresh (2026-10-01)

A concurrent ordinary child advanced this Draft PR from 76aa57533b1332af2d3976e32437dba70fafe47a to exact head 37ce2e405ab96963a59c96e4e98024ab1a51af32 (tree 6c8b45a22253260c5d25e1ce1683115bec37f4ef) to restore the queue-health credential fallback. The current head is a one-parent fast-forward, and canonical base #2536@18c886cbb8c41f5f9c2bcae99640b97d53a4806e is an ordinary ancestor. GitHub re-evaluation reports mergeable.

The effective base-relative delta remains bounded to the ten queue-retirement paths (+175/-18). Draft-event Security 36779949657, SAST 36779949766, and CodeQL 36779949579 are skipped and are not passing evidence. Keep Draft/Proposed; retarget or integrate the newer #2040 successor only after its exact-head Checks and ownership boundary stabilize, without copying mutable source or force-updating this active writer.

Latest review-authority restack (2026-10-01)

Exact head 0dacd79d00907c9c310a86f3a0cc423955eeb79c (tree 1d5a611b1c1ad63a451b5c0114ee4146c5993981) is an ordinary two-parent merge of prior #2537 head 37ce2e405ab96963a59c96e4e98024ab1a51af32 and current base owner #2536 head 6a37e4cdfbd8bf6f3a60645a0ad7c13e1b9c1ae3. The merge was conflict-free and preserves the complete ten-path queue-retirement delta while adopting the latest review-authority, Noema, Strix, and security contracts.

Fresh local exact-tree evidence: queue-retirement contracts 25 passed; integrated parent contracts 62 passed, 2 optional skips; unfiltered warning-fatal repository suite 5,261 passed, 5 optional skips, 40 subtests; the only changed production module reports 10/10 documented (100%); compileall, diff-check, conflict-index, ordered-parent, and tree-SHA verification pass. The branch remains Draft/Proposed. Fresh hosted exact-head evidence and an independent current-head approval are required before Ready or merge. No force update, bypass, manual rerun, synthetic status, or predecessor-GREEN transfer was used.

@coderabbitai

coderabbitai Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobae seonghobae left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head COMMENT review for f1823128a51d74590eec524e183a09fcbdd47f0f against main@37b10243cec3d160ecc9c1be75c71428b160a703. The four required workflows now subscribe to converted_to_draft, enter their existing per-PR concurrency group, and skip every ordinary entry job on a Draft payload before runner admission. Ready, push, schedule, and manual behavior remains governed by the existing conditions; scanner thresholds and permissions are unchanged. The executable contract covers all four workflow owners. I found no source-backed blocker in this exact diff. This COMMENT is not approval: four current exact-head runs are queued, earlier skipped replacements are not GREEN evidence, and qualifying independent approval is absent.

@seonghobae seonghobae added area: ci-cd CI, GitHub Actions, checks, release, or supply chain bug Something isn't working priority: high High-priority or P1 work status: needs-review Open pull request requiring current-head review or checks labels Sep 30, 2026 — with ChatGPT Codex Connector
@seonghobae
seonghobae changed the base branch from main to codex/security-baseline-final-20260930 September 30, 2026 11:36

Copy link
Copy Markdown
Contributor Author

Fresh live confirmation of the incident boundary (2026-09-30): fast-mlsirm #2299 and #2301 were converted from Ready to Draft at unchanged exact heads ded3ad85e19061ff38a4f84604ed032ebbcf44f0 and e51a4d94244f3ae34688a2353f7483cc784c548c. Each conversion produced only a new skipped CI run (36710535442, 36710532709). It did not create a replacement CodeQL PR, SAST Semgrep, Security Scan, or Python Security run, so the prior failed CodeQL PR runs (36688833003, 36696423987) remain and their exact central dispatch runs (36708176563, 36709991599) are still queued. This is current operational evidence for the missing converted_to_draft admission path repaired here; it is not merge approval. Post-merge acceptance still requires observing all four replacement workflows retire stale queued work without runner admission.

@seonghobae
seonghobae marked this pull request as draft September 30, 2026 13:39

Copy link
Copy Markdown
Contributor Author

stack-ordering을 current evidence에 맞춰 Ready→Draft/Proposed로 교정했습니다.

  • child exact head: bf8acdc6855e854692b4bf940b97490400a9b33d
  • prerequisite: fix(security): refresh shared PyJWT and PyO3 locks #2531 d1aa3659fca527a6c7330151f3ab4df3d7578391
  • prerequisite는 exact-head CodeQL Python/Actions terminal failure와 미도착 후속 verdict 때문에 Draft입니다.
  • 이 PR의 converted_to_draft queue-retirement delta와 tests/ancestry는 그대로 보존합니다.

부모가 ordinary protected merge 및 terminal GREEN을 획득하기 전에는 이 자식도 Ready/merge 후보가 아닙니다. Close, bypass, Force Push, destructive rebase, manual rerun은 수행하지 않았습니다.

Preserve the queue-retirement delta while adopting the exact owner tree from #2536, including the PyO3, PyJWT, and urllib3 security fixes. This is a non-destructive two-parent integration; exact-head checks must rerun.
@seonghobae
seonghobae changed the base branch from codex/security-baseline-final-20260930 to seonghobae/coverage-incomplete-approval-20260930 September 30, 2026 19:05

Copy link
Copy Markdown
Contributor Author

Exact-head verification and non-force publication receipt

  • predecessor head: cf8fa367a590c97a440e434731d469909920e4bc
  • current head: 76aa57533b1332af2d3976e32437dba70fafe47a
  • exact tree: 536bc45136c2c9ad406fac0dd54ec5caeb2f2ab2
  • root cause: tests/test_docs_only_pr_runner_admission.py still encoded the old SAST closed-only job guard after the production workflow added the Draft admission predicate
  • RED: the affected workflow-contract suite failed exactly that stale assertion after 244 prior passes
  • GREEN: focused regression 1 passed; affected workflow-consumer suite 672 passed; warnings-fatal repository suite 5,259 passed, 5 optional-platform skips, 40 subtests
  • durable evidence: docs/doctoring/draft-transition-queue-retirement.md records the audit and exact counts
  • publication: ordinary single-parent Git-data fast-forward, force=false

Ready is review admission only. Fresh hosted exact-head Checks and a qualifying independent APPROVED review remain mandatory. No bypass, synthetic status, manual rerun, auto-merge, or merge was used.

@seonghobae
seonghobae marked this pull request as ready for review September 30, 2026 20:12

Copy link
Copy Markdown
Contributor Author

Hosted Security RCA and canonical owner repair (2026-10-01)

Fresh exact-head revalidation confirms this PR is still open and mechanically mergeable at 76aa57533b1332af2d3976e32437dba70fafe47a over 18c886cbb8c41f5f9c2bcae99640b97d53a4806e.

The Security failure in run 36770959254, job 110076981183, is substantive inherited base-tree evidence:

Canonical owner repair: #2545 at exact head 08d8506468ded357b7b1a2493dfad40fedb6758c, based directly on protected main@37b10243cec3d160ecc9c1be75c71428b160a703. It updates only those transitive lock entries, adds a deterministic regression contract, and records RED→GREEN evidence (npm audit: 2 moderate vulnerable packages → 0; full warnings-fatal suite: 5164 passed, 6 skipped, 40 subtests).

CodeQL run 36770959205 is the designed pending handoff: both shards dispatched successfully and reported VERDICT_STATE=pending; bound dispatch 36771086349 is queued for this exact PR head/base/run/merge tuple.

Merge HOLD: do not bypass the Security defect. Merge #2545 through ordinary protection, then refresh/restack this PR and regenerate exact-current-head Security evidence. CodeQL completion and independent approval also remain required.

@seonghobae
seonghobae marked this pull request as draft September 30, 2026 21:04

Copy link
Copy Markdown
Contributor Author

Exact-head stack correction for 76aa57533b1332af2d3976e32437dba70fafe47a:

This PR is stacked on #2536, whose exact-head CodeQL run is terminal failure with missing authenticated rerun. This head also has terminal Security failure 36770959254 and CodeQL failure 36770959205; only SAST is GREEN. It is now Draft/Proposed until the canonical #2040 admission repair lands, the stack is reconciled non-destructively, and fresh exact-head Checks plus independent review pass.

No leaf copy, manual rerun, synthetic status, bypass, force update, or predecessor-GREEN transfer was used.

Use the existing job-bound OpenCode OIDC exchange when legacy cross-repository secrets are absent, so the scheduled read-only queue report can emit evidence instead of failing before collection.

Copy link
Copy Markdown
Contributor Author

Queue-health RCA and owner repair (2026-10-01)

Exact head: 37ce2e405ab96963a59c96e4e98024ab1a51af32

RCA

A fresh sample of the 100 newest repository_dispatch runs contained 52 CodeQL handler runs: 28 were queued, 24 were superseded/cancelled, and 0 were in_progress. The handler concurrency key is already scoped by target repository and PR number, so unrelated PRs are not cancelling or serializing each other. Every queued handler instead requires the shared self-hosted CWL central CodeQL runner group; the evidence is consistent with no eligible runner being available, not a per-PR concurrency collision.

The scheduled diagnostic that should distinguish offline/capacity/stale-demand causes also failed before collection: run 36763232032, job 110050816328, exited because both legacy cross-repository secrets were empty. No JSON/HTML evidence artifact was uploaded.

Repair

The existing queue-owner branch now gives the read-only queue-health collector the same job-bound OpenCode OIDC exchange already used by central schedulers. Legacy secrets remain preferred; the exchanged GitHub App token is only a fallback. The job adds only id-token: write; repository permissions remain contents: read and actions: read. No cancellation, merge, status synthesis, or other mutation was added.

TDD evidence:

  • RED: the queue-health workflow contract failed on the missing id-token: write / OIDC fallback.
  • GREEN: all 3 queue-health contract functions passed; workflow YAML parsed; git diff --check passed.
  • Published delta: exactly .github/workflows/actions-queue-health.yml and tests/test_actions_queue_health_contract.py.
  • Publication: ordinary single-parent commit on prior exact head 76aa57533b1332af2d3976e32437dba70fafe47a, tree 6c8b45a22253260c5d25e1ce1683115bec37f4ef, force=false.

The PR remains Draft. The live scheduled OIDC exchange and resulting queue artifact can only be proven after protected integration, and the shared CodeQL runner-group availability itself is still unresolved. Current Draft-triggered Security Scan, SAST Semgrep, and CodeQL PR runs are intentionally skipped and are not merge evidence.

Copy link
Copy Markdown
Contributor Author

Concurrent head re-fetch:

  • current exact head: 37ce2e405ab96963a59c96e4e98024ab1a51af32, tree 6c8b45a22253260c5d25e1ce1683115bec37f4ef
  • parent: prior head 76aa57533b1332af2d3976e32437dba70fafe47a
  • base: #2536@18c886cbb8c41f5f9c2bcae99640b97d53a4806e, verified ordinary ancestor
  • effective delta: 10 queue-retirement paths, +175/-18
  • GitHub mergeability re-evaluated to true
  • Draft-event Security 36779949657, SAST 36779949766, and CodeQL 36779949579 are skipped, not passing evidence

No branch update was made by this review, preserving the active writer's ordinary child. The PR remains OPEN / Draft / Proposed; later owner-successor integration must be non-force and exact-head verified.

Copy link
Copy Markdown
Contributor Author

Ordinary non-force restack published at exact head 0dacd79d00907c9c310a86f3a0cc423955eeb79c (tree 1d5a611b1c1ad63a451b5c0114ee4146c5993981; ordered parents 37ce2e405ab96963a59c96e4e98024ab1a51af32, 6a37e4cdfbd8bf6f3a60645a0ad7c13e1b9c1ae3). The complete ten-path queue-retirement delta is preserved on the latest canonical review-authority base.

Fresh exact-tree evidence: 25 queue-retirement tests; 62 integrated-parent tests plus 2 optional skips; 5,261 warning-fatal repository tests plus 5 optional skips and 40 subtests; changed production module docstrings 10/10 (100%); compileall, diff-check, conflict-index, parent order, and tree identity pass. GitHub reports Draft/open/mergeable with zero unresolved inline threads. Fresh hosted Checks and an independent exact-head approval remain required; no force update, bypass, manual rerun, synthetic status, merge, or predecessor-GREEN transfer was used.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci-cd CI, GitHub Actions, checks, release, or supply chain bug Something isn't working priority: high High-priority or P1 work status: needs-review Open pull request requiring current-head review or checks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant