Make central PR validation recoverable with exact-head evidence

[seonghobae]

Goal

Make central PR validation recoverable and auditable for fast-mlsirm and other consumers. Each required Strix, Noema, OpenCode, and CodeQL verdict must be tied to the current PR head and real producer evidence. Provider and runner-capacity failures must remain non-passing, with bounded same-head continuation where the existing contract permits it.

Current evidence

• feat(two-tier): orthogonal primary identification with fixed Phi fast-mlsirm#2114 was admin-merged as 2d80e8d85d40c665a8000020b78723f701f878c8 while five required checks were failing. This is a bypass record, not a clean validation record.
• 🛡️ Sentinel: [HIGH] CSP 'unsafe-inline' 제거 및 동적 해시 적용 fast-mlsirm#2238 at head 8053586b33145d9f4a4231f8193e89a4e73d1068 is the current canary. Central Strix run 36361585404 failed after a provider connection timeout without a vulnerability report, and its continuation was skipped. Noema run 36360063791 passed the native 413 self-test, then failed HTTP 504 provider_capacity_unavailable; its bounded continuation was queued. OpenCode coverage and CodeQL validation were queued at the same snapshot.

Acceptance

1. Diagnose each canary check from its exact-head run and artifact, separating code defects, provider outages, and scheduler delay.
2. Repair confirmed central workflow defects with focused tests. Keep security gates fail-closed, preserve findings, and use bounded same-head continuation only for explicit transport failures.
3. Verify the five checks on a live current-head canary or record a typed terminal infrastructure result. Do not describe a queued, skipped, synthetic, or bypassed check as green.
4. Keep the operator receipt current as runs advance; close only after a genuine hosted validation cycle proves the contract.

Active owner

Codex, 2026-09-28T03:49:36Z. First increment: restore bounded continuation for Strix runtime transport failures.

[seonghobae]

2026-09-28 UTC receipt for the active goal:

• First repair: fix(ci): continue Strix runtime transport failures on same PR head #2479 at cd04e6dacd97e735c8a44fd6bef3d24ab8aa4269 connects Strix runtime connection/rate-limit failures to the existing bounded same-head continuation. The scan still fails closed. A review-identified mixed sandbox/connection log is excluded. Focused tests: 83 passed under GITHUB_ACTIONS=true; the new module has 100% focused statement/branch coverage. Hosted checks are queued, so this is not hosted acceptance.
• 🛡️ Sentinel: [HIGH] CSP 'unsafe-inline' 제거 및 동적 해시 적용 fast-mlsirm#2238 remains at 8053586b33145d9f4a4231f8193e89a4e73d1068. Its earlier direct CodeQL compatibility jobs (108701353093/108701353056) deliberately failed after dispatch while waiting for authenticated central verdicts; its direct OpenCode job 108701609296 failed while waiting for an exact-head reviewer verdict. Those logs do not identify a source defect.
• Central Noema run 36360063791 continuation job 108765382600 was still queued with runner_id 0. At the runner snapshot, six self-hosted runners were online and busy; the central repository had 619 queued runs. This supports a scheduling-capacity diagnosis but does not establish that a runner label is misconfigured.
• The local full Strix fixture script was stopped after more than eight minutes to avoid duplicate resource use while another worktree ran the same broad script. Its terminal result is unverified. The focused tests above are the completed local evidence.

Next: follow exact-head hosted checks and reviews on #2479, then verify deployed continuation on a genuine current-head canary. Keep failed, queued, and skipped results distinct.

[seonghobae]

2026-09-28 follow-up: The #2479 superseded Strix and runtime-quality runs for heads d73e0ee2a and e770858cd reached completed/cancelled; only current head cd04e6dac remains queued. The CodeRabbit mixed sandbox/connection thread is resolved after commit cd04e6dac.

For ContextualWisdomLab/fast-mlsirm#2238, central CodeQL run 36355503206 completed validate-dispatch successfully at 04:04:40 UTC, then queued real CodeQL dispatch scan (python) job 108782399240 and (actions) job 108782399244. This is admission progress, not a terminal CodeQL verdict. Central Noema continuation job 108765382600 and OpenCode coverage-evidence job 108764444766 remain queued. All six org self-hosted runners were online and busy at the latest snapshot; 573 central runs were queued. No runner-label defect is established by that evidence.

[seonghobae]

2026-09-28 control-runner RCA snapshot: continue-noema-transport job 108765382600 has been queued since 02:35:39 UTC with labels [self-hosted,linux,x64,cwlab-control] and runner_id 0. Live org runner group 6 (CWL central control) is visibility=all, allows_public_repositories=true, restricted_to_workflows=true, and includes the central main noema-review.yml path. Its one registered worker is online and busy. The workflow and runner labels match. This evidence supports a group-capacity wait; it does not support widening the security boundary or changing labels.

The apparently long-running central jobs were admitted only recently: Noema #2464 model job started 03:46, OpenCode appguardrail coverage started 04:04, and naruon CodeQL's three analysis jobs started 04:13. Their earlier run timestamps mostly reflect queue age. All three CodeQL workers were active at the snapshot. Recheck job completions before changing allocation.

[seonghobae]

Correction to my preceding control-runner RCA: at one live read immediately after clearfolio job 108750194665 completed successfully at 04:22:15 UTC, cwlab-s2-01 was online idle while #2238 Noema continuation job 108765382600 remained queued. A later read found the runner busy again and the Noema job still queued. The idle observation was transient, so it does not prove persistent runner eligibility failure; it does disprove treating a busy snapshot alone as a complete explanation. The job has no pending deployment approval, group 6 permits the central main Noema workflow, and its labels match. Workflow-level concurrency is keyed by target repo and PR with cancel-in-progress. Current cause is admission/scheduling unresolved. I will not change group access, labels, or cancel another PR's job without stronger evidence.

[seonghobae]

2026-09-28 local verification update for #2479 head cd04e6dacd97e735c8a44fd6bef3d24ab8aa4269: the exact central Strix quality script used by Agent Review Runtime Quality CI completed with exit code 0 and test_strix_quick_gate: PASS under STRIX_TEST_PROCESS_TIMEOUT_SECONDS=3 STRIX_TEST_FAKE_SLEEP_SECONDS=5. This extends the earlier 83 affected pytest passes and the new module's 100% focused statement/branch coverage. It is local contract evidence only; hosted current-head checks and a complete vulnerability report are still required.

[seonghobae]

2026-09-28 04:49–04:51 UTC admission follow-up for fast-mlsirm #2238 (8053586b33145d9f4a4231f8193e89a4e73d1068): central Noema continuation job 108765382600 remained queued, runner_id=0, since 02:35:39 UTC. The CWL central control group includes online cwlab-s2-01; it was idle in both samples (~45 s apart). The job requests self-hosted,linux,x64,cwlab-control; the runner has those labels; the group is visible to all public repositories and allowlists central noema-review.yml@refs/heads/main. Thus continuous runner busyness is not a sufficient RCA. This is an observed admission/scheduling gap under backlog; GitHub's assignment cause is still unproven. No broad rerouting or duplicate dispatch has been made.

Central CodeQL dispatch jobs python 108782399240 and actions 108782399244 were also queued. CWL central CodeQL briefly had an idle online runner at 04:49 UTC but both group members were busy at 04:51 UTC. Their workflow and runner labels match the group allowlist; this sample alone does not prove an eligibility defect. All five canary gates remain without terminal exact-head evidence.

[seonghobae]

2026-09-28 04:56–04:57 UTC milestone: central Strix runtime-continuation repair PR #2479 was admin-squash-merged at 1d908b76a984518393dc46d7e0181b19b97c50bc while hosted required checks were still queued. This is an authorized queue bypass after the local 83-test contract suite and test_strix_quick_gate.sh passed; it is not a clean hosted-validation pass.

I dispatched one new default-branch Strix canary for open, ready fast-mlsirm #2238, exact head 8053586b33145d9f4a4231f8193e89a4e73d1068, base 8c1361a794176f040e7b999ca8ffc1fb4c0b5607, retry attempt 0. Central run 36379830577 was created from the repaired central main commit 1d908b76a984518393dc46d7e0181b19b97c50bc and is queued. No scan result, report, or continuation verdict exists yet. Do not count the pre-fix run 36361585404 as validation of the repair; its scan failed on a provider connection timeout and the continuation skipped.

[seonghobae]

2026-09-28 05:00 UTC Strix evidence audit for fast-mlsirm #2238, exact head 8053586b33145d9f4a4231f8193e89a4e73d1068: native run 36343272866 ended success and uploaded strix-reports artifact 10941945761. The scan metadata says completed/success and SARIF has zero results, but the generated report asserts a generic “Python OpenSSH client RCE vulnerability” with no changed path or concrete finding. The scanner log confirms zero vulnerability reports; this is contradictory narrative, not proof of a real OpenSSH defect or a clean, scoped review.

RCA: that native run checked out trusted central source f5864568aee369f7567bd7b8b0ac249f9ec4c513, before the current strix_report_scope.py guard was added. Replaying its downloaded run.json and report against the current guard with #2238's changed paths (python/fast_mlsirm/report.py, python/fast_mlsirm/scoring/essay/report_html.py) fails as intended: scan report does not identify a changed source file. Therefore the old green check is historical execution evidence but does not establish a scoped clean verdict under today's contract. The repaired-main canary run 36379830577 is still queued; await its own artifact and terminal result before counting Strix.

[seonghobae]

2026-09-28 05:07 UTC consumer-failure log audit for open fast-mlsirm #2238 at 8053586b33145d9f4a4231f8193e89a4e73d1068:

• OpenCode consumer job 108701609296 failed because no authenticated APPROVED or CHANGES_REQUESTED review from opencode-agent existed on that exact head. This is intentional fail-closed behavior while the central coverage-evidence job 108764444766 is still queued, not evidence of a fast-mlsirm source defect.
• CodeQL Python job 108701353056 and Actions job 108701353093 both logged DISPATCH_OUTCOME=success, VERDICT_STATE=pending; their failed compatibility results explicitly await the authenticated central scan and subsequent rerun. Central scan jobs 108782399240/108782399244 remain queued. Neither consumer failure proves a CodeQL finding.
• Native Noema job 108700905202 crashed in the offline loopback sidecar body-limit self-test: CPython 3.12.14 reported Fatal Python error: Segmentation fault, with a request thread in contextual_orchestrator.server._send_error / logging, main thread waiting in http.client.getresponse, and _cffi_backend loaded. This is a local sidecar process crash before a model review, not a fast-mlsirm source verdict or proof of provider 504. A separate later central Noema run 36360063791 reached the model and failed on HTTP 504 capacity; its bounded continuation job 108765382600 remains queued. The native crash is one observed occurrence; a deterministic code or runner cause is not yet established.

[seonghobae]

2026-09-28 05:11 UTC central .github queued-run census: one paginated read returned 589 queued runs (six pages). Major workflow IDs/counts: Code Quality dynamic 109; central CodeQL dispatch 72; CodeQL PR 62; OpenCode dispatch 58; agent-review-runtime-quality 52; Security Scan 37; Python Security 33; Semgrep 30; merge scheduler 25; required Noema 25; required OpenCode 16; Strix 16.

Of 183 queue groups whose run names explicitly bind workflow + target repository + PR number + target head, zero had duplicate runs for the same workflow/repository/PR. That does not prove all queue entries are deduplicated: the GitHub-managed Code Quality: PR #... dynamic workflow had 109 queued runs, with 37 runs spread over 15 PRs that had multiple queued heads (22 runs beyond one per PR). This is a bounded queue inefficiency, but even removing every surplus dynamic run would reduce the total by only ~3.7%, and those jobs have no verified impact on the current #2238 gate admission. No run was canceled or workflow routing changed from this census. The direct #2238 Strix, Noema, OpenCode, and CodeQL handles remain live and queued; the dominant bottleneck is still unproven at the GitHub assignment/organization-capacity boundary.

[seonghobae]

2026-09-28 05:17 UTC correction/follow-up to the native Noema crash classification: #2238's failed native job used older central control source f5864568aee369f7567bd7b8b0ac249f9ec4c513. Current central main already contains two relevant changes absent from that source: 0bea6aeee binds the sidecar process to the selected CPython interpreter's matching shared library, and PR #2473, merged as d1785f7c553d877f1a88c6dbdbfab9be8988c5e9, replaced process-global stderr redirection in the threaded HTTP 413 self-test with a temporary handler on the server logger. #2473's own verification did not prove the native segfault's precise cause or deterministic resolution.

The later central Noema job 108748091705 executed on d1785f7c5: Provision contextual-orchestrator review sidecar succeeded (02:11:38–02:18:05 UTC), then Prepare Noema model verdict failed at 02:35:27 UTC on HTTP 504 provider_capacity_unavailable. This proves the updated native Linux path got past the previous sidecar-startup failure in that run. The model verdict is still missing; bounded same-head continuation job 108765382600 remains queued. No new code patch is justified by the historical native crash alone.

[seonghobae]

2026-09-28 follow-up: fast-mlsirm #2238 was closed by its author at 06:25 UTC as a CSP regression: the policy removes inline-style permission while report bar widths still use style attributes. This canary must not be used as acceptance evidence or merged.

Its central Noema run 36360063791 is now cancelled. Before cancellation, continuation job 108765382600 succeeded at 05:53 UTC in scheduling bounded same-head retry after HTTP 504; that is not a model verdict. OpenCode coverage job 108764444766 failed at 05:38 UTC: the log reports it could not materialize base Rust dependencies because trusted cargo vendor raised FileNotFoundError for Cargo.lock, crates/fast-mlsirm-py/Cargo.lock, and fuzz/Cargo.lock. Reproduce this coverage-preparation failure on the next valid PR.

To limit obsolete queue load, cancellation was requested for exact #2238 central runs Strix 36379830577, OpenCode 36351939775, and CodeQL 36355503206. Requests are not terminal conclusions. The durable goal remains exact-head terminal Strix, Noema, OpenCode, and CodeQL Python/Actions evidence on a valid future PR, with provider, runner, and source failures classified from logs. #2114's earlier admin merge is not a clean-green CI result.