Skip to content

ci: plan affected validation instead of running every native suite - #18

Merged
novelKR merged 7 commits into
mainfrom
codex/ci-affected-validation
Sep 28, 2026
Merged

ci: plan affected validation instead of running every native suite#18
novelKR merged 7 commits into
mainfrom
codex/ci-affected-validation

Conversation

@novelKR

@novelKR novelKR commented Sep 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

Select the minimum sufficient CI/qualification coverage for a change while failing closed for unknown, CI-sensitive or untrusted changes.

This is an independent CI-hardening change. It is not CS-RG implementation, changes no DevGuard resource semantics, and does not modify or amend #17.

Base: 9e21cc8f707f16b7da98490293b5ea7e4548916d
Head: 094b0b7b46c7ebe6f1a37ccdaaa08edb5f7c9ca0

Motivation

#17 changes only docs/handoff/2026-09-28-w3-decision-packet.md. Its pull-request macOS run 36408760157 nevertheless ran cargo test --workspace and failed the unrelated native upgrade test only at its < 10 s timing assertion, after the expected ResourceUnavailable / did not finish result. Its push run 36408700221 passed on the same macOS image. The PR merge tree and branch head have the same Git tree.

That is not treated as a product regression. It shows that running all native qualification for every documentation record produces a poor merge signal.

What changes

  • scripts/ci-policy.json is the reviewable source of path classes, component-to-suite dependencies, platforms, full-coverage triggers, whole-tree whitespace pins and exact hosted-runner incomplete allowances.
  • scripts/ci_plan.py binds a plan to the source SHA, event, policy digest, run ID and attempt.
  • scripts/ci_run.py runs only one job's planned stages/suites and records the binding and leg result.
  • scripts/check_ci_results.py is the final gate. Required checks always runs, derives the plan again and accepts only the exact current-attempt plan, changed paths, leg records and reports.
  • scripts/validate.py can run selected stages without probing Rust. Its default invocation preserves the existing command order and scope; the protocol-analysis tests are only split into their own recorded stage.
  • git diff --check is now an actual CI stage.
  • docs/planning/verification.md and its reviewed Korean counterpart describe the affected-check model. The stale CodeSpace documentation-CI sentence now matches CodeSpace main at ab0341b and observed docs-only run 36336288225.

Profiles

Change Repository checks Rust / functional qualification
docs/handoff/** historical record whitespace + documentation not selected
Normative design/contract/planning/translation/ledger input whitespace + documentation; source-contract for its three inputs not selected
Rust/component source whitespace full workspace fmt/clippy/tests on macOS and Ubuntu; only affected DG1 suites
Workflow, planner/policy/gate, qualification framework, Cargo/lock, toolchain, Git inputs, unknown path or untrusted base all non-Rust checks full validator and every suite

The suite map is recomputed in tests from qualify.py's package selectors, the explicit workspace dependency graph and its non-Cargo PREBUILD launch-helper edges.

The #17 one-file diff is a policy regression case: it selects whitespace and documentation, no Rust and no functional suite, including no dg1-upgrade.

Events and full-coverage control

  • pull_request: affected plan against the checked-out merge's trusted first parent.
  • push: main only, affected plan from trusted before to GITHUB_SHA.
  • schedule (daily 18:17 UTC): full.
  • workflow_dispatch: full.
  • Missing/untrusted base, forced push, empty diff, planning-input change or unclassified path: full/fail closed.
  • Any unsupported event, including merge_group, has no plan and fails. Read-only repository queries found no classic branch protection, no rulesets/effective rules and no merge queue or past merge_group run.

The new daily full run is the compensating control for affected PR/main checks: it continues qualification of the full repository on both platforms, all native suites included. This PR does not activate that schedule unless merged.

Evidence integrity

  • Current-attempt artifact names carry github.run_attempt; every report is bound to the exact source/event/policy/run/attempt.
  • Artifacts from an earlier attempt are ignored and can never satisfy a later attempt. Re-run all jobs, not only failed jobs.
  • Unexpected artifact names, missing/corrupt changed-paths.json, mismatched leg records, unplanned suites or a stage reported as passed when not selected fail the gate.
  • An incomplete suite is accepted only when each not-run case occurs in its policy-named stage and exactly matches its platform/suite/case/path/reason allowance. Duplicate, moved or unexplained cases fail.

Validation

At exact head 094b0b7b46c7ebe6f1a37ccdaaa08edb5f7c9ca0:

  • python3.12 -B -m unittest discover -s scripts -p 'test_ci_*.py': 95 passed (also exercised on Python 3.10 before the final evidence-tightening commit).
  • python3 scripts/check_docs.py: 17 reviewed pairs, 48 work units, 25 logical groups.
  • git diff --check origin/main HEAD: passed.
  • python3 scripts/ci_plan.py --base origin/main --head HEAD: full, 14 paths, zero unclassified, 12 macOS suites and 2 Ubuntu suites; reasons planning-changed, .github/**, scripts/**.
  • docs(handoff): record the W3 experiments and decision packet #17 preview: affected, no Rust, no suite.
  • PR test(launcher): close the survivor release race (DG1-C06) #9 launch-crate preview: full workspace Rust checks plus exactly six affected macOS suites, including the PREBUILD edges.
  • PR docs(planning): state the CSP-D04 entry condition and evidence order (DGP-D07) #10 planning-document preview: whitespace + documentation only, no Rust.
  • Rust 1.95.0: python3.12 -B scripts/validate.py --offline --output target/qualification/ci-hardening-task8-full-094b0b7: passed, including source contract, docs, 62 protocol-analysis tests, dependency boundary, fmt, workspace Clippy and 318 workspace test results.

Pull-request run 36443987640 passed at the exact branch head. GitHub tested merge source d770796edb3f28b6bdd35d219d5481fc5ab25b8d, whose parents are the base and 094b0b7… and whose tree exactly equals the branch head's tree.

  • Plan: full, the same 14 paths/reasons/suite lists as the local preview, policy f81f54c…, run 36443987640 attempt 1.
  • Repository checks: passed; whitespace, source-contract, documentation and protocol-analysis passed, every Cargo stage recorded not_selected_by_plan.
  • contracts (ubuntu-24.04): passed; complete validator (198 test results), dg1-authority and dg1-auth passed.
  • contracts (macos-14): passed; complete validator (318 test results), all 12 suites ran. Nine passed; dg1-scopes, dg1-launch and dg1-cargo were accepted as incomplete only for their exact policy-listed hosted-runner cases.
  • Required checks: passed after downloading and validating all four attempt-1 artifacts.

Repository settings and limitations

Read-only preflight found no required checks and no merge queue. No repository setting changes are in this task. If this is later approved for merge, making Required checks required is a separate owner action.

This first version deliberately does not narrow workspace Clippy or workspace tests by package. Rust source still gets those complete checks. Package-level narrowing requires a separately proven model for test-only, feature, platform and runtime-binary edges.

Policy self-tests reject a newly tracked unclassified path before planning, so a new path class must be reviewed in the same PR. The planner itself also treats unclassified changed paths as full.

Rollback

Revert this PR. It changes only CI/planning code and verification documentation; no runtime state, service, credential, journal, release or host setting changes.

Evidence

Local ignored evidence: evidence/ci-hardening-2026-09-28/ (Task 0 settings/merge-queue queries, #17 baseline runs, CodeSpace readback and Task 8 hashes). No private payload or credential scan hit was found.

novelKR and others added 7 commits September 28, 2026 22:41
@novelKR
ci(validate): select validation stages without the Rust toolchain
4d605ed 
`scripts/validate.py --stages LIST` runs only the named stages, always in
the fixed order whitespace, source-contract, documentation,
protocol-analysis, dependency-boundary, format, clippy, contracts. The
Rust toolchain is probed only when a Cargo stage is selected, so the
documentation and source-contract checks run without Rust. A stage left
out is recorded as not_selected_by_plan, never as passed, and a run of
only some stages does not claim the DG-0 milestone.

The default run is unchanged: every stage except whitespace, the same
commands in the same order. test_measure.py is its own stage
(protocol-analysis) instead of part of documentation.

The new whitespace stage is `git diff --check` of `--diff-base`..HEAD,
or of the whole tree at HEAD when no base is given. The whole tree
exempts only files whose bytes match the SHA-256 the CI policy pins.

The workspace graph moves to module level (WORKSPACE_GRAPH) so the CI
planner can derive the affected functional suites from it.
@novelKR
ci: classify changed paths and plan affected checks
73991ed 
scripts/ci-policy.json classifies every changed path, first match in
this order:

- full: .github, scripts, Cargo manifests and locks, toolchain files,
  Git ignore/attribute files and .devguard.toml;
- historical: docs/handoff records, which run whitespace and
  documentation checks only;
- normative: design, contract, operations, planning, translation and
  ledger inputs, AGENTS.md, README.md, LICENSE and NOTICE, which run
  whitespace and documentation, plus source-contract for the approved
  design, its source record and milestones.json;
- a workspace crate, which runs the complete validator on both
  platforms and the functional suites whose packages, or the binaries
  they build first (qualify.py PREBUILD), depend on it.

A path in no class, a missing or untrusted base, an empty diff and any
change to a planning input make the plan full. scripts/ci_plan.py binds
the plan to GITHUB_SHA, the event, the policy digest, the run id and the
attempt. pull_request plans the merge against its first parent and push
plans main's before..after; schedule and workflow_dispatch are full; any
other event, merge_group included, has no plan.

PR #17's one-file diff (a docs/handoff record) is a regression case: it
selects whitespace and documentation only, no Rust and no suite.
Drift tests tie the policy to the workspace members, the suite map to
the dependency closure recomputed from qualify.py, the native set to
qualify.NATIVE, the source-contract inputs to what source_contract()
reads, and every tracked path to a class.
@novelKR
ci: run planned jobs and gate on the current attempt's evidence
02d2167 
scripts/ci_run.py runs one job's share of the plan after refusing any
plan not made for this commit, event, policy, run and attempt, and
records that binding beside the job's evidence:

- repository: whitespace and the plan's other non-Rust validator stages;
- bind contracts / contracts --os OS: the functional suites the plan
  selects for OS, in policy order, continuing after a failure and never
  starting an unplanned suite. The hosted-runner exceptions moved into
  the policy: a suite listed there runs with --allow-incomplete, and each
  case its report records as not run must match an allowance's suite,
  stage, case, path and reason exactly;
- summary --os OS: the job summary, with today's statement of scope.

scripts/check_ci_results.py derives the plan again and requires the
plan job's to equal it, the jobs to have the planned results, and
evidence only from artifacts of the current attempt whose bindings name
this run and attempt. An earlier attempt's artifact never satisfies the
current one, so re-running only the failed jobs cannot pass.
@novelKR
ci: drive the workflow from the plan with a required gate
6739142 
The workflow keeps its name, the contracts matrix (contracts (macos-14),
contracts (ubuntu-24.04)), the toolchain step, the exact validator step
and the dg0-<os>-<attempt> artifacts, and adds:

- Plan: runs the CI policy tests, then scripts/ci_plan.py;
- Repository checks: whitespace (git diff --check, now a real CI check)
  and the planned non-Rust stages, without Rust;
- contracts: only when the plan selects Rust; the suites and the
  hosted-runner exceptions come from the policy;
- Required checks: always runs and passes only when this run and
  attempt did what the plan requires.

Pull requests and pushes to main are planned; branch pushes no longer
duplicate pull-request runs. A daily scheduled run and every manual run
are full: the scheduled run is the compensating control that keeps
qualifying the whole repository while pull requests and main pushes run
only affected checks. A newer push to a pull request cancels its older
run. No expression is expanded inside a script, and every checkout
drops its credentials.
@novelKR
docs(planning): describe affected-check CI in the verification plan
de87e2b 
The verification plan's commands, status list, suite paragraph and
evidence rules now describe the affected-check model: path classes,
full-coverage triggers, the daily scheduled full run as its compensating
control, not_selected_by_plan, policy-listed hosted exceptions and the
current-attempt gate. The CS-RG sections are unchanged.

The sentence that CodeSpace's existing CI checks still run for
documentation PRs is out of date since CodeSpace #70; it now says what
CodeSpace main (ab0341b) runs for a documentation-only change, as its
workflow, policy and docs-only run 36336288225 show.

The Korean counterpart was reviewed and only the planning-verification
pair's hashes were recorded.
@novelKR
ci: validate exact leg and allowance evidence
094b0b7 
Require changed-paths.json, both leg records and every current-run artifact name to match the current attempt's plan. Earlier attempts remain ignored and can never satisfy it; unexpected unsuffixed artifacts now fail instead of being mislabeled as earlier evidence.

Validate incomplete suites stage by stage: each allowed not-run case must appear in its policy-named incomplete stage, every other stage must pass, and duplicate or unexplained cases fail.
@novelKR
Merge main into codex/ci-affected-validation after #19
556651c
@novelKR
novelKR merged commit 27fafeb into main Sep 28, 2026
5 checks passed
@novelKR novelKR mentioned this pull request Sep 28, 2026
Open
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@novelKR