Restore RFC 008 validation contracts - #1111
Conversation
burtenshaw
added
feature
size: extra-large
Cursor
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Alignment Review Report
Two-tier review of the RFC 008 "contracts" slice. Net: clean, faithful to the RFC, no mechanical fixes required. One process flag for a human.
Automated Checks
- Lint: PASS (this PR's code).
ruff format --check,usort check, andruff check src/ tests/all pass on the changed files undersrc/openenv/validation/andtests/test_validation/..claude/hooks/lint.shexits non-zero, but only because itsruff format … envs/pass flags ~26 pre-existing unformatted files underenvs/that this PR does not touch. - Debug code: CLEAN (this PR's code).
.claude/hooks/check-debug.shreportsprint(/TODO hits, but every one is in pre-existing files (src/openenv/core/**,cli/**,auto/**) — none insrc/openenv/validation/. Nobreakpoint()/pdb. - Tests: PASS.
tests/test_validation/65 passed; full suite 1597 passed / 133 skipped (expected optional-dep skips);scripts/sync_validation_schemas.py --checkreports schemas in sync.
Open RFCs Context
- RFC 008 — Environment Auto-Validation (
rfcs/008-environment-auto-validation.md, Status: In Review, author @zkwentz). This PR is the RFC's "PR2 — contracts": signature detection surface, parser/grader/provider protocols, normalized-manifest + report JSON Schemas, and severity policy v1. - Other open RFCs (000/001/002/003/005 In Review; 010 Draft) don't intersect these changes.
Tier 1: Fixes Required
None. New modules are import-clean (no circular imports; every __all__ name resolves; from openenv.validation import * works), fully typed, and lint-clean. Parsers are declared pure-reads and there's no credential handling, so no security concerns.
Tier 2: Alignment Discussion
Principle Conflicts
None identified. Checked against INVARIANTS.md:
- Client-server separation — validation is a standalone subsystem; no
server/imports. - Rewards in environment — the contracts validate verifier sanity (oracle→max, floor gap); they do not externalize reward computation.
- Agent isolation / "agents cannot reset" —
set_stateis an infra-side validation capability, andruntime.oracle_containmentstays a required check so the oracle is withheld from the agent at serve time. - Pydantic serialization — all wire types (
NormalizedManifest,ValidationReport,CheckResult, policy models) are Pydantic withextra="forbid".
RFC Conflicts
No design conflict — the implementation matches RFC 008 exactly. I cross-checked committed policies/severity-v1.json against the RFC's stated rollup and it agrees to the number: 45 check ids = 34 fail / 10 warn / 1 advisory, 33 local-lane, with levels/lanes/severities matching the 44-test mapping (and EXPECTED_POLICY in conftest.py mirrors it as a drift guard).
One process flag:
ALIGNMENT FLAG: Freezing normative contracts while the RFC is still In Review
- RFC at stake: RFC 008 (Status: In Review)
- The concern: This PR commits hard-to-revise artifacts — the report/manifest JSON Schemas (
report_schema_version"1"), the check-id taxonomy, and the v1 severity mapping — before the RFC that defines them is finalized. Any review-driven change to a check id, severity, or schema field must then move in lockstep acrossseverity-v1.json,report.schema.json, and theEXPECTED_POLICYmirror. Worth an explicit "we accept freezing these now" from the RFC owner. - Suggested reviewer: @zkwentz (RFC 008 author); cc @Darktex (owns the principles/invariants this touches)
Summary
- 0 mechanical issues to fix (Tier 1)
- 1 alignment point for human review (Tier 2, process): freezing RFC-008 contracts while In Review
- 0 RFC design conflicts — severity policy and schemas match RFC 008 exactly
Sent by Cursor Automation: Pre-review
| severity: Severity | ||
|
|
||
|
|
||
| class DeclarationBounds(BaseModel): |
There was a problem hiding this comment.
FYI (non-blocking): DeclarationBounds ships here and is loaded as SeverityPolicy.bounds, but nothing enforces these ceilings against a manifest's declared oracle_tolerance / floor_margin / variance_tolerance / episode_timeout_s yet — these field names appear only in this class. Enforcement belongs to the static.manifest grader in a later implementation PR, which is consistent with RFC 008's stacked-delivery plan. Flagging so the wiring isn't lost between the contracts PR and the implementation PR.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
| signature: SignatureKind | ||
|
|
||
| def parse(self, package_root: Path) -> NormalizedManifest: ... |
| requires_provider: frozenset[ProviderCapability] | ||
| depends_on: tuple[str, ...] | ||
|
|
||
| def applies_to(self, manifest: NormalizedManifest) -> bool: ... |
|
|
||
| def applies_to(self, manifest: NormalizedManifest) -> bool: ... | ||
|
|
||
| def run(self, subject: Subject) -> CheckResult: ... |
|
|
||
| base_url: str | ||
|
|
||
| def exec(self, argv: list[str], timeout_s: float) -> ExecResult: ... |
|
|
||
| def exec(self, argv: list[str], timeout_s: float) -> ExecResult: ... | ||
|
|
||
| def stop(self) -> None: ... |
| *, | ||
| network: NetworkPolicy | None = None, | ||
| env_vars: dict[str, str] | None = None, | ||
| ) -> RunningSubject: ... |
2 participants
This PR reapplies #1045 on #1110. Stack: #1110 → #1111 → #1112.