Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/HIERARCHY_LIVE_PREPARATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ rechecked quotes, the enforced cap, and the consents, then bind it with
matches the protocol, the current software parameters, and the same teacher
endpoint. See [the study guide](HIERARCHY_STUDY.md) for the exact commands.

## Prepared proposal for issue #20
## Prepared live proposal tracked in issue #74

The 2026-09-30 local preparation retained 14,960 rows: 10,451 train, 917
validation, 902 calibration, and 2,690 test. It excluded 484 rows without the
Expand Down Expand Up @@ -104,6 +104,11 @@ sharing; validation, calibration, and test remain excluded. Selection approval
does not authorize the held-out phase. The teacher's moving alias must be
rechecked and recorded before selection; native Jev remains pinned.

The 2026-10-03 owner decision separates delivered software in #20/#21 from the
original live requirements retained in #74 and epic #6. The proposal above remains
unapproved for execution. Closing the software trackers grants no spending, data
sharing, semantic review or held-out test permission.

## Consuming prior-holdout export pairs

New exact-input exports record each task's `declared_input_fields`. Use
Expand Down
10 changes: 6 additions & 4 deletions docs/HIERARCHY_RELEASE_VERIFICATION.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Hierarchy v1 release verification

This is the software evidence map for epic [#6](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/issues/6), current through 2026-09-30. It does not approve deployment or claim a measured Jev gain. Check the linked issues and PRs for current hosted status before release.
This is the software evidence map for epic [#6](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/issues/6), tracking refreshed on 2026-10-03. It does not approve deployment or claim a measured Jev gain. Check the linked issues and PRs for current hosted status before release.

| Scope | Issue | Merged implementation |
|---|---|---|
Expand All @@ -17,13 +17,15 @@ This is the software evidence map for epic [#6](https://github.com/CompleteTech-
| GEPA wording search | [#17](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/issues/17) | [#32](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/pull/32) |
| Public CLI, API, and package | [#18](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/issues/18) | [#33](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/pull/33) |
| Synthetic examples and traces | [#19](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/issues/19) | [#34](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/pull/34) |
| Offline preregistered study runner and holdout gates | [#20](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/issues/20), **open** | [#35](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/pull/35), [#37](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/pull/37), [#38](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/pull/38), [#41](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/pull/41); a completed independent review of a real live manifest remains open |
| Integration and final release gate | [#21](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/issues/21), **open** | [#36](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/pull/36), merged; final all-child acceptance remains open |
| Offline preregistered study runner and holdout gates | [#20](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/issues/20), **software-complete by owner decision** | [#35](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/pull/35), [#37](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/pull/37), [#38](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/pull/38), [#41](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/pull/41); a completed independent review of a real live manifest remains open in [#74](https://github.com/Jev-Engineering/TypeWright/issues/74) |
| Integration and final release gate | [#21](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/issues/21), **software-complete by owner decision** | [#36](https://github.com/CompleteTech-LLC-AI-Research/systemone-compiler/pull/36), merged; final empirical all-child acceptance remains open in [#74](https://github.com/Jev-Engineering/TypeWright/issues/74) and epic [#6](https://github.com/Jev-Engineering/TypeWright/issues/6) |

## Verification boundaries

`tests/test_hierarchy_validation.py` covers cycles, typed references, contract drift, and conditional dominance. `tests/test_hierarchy_artifact.py` covers size limits, path-like definition references, format dispatch, and flat hash compatibility. Runtime, budget, data, evidence, architect, GEPA, compiler, metrics, examples, and study test modules cover malformed responses, model drift, split leakage, small-branch review, cancellation, retries, lost attempt receipts, strict replay, and held-out freeze order. `tests/test_hierarchy_security.py` guards the hierarchy modules against dynamic execution and unsafe serialization/process imports. These are software tests; model prompt-injection resistance is not established by them.

The CI matrix runs pytest, Ruff, wheel build, installed-wheel flat and hierarchy no-key demos, and runtime import isolation on Python 3.11–3.13. Its optional job installs the pinned SDK/DSPy/GEPA extras and runs interface tests with fake providers. No CI job uses a live key. The exact results belong to the PR's check runs; a green configuration file alone proves nothing.

Prior flat-study test-input and normalized-text exclusion lists were generated in the original local checkout under ignored `runs/hierarchy-issues-20260930/`; they are not published because hashes of low-entropy text can be sensitive. A new checkout needs independently verified exclusion lists. Live registration checks both lists against every split; a human must still review semantic and near-duplicate independence. `typewright-study manifest` now writes the complete pre-spend live manifest from a registered live protocol, `typewright-study review` binds a human review of the required attestations to that exact digest, and live selection refuses to start unless the reviewed manifest matches the protocol and current software parameters. The software can therefore produce and enforce a complete manifest, but issue #20 still needs the human inputs: a new independently human-reviewed four-split dataset, a completed review with verified price quote and externally enforced billing cap, paid and teacher-sharing approvals, and after live selection the frozen graph digests and semantic review. Paid calls and teacher-example sharing each require separate explicit authorization. Issue #21 and epic #6 remain open until their acceptance criteria are verified. The existing flat benchmark is not the new hierarchy holdout.
Prior flat-study test-input and normalized-text exclusion lists were generated in the original local checkout under ignored `runs/hierarchy-issues-20260930/`; they are not published because hashes of low-entropy text can be sensitive. A new checkout needs independently verified exclusion lists. Live registration checks both lists against every split; a human must still review semantic and near-duplicate independence. `typewright-study manifest` now writes the complete pre-spend live manifest from a registered live protocol, `typewright-study review` binds a human review of the required attestations to that exact digest, and live selection refuses to start unless the reviewed manifest matches the protocol and current software parameters. The software can therefore produce and enforce a complete manifest, but the live scope retained in issue #74 still needs the human inputs: a new independently human-reviewed four-split dataset, a completed review with verified price quote and externally enforced billing cap, paid and teacher-sharing approvals, and after live selection the frozen graph digests and semantic review. Paid calls and teacher-example sharing each require separate explicit authorization. The owner explicitly approved splitting software and live tracking on 2026-10-03. Issues #20/#21 are closed for their delivered software scope; their original reviewed-live-manifest and empirical all-child requirements remain assigned to #74 and epic #6. Both live trackers stay open until their acceptance criteria are verified. This decision supersedes the earlier conflicting close/reopen comments and does not authorize any live execution. The existing flat benchmark is not the new hierarchy holdout.

The owner decision and retained requirements are recorded on [#20](https://github.com/Jev-Engineering/TypeWright/issues/20#issuecomment-5968647207), [#21](https://github.com/Jev-Engineering/TypeWright/issues/21#issuecomment-5968647383), [#74](https://github.com/Jev-Engineering/TypeWright/issues/74#issuecomment-5968647572) and [#6](https://github.com/Jev-Engineering/TypeWright/issues/6#issuecomment-5968647770). Fresh software evidence on main `e259dac3028ba110dabe584423d3d3a77de91e6a`: five hosted checks succeeded (core Python 3.11–3.13, optional DSPy 3.3.1/3.4.0). This observes software checks only.
Loading