experiment: benchmark-driven evolvable local router tree

.github/workflows/validate.yml

@@ -2,9 +2,12 @@ name: Validate skill
 
 on:
   push:
-    branches: [main]
+    branches: [main, experiment/evolvable-router-tree]
   pull_request:
 
+permissions:
+  contents: read
+
 jobs:
   validate:
     runs-on: ubuntu-latest
@@ -13,20 +16,122 @@ jobs:
         with:
           # The spec requires the skill name to match its directory name.
           path: practical-coding
+          fetch-depth: 0
+          persist-credentials: false
       - uses: actions/setup-python@v5
         with:
           python-version: "3.12"
       - name: Install skills-ref
         run: pip install "git+https://github.com/agentskills/agentskills.git#subdirectory=skills-ref"
       - name: Validate SKILL.md
         run: skills-ref validate ./practical-coding
-      - name: Run benchmark harness tests
+      - name: Run benchmark harness unit tests
+        working-directory: practical-coding
+        run: >-
+          python -m unittest
+          benchmarks.test_benchmarks
+          benchmarks.test_stability
+          benchmarks.test_catalog
+          benchmarks.test_ladder_analysis
+          benchmarks.test_progressive_validation
+          benchmarks.test_tree_benchmarks
+          benchmarks.test_capability_environment
+          benchmarks.test_dependency_tree_validation
+          benchmarks.test_retrieval_analysis
+          benchmarks.test_retrieval_integrity
+          benchmarks.test_evolution_workflow
+          benchmarks.test_measured_transcript
+          benchmarks.test_measured_process
+          benchmarks.test_delivery_cases
+          benchmarks.test_active_runner
+          benchmarks.test_release_gate
+      - name: Validate execution and retrieval topology contracts
+        working-directory: practical-coding
+        run: |
+          python benchmarks/tree_validation.py --self-test
+          python benchmarks/dependency_tree_validation.py --self-test
+          python benchmarks/retrieval_validation.py --self-test
+          python benchmarks/retrieval_analysis.py /dev/null --self-test
+      - name: Validate explicit evolution workflow contract
         working-directory: practical-coding
-        run: python -m unittest benchmarks.test_benchmarks benchmarks.test_stability benchmarks.test_catalog benchmarks.test_ladder_analysis benchmarks.test_progressive_validation
+        run: python benchmarks/evolution_workflow_validation.py --self-test --output benchmark-results/evolution-workflow-contract.json
+      - name: Check manual-only Decision layout
+        run: |
+          test ! -e practical-coding/references/decision.md
+          test -e practical-coding/references/manual/decision.md
+      - name: Check progressive Retrieval layout
+        run: |
+          test -e practical-coding/references/retrieval/SKILL.md
+          test -e practical-coding/references/retrieval/direct.md
+          test -e practical-coding/references/retrieval/discovery.md
+          test -e practical-coding/references/retrieval/evidence.md
+          test -e practical-coding/references/retrieval/structural.md
+          test -e practical-coding/benchmarks/capability_manifest.json
+      - name: Ensure removed ranked-search integration is absent from active surfaces
+        run: |
+          ! grep -R -i -E 'fff-style|pi-fff|\bFFF\b' \
+            practical-coding/SKILL.md \
+            practical-coding/AGENTS.md \
+            practical-coding/README.md \
+            practical-coding/README_zh.md \
+            practical-coding/CONTRIBUTING.md \
+            practical-coding/examples \
+            practical-coding/references \
+            practical-coding/agents \
+            practical-coding/docs/CAPABILITY_LAYER.md \
+            practical-coding/benchmarks/README.md \
+            practical-coding/benchmarks/capability_manifest.json
       - name: Check Codex default_prompt references the skill as $skill-name
         run: grep -qF '$practical-coding' practical-coding/agents/openai.yaml
+      - name: Ensure retired execution-state experiment is absent
+        run: |
+          retired_files="$({
+            find practical-coding/runtime practical-coding/tests practical-coding/benchmarks \
+              -maxdepth 1 -type f \
+              \( -name '*skill_state*' -o -name 'test_skill_state*' \) \
+              -print 2>/dev/null || true
+            find practical-coding/evolution/experiments \
+              -maxdepth 1 -type f -name 'skill-state-*' -print 2>/dev/null || true
+          })"
+          if test -n "$retired_files"; then
+            printf 'retired execution-state files remain on an active surface:\n%s\n' "$retired_files"
+            exit 1
+          fi
+          test ! -e practical-coding/docs/SKILL_STATE.md
+          test ! -e practical-coding/docs/SKILL_STATE_HOST.md
+          test ! -e practical-coding/docs/SKILL_STATE_INVARIANTS.md
+          test -e practical-coding/evolution/rejected/execution-state/README.md
+          ! grep -R -E \
+            'runtime/skill_state|SKILL_STATE_MODEL_GATE|state-history-free|Execution State Projection|"execution_state"' \
+            practical-coding/SKILL.md \
+            practical-coding/AGENTS.md \
+            practical-coding/README.md \
+            practical-coding/README_zh.md \
+            practical-coding/agents/openai.yaml \
+            practical-coding/benchmarks/README.md \
+            practical-coding/benchmarks/tree_topology.json
       - name: Ensure legacy local graph runtime is not reintroduced
         run: |
           test ! -e practical-coding/runtime/codebase_memory.py
           test ! -e practical-coding/runtime/_codebase_memory_impl.py
           ! grep -R "runtime/codebase_memory.py" practical-coding/SKILL.md practical-coding/AGENTS.md practical-coding/README.md practical-coding/references
+      - name: Run pinned evaluator regression benchmark
+        working-directory: practical-coding
+        run: python benchmarks/benchmark_retrieval_integrity.py --output benchmark-results/evaluator-regression.json
+      - name: Run prompt and executable-oracle controls
+        working-directory: practical-coding
+        run: python benchmarks/benchmark_readiness.py --output benchmark-results/readiness.json
+      - name: Package tracked source and revision
+        working-directory: practical-coding
+        run: |
+          mkdir -p benchmark-results
+          git rev-parse HEAD > benchmark-results/source-revision.txt
+          git archive --format=tar --output=benchmark-results/source.tar HEAD
+          sha256sum benchmark-results/source.tar > benchmark-results/source.sha256
+      - name: Upload auditable validation artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: validation-${{ github.sha }}
+          path: practical-coding/benchmark-results/
+          if-no-files-found: error
+          retention-days: 7

AGENTS.md

@@ -1,33 +1,23 @@
-# Practical Coding
+# Practical Coding maintenance
 
-This repository is an Agent Skill. Apply [`SKILL.md`](SKILL.md) when working from this checkout.
+This repository is an Agent Skill. Apply [`SKILL.md`](SKILL.md); it is the runtime authority. Do not duplicate its routers here or preload reference siblings. Execution, Retrieval, provider capabilities, output transport, and maintenance are separate concerns.
 
-## Runtime model
+## Boundaries
 
-1. Apply the Core and stay Direct unless one present unresolved event matches the Router.
-2. Route only Debugging, Decision, or Implementation; load at most one reasoning reference for the current event.
-3. Complete routing before diagnostic, decision-research, or change-mapping source work. The selected reference is the next read.
-4. Keep retrieval orthogonal. Unknown paths, callers, consumers, and data flow are retrieval questions, not Implementation events.
-5. Contract to the smallest affected surface as soon as the cause, choice, contract, invariant, or evidence boundary is established.
+Core knows only its immediate Debugging and Implementation children. They remain leaves until independent capability ablation earns a child. Retrieval expands through each loaded node's immediate-child router. Unknown locations or relationships alone do not activate Implementation. Decision and clarification stay explicit-only manual modes; alternatives found while coding do not activate them.
 
-## Event Router
+No automatic route opens maintenance workflows. Ordinary runtime does not read `evolution/`. Preserve the retired execution-state proposal and rejected experiments as historical evidence; do not reactivate them through synonyms or compatibility wrappers. Never add a provider-named node or restore the removed ranked-search integration.
 
-| Present unresolved event | Reference |
-|---|---|
-| Observed failure still lacks an evidenced cause | [`references/debugging.md`](references/debugging.md) |
-| Material user-owned implementation choice changes the next action | [`references/decision.md`](references/decision.md) |
-| Unknown contract/invariant, coordinated guarantee, material risk boundary, or evidence plan blocks safe execution | [`references/implementation.md`](references/implementation.md) |
+## Changes and evidence
 
-A known target and settled behavior/boundary/check stay Direct even when risk nouns are present. A read-only mapping request is Direct plus Retrieval.
+Read `CONTRIBUTING.md` for maintenance policy. Before changing runtime behavior, record evidence, one attributable hypothesis, and frozen validation under `evolution/`; consolidate repeated mechanisms in its wiki. Do not put benchmark case nouns or expected numeric routes into runtime wording. Promote/split/merge/collapse/remove nodes only after quality-qualified ablation, not symmetry or file count.
 
-Requirements interviewing is explicit-only through [`references/manual/clarification.md`](references/manual/clarification.md).
+Iteration uses n=1. Freeze the candidate before the complete n=3 baseline/no-skill comparison. Keep historical results immutable. A scorer fix invalidates both comparison arms, not just the candidate. Public regressions are not held-out generalization evidence.
 
-## Retrieval
+## Active evaluation
 
-Use known source, then bounded/ranked search, then an already-available structural capability when it materially reduces relationship discovery. Use exhaustive coverage or external authoritative sources only when the claim requires them. Source remains authoritative.
+`benchmarks/retrieval_validation.py` owns the measured runner for source analysis and executable delivery. `dependency_tree_validation.py` selects the execution axis; historical monkey-patch adapters are not active entry points. `release_gate.py` checks the complete engineering matrix and raw receipts. Commands, acceptance targets, and limitations are in `benchmarks/DELIVERY_READINESS.md`.
 
-Read [`references/navigation.md`](references/navigation.md) only for substantial retrieval. Missing graph/ranked capabilities fall back without installing or persisting tooling solely for retrieval.
+Every measured profile requires all pinned providers. Initialize indexes, download models, resolve dependencies, and warm first builds before measurement. Missing providers abort; never silently use mocks or fallback as provider evidence. Normal runtime still supports bounded native-source fallback.
 
-## Evolution
-
-`evolution/` is maintainer knowledge and must not enter ordinary runtime context. During Skill maintenance, record mechanisms and failed changes there before modifying another runtime rule. Iterations use n=1; only a frozen release candidate receives the complete n=3 matrix.
+Bind candidate, baseline, harness, model, provider versions, task matrix, and raw artifacts to the run. Missing telemetry is unknown, not zero. A path mentioned in a command is not a verified read. Preserve failures, exits, and material evidence through output compaction. Do not claim model performance from deterministic tests or label targets as measurements.

CONTRIBUTING.md

@@ -4,14 +4,37 @@ Practical Coding optimizes for the smallest quality-qualified runtime contract,
 
 ## Runtime boundaries
 
-- Keep `SKILL.md` compact and route-agnostic outside the Event Router.
-- Route only a present unresolved Debugging, Decision, or Implementation event.
-- Load at most one reasoning reference for the current event.
-- Keep source/context retrieval independent from reasoning selection.
-- Requirements interviewing remains explicit-only.
+- Keep `SKILL.md` compact. Core knows only the immediate automatic execution children.
+- Route only a present unresolved Debugging or Implementation blocker.
+- Decision and requirements interviewing remain explicit-only manual modes.
+- Keep the execution tree, Retrieval tree, capability providers, output transport, and maintenance workflows separate.
+- A loaded node may name only its immediate child. Do not place every depth decision in Core.
 - Do not introduce mandatory plans, reviews, tests, documents, Git workflows, workers, or lifecycle ceremony.
 
-Unknown locations, callers, consumers, or relationships are retrieval questions. They become Implementation only when a requested coordinated change has an unresolved governing contract or material risk/evidence boundary.
+Unknown locations, callers, consumers, or relationships are Retrieval questions. They become Implementation only when a requested coordinated change has an unresolved governing contract or material risk/evidence boundary.
+
+## Retrieval-tree discipline
+
+The active Retrieval path is:
+
+`Retrieval Root -> R0 Direct Locate -> R1 Ranked Discovery -> R2 Evidence Expansion -> R3 Structural Trace`.
+
+- R0 handles known or narrowly identifiable targets.
+- R1 finds ranked candidates when intent is known but location is not.
+- R2 builds the smallest distributed evidence set required by unresolved claims.
+- R3 resolves relationship, flow, dependency, ownership, and impact questions.
+
+Depth represents the unresolved information problem. Tool names, brands, and installation methods do not belong in node identities or routing conditions. Every material conclusion must be verified in current source.
+
+Navigation has one narrower job: answer which bounded repository area should be searched. It must not absorb semantic search, evidence expansion, or graph tracing.
+
+## Capability providers
+
+Normal runtime use remains portable: use the strongest already-available provider appropriate to the current node, then fall back losslessly to bounded source search.
+
+Provider-enabled benchmark claims use `benchmarks/capability_manifest.json` and `benchmarks/dependency_tree_validation.py`. That profile requires `zg`, `codebase-memory-mcp`, and `rtk`; missing or failed providers abort the run. Provider probes, indexes, dependency resolution, and first-build warm-up occur before Codex starts and are never merged into compared token, duration, or tool-call fields.
+
+Do not weaken the benchmark by adding an allow-missing flag, silently falling back, estimating setup tokens, or resuming a measured cell without its matching `capability-setup.json` receipt.
 
 ## Evolution before wording
 
@@ -26,18 +49,16 @@ Do not add benchmark case nouns to runtime wording. A module must have an observ
 
 ## Benchmark discipline
 
-- Quality, safety, and build/reachability precede routing and cost.
+- Quality, safety, build/reachability, and environment parity precede routing and cost.
 - Iteration runs use n=1. Run n=3 only for a frozen candidate believed ready to release.
 - Current-only runs may compare against prior published reports offline, but are not paired ranking evidence.
 - Keep deterministic prompts and oracles contract-consistent; do not reward behavior the prompt forbids.
+- Every arm in a paired task sees the same initialized providers and repository warm-up.
+- Setup is separately auditable but excluded from comparison; measured execution begins only after setup succeeds.
 - Add a test when a newly discovered mechanism or scorer invariant would otherwise regress.
 
-Public regression covers Delivery, Debug, Decision, Router, and Native Behavior. Real-repository held-out coverage validates delivered evidence, zero spontaneous requirements interviewing, event selection, and retrieval scope.
-
-## Retrieval capabilities
-
-Known source → bounded/ranked search → structural capability when useful → bounded exhaustive or authoritative external evidence only when required. FFF-style search, ordinary search, LSP/AST, and Codebase Memory are optional capabilities. Verify material claims against current source and disclose coverage gaps.
+Public regression covers Delivery, Debug, Decision, Router, and Native Behavior. Real-repository held-out coverage validates delivered evidence, zero spontaneous manual activation, execution-node selection, progressive Retrieval disclosure, and provider use.
 
 ## Mature implementation first
 
-For a non-trivial new capability, inspect maintained prior art, extract the smallest fitting mechanism, verify maintenance/license/API fit, and keep the result removable. Do not copy an entire expert workflow into the Core.
+For a non-trivial new capability, inspect maintained prior art, extract the smallest fitting mechanism, verify maintenance/license/API fit, and keep the result removable. Do not copy an entire expert workflow into Core.