diff --git a/docs/commands-by-level.md b/docs/commands-by-level.md index c10ac26..04486dc 100644 --- a/docs/commands-by-level.md +++ b/docs/commands-by-level.md @@ -13,3 +13,4 @@ Generated by `npx tsx scripts/commands.ts`. Do not edit by hand. | delivery | `make levels LEVEL=delivery` | `make prove LEVEL=delivery` | `make exercise LEVEL=delivery TRACK=a` | | sharing | `make levels LEVEL=sharing` | `make prove LEVEL=sharing` | `make exercise LEVEL=sharing TRACK=a` | | workflows | `make levels LEVEL=workflows` | `make prove LEVEL=workflows` | `make exercise LEVEL=workflows TRACK=a` | +| capstone | `make levels LEVEL=capstone` | `make prove LEVEL=capstone` | `make exercise LEVEL=capstone TRACK=a` | diff --git a/docs/factory/f9-capstone.md b/docs/factory/f9-capstone.md new file mode 100644 index 0000000..3bda09b --- /dev/null +++ b/docs/factory/f9-capstone.md @@ -0,0 +1,105 @@ +# F9 Capstone: rebuild the workbook and template features through your factory + +**Stage:** all of them. **Layer:** every level. **Workflows:** `feature-bdd` for the cache and templates, `feature-sdd` for the workbook, `bug-tdd` for the template id boundary. + +## Goal + +You do not write the workbook, template or cache code. Your factory does. You supply the red suites, the factory builds until they are green, and you compare its tree with the reference answer. The score is whether the gates held, not whether code appeared. + +## What is built and what is design + +- Built and tested: the red suites in `specs/34`, `specs/35` and `specs/37`, the workflows that run them, and the reference code the capstone level owns (see `docs/levels/capstone.md` for the paths). +- Built, not run for you: the factory runs against your own GitHub issues and your own API key. This guide gives the steps. No run log from a capstone is committed, so do not expect one in this repo. +- Design: the order of tickets below is a suggestion. Any order works if each ticket's red suite fails first. + +## The setup + +1. Generate the two snapshots: + + ``` + make levels LEVEL=workflows + make levels LEVEL=capstone + ``` + +2. Work in a stripped copy of `supplied/workflows`. The copy keeps the engine, the workflows, the specs and the golden tests, and loses the workbook, template and cache code. +3. Install and confirm the suites are red: + + ``` + cp -R supplied/workflows /tmp/capstone-start + npx tsx scripts/capstone-strip.ts /tmp/capstone-start + cd /tmp/capstone-start + npm ci + npx vitest run specs/34 specs/35 specs/37 + ``` + + Every file fails to import code that is not there yet. That is the starting line. If a suite is already green, stop: the strip did not run. + +4. Keep `supplied/capstone` closed. It is the reference answer, and reading it first turns the capstone into copying. + +## The tickets + +File one GitHub issue per row. The issue names the spec folder and says nothing about how to build it. + +| Ticket | Spec folder | Workflow | First red | +| ------------------------------------ | ------------------------------------------ | ------------- | ------------------------------------ | +| Workbook plan, schema, check and PDF | `specs/34` | `feature-sdd` | a property test over any plan | +| Exact-prompt cache | `specs/35/golden/cache.golden.test.ts` | `feature-bdd` | a repeat prompt spends zero tokens | +| Templates from good renders | `specs/35/golden/templates.golden.test.ts` | `feature-bdd` | a matching request fills only slots | +| Template id boundary | `specs/37` | `bug-tdd` | an id with a path meaning is refused | + +Run them in this order. The template tickets use the cache, and the id ticket guards the route the templates ticket adds. + +## Run each ticket + +``` +npx tsx factory/engine/main.ts --workflow=feature-bdd +``` + +Use the workflow named in the table. Then read, in order: + +1. `specs//run.log` for each step and round. +2. The `red` result: it must say the golden test failed on the base commit. +3. The `lock` result: it must say no test file changed during the build. +4. The `plan-drift` result: it must say the diff stayed inside `plan.md`. + +A ticket is done only when all three held and the spec folder is green. + +## What the red suites teach + +- **Zero model calls on a hit.** The golden test counts calls to a fake client. A cache that returns early but still calls the model fails it. +- **Slots only.** A template holds slots and no filled content. The test checks the schema the model is offered, so a template that stores a whole sheet fails. +- **Layout is code.** A template fill is rendered with the band rules. A model that returns more items than the band allows is stopped by the check, not by the prompt. +- **Customer errors are generic.** A provider overload never reaches the response body. +- **A strict id.** The mark-good route takes the id the api issued and nothing else. A traversal id is refused before any lookup. + +## Compare with the reference + +When all four tickets are done, in the repo you worked in: + +``` +diff -r shared/src /supplied/capstone/shared/src +diff -r api/src /supplied/capstone/api/src +make prove LEVEL=capstone +``` + +Differences are expected. A different name or file split is fine. A missing guard is not: for each boundary the golden tests cover, find where your tree enforces it, and where the reference does. + +## Proof that bites + +Pick one guard and break it in your tree. + +- In the cache key, drop the band. A golden test named for the band fails. +- In the id schema, allow any string. The traversal cases in `specs/37` fail. +- In the template store, save the filled sheet. The slots-only test fails. + +Restore each one and confirm the suite is green again. A guard you broke without seeing a failure is a guard no test holds, which means your build has a hole the reference does not. + +## Exercise + +Track A: run all four tickets and record, for each, how many rounds the loop took and where it stopped for a person. Track B: delete the `lock` check from one workflow, run the template ticket, and read whether the builder edited a golden test to get green. + +## Discussion + +1. Which ticket needed a person, and which gate told you so? +2. Your tree is green and differs from the reference. What evidence would make you trust it anyway? +3. Which red suite would you write first for the next feature, before any code exists? diff --git a/docs/levels/capstone.md b/docs/levels/capstone.md new file mode 100644 index 0000000..bd0d807 --- /dev/null +++ b/docs/levels/capstone.md @@ -0,0 +1,48 @@ +# Capstone: rebuild a feature through your own factory + +**Layer:** everything before it. **Proof:** `make prove LEVEL=capstone`. + +## Goal + +Take the specs that go red when the feature is stripped out and turn them green by running your own factory, not by typing the code. The finished tree is the reference answer you compare against afterwards. + +## The problem + +Every earlier level taught one part of the factory on a small task. None of them showed the parts working together on a feature that spans the shared package, the api and a cache. Until one real feature goes through the whole line, nobody knows which gate is missing. + +## Without + +Ask a model to "add workbooks and templates" in one session. It writes code and tests together, so the tests agree with the code whatever the code does. Nothing was red first, and nothing says the template path really skips the model. + +## The code + +- `shared/src/workbook-*.ts`, `shared/src/render-workbook.ts` and `shared/src/template.ts` plan, validate, check and render a multi-chapter workbook and a template made of slots. +- `api/src/cache/` holds the exact-prompt cache. `api/src/templates/` distills a good sheet into a template, names it with a strict id and stores it. +- `api/src/agent/workbook-agent.ts`, `fill-template.ts` and `template-eval.ts` run the workbook agent, fill only a template's slots, and compare output tokens with and without a template. +- `specs/34`, `specs/35` and `specs/37` hold the red suites. They belong to the workflows level and fail once the code is stripped. +- `scripts/capstone-strip.ts` lists those files in one place and deletes them from a copy of the tree. The earlier levels own the same files, because the api server imports them. + +## With + +On a stripped copy the three spec folders fail because the code they import is missing. On `supplied/capstone` they pass, and the unit tests beside the code pass too. + +## Run it + +1. Type `make levels LEVEL=workflows`, then `cp -R supplied/workflows /tmp/capstone-start`, `npx tsx scripts/capstone-strip.ts /tmp/capstone-start`, `cd /tmp/capstone-start` and `npm ci`. +2. Type `npx vitest run specs/34 specs/35 specs/37`. Expect red, and read the first failure. +3. Run your factory on a ticket that names the spec folder. Follow [the capstone factory guide](../factory/f9-capstone.md). +4. Type `make levels LEVEL=capstone` and diff your result against `supplied/capstone` for the files listed above. + +## Prove it + +`make prove LEVEL=capstone` runs the three spec folders and the unit tests beside the workbook, template, cache and route code. It passes only when all of them pass. + +![The proof passing](../recordings/capstone.gif) + +## Exercise + +Track A: from a clean `supplied/workflows`, get `specs/37` green through the bug workflow and read the run log for the step that first produced a red test. Track B: get `specs/35` green, then change the cache key so it ignores the band and watch which golden test fails. + +## Discussion + +Where did your factory stop and ask a person, and was that the right place? Which gate would you add before running it on a feature you could not check yourself? diff --git a/docs/recordings/capstone.cast b/docs/recordings/capstone.cast new file mode 100644 index 0000000..abb497b --- /dev/null +++ b/docs/recordings/capstone.cast @@ -0,0 +1,4 @@ +{"version": 3, "term": {"cols": 80, "rows": 24}, "timestamp": 1793396600, "command": "make prove LEVEL=capstone", "env": {"SHELL": "/bin/zsh"}} +[0.012, "o", "$ make prove LEVEL=capstone\r\n"] +[6.0, "o", "$ make prove LEVEL=capstone\r\nnpm run prove -- capstone\r\n\r\n> claude-code-workshop@0.1.0 prove\r\n> tsx scripts/prove/run.ts capstone\r\n\r\nPASS capstone: the workbook and template specs are green on the finished tree (workbook plan, schema, check, pdf, template store, cache and routes all pass their specs)\r\n"] +[0.0, "x", "0"] diff --git a/docs/recordings/capstone.gif b/docs/recordings/capstone.gif new file mode 100644 index 0000000..9326bd1 Binary files /dev/null and b/docs/recordings/capstone.gif differ diff --git a/levels.json b/levels.json index 88ce165..d17c59c 100644 --- a/levels.json +++ b/levels.json @@ -210,6 +210,29 @@ "docs/recordings/workflows.gif", "stryker.config.json", "stryker.vitest.config.ts", - "docs/factory/**" + "docs/factory/f0-1-anatomy.md", + "docs/factory/f0-overview.md", + "docs/factory/f1-plan.md", + "docs/factory/f10-assignments.md", + "docs/factory/f11-cloud-sessions.md", + "docs/factory/f12-model-routing.md", + "docs/factory/f13-spend-and-safety.md", + "docs/factory/f2-design.md", + "docs/factory/f3-build.md", + "docs/factory/f4-test.md", + "docs/factory/f5-review.md", + "docs/factory/f6-deploy.md", + "docs/factory/f7-maintain.md", + "docs/factory/f8-workflows.md", + "docs/factory/faq.md" + ], + "capstone": [ + "scripts/prove/capstone.ts", + "docs/levels/capstone.md", + "docs/factory/f9-capstone.md", + "docs/recordings/capstone.cast", + "docs/recordings/capstone.gif", + "scripts/capstone-strip.ts", + "scripts/capstone-strip.test.ts" ] } diff --git a/scripts/capstone-strip.test.ts b/scripts/capstone-strip.test.ts new file mode 100644 index 0000000..97f3b76 --- /dev/null +++ b/scripts/capstone-strip.test.ts @@ -0,0 +1,22 @@ +import { existsSync, mkdirSync, mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { CAPSTONE_CODE, stripCapstone } from "./capstone-strip.ts"; + +describe("capstone strip", () => { + it("names only paths that exist in the finished tree", () => { + const missing = CAPSTONE_CODE.filter((p) => !existsSync(p)); + expect(missing).toEqual([]); + }); + + it("removes the feature and leaves everything else", () => { + const root = mkdtempSync(join(tmpdir(), "strip-")); + mkdirSync(join(root, "api/src/cache"), { recursive: true }); + writeFileSync(join(root, "api/src/cache/prompt-cache.ts"), "x"); + writeFileSync(join(root, "api/src/server.ts"), "keep"); + expect(stripCapstone(root)).toEqual(["api/src/cache"]); + expect(existsSync(join(root, "api/src/cache"))).toBe(false); + expect(existsSync(join(root, "api/src/server.ts"))).toBe(true); + }); +}); diff --git a/scripts/capstone-strip.ts b/scripts/capstone-strip.ts new file mode 100644 index 0000000..4b1dcfd --- /dev/null +++ b/scripts/capstone-strip.ts @@ -0,0 +1,42 @@ +import { existsSync, rmSync } from "node:fs"; +import { join } from "node:path"; + +// The feature the capstone rebuilds. Stripping these from a copy of the tree gives the red start: +// the specs in specs/34, 35 and 37 import them, so they fail until your factory writes them back. +export const CAPSTONE_CODE = [ + "shared/src/workbook-check.ts", + "shared/src/workbook-check.test.ts", + "shared/src/workbook-pdf.ts", + "shared/src/workbook-pdf.test.ts", + "shared/src/workbook-plan.ts", + "shared/src/workbook-plan.test.ts", + "shared/src/workbook-schema.ts", + "shared/src/workbook-schema.test.ts", + "shared/src/render-workbook.ts", + "shared/src/render-workbook.test.ts", + "shared/src/template.ts", + "shared/src/template.test.ts", + "api/src/cache", + "api/src/templates", + "api/src/agent/fill-template.ts", + "api/src/agent/template-eval.ts", + "api/src/agent/workbook-agent.ts", + "api/src/agent/workbook-agent.test.ts", + "api/src/templates-good-route.test.ts", + "api/src/workbook-server.test.ts", +]; + +export function stripCapstone(root: string): string[] { + const removed = CAPSTONE_CODE.filter((p) => existsSync(join(root, p))); + for (const p of removed) rmSync(join(root, p), { recursive: true, force: true }); + return removed; +} + +if (import.meta.url === `file://${process.argv[1]}`) { + const target = process.argv[2]; + if (!target) { + console.error("Usage: npx tsx scripts/capstone-strip.ts "); + process.exit(1); + } + console.log(`removed ${stripCapstone(target).length} paths from ${target}`); +} diff --git a/scripts/levels.ts b/scripts/levels.ts index bb99d43..847f2ab 100644 --- a/scripts/levels.ts +++ b/scripts/levels.ts @@ -14,6 +14,7 @@ export const LEVEL_ORDER = [ "delivery", "sharing", "workflows", + "capstone", ] as const; export type LevelName = (typeof LEVEL_ORDER)[number]; diff --git a/scripts/prove/capstone.ts b/scripts/prove/capstone.ts new file mode 100644 index 0000000..5946da5 --- /dev/null +++ b/scripts/prove/capstone.ts @@ -0,0 +1,33 @@ +import { spawnSync } from "node:child_process"; +import type { Proof } from "./hand-built.ts"; + +// Claim: the workbook and template features are green against the specs that were red on the workflows snapshot. +// The golden and property suites are the red half; the tests beside the code are the unit half. +const SUITES = [ + "specs/34", + "specs/35", + "specs/37", + "shared/src/workbook", + "shared/src/render-workbook", + "shared/src/template", + "api/src/cache", + "api/src/templates", + "api/src/agent/workbook-agent", + "api/src/workbook-server", + "api/src/templates-good-route", +]; + +export function proveCapstone(): Proof { + const run = spawnSync("npx", ["vitest", "run", ...SUITES, "--reporter=dot"], { + encoding: "utf8", + }); + return { + level: "capstone", + claim: "the workbook and template specs are green on the finished tree", + passed: run.status === 0, + detail: + run.status === 0 + ? "workbook plan, schema, check, pdf, template store, cache and routes all pass their specs" + : (run.stdout + run.stderr).split("\n").slice(-8).join(" | "), + }; +}