diff --git a/docs/factory/f10-assignments.md b/docs/factory/f10-assignments.md new file mode 100644 index 0000000..c67ea8b --- /dev/null +++ b/docs/factory/f10-assignments.md @@ -0,0 +1,96 @@ +# F10. Assignments: pick one, file it, run it + +**Stage:** all of them. **Workflows shown:** whichever you choose below. + +## Status: not built yet + +None of these assignments exists in the repo today. There are no issue templates (`.github/ISSUE_TEMPLATE` is absent), no red tests under `tests/assignments/`, and no structural test that checks both exist. This page is the brief for each one. The paths below named "proposed" are where the red test should go, and you create it. Paths without that word exist now. + +## Goal + +Choose one assignment, file it as an issue, write its red test first, and make your factory turn it green. Pick the workflow from F8. + +## Steps (the same for every assignment) + +1. Check you can run the factory: `make doctor`, then `npx vitest run factory/engine`. +2. File the issue with the text in your assignment's section: `gh issue create --title "" --body-file <file>`. Label it `factory:ready` (the label in `.factory/config.json`). `npx tsx factory/engine/main.ts --watch` lists what carries it. +3. Write the red test and run it. It must fail, and for the reason the issue names. A test that passes before any work proves nothing, which is what the `red` check in `factory/engine/checks/red.ts` refuses. +4. Run the workflow: `npx tsx factory/engine/main.ts <issue> --workflow=<name>`. The script loads the API key inside its own process, so keep the key off the command line. +5. Read `specs/<issue>/run.log` and `.factory/runs/<issue>/metrics.json`. If the run ended `needs-human`, the log says which check capped out. +6. Finish with `make check`, zero errors and zero warnings. Revert the fix once and watch your test fail again. + +Each red test below is skipped by default in the finished design, as `make prove-homework` is for the take-home (`HOMEWORK=1`). Use an environment variable of your own, such as `ASSIGNMENT=1`, so `make check` stays green until you start. + +## 1. A worksheet video with the HyperFrames timeline + +- **Goal:** `shared/src/render.ts` already turns a worksheet into a HyperFrames still composition (its header comment says so), and `hyperframes` is a dependency in `package.json`. Add a timeline so a worksheet plays as a short video: items appear in order, the answer key last. +- **Workflow:** `ui-visual`. +- **Issue:** "Render a worksheet as a HyperFrames video. Acceptance: every item appears once, in page order; the answer key is the final scene; `hyperframes lint` passes on the output." +- **Red test (proposed `tests/assignments/worksheet-video.test.ts`):** render a fixture from `fixtures/worksheets.json` with the new function, assert the scene count equals the item count plus one for the key, and assert the key scene is last. It fails because the function does not exist. + +## 2. Templates in a database + +- **Goal:** `api/src/templates/template-store.ts` holds templates today. Replace its storage with a database after a spike on which one. +- **Workflow:** `spike` first, then `feature-sdd`. +- **Issue:** "Spike: which store for worksheet templates, and what it rules out. Then move `template-store` onto it." File the spike alone first and read its decision record before filing the build. +- **Red test (proposed `tests/assignments/template-db.test.ts`):** save a distilled template, build a second store object on the same database file, and assert the first template is found. It fails while the store is in memory only. Read `api/src/templates/template-store.test.ts` for the current contract it must keep. + +## 3. A canary prompt with rollback when the eval drops + +- **Goal:** serve a changed prompt to a small share of requests, compare its eval score to the current one, and roll back by itself when it drops. +- **Workflow:** `llm-eval`. +- **Issue:** "Canary for `api/src/agent/prompt.ts`. A candidate prompt takes a configured share of traffic. If its pass rate falls below the current prompt's by more than a stated margin, traffic returns to the current one and the event is logged." +- **Red test (proposed `tests/assignments/canary.test.ts`):** feed a pure decision function scores for a good and a worse candidate with a fixed seed, assert the share routed, and assert the worse one is rolled back. Test the decision with recorded scores so it spends no tokens. `make eval` is the live path (`api/src/eval.ts`) and shares the cap in `api/src/agent/live-cap.ts`. + +## 4. Fuzzing the tool input + +- **Goal:** the model's tool input is validated with zod at the boundary. Fuzz that boundary and find an input that gets through and breaks a page. +- **Workflow:** `security`. +- **Issue:** "Property-test the worksheet tool input. Any input the schema accepts must render and pass `checkBand`; any it rejects must be rejected with a generic error." +- **Red test (proposed `tests/assignments/fuzz-tool-input.test.ts`):** `fast-check` is already in `package.json`. Generate arbitrary objects against the schema in `shared/src`, and assert accepted ones render. It stays red until you find and fix a case, so plant one first (a mutant that accepts an empty item list) and show the test catches it. + +## 5. The label state machine as a property + +- **Goal:** the labels in `.factory/config.json` (`factory:ready`, `factory:planned`, `plan:approved`, `needs-human`, `factory:protected-ok`) form a state machine that `factory/run.ts` drives. State its rules and check them with generated sequences of events. TLA+ is the heavier option; a `fast-check` model-based test is enough. +- **Workflow:** `feature-atdd`. +- **Issue:** "Specify the label transitions. Invariants: a build never starts without `plan:approved`; a new plan always clears `plan:approved`; `needs-human` is never cleared by the factory." +- **Red test (proposed `tests/assignments/label-machine.test.ts`):** generate event sequences against the fake deps in `factory/fake-deps.ts` and assert the invariants. Existing single cases live in `factory/run.test.ts`; yours must cover sequences, not one path. + +## 6. A new activity kind + +- **Goal:** add a new kind of activity to a worksheet, for example a clock face. +- **Workflow:** `feature-bdd`. +- **Issue:** "Add the `clock` activity kind: the model supplies a time, the code draws the face. Acceptance: it renders in the bands that allow it and `checkBand` rejects it in the others." +- **Red test (proposed `tests/assignments/clock-activity.test.ts`):** `ACTIVITY_KINDS` in `shared/src/activity-kinds.ts` is the list, and `shared/src/render-activity.ts` draws each. Assert `clock` is in the list, renders markup, and that `.claude/rules/shared.md` is honoured: a new band needs a fixture and a passing `checkBand`. + +## 7. A workflow from your own company + +- **Goal:** encode how your team really ships one kind of change. +- **Workflow:** the one you write (see F8). +- **Issue:** "Add `.factory/workflows/<name>.yaml` for <task>. Acceptance: it names its method and source, uses only checks from `CHECK_NAMES`, and dry-runs." +- **Red test:** none to author. `factory/engine/workflows.test.ts` already runs every file in the folder, so the red state is your YAML failing it. Add one check of your own for the step that matters most, with a failing input first, following `factory/engine/checks/plan-drift.test.ts`. + +## 8. A new review lens + +- **Goal:** add a reviewer that hunts one kind of defect. `reviewRound` in `factory/engine/review.ts` takes a list of lens agents, runs them in parallel, and keeps a finding only when two of three verifiers vote yes. +- **Workflow:** `llm-eval`. +- **Issue:** "Add a `performance-reviewer` agent. It reads the diff and replies with a JSON array of findings with file, line, severity and evidence." +- **Red test (proposed `tests/assignments/review-lens.test.ts`):** copy the shape of `.claude/agents/security-reviewer.md`, then using the fake host in `factory/engine/fake-host.ts`, assert a one-vote finding is dropped, as `factory/engine/review.test.ts` does for existing lenses. Also feed a diff with a planted defect and assert the lens reports it with evidence. + +## 9. An LLM-judge pre-commit hook + +- **Goal:** the opt-in paid pattern. A hook asks a model to judge a staged change, and only runs when the person turns it on. +- **Workflow:** `feature-atdd`. +- **Issue:** "Add an opt-in judge hook. Default off. When `JUDGE=1`, it sends the staged diff to a small model and blocks the commit on a `blocking` verdict. It must never send a file the secrets hook would block." +- **Red test (proposed `tests/assignments/judge-hook.test.ts`):** export the decision as a pure function like the hooks in `.claude/hooks/`, and assert: off by default, blocks on a blocking verdict, passes on a clean one, and refuses to send a diff containing a secret. `tests/structure/hook-catalogue.test.ts` shows the shape, one hook fed one bad input. + +## 10. A mod of your own + +- **Goal:** build a small change to the Claude interface. `plugins/factory-status` is the working example in this repo. +- **Workflow:** `feature-atdd`. +- **Issue:** "Add a mod that shows <something about the run> above the prompt. Acceptance: `claude plugin validate` passes and a test feeds it a fixed state and checks what it draws." +- **Red test (proposed `tests/assignments/my-mod.test.ts`):** follow `tests/structure/plugin.test.ts` and `tests/structure/commands-and-mod.test.ts`. Assert your mod's manifest is valid and its render function turns a fixed state into the text you expect. + +## Discussion + +Which assignment would change what your team ships next month? Which red test would you be unable to write, and what does that say about the issue? diff --git a/docs/factory/f8-workflows.md b/docs/factory/f8-workflows.md new file mode 100644 index 0000000..23ef6d0 --- /dev/null +++ b/docs/factory/f8-workflows.md @@ -0,0 +1,101 @@ +# F8. Workflows: write one, and choose one per task + +**Stage:** all of them. **Layer:** delivery, made configurable. **Workflows shown:** all thirteen in `.factory/workflows/`. + +## Goal + +Write a workflow file from scratch, make the repo's own tests accept it, run it on an issue, and pick the right one for each kind of ticket. + +## The schema + +`factory/engine/workflow.ts` is the whole schema. Read it before writing a line of YAML. + +- A workflow has `name`, `method`, `source`, `for`, and a list of `steps`. `manifests` defaults to false and only a dependency update needs it. +- A step has `stage` and `agent`. Optional fields: `brief`, `writes` (one of the artifact names in `factory/engine/artifacts.ts`), `startsWith`, `checks`, `loop`, `gate` (`human` or `auto`) and `phase` (`build` or `fix`). +- `checks` may only name entries of `CHECK_NAMES`: `red`, `green`, `lock`, `holdout`, `acceptance`, `properties`, `behaviour-unchanged`, `mutation`, `contract`, `visual`, `eval`, `plan-drift`, `gate`, `prepush`, `ci`, `prose`, `manifest-deps`. +- `loop: { until: [...], max: n }` repeats the agent until those checks pass. `max` is 1 to 5, and every `until` check must also be in the step's `checks`, or `loadWorkflow` throws. + +## Steps + +1. Read `.factory/workflows/docs.yaml`. It is the smallest workflow that plans, changes files and checks the result. +2. Create `.factory/workflows/typo.yaml` for a one-line docs fix. The first line must start with `# Method:`, because `factory/engine/workflows.test.ts` reads it. + +```yaml +# Method: a one-line docs fix, held to the prose rules and nothing else. +name: typo +method: prose check only +source: engineering standard +for: one-line documentation fixes +steps: + - { + stage: plan, + writes: plan.md, + agent: planner, + brief: "Reply with the plan and nothing else: one repo path per line for every file you will create or change, never a script you only run. Do not list files under specs/<issue>/; the runner allows those. A new file needs an owner in levels.json, so list levels.json whenever you add a file. Commentary of any kind makes the plan unusable. The runner saves your reply as specs/<issue>/plan.md, and later stages may touch only the files listed.", + } + - { + stage: fix, + agent: builder, + brief: "Fix the typo the ticket names and change nothing else. No em dashes, no filler.", + checks: [prose], + loop: { until: [prose], max: 2 }, + } + - { stage: review, agent: reviewer, checks: [plan-drift] } +``` + +3. Run `npx vitest run factory/engine/workflows.test.ts`. Every file in the folder gets the same tests, so `typo.yaml` is tested the moment it exists. +4. Break it on purpose, one edit at a time, and read each failure: + - change `agent: builder` to `agent: coder`: the "uses only agents that exist" test fails; + - add `loop: { until: [red], max: 2 }` to the fix step: `loadWorkflow` rejects a loop on a check the step does not run; + - delete the `plan` step: the "plan-drift needs a plan" test fails. +5. File an issue, then run it: `npx tsx factory/engine/main.ts <issue> --workflow=typo`. It needs `gh` logged in. The agents need `ANTHROPIC_API_KEY`, which the script loads inside its own process, so never put the key on a command line. +6. Open `specs/<issue>/run.log` and `.factory/runs/<issue>/metrics.json`. Each step logs its rounds, its cost and every check as `ok` or `FAIL`. + +## Why each rule in the brief exists + +The plan brief repeats three sentences in every workflow. Each is pinned by a test in `factory/engine/workflows.test.ts` that names the run that went wrong: + +- "Do not list files under `specs/<issue>/`": run 31 listed its golden test and the plan was refused. +- "List `levels.json` whenever you add a file": a new file needs an owner there, and `plan-drift` failed without it. +- "Commentary of any kind makes the plan unusable": run 29's plan held a note, and `badPlanLines` in `factory/engine/artifacts.ts` refuses lines that are not repo paths. + +## The golden set + +One workflow per kind of task. The test "covers every task type the plan names" fails if one is deleted. + +| task | workflow | what it refuses to skip | +| ----------------------------------- | -------------- | --------------------------------------------------------- | +| a bug | `bug-tdd` | `red` on the base commit, then `lock`, then `mutation` | +| a small feature with examples | `feature-atdd` | one test per Given/when/then example, then `holdout` | +| user-facing behaviour | `feature-bdd` | Gherkin scenarios, a human gate after the `amigos` review | +| a large feature or epic | `feature-sdd` | phase artifacts with human gates | +| a refactor | `refactor` | characterization tests first, then `behaviour-unchanged` | +| a UI change | `ui-visual` | a design contract, `visual`, the design lens | +| a prompt, model or `.claude` change | `llm-eval` | `eval` red on main, a loop to the pass rate | +| the api and web boundary | `contract` | the contract test first, both sides move | +| a dependency bump | `deps` | `manifest-deps` and the whole gate, never the bump alone | +| a security fix | `security` | a failing exploit test and a PR naming the class of bug | +| an open question | `spike` | the output is a decision record behind a human gate | +| documentation | `docs` | `prose` and `gate` | +| an incident or band breach | `incident` | the incident becomes an eval before the fix | + +## Choosing one + +Ask what evidence the task can honestly give. + +- Can you write a test that fails today? Use `bug-tdd`. If the change is meant to leave behaviour alone, use `refactor`, where the pinned tests pass on both sides. +- Is "done" something a person must judge? Use a workflow with a `gate: human`, and put the gate before the build. +- Is there no code to write yet? Use `spike`. Its two steps explore and decide, and have no build. +- Does the thing under test answer differently each time? Use `llm-eval`, which grades a pass rate instead of one run. + +`factory/engine/main.ts` does not pick for you: you name the workflow with `--workflow=`. The older runner in `factory/route-ticket.ts` classifies a ticket for its own stages; read it to see how a type is chosen there. + +## Proof + +- `npx vitest run factory/engine/workflows.test.ts` parses and dry-runs every workflow, including yours. +- `npx vitest run factory/engine/engine.test.ts` covers the schema rejections and the loop cap ("stops at the cap and writes run.log"). +- `make prove LEVEL=workflows` runs the engine tests as one gate (`scripts/prove/workflows.ts`). + +## Discussion + +Which of these would your team delete first, and what evidence would you lose? Which task in your backlog fits none of them? diff --git a/docs/factory/faq.md b/docs/factory/faq.md new file mode 100644 index 0000000..558a2a5 --- /dev/null +++ b/docs/factory/faq.md @@ -0,0 +1,58 @@ +# Factory FAQ + +Each answer starts from a failure someone hit, and points at the file or test that proves what the code does. + +## The plan forgot a file. Why did the run stop? + +The plan is an allowlist. `planDrift` in `factory/engine/artifacts.ts` fails any changed file that the plan does not list, apart from files under `specs/<issue>/` and regenerated snapshots. The `plan-drift` check turns that into evidence the loop can read: "files outside plan.md: ... Update plan.md in the same change." + +The common miss is a new source file. Each file needs an owner in `levels.json`, so the plan must list `levels.json` too. The plan brief in every workflow says so, and `factory/engine/workflows.test.ts` fails a workflow whose planner brief drops that sentence. + +Proof: `factory/engine/checks/plan-drift.test.ts`, and "plan-drift fails on a stray file, even when plan.md is in the diff" in `factory/engine/engine.test.ts`. + +## I relaunched the run and my approval no longer counted. Is that a bug? + +No. An approval binds to a hash, not to an issue. `artifactDigest` in `factory/engine/artifacts.ts` hashes the stage and the files it was shown, one hash per file so a moved boundary changes the result. Edit the artifact and the old approval does not match. + +The spec and plan gates sign only the documents written so far. Run 34 showed the other way: hashing the whole tree made every later stage void an earlier approval. On resume, `runWorkflow` in `factory/engine/interpret.ts` skips every step up to the last approved gate, so the signed artifacts are not regenerated. + +In the older runner, posting a new plan clears the `plan:approved` label and never builds in that run. + +Proof in `factory/engine/engine.test.ts`: "binds an approval to the artifact digest, so edited artifacts need a new one", "keeps a spec or plan approval valid after later stages change other files", and "on resume skips every step before the last approved gate". The label case is "clears a stale approval when it posts a new plan" in `factory/run.test.ts`. + +## The runner could not call `gh`. Is something misconfigured? + +By design. The gate runs the edited code with a throwaway `HOME`, `GIT_CONFIG_GLOBAL=/dev/null` and an empty `GH_CONFIG_DIR` (`scrubbedEnv` in `factory/stages.ts`), so code the agent wrote finds no credential to read. `.claude/settings.json` also denies `Bash(git push *)`. + +So a stage cannot fetch its own issue. The runner fetches it and puts the text in the prompt, inside `<issue>` tags with a warning that it is untrusted data (`runIssue` in `factory/engine/cli.ts`). If a stage needs GitHub data, add it to the prompt in the runner. Do not hand a credential to the stage. + +Proof: "puts the issue text in the prompt, since stages cannot reach GitHub" in `factory/run.test.ts`, and the escaping test in `tests/structure/factory-workflow.test.ts`. + +## A test contradicts the ticket. Who is right? + +The ticket, but the builder may not decide that. `lock` hashes the test files in runner state the agent cannot write, and `.claude/hooks/pretooluse-test-lock.mjs` blocks test edits in a fix phase with the message "Change the code until the test passes, or stop and say the test is wrong." + +A person fixes a wrong test by hand. The plan may then list that older spec's golden test (the comment above `GATE_OWNED` in `factory/engine/artifacts.ts` says why), and `lock` still fails any later edit to it. Other golden files stay gate-owned. + +Proof in `factory/engine/engine.test.ts`: "lets a plan list an older spec's golden test, which the lock check already protects", "still refuses golden snapshots and other golden files", and "lock catches an edited test". + +## What stops a run from spending without limit? + +Four limits, none of them the model's judgment: + +- A loop has `max` from 1 to 5, enforced by the schema in `factory/engine/workflow.ts`. At the cap the run ends as `needs-human` ("stops at the cap and writes run.log" in `factory/engine/engine.test.ts`). +- Each agent call has a ceiling: `maxBudgetUsd: o.maxUsdPerAgent ?? 3` in `factory/engine/real-host.ts`, plus `maxTurns` from the agent file. The older runner takes `maxBudgetUsd` and `maxTurns` per stage from `.factory/config.json`, and `npx tsx factory/main.ts <issue> --dry-run` prints them. +- The review loop stops after `reviewLoop.maxRounds` and labels the issue `needs-human` ("caps the review loop at maxRounds" in `factory/run.test.ts`). +- Live eval and smoke runs share one cap, `withinLiveCap` in `api/src/agent/live-cap.ts`. + +What the repo does not do is cap the total of one workflow run. `factory/cost-ledger.ts` reports spend by stage and model, the run log shows the cost of every round, and `npx tsx factory/engine/main.ts --report` rolls up the saved `metrics.json` files. + +## Why Sonnet and not Opus for build? + +Because build is the longest stage and `.factory/config.json` prices it. Per million tokens Opus is $4 in and $20 out, Sonnet is $2 and $10. The `build` stage is Sonnet with 40 turns and a $3 budget. Opus appears where one plan shapes everything after it: `routes.epic.plan` and the `delivery` stage. + +The ledger measures the choice instead of asserting it. `buildLedger` re-prices the same tokens at `baselineModel` (Opus) and reports `savedUsd`, so a run shows what an all-Opus run would have cost. + +What the repo does not contain is a measured quality comparison between the two models on build. To test the choice, change `stages.build.model`, run the same issue twice and compare `metrics.json` and the check results. + +Proof: "splits spend by stage and model and sums to the reported total" in `factory/cost-ledger.test.ts`, and `model: sonnet` in `.claude/agents/builder.md`.