diff --git a/docs/factory/f0-1-anatomy.md b/docs/factory/f0-1-anatomy.md new file mode 100644 index 0000000..3f719bb --- /dev/null +++ b/docs/factory/f0-1-anatomy.md @@ -0,0 +1,101 @@ +# F0.1 Anatomy: every file the factory reads + +**Stage:** setup. **Layers:** context, boundary, skills, execution, sharing. **Workflows:** all of them use these files. + +## Goal + +Know what each file in `.claude/`, `.factory/` and the repo root does, which one wins when two disagree, and which hook pattern guards what. By the end you can open an unfamiliar repo and say which file would change a given behaviour. + +## The `.claude` files in this repo + +| File or folder | Does | Look at | +| ---------------------------------------- | ----------------------------------------------------------------------- | --------------------------------------------------------------------- | +| `AGENTS.md`, loaded by `CLAUDE.md` | facts every session needs: commands, map, standards, done | `CLAUDE.md` is `@AGENTS.md` plus a compaction line | +| `.claude/rules/*.md` | a rule that loads only when a file matching its `paths` is touched | `.claude/rules/api.md` has `paths: api/**` | +| `.claude/settings.json` | permissions, sandbox, hooks, shared with the team | denies reading `.env` files and `.factory/holdout/**`, and `git push` | +| `.claude/settings.local.json` | your own overrides, gitignored | listed in `.gitignore` and `.worktreeinclude` | +| `CLAUDE.local.md` | your own notes, gitignored | example in `docs/examples/CLAUDE.local.md` | +| `.claude/skills//SKILL.md` | a procedure Claude chooses to run, from its description | `issue-triage`, `test-driven-development`, `factory-retro` | +| `.claude/commands/*.md` | a prompt you run by name, one file | `/gate` runs `make check` | +| `.claude/agents/*.md` | a worker with its own tools and cap | see the frontmatter below | +| `.claude/agent-memory//MEMORY.md` | notes an agent keeps between sessions | `reviewer/MEMORY.md` | +| `.claude/output-styles/*.md` | how replies read | `teaching.md` | +| `.claude/workflows/factory.js` | a scripted order of stages inside a session | pinned by `tests/structure/factory-workflow.test.ts` | +| `.claude/hooks/*.mjs` | code the harness runs on an event | registered in `settings.json` | +| `.claude/teams.json` | agent-team settings and the `TeammateIdle` hook | `teammateMode: in-process` | +| `.mcp.json` | an MCP server; here GitHub, with the token read from `GITHUB_MCP_TOKEN` | no secret in the file | +| `.worktreeinclude` | gitignored files a new worktree still gets | `CLAUDE.local.md`, `.claude/settings.local.json` | +| `REVIEW.md` | what a failure looks like here, read by the reviewer first | nit cap, `file:line` evidence, round rules | +| `docs/managed-settings.example.json` | rules nobody on the team can override | an example, not installed by this repo | + +`docs/when-to-use-what.md` is the same table by job, and `tests/structure/when-to-use-what.test.ts` keeps it honest. + +### Agent frontmatter used here + +- `builder`: `isolation: worktree`, `permissionMode: acceptEdits`, `maxTurns: 40`, tools `Read, Edit, Write, Bash, Grep, Glob`. +- `verifier`: `permissionMode: plan`, no `Edit` or `Write` in its tools. +- `reviewer`: `memory: project`, and no `Edit` or `Write` in its tools. +- `planner`: tools `Read, Grep, Glob` only. + +A workflow step names an agent from this folder, so these fields are what limit a stage. `tests/structure/agent-frontmatter.test.ts` asserts the verifier's `permissionMode`, the builder's `isolation`, the reviewer's `memory` and a `maxTurns` on every agent. The tool lists are not asserted. + +## Precedence + +When two settings files disagree, Claude Code's documentation orders them: managed settings, then command-line flags, then `settings.local.json`, then project `settings.json`, then your user settings. A `deny` rule in any layer wins over an `allow`. This repo does not test precedence, so treat the order as the docs' claim and check it with `/permissions` in a session. `CLAUDE.local.md` is read after `CLAUDE.md`, so a local note can add to a team rule but should not be relied on to cancel one. + +## `.factory/` + +- `config.json`: per stage model, `maxTurns`, `maxBudgetUsd` and `permissionMode`; the allowed model list; prices; the review round cap; the labels (`factory:ready`, `plan:approved`, `needs-human`). +- `workflows/.yaml`: one workflow each, parsed by `factory/engine/workflow.ts`. +- `bands.yaml`: metrics with a direction and a window, read by `factory/engine/bands.ts`. +- `holdout//`: cases the builder must not read. `settings.json` denies reading them, but no `.factory/holdout/` folder is committed in this checkout, so the deny is wiring waiting for a case. +- `runs//`: `cost.json`, `run.log` and `metrics.json` per run. Gitignored. + +## Hook patterns + +Each hook in `.claude/hooks/` is one pattern, and `tests/structure/hooks.test.ts` and `hook-catalogue.test.ts` feed each one bad input. + +| Pattern | Hook | Behaviour | +| -------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------ | +| Block before the tool runs | `pretooluse-secrets.mjs`, `pretooluse-file-guard.mjs` | refuses the call, with a reason | +| Block tests during a fix | `pretooluse-test-lock.mjs` | active only when `FACTORY_PHASE=fix` | +| Gate the finish | `stop-gate.mjs` | blocks stopping while the gate fails; lets the second stop through so it cannot loop | +| Warn only | `posttooluse-length-warn.mjs`, `posttooluse-output-scan.mjs` | tells the model, never blocks | +| Fail safe | `posttooluse-format.mjs` | a missing formatter does not fail the edit | +| Integrity hash | `sessionstart-integrity.mjs` | names a hook or settings file whose hash moved since last session | +| Route a prompt | `userpromptsubmit-route.mjs` | adds a skill hint, never blocks | +| Log | `posttooluse-session-log.mjs`, `teammateidle-log.mjs` | records, changes nothing | + +Hooks are code the harness runs, so they enforce what a line in `AGENTS.md` can only ask for. + +## Mods + +`plugins/factory-status` is a mod: it draws the newest `.factory/runs/` ticket, spend and last log line above the prompt, and shows nothing when no run exists. Its `register(on)` handles `session.start`, `turn.complete` and `ui.render`. Its README gives `claude plugin validate plugins/factory-status` and `claude plugin test plugins/factory-status`. `plugins/factory-kit` packages agents, skills and a secrets guard for other repos. + +## Steps you run + +1. `cat .claude/settings.json` and match each hook entry to a file in `.claude/hooks/`. +2. Start `claude` in this repo and run `/memory`, then `/context`. Expect `AGENTS.md` loaded through `CLAUDE.md`. +3. Open `.claude/agents/verifier.md` and `builder.md` side by side and list which tools each lacks. +4. `npx vitest run tests/structure` runs every structural test. +5. `claude plugin validate plugins/factory-status`. + +## Proof that bites + +In `.claude/hooks/pretooluse-test-lock.mjs`, change `FACTORY_PHASE !== "fix"` to `FACTORY_PHASE !== "never"` and run: + +``` +npx vitest run tests/structure/review-and-test-lock.test.ts +``` + +Expect `test lock > blocks an edit to api/src/a.test.ts during a fix` to fail, because the hook now lets the edit through. Restore it and the test passes. It runs in `make check` through `npm test`. + +## See it on a real run + +`specs/26/` ran `bug-tdd`. Its `reproduce` step wrote `specs/26/golden/font-face.golden.test.ts`, and `specs/26/lock.json` lists the hash of that file. The `fix` step has `phase: fix`, which sets `FACTORY_PHASE=fix` for the hook above, and `specs/26/fix.md` says the locked test "passes", with no edit to it. The same note records that `make check` lint still failed on `specs/26/lock.json` formatting, a runner-owned file the agent left alone. No `run.log` is committed because `.gitignore` excludes `*.log`. + +## Discussion + +1. Which of your team's rules sit only in a doc, and which file type above would make each one a check? +2. The test lock is a hook and `lock` is a check. What does each catch that the other misses? +3. Who in your org should own managed settings, and what would you put in the deny list first? diff --git a/docs/factory/f0-overview.md b/docs/factory/f0-overview.md new file mode 100644 index 0000000..58fbc62 --- /dev/null +++ b/docs/factory/f0-overview.md @@ -0,0 +1,86 @@ +# F0 Overview: from a ticket to a merged pull request + +**Stage:** the whole path. **Layers:** all of them. **Workflows:** every file in `.factory/workflows/`. + +## Goal + +Say in one page what the factory does, which parts are code and which are a model, and how it compares with the two public descriptions of an agentic software factory. By the end you can name the artifact each stage leaves behind and the check that decides whether the next stage may start. + +## The path + +A ticket moves through stages. Each stage commits one artifact under `specs//` and the next stage reads only what was committed, so the commit history is the audit trail. + +| Stage | Artifact | What decides "next" | +| -------- | --------------------------------- | -------------------------------------- | +| Plan | `intent.md`, `spec.md`, `plan.md` | a human gate, and `plan-drift` later | +| Build | tests, then the diff, `lock.json` | `red`, `lock`, `green`, `acceptance` | +| Test | more tests, mutants | `mutation`, `properties`, `holdout` | +| Review | `review.json` | lenses, then verifiers that vote | +| Deploy | the pull request, CI | `ci`, risk tier approvals | +| Maintain | `incident.md`, a new `intent.md` | sigma bands from `.factory/bands.yaml` | + +The names come from `ARTIFACTS` in `factory/engine/artifacts.ts` and `CHECK_NAMES` in `factory/engine/workflow.ts`. A workflow is a YAML file. `factory/engine/interpret.ts` runs its steps in order, repeats a step while its loop checks fail up to a cap, and writes `run.log` when the run finishes or stops at a gate or a cap. + +## What is code and what is a model + +A model writes `intent.md`, `spec.md`, `plan.md`, the tests and the fix. Code decides everything that gates them: + +- `red` runs the new tests on the base commit and fails if they pass. +- `lock` hashes every test file in state the agent cannot write. +- `plan-drift` fails a diff that touches a file `plan.md` does not list. +- `review.ts` keeps a finding only when at least two of its verifiers vote for it. +- `risk.ts` turns changed paths into a tier, and an agent's approval never counts. +- `bands.ts` turns a metric breach into an action with no model involved. + +The rule behind the list: a gate reads an exit code or a hash, never the model's word that it is done. + +## Public factories, compared + +The workshop is built from published descriptions. This table says where each idea lands here and where this repo does less. + +| Idea | Anthropic AI-native SDLC playbook and review post | OpenAI software factory (Pragmatic Engineer write-up) | This repo | +| -------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------- | +| Start | an `intent.md` per stage, committed | an outcome, then context | `intent.md` from the `intent` step, or from a sigma breach | +| Verify | tests first, hooks that block test edits during a fix, evals gate prompt changes | CI is watched until it is green | `red` and `lock`, `.claude/hooks/pretooluse-test-lock.mjs`, the `ci` check | +| Review | layered review, parallel hunters, agents that vote on each finding | specialist reviewers | `factory/engine/review.ts`, `REVIEW.md` | +| Deploy | autonomy tiered by environment, hooks as approval gates | an agent assists the deploy after a human approves | `factory/engine/deploy.ts`: production refuses without the tier's human approvals | +| Risk | the tier comes from the change, not the author | risk classification picks the path | `classify` in `factory/engine/risk.ts` | +| Maintain | sigma bands turn a breach into a new intent | observe, feed regressions back, an incident bot that acts only when told | `.factory/bands.yaml`, `detect` and `actionFor` in `factory/engine/bands.ts` | + +This repo has no deploy target of its own: `release` calls `make deploy ENV=...`, and the Makefile here does not define that target. Treat F6 as a pattern to copy, not a pipeline that ships this app. + +## The Founder's playbook + +Anthropic also publishes a Founder's playbook. Its page holds only a table of contents (Idea, MVP, Launch, Scale); the body is a long PDF. Nothing in this workshop is built from it. This is the only mention: the factory here starts after the idea is already a ticket. + +## Steps you run + +1. Read the comment line above `name:` in `.factory/workflows/bug-tdd.yaml` and `.factory/workflows/feature-sdd.yaml`. Each names its method and source. +2. List the workflows: `ls .factory/workflows`. +3. Print the stages the legacy runner would run, with no spend: `make factory ISSUE=1 FLAGS=--dry-run`. It reads `.factory/config.json` and prints model, turns and budget per stage. +4. Run the engine's offline tests, which run every workflow against a fake host: `npx vitest run factory/engine`. +5. With `gh` signed in, `npx tsx factory/engine/main.ts --watch` lists issues labelled `factory:ready`. `npx tsx factory/engine/main.ts --report` rolls up saved `metrics.json` files and needs a `.factory/runs` folder from an earlier run. + +A real run is `npx tsx factory/engine/main.ts --workflow=`. It needs `gh` and the API key file the api workspace holds, and it spends tokens, so read F1 first. + +## Proof that bites + +`factory/engine/review.test.ts` holds the risk and review rules. Open `factory/engine/risk.ts`, change the first HIGH pattern `/^\.github\//` to one that never matches, and run: + +``` +npx vitest run factory/engine/review.test.ts +``` + +Expect `risk > is high for a CI workflow and low for docs` to fail with `expected 'medium' to be 'high'`. Restore the pattern and it passes. The test is wired into `make check` through `npm test`. + +## See it on a real run + +`specs/26/` is a bug run through `bug-tdd`. `specs/26/golden/font-face.golden.test.ts` is the test written to fail first, `specs/26/lock.json` is the hash of every test file at that point, `specs/26/fix.md` is the fix stage's note, and `specs/26/plan.md` is the allowlist the diff was held to. + +`run.log` is the file the engine writes last, but `.gitignore` excludes `*.log`, so none is committed here. `specs/35/ci.md` quotes one line from its run's log: `check plan-drift FAIL: files outside plan.md: api/src/agent/template-eval.ts`. That is `plan-drift` stopping a run, and `ci.md` records that the path was then listed in `plan.md`. + +## Discussion + +1. Which stage in your team's flow commits nothing the next stage can read, and what would you commit? +2. The OpenAI description has an agent assist the deploy after a human approves. Where in `deploy.ts` is the approval, and what would you add before you let an agent run `make deploy`? +3. Which gate here would you most want to turn off under deadline, and what does its failing test say you would lose? diff --git a/docs/factory/f1-plan.md b/docs/factory/f1-plan.md new file mode 100644 index 0000000..1ed83de --- /dev/null +++ b/docs/factory/f1-plan.md @@ -0,0 +1,56 @@ +# F1 Plan: intent, triage and the risk tier + +**Stage:** Plan. **Layers:** skills, boundary. **Workflows:** `feature-atdd`, `feature-sdd`, `deps`, `incident`, `security` and `ui-visual` open with an `intent` step; every workflow with a `plan` step ends the stage with `plan.md`. + +## Goal + +Turn a GitHub issue into decisions before any code exists: what outcome counts (`intent.md`), what kind of work it is (triage), and how much human approval it needs (the risk tier). By the end you can say which of the three is decided by a model and which by code. + +## The pieces + +**`intent.md`.** A short statement of the outcome and how you will know. In the workflows that have an `intent` step, `agent: planner` and `writes: intent.md`, and the brief says "State the outcome ... in a few lines. Reply with the intent only." The planner agent has `Read, Grep, Glob` and cannot write files, so the runner saves its reply as `specs//intent.md` (`factory/engine/interpret.ts`). A sigma breach can also start a run: `intentFor` in `factory/engine/bands.ts` writes an `intent.md` that says the metric is outside its band and asks for an eval so a repeat is caught. + +**Triage.** The `issue-triage` skill (`.claude/skills/issue-triage/SKILL.md`) tells the model to read the issue as data, pick a TYPE (feature, enhancement, bug, epic or docs), a SIZE and a RISK, and end with `TYPE:`, `SIZE:`, `RISK:` and `DECISION: proceed|refuse`. `factory/run.ts` sends the same TYPE, SIZE, RISK and DECISION contract in its triage prompt, and `parseTriage` in `factory/route-ticket.ts` reads SIZE and RISK back. A missing or odd value becomes medium size and high risk, the costly case. `routeTicket` then picks the model: an epic or a high risk ticket gets the top plan model, docs or a small low-risk ticket gets the cheapest, everything else the middle one. A `refuse` posts the reply and stops. + +Triage here picks a model and a refusal, not the workflow. The engine takes the workflow from the command line, `--workflow=`. Matching a ticket to `bug-tdd` or `feature-bdd` is a human or orchestrator choice in this repo, and F8 returns to it. + +**The risk tier.** `classify` in `factory/engine/risk.ts` takes the changed paths and returns `low`, `medium` or `high`. It looks only at paths, never at who wrote the diff, and the worst path wins: + +- Anything under `.github/`, `.claude/`, `.factory/`, `factory/`, `scripts/`, plus `CLAUDE.md`, `AGENTS.md`, `REVIEW.md`, the Makefile, `package.json`, the lockfile and any `migrations/` folder is high. Markdown that steers an agent is high on purpose. +- A diff made only of `docs/` and `fixtures/` is low. +- The run's own files in `specs//` (`intent.md`, `spec.md`, `plan.md`, `run.log`, `review.json`, `lock.json`, `incident.md`) are ignored. Any other file there, such as a script, is classified normally. +- Everything else is medium, and an empty diff is medium. + +`APPROVALS` maps the tier to human approvals: low 0, medium 1, high 2. `approvalsMet` counts only approvals of kind `human` on the current head commit, so an agent never counts and a push voids earlier approvals. + +Be exact about what runs today. `classify` and `approvalsMet` are tested, and `release` in `factory/engine/deploy.ts` takes a `tier` argument and refuses production without the approvals. No code in this checkout calls `classify` for you, so the caller computes the tier. The triage RISK and the path tier are different values: the first is the model's guess used to route models, the second is code over the diff used to gate approvals. + +## Steps you run + +1. Read `.claude/skills/issue-triage/SKILL.md`, then `parseTriage` and `routeTicket` in `factory/route-ticket.ts`. +2. Run `npx vitest run factory/route-ticket.test.ts` and read what each case routes to. +3. Compute a tier by hand: `npx tsx -e 'import {classify} from "./factory/engine/risk.ts"; console.log(classify([".github/workflows/ci.yml"]), classify(["docs/a.md"]), classify(["shared/src/bands.ts"]))'`. Expect `high low medium`. +4. File a small docs issue and read its plan without spending: `make factory ISSUE= FLAGS=--dry-run` lists the stages, models and budgets. +5. For a real plan, label the issue `factory:ready` and run `npx tsx factory/engine/main.ts --workflow=feature-atdd`. It needs `gh` signed in and an API key available to the api workspace, and it spends tokens against the per stage caps in `.factory/config.json`. It stops at the `spec` human gate. Read `specs//intent.md` and `specs//spec.md` before you approve. + +## Proof that bites + +Both tests are in `make check` through `npm test`. For triage, in `factory/route-ticket.ts` change the RISK fallback from `"high"` to `"low"` and run: + +``` +npx vitest run factory/route-ticket.test.ts +``` + +Expect `treats a missing SIZE or RISK as medium size and high risk` to fail with `expected { size: 'medium', risk: 'low' } to deeply equal { size: 'medium', risk: 'high' }`. For the tier, see the `.github/` change in F0: `factory/engine/review.test.ts` fails with `expected 'medium' to be 'high'`. Restore both files afterwards. + +## See it on a real run + +`specs/34/intent.md` is what an `intent` step actually returned for a workbook epic. It is not a three-line outcome. The agent reported that the gate was red because `shared/src/workbook-pdf.test.ts:34` needs headless Chromium, which was missing from the run's temporary home, and it said it had not installed a browser or skipped the test. That is a real failure shape: the stage can answer with a status report instead of the artifact. `feature-sdd` pins the `spec` step with `startsWith: "R1."` to refuse chat, and its `intent` step has no such pin. + +`specs/36/intent.md` is the same step in a dependency run. It tabulates three packages with their locked and target versions and says the registry could not be reached, so publish dates and maintainers were not checked. Both files show what a reader gets before the human gate: evidence and its limits, in the artifact. + +## Discussion + +1. Triage fails closed to high risk. Which fallback would you choose for a field the model left out, and what does it cost on a busy day? +2. `scripts/` and `.claude/` are high even when only a comment changed. What would you gain and lose by classifying inside the file instead? +3. If the tier came from the model's RISK line instead of the paths, which diff would slip through?