Skip to content

Factory workshop modules F0, F0.1 and F1 #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Param-Harrison merged 3 commits into from
Oct 30, 2026
Merged

Factory workshop modules F0, F0.1 and F1 #82

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3d6f678
Factory workshop modules F0, F0.1 and F1
Param-Harrison Oct 30, 2026
792b42b
Rename factory docs to kebab-case
Param-Harrison Oct 30, 2026
a73d5e5
Format factory docs
Param-Harrison Oct 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 101 additions & 0 deletions docs/factory/f0-1-anatomy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# F0.1 Anatomy: every file the factory reads

**Stage:** setup. **Layers:** context, boundary, skills, execution, sharing. **Workflows:** all of them use these files.

## Goal

Know what each file in `.claude/`, `.factory/` and the repo root does, which one wins when two disagree, and which hook pattern guards what. By the end you can open an unfamiliar repo and say which file would change a given behaviour.

## The `.claude` files in this repo

| File or folder | Does | Look at |
| ---------------------------------------- | ----------------------------------------------------------------------- | --------------------------------------------------------------------- |
| `AGENTS.md`, loaded by `CLAUDE.md` | facts every session needs: commands, map, standards, done | `CLAUDE.md` is `@AGENTS.md` plus a compaction line |
| `.claude/rules/*.md` | a rule that loads only when a file matching its `paths` is touched | `.claude/rules/api.md` has `paths: api/**` |
| `.claude/settings.json` | permissions, sandbox, hooks, shared with the team | denies reading `.env` files and `.factory/holdout/**`, and `git push` |
| `.claude/settings.local.json` | your own overrides, gitignored | listed in `.gitignore` and `.worktreeinclude` |
| `CLAUDE.local.md` | your own notes, gitignored | example in `docs/examples/CLAUDE.local.md` |
| `.claude/skills/<name>/SKILL.md` | a procedure Claude chooses to run, from its description | `issue-triage`, `test-driven-development`, `factory-retro` |
| `.claude/commands/*.md` | a prompt you run by name, one file | `/gate` runs `make check` |
| `.claude/agents/*.md` | a worker with its own tools and cap | see the frontmatter below |
| `.claude/agent-memory/<agent>/MEMORY.md` | notes an agent keeps between sessions | `reviewer/MEMORY.md` |
| `.claude/output-styles/*.md` | how replies read | `teaching.md` |
| `.claude/workflows/factory.js` | a scripted order of stages inside a session | pinned by `tests/structure/factory-workflow.test.ts` |
| `.claude/hooks/*.mjs` | code the harness runs on an event | registered in `settings.json` |
| `.claude/teams.json` | agent-team settings and the `TeammateIdle` hook | `teammateMode: in-process` |
| `.mcp.json` | an MCP server; here GitHub, with the token read from `GITHUB_MCP_TOKEN` | no secret in the file |
| `.worktreeinclude` | gitignored files a new worktree still gets | `CLAUDE.local.md`, `.claude/settings.local.json` |
| `REVIEW.md` | what a failure looks like here, read by the reviewer first | nit cap, `file:line` evidence, round rules |
| `docs/managed-settings.example.json` | rules nobody on the team can override | an example, not installed by this repo |

`docs/when-to-use-what.md` is the same table by job, and `tests/structure/when-to-use-what.test.ts` keeps it honest.

### Agent frontmatter used here

- `builder`: `isolation: worktree`, `permissionMode: acceptEdits`, `maxTurns: 40`, tools `Read, Edit, Write, Bash, Grep, Glob`.
- `verifier`: `permissionMode: plan`, no `Edit` or `Write` in its tools.
- `reviewer`: `memory: project`, and no `Edit` or `Write` in its tools.
- `planner`: tools `Read, Grep, Glob` only.

A workflow step names an agent from this folder, so these fields are what limit a stage. `tests/structure/agent-frontmatter.test.ts` asserts the verifier's `permissionMode`, the builder's `isolation`, the reviewer's `memory` and a `maxTurns` on every agent. The tool lists are not asserted.

## Precedence

When two settings files disagree, Claude Code's documentation orders them: managed settings, then command-line flags, then `settings.local.json`, then project `settings.json`, then your user settings. A `deny` rule in any layer wins over an `allow`. This repo does not test precedence, so treat the order as the docs' claim and check it with `/permissions` in a session. `CLAUDE.local.md` is read after `CLAUDE.md`, so a local note can add to a team rule but should not be relied on to cancel one.

## `.factory/`

- `config.json`: per stage model, `maxTurns`, `maxBudgetUsd` and `permissionMode`; the allowed model list; prices; the review round cap; the labels (`factory:ready`, `plan:approved`, `needs-human`).
- `workflows/<name>.yaml`: one workflow each, parsed by `factory/engine/workflow.ts`.
- `bands.yaml`: metrics with a direction and a window, read by `factory/engine/bands.ts`.
- `holdout/<n>/`: cases the builder must not read. `settings.json` denies reading them, but no `.factory/holdout/` folder is committed in this checkout, so the deny is wiring waiting for a case.
- `runs/<n>/`: `cost.json`, `run.log` and `metrics.json` per run. Gitignored.

## Hook patterns

Each hook in `.claude/hooks/` is one pattern, and `tests/structure/hooks.test.ts` and `hook-catalogue.test.ts` feed each one bad input.

| Pattern | Hook | Behaviour |
| -------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------ |
| Block before the tool runs | `pretooluse-secrets.mjs`, `pretooluse-file-guard.mjs` | refuses the call, with a reason |
| Block tests during a fix | `pretooluse-test-lock.mjs` | active only when `FACTORY_PHASE=fix` |
| Gate the finish | `stop-gate.mjs` | blocks stopping while the gate fails; lets the second stop through so it cannot loop |
| Warn only | `posttooluse-length-warn.mjs`, `posttooluse-output-scan.mjs` | tells the model, never blocks |
| Fail safe | `posttooluse-format.mjs` | a missing formatter does not fail the edit |
| Integrity hash | `sessionstart-integrity.mjs` | names a hook or settings file whose hash moved since last session |
| Route a prompt | `userpromptsubmit-route.mjs` | adds a skill hint, never blocks |
| Log | `posttooluse-session-log.mjs`, `teammateidle-log.mjs` | records, changes nothing |

Hooks are code the harness runs, so they enforce what a line in `AGENTS.md` can only ask for.

## Mods

`plugins/factory-status` is a mod: it draws the newest `.factory/runs/<n>` ticket, spend and last log line above the prompt, and shows nothing when no run exists. Its `register(on)` handles `session.start`, `turn.complete` and `ui.render`. Its README gives `claude plugin validate plugins/factory-status` and `claude plugin test plugins/factory-status`. `plugins/factory-kit` packages agents, skills and a secrets guard for other repos.

## Steps you run

1. `cat .claude/settings.json` and match each hook entry to a file in `.claude/hooks/`.
2. Start `claude` in this repo and run `/memory`, then `/context`. Expect `AGENTS.md` loaded through `CLAUDE.md`.
3. Open `.claude/agents/verifier.md` and `builder.md` side by side and list which tools each lacks.
4. `npx vitest run tests/structure` runs every structural test.
5. `claude plugin validate plugins/factory-status`.

## Proof that bites

In `.claude/hooks/pretooluse-test-lock.mjs`, change `FACTORY_PHASE !== "fix"` to `FACTORY_PHASE !== "never"` and run:

```
npx vitest run tests/structure/review-and-test-lock.test.ts
```

Expect `test lock > blocks an edit to api/src/a.test.ts during a fix` to fail, because the hook now lets the edit through. Restore it and the test passes. It runs in `make check` through `npm test`.

## See it on a real run

`specs/26/` ran `bug-tdd`. Its `reproduce` step wrote `specs/26/golden/font-face.golden.test.ts`, and `specs/26/lock.json` lists the hash of that file. The `fix` step has `phase: fix`, which sets `FACTORY_PHASE=fix` for the hook above, and `specs/26/fix.md` says the locked test "passes", with no edit to it. The same note records that `make check` lint still failed on `specs/26/lock.json` formatting, a runner-owned file the agent left alone. No `run.log` is committed because `.gitignore` excludes `*.log`.

## Discussion

1. Which of your team's rules sit only in a doc, and which file type above would make each one a check?
2. The test lock is a hook and `lock` is a check. What does each catch that the other misses?
3. Who in your org should own managed settings, and what would you put in the deny list first?
86 changes: 86 additions & 0 deletions docs/factory/f0-overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# F0 Overview: from a ticket to a merged pull request

**Stage:** the whole path. **Layers:** all of them. **Workflows:** every file in `.factory/workflows/`.

## Goal

Say in one page what the factory does, which parts are code and which are a model, and how it compares with the two public descriptions of an agentic software factory. By the end you can name the artifact each stage leaves behind and the check that decides whether the next stage may start.

## The path

A ticket moves through stages. Each stage commits one artifact under `specs/<n>/` and the next stage reads only what was committed, so the commit history is the audit trail.

| Stage | Artifact | What decides "next" |
| -------- | --------------------------------- | -------------------------------------- |
| Plan | `intent.md`, `spec.md`, `plan.md` | a human gate, and `plan-drift` later |
| Build | tests, then the diff, `lock.json` | `red`, `lock`, `green`, `acceptance` |
| Test | more tests, mutants | `mutation`, `properties`, `holdout` |
| Review | `review.json` | lenses, then verifiers that vote |
| Deploy | the pull request, CI | `ci`, risk tier approvals |
| Maintain | `incident.md`, a new `intent.md` | sigma bands from `.factory/bands.yaml` |

The names come from `ARTIFACTS` in `factory/engine/artifacts.ts` and `CHECK_NAMES` in `factory/engine/workflow.ts`. A workflow is a YAML file. `factory/engine/interpret.ts` runs its steps in order, repeats a step while its loop checks fail up to a cap, and writes `run.log` when the run finishes or stops at a gate or a cap.

## What is code and what is a model

A model writes `intent.md`, `spec.md`, `plan.md`, the tests and the fix. Code decides everything that gates them:

- `red` runs the new tests on the base commit and fails if they pass.
- `lock` hashes every test file in state the agent cannot write.
- `plan-drift` fails a diff that touches a file `plan.md` does not list.
- `review.ts` keeps a finding only when at least two of its verifiers vote for it.
- `risk.ts` turns changed paths into a tier, and an agent's approval never counts.
- `bands.ts` turns a metric breach into an action with no model involved.

The rule behind the list: a gate reads an exit code or a hash, never the model's word that it is done.

## Public factories, compared

The workshop is built from published descriptions. This table says where each idea lands here and where this repo does less.

| Idea | Anthropic AI-native SDLC playbook and review post | OpenAI software factory (Pragmatic Engineer write-up) | This repo |
| -------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------- |
| Start | an `intent.md` per stage, committed | an outcome, then context | `intent.md` from the `intent` step, or from a sigma breach |
| Verify | tests first, hooks that block test edits during a fix, evals gate prompt changes | CI is watched until it is green | `red` and `lock`, `.claude/hooks/pretooluse-test-lock.mjs`, the `ci` check |
| Review | layered review, parallel hunters, agents that vote on each finding | specialist reviewers | `factory/engine/review.ts`, `REVIEW.md` |
| Deploy | autonomy tiered by environment, hooks as approval gates | an agent assists the deploy after a human approves | `factory/engine/deploy.ts`: production refuses without the tier's human approvals |
| Risk | the tier comes from the change, not the author | risk classification picks the path | `classify` in `factory/engine/risk.ts` |
| Maintain | sigma bands turn a breach into a new intent | observe, feed regressions back, an incident bot that acts only when told | `.factory/bands.yaml`, `detect` and `actionFor` in `factory/engine/bands.ts` |

This repo has no deploy target of its own: `release` calls `make deploy ENV=...`, and the Makefile here does not define that target. Treat F6 as a pattern to copy, not a pipeline that ships this app.

## The Founder's playbook

Anthropic also publishes a Founder's playbook. Its page holds only a table of contents (Idea, MVP, Launch, Scale); the body is a long PDF. Nothing in this workshop is built from it. This is the only mention: the factory here starts after the idea is already a ticket.

## Steps you run

1. Read the comment line above `name:` in `.factory/workflows/bug-tdd.yaml` and `.factory/workflows/feature-sdd.yaml`. Each names its method and source.
2. List the workflows: `ls .factory/workflows`.
3. Print the stages the legacy runner would run, with no spend: `make factory ISSUE=1 FLAGS=--dry-run`. It reads `.factory/config.json` and prints model, turns and budget per stage.
4. Run the engine's offline tests, which run every workflow against a fake host: `npx vitest run factory/engine`.
5. With `gh` signed in, `npx tsx factory/engine/main.ts --watch` lists issues labelled `factory:ready`. `npx tsx factory/engine/main.ts --report` rolls up saved `metrics.json` files and needs a `.factory/runs` folder from an earlier run.

A real run is `npx tsx factory/engine/main.ts <issue> --workflow=<name>`. It needs `gh` and the API key file the api workspace holds, and it spends tokens, so read F1 first.

## Proof that bites

`factory/engine/review.test.ts` holds the risk and review rules. Open `factory/engine/risk.ts`, change the first HIGH pattern `/^\.github\//` to one that never matches, and run:

```
npx vitest run factory/engine/review.test.ts
```

Expect `risk > is high for a CI workflow and low for docs` to fail with `expected 'medium' to be 'high'`. Restore the pattern and it passes. The test is wired into `make check` through `npm test`.

## See it on a real run

`specs/26/` is a bug run through `bug-tdd`. `specs/26/golden/font-face.golden.test.ts` is the test written to fail first, `specs/26/lock.json` is the hash of every test file at that point, `specs/26/fix.md` is the fix stage's note, and `specs/26/plan.md` is the allowlist the diff was held to.

`run.log` is the file the engine writes last, but `.gitignore` excludes `*.log`, so none is committed here. `specs/35/ci.md` quotes one line from its run's log: `check plan-drift FAIL: files outside plan.md: api/src/agent/template-eval.ts`. That is `plan-drift` stopping a run, and `ci.md` records that the path was then listed in `plan.md`.

## Discussion

1. Which stage in your team's flow commits nothing the next stage can read, and what would you commit?
2. The OpenAI description has an agent assist the deploy after a human approves. Where in `deploy.ts` is the approval, and what would you add before you let an agent run `make deploy`?
3. Which gate here would you most want to turn off under deadline, and what does its failing test say you would lose?
Loading
Loading