feat: turn check — a finished turn that needs you reads as waiting

[Stener1]

Closes #6. Refs #8 (its hook findings; the shared-daemon part of #8 is separate).

Split as asked: the Codex half is its own PR, #71. This branch is now on main without it, so the diff is turn check only.

Choose who checks turns (new, last commit)

Point 3 asked for no default vendor model. A model field alone wasn't a real choice, because AI Gateway's /v1/evaluate only serves Jev, so Settings → Agents now starts with a provider picker. Nothing is picked for you:

Provider	Endpoint	Key	Model
Jev on AI Gateway	https://ai-gateway.vercel.sh/v1/evaluate	AI Gateway key	typesafe-ai/jev
OpenAI Decisions	https://api.openai.com/v1/decisions (public beta)	OpenAI key	gpt-6-luna
Laya, your own server	the laya-serve URL you give (/v1/systemone, Jev's shape)	none	optional (english, multilingual, typed-decisions)

• OpenAI gets the same instructions and criteria as one choice question over the turn as one text. Its answer is read for the probability of its choice, against the same 0.65 threshold, and a refusal leaves the turn finished.
• Keys are kept per provider (turncheck.key, turncheck-openai.key) and sent only to that provider's own fixed URL. A url in turncheck.json never changes where a key goes, and Laya never gets one. A reference set for one provider is dropped when the provider changes, and switching provider turns the check off until you turn it on again.
• Tests: TestTurnCheckAsksOpenAIDecisions, TestDecisionsVerdictReadsOnlyItsAnswer, TestTurnCheckKeepsAKeyPerProvider (a pasted AI Gateway key is never sent to OpenAI), TestTurnCheckSendsAKeyOnlyToItsProvider; e2e settings-turn-check covers both flows (OpenAI with a key, then switching; Laya with only an address).
• Laya, live: I ran the official laya 0.4.6 (laya-serve, CPU) on a box, and berthd's turn check reached it and read its answers. Its quality on this task is weak, though. On 4 hand-made turns (a question, a finished fix, a running deploy, a blocked migration) it got 1–2 right per checkpoint and was never above 0.52 sure, so with the 0.65 threshold it left every turn Done: safe, but it changes nothing in practice. The Settings text doesn't oversell it.
• OpenAI, not live: I had no OpenAI key here, so the Decisions request and answer are tested against a stand-in server built from OpenAI's documented shape.

Changes from the review

1. Laptop-only settings, and a key only to its own provider. GET and PUT /v1/turncheck now refuse wire.IsLocal (403, "turn check is set from a paired laptop"), so an agent on the box can't read or change where turns go. A key, pasted or a secret reference, is sent only to its provider's own fixed URL (AI Gateway's or OpenAI's); url is used only for a Laya server, which never gets a key. PUT refuses a key with Laya and a url with a hosted provider. Since turncheck.json is still a file an agent on the box could edit, the check refuses too: a reference beside a Laya url sends nothing at all, and a url beside a hosted provider is ignored. TestTurnCheckIsSetFromALaptopOnly drives both routes over a real Unix socket; TestTurnCheckSendsAKeyOnlyToItsProvider writes your exact scenario into the file (own server, key: env://…, over http and https) and asserts that server is never asked.
2. Capped and redacted. The prompt is capped at 4,000 characters (its start), the title at 300, and the reply stays at its last 12,000. Title, prompt and reply go through RedactSecrets first, on the full text before cutting: the same secretPatterns that refuse an artifact (artifactcheck.go), each match replaced with [redacted: <what>], plus a private key taken out whole, header to footer, since the artifact pattern only needs the header to refuse. If LooksLikeSecret still finds anything after that, the turn isn't sent. TestTurnCheckSendsNoSecretAndAShortPrompt covers an AWS key, a GitHub token, an Anthropic key in the title, an .env line and a PEM block.
3. No default model. jevModel is gone. Turning it on needs a provider and, for the hosted ones, a model (400 otherwise), and a check with none sends nothing. See the provider picker above.

Docs (security.mdx, sessions-and-agents.mdx, app-api.mdx, config.mdx, events.mdx, changelog) are updated to match.

Rebased on main after #73–#77. Turn check reads a session's transcript through transcriptFile(ctx, …) like the other callers now. Structured Labs agents (the new Claude driver) aren't turn-checked yet: it only reads Claude Code's and Codex's own transcript files. Happy to add the driver's record if you want it.

After the rebase: go vet ./... and go test -race ./... pass, except TestAStructuredSessionGoesToItsTerminalAndBack ("the terminal's turn in the chat never happened"). That test fails the same way on a clean main here, so it isn't this branch. The turn-check e2e specs pass.

Tested before the rebase: go vet ./..., go test -race ./..., pnpm test, pnpm build, typecheck:plugins, check:titles, check:csp, check:themes, tsc -p e2e; e2e settings-turn-check, dashboard-turn-check and skills-notice (same Settings page), 66 of 66 over 3 repeats. Live: Laya as above. Not re-run live against Jev or OpenAI after these changes.

---

An opt-in turn check: when an agent's turn finishes, berthd asks a calibrated classifier what the turn's end means, and if it is confident, the session shows as Needs you (it asks you something or is blocked) or Working (its work goes on without it: a build, a deploy, a later time) instead of Done. Off by default; nothing is sent unless it is turned on.

What it does

• On agent.finished (not error or interrupted) for Claude Code or Codex, berthd waits for the reply to reach the transcript (the notifier's settle), reads the turn's last prompt and reply, and asks the model you chose through Vercel AI Gateway's evaluate endpoint (POST /v1/evaluate, Jev's request and answer, prompt training disallowed). There is no default model.
• With probability ≥ 0.65: needs_input or blocked publishes agent.waiting (Needs you); working publishes agent.ready, which keeps the turn finished and marks the session unfinished (shown as Working; see the last section), both with source: "turncheck", the choice and the finished event's seq. working after a finished turn means the agent stopped with its work unfinished: a build still compiling, something to happen first, work parked until a later day, or work it resumes by itself. The instructions sent to Jev say so explicitly; on 15 real finished turns that wording put every one in the column we would put it in by hand (it moved a "we'll wait until Monday" turn from Done to Idle and one "paused until the import finishes" turn from Needs you to Idle, and changed nothing else) (Codex does when a background terminal finishes, with no new prompt hook). That is neither done nor a question, and Idle ("agents open with nothing to do") was otherwise almost always empty. An earlier version sent it to Needs you, which put busy Codex sessions there. done, ready_to_ship and idle leave the turn Done. The phone app labels Idle "ready for a first prompt", which reads a little oddly for these; happy to adjust if you prefer a different column or wording.
• No fallback. No key, an error, a timeout (10 s), an unusable or unsure answer: the session stays finished and berthd logs one line with the reason (never the prompt, reply or key).
• A turn-check wait is soft. The dashboard, phone and notifications show Needs you, but sends are not held: idle() and the now/idle send gates treat it as finished, since the agent is at its prompt, not at an approval. (Otherwise a lead that ends with "teammates are on it" would hold its teammates' reports.)
• Late answers are dropped. Both events carry the finished event's seq; the ledger applies them only if the session is still finished at that seq, inside Publish, so a prompt you send meanwhile always wins.

Settings → Agents → Turn check

Per box: paste an AI Gateway key and Save ("Key saved on ", Remove), then switch it on. The key is kept on the box in the state directory (turncheck.key, 0600), is never returned by the API, logged or put in an event, and goes only to the AI Gateway over https. Use a secret reference instead takes op://… or env://NAME. A Model field comes first; the switch stays off until one is saved. url in turncheck.json can point it at another server that answers the same request (e.g. a self-hosted laya-serve); that server never gets a key.

Privacy

This is the first feature that sends transcript text off the box, so it is opt-in per box and documented next to the existing promise in security.mdx and sessions-and-agents.mdx. Hooks stay content-free; TestTranslateNeverCopiesContent is unchanged.

Tested

• go vet ./...; go test -race ./internal/box/ (incl. 12 turn-check tests: needs_input/blocked → waiting, working → idle, stays finished for done/low confidence/server errors/timeouts, no request when off, late answer dropped, soft wait lets idle and now sends through, request shape, key only over https and only to the gateway, stored key 0600 and never returned); pnpm -C app build, pnpm -C app test (160 pass).
• Live on a Linux box (berthd from this branch, real Claude Code sessions, real Jev; the key first as an env:// reference, and for the last runs pasted in the new Settings screen with the environment variable removed): turns ending on a plain-text question went to Needs you 9 of 10 times (Jev 0.78–0.95); the one miss was on an earlier build of this branch, gave no log line, and did not recur in the 9 runs after it. Every turn that was really finished stayed Done (5 of 5); a send to a flipped session was delivered at once.
• Not run: the Playwright e2e (its Chromium isn't installed here); the Settings section was clicked through in mock mode and against the live box instead.

Unrelated to this change, TestATeamSetupResumesAfterBerthdRestarts failed under -race on a clean main on one run here and passed on another.

working shows as Working, not Idle

Running the earlier version on a busy box, about 40% of finished turns (65 of ~170 over three days) went to Idle. Idle's own text is "Agents open with nothing to do", so finished work looked like it had disappeared from Done. Replaying all 146 verdicts through this branch's own transcript code showed Jev was mostly right: about 43 of 53 working turns really were waiting on a background job, CI, a deploy watch or a scheduled run. The problem was where they went.

• Ledger: turn check's agent.ready no longer moves the session to idle. The turn stays finished, and the session gets Unfinished, which the next state change clears. So the turn still takes prompts, ends waits and delivers held prompts as done. The seq guard is the same as for the waits.
• API: Session.unfinished (also in @berth/plugin's Session).
• App: sessionState reads finished+unfinished as running, so the dashboard's Working column, the sidebar, counts and Home all agree. The card shows "Waiting on its work" with a tip saying it picks up by itself. Done holds only work that is done.
• Review inbox: leaves an unfinished session out unless all is asked for.

Tested

• TestTurnCheckMarksWorkStillGoingOnUnfinished (stays finished and ready for a prompt, enrich says unfinished, cleared by the next turn, a verdict on an older turn is ignored); the Review test extended, and it fails without the filter; derive.test.ts; new e2e/dashboard-turn-check.spec.ts (the card is in Working with the marker and not in Done; a done card stays in Done), 5 of 5 runs.
• go vet ./..., go test -race ./...; typecheck:plugins, check:titles, check:csp, check:themes, pnpm test, pnpm build, tsc -p e2e, the docs build; e2e: rail, home-widgets, first-run, idle, pr-review and the new spec (26 passed).
• TestReviewRoutesOverTheWire and TestPaneDeadOfASignalIsItsShellStatus failed once each in the full -race run and passed 10 of 10 alone, on this branch and on the commit before it. They're flaky under load and unrelated to this change.

What Jev is given

Jev was often judging a turn on its reply alone. user_request came from the last 60 transcript items, so a long turn, or one the agent resumed by itself when its background work reported back, sent it empty: 45 of 189 checked turns on a real box. session_objective was always empty.

• The check looks back up to 10 pages for the turn's prompt (10 of 189 still have none) and sends the session's title as session_objective. Docs, security.mdx and the Settings text say the title is sent.
• Instructions: a wrap-up, a handoff or "the queue is empty, send the next one" is done, even with follow-ups listed. A reply that asks, waits for the user's go-ahead or leaves something waiting on them is needs_input, even with work going on in the background. working needs the agent itself to be waiting on or going on with something specific.
• Replayed against Jev on all 189 real turns (old input and instructions vs new): about 10 move to the right column ("Waiting for the go-ahead to continue": Working → Needs you; "The queue is empty": Working → Done; "Shall I check for a reply later today?": Done → Needs you) and about 3 to a worse one (e.g. "the four points still wait for you": Needs you → Done at 0.56). Many verdicts sit near the 0.65 threshold either way.
• TestTurnCheckSendsTheLongTurnsPromptAndTheObjective (200 tool calls between the prompt and the reply; it fails without the lookback). go vet, go test -race ./... (only the known flake TestPaneDeadOfASignalIsItsShellStatus), pnpm test, pnpm build, the docs build.

🤖 Generated with Claude Code

https://claude.ai/code/session_01V7UV6sY16AabsdHtcyxyjL

[vercel]

@Stener1 is attempting to deploy a commit to the seanbrydonio's projects Team on Vercel.

A member of the Team first needs to authorize it.

[sean-brydon]

Thanks, this is a really thorough PR. Could you split it in two?

Codex turns: happy to take this. notify firing per message explains the Codex turns we've seen showing Done while still working, and the fix looks right. Please open it on its own, rebased on main (it conflicts now).

Turn check: holding for now. The key handling is careful, but:

1. It can send secrets out. PUT /v1/turncheck is also reachable on berthd's local socket, so any agent on the box can call it. It could point url at its own server with key: op://… and receive the secret plus later transcripts. It needs: laptop-only settings (refuse wire.IsLocal), and secret references only sent to the default gateway.
2. Transcripts go out unredacted. Please cap the prompt and run both the prompt and the reply through the same secret detection artifacts use.
3. I'd rather not default to one vendor's model for users' transcripts. Let the user choose the model.

Send the Codex half and I'll merge it quickly. Thanks again!

[Stener1]

@sean-brydon Great! Apologies for the long time it took, I didn't realize a comment had come in, but I'm very happy to be contributing now that I saw it. Will fix straight away.

[Stener1]

@sean-brydon the Codex half is now its own PR: #71 (rebased on main, ready to merge). This PR is turn check only, with your three points addressed; see the description.

[Stener1]

@sean-brydon updated: this is now turn check only, rebased straight on main (the Codex part is #71).

On point 3, I went further than a free-text model field, since AI Gateway's evaluate endpoint only serves Jev. Settings → Agents now asks who checks turns, with nothing picked by default: Jev on AI Gateway, OpenAI Decisions (gpt-6-luna), or Laya on a server you run (no key, and turns stay on your machines). Each provider's key is kept apart and only ever sent to that provider's own URL. Details, including how Laya actually did in a live run, are in the description.

[sean-brydon]

@sean-brydon updated: this is now turn check only, rebased straight on main (the Codex part is #71).

On point 3, I went further than a free-text model field, since AI Gateway's evaluate endpoint only serves Jev. Settings → Agents now asks who checks turns, with nothing picked by default: Jev on AI Gateway, OpenAI Decisions (gpt-6-luna), or Laya on a server you run (no key, and turns stay on your machines). Each provider's key is kept apart and only ever sent to that provider's own URL. Details, including how Laya actually did in a live run, are in the description.

Thanks man! Appreciate that - I'll look into reviewing this later today

[Stener1]

@sean-brydon a heads-up before you review: I rebased on main (only a changelog conflict) and added three commits on top.

• 14b6156 turn check tells parked work from work still running. Using it, "working" turned out to cover two different things: something actually running (a deploy, a build, a watcher) and an agent paused until someone replies or a set time ("watching for an email, checking again in 24 h"). Both showed in Working. Now there's a separate parked answer: the turn stays finished, marked parked (like unfinished), and shows in Idle. It's out of Review and never notifies. The two real sessions that showed the problem are in turncheck_test.go as anonymised fixtures.
• 0a0249b a Home widget's "more" button is a full-size touch target. The PR's demo agents made that button appear, and it failed axe target-size on 5 a11y screens. It's 24 px now. With this, the dark and light a11y runs fail only what main already fails (settings-boxes, fixed in fix(app): a box with no browsers says so outside the list #72, and database-review).
• f1af652 Clean up never ticks a parked agent for you. Parked agents now sit in Idle, so the clean-up dialog would otherwise pre-tick an agent that's waiting on purpose.

Checks: Go build, vet and the turn-check and review tests pass; the app builds and its unit tests pass (392/392); dashboard-turn-check, settings-turn-check and home-widgets e2e pass. go test ./internal/box/ also fails three structured-agent tests and a flaky review-routes test, and those fail the same way on a clean main.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LQ7gUBM8j8uAh3PWTFSYQa