Outcome bench on Claude Code, and the Claude fixes it surfaced

[3dgiordano]

What

Runs the outcome bench on Claude Code (Sonnet 5, Opus 5, Opus 5.5, all High), then fixes what those runs showed: host bugs in the Claude adapters, lexicon misses on Claude's closes, one plugin message that ended Opus 5's runs, a new anchored signal for executive, and grader/audit defects that penalised Claude's formats.

Tooling (bench, not CI)

• scripts/claude-bench.js: outcome driver for Claude (driver claude in bench/suite.json). Each invocation gets a scratch HOME and an allowlisted environment, so the baseline has no installed plugin, no user hooks and no inherited session id. Grants live in that HOME, node is fenced to the workspace, and the account's 5h/7d quota is read from the stream (the run waits or stops before exhausting it). Options: --rescore (re-audit and re-grade stored runs, no calls), --plugin-env (measure an opt-in strict gate).
• Each run keeps its session transcript. scripts/claude-hooks.js reads it to show what each hook said, when, after which tool, plus skill loads, blocks and Stop verdicts.
• bench.js picks the driver per suite row and re-reads the session before storing a slot. A case that is run again starts from a clean directory (both drivers).

Plugins (Claude Code)

• Every plugin: a <task-notification> prompt no longer counts as a user turn (measured: 20 of 23 prompts in one cloud session were notifications).
• Every plugin: a resumed session keeps its state. SessionEnd only sweeps by age now, and a new SessionStart(resume|compact) reloads the discipline, as Cursor's sessionStart does. Before this, the retrospective the Stop hook parked (and promised to the user) was lost on resume.
• executive 1.7.0: a document the agent read that later changed on disk, not by the agent, is named on the next prompt. Only the path and mtime are used, never the text; documented in SECURITY.md.
• persistence 0.3.3: the load no longer lists the block's fields. With the field list, Opus 5 stopped on [reasoning_extraction] in 11 of 12 runs; without it, 0 of 3.
• handoff 0.3.2: the "blocked without Blocked-by" finding now carries the skill's rule 5, and a status ended by a full stop is read as the status.
• termination 0.3.1: "under time pressure" counts only when it is the agent's own.
• coverage 0.4.2: accepts bold, numbered and table part lines; hypotheses "still open" are not treated as deferred work.
• epistemic, integrity, progress: patch bumps for the host changes above.

Grader and audit

• epistemic/the-cause-i-named-first: joins hard-wrapped prose, reads **Status:** conjecture, and treats quotes, asides and decision-table arrows as non-claims. Fixtures were added first. Every changed stored verdict was read by hand.
• Audit (POSIX): code carried by a command (heredocs, multi-line scripts, relative module specifiers) is not read as paths, and shell quotes are paired by scanning. Real searches (find /) still void a run.

Measured (n=3 per arm per model, two sessions pooled, n=18 per arm):

• progress: 0/18 → 18/18
• executive: 7/18 → 15/18 before the new signal. With it, 18/18 with the plugin (n=6 per model); that run measured only the plugin arm.
• termination and coverage: at ceiling.
• handoff/decision-file: does not separate the arms on Claude (0/18 → 1/18). The models take the offered exit in both arms, and an opt-in strict gate enforces the block's form, not the delivery.

Details and numbers are in CHANGELOG.md under Unreleased.

Open, left for a maintainer decision:

• handoff/decision-file: the case design on Claude.
• integrity: Opus 5 edits the acceptance test to inject rates. Tests are out of scope for the detector by design.
• epistemic: its every-6-commands cadence is noisy in long sessions.
• persistence: the threshold nudge still names the fields; its effect on Opus 5 is unmeasured.

Checklist

• node scripts/test.js passes locally (149/149; also hosts.js, samples.js, corpus.js, bench-check.js)
• node scripts/version.js --check passes; plugin version bumped if behaviour changed
• CHANGELOG.md has a line under Unreleased (skip for docs-only)
• If a hook's input/output/log changed: plugin README table updated, SECURITY.md contract still true
• Both hosts covered (Claude Code + Cursor), or the README explains why not. The resume/notification fixes and the executive document signal are Claude-side (Cursor has no such events headless). hosts.js --check passes, and the Cursor driver got the same clean-case fix.