feat(eval): introduce new reporter API

.prettierignore

@@ -0,0 +1,4 @@
+tests/golden/
+bt-reporter-compliance-plan.md
+bt-reporter-test-plan.md
+eval-reporter-design.md

bt-reporter-compliance-plan.md

@@ -0,0 +1,103 @@
+# Plan: Make `bt` Compliant with the Eval Reporter Design
+
+Target: `eval-reporter-design.md`. This plan covers the `bt` side only — the reporter core, built-in reporters, the runner-script wire protocol, and the devserver. SDK-side work (emission hooks for full case fidelity) is a dependency of the final phase, not of this plan. Supersedes `eval-reporters-plan.md`.
+
+**Phase 0 — characterization tests — is specified separately in `bt-reporter-test-plan.md` and must be green on `main` before Phase 1 begins.**
+
+## Where `bt` is today
+
+- `src/eval.rs` parses runner SSE (`handle_sse_event`, ~line 2737) into a legacy `EvalEvent` enum (~2602): `Processing`, `Start`, `Summary`, `Progress` (bar kinds `start`/`increment`/`set_total`/`stop`), `Console`, `Error`, `Dependencies`, `Done`.
+- `EvalUi` (~2797) is a monolithic renderer: progress bars via `MultiProgress`, summary tables, console echo/suppression, deferred errors, JSONL mode, all keyed off constructor flags.
+- `run_eval_attempt` (~725) wires `EvalUi` directly into `drive_eval_runner` (~964).
+- The devserver endpoints drive `drive_eval_runner` with three ad-hoc closures (~1550, ~1679, ~1745): manifest-stdout collection, SSE re-encoding for the browser UI, summary/error collection.
+- The runner scripts (`scripts/eval-runner.py`, `scripts/eval-runner-impl.ts`) ship inside the `bt` binary, so **both ends of the SSE protocol change atomically in one `bt` release**. The only cross-release boundary is runner ↔ installed SDK.
+- Exit/retry logic is independent of rendering: `drive_eval_runner` collects `error_messages` and buffered stderr itself (~980–1026), and the ESM-retry path consumes those. Reporter work cannot break exit codes.
+
+## Compliance gaps, mapped to the design
+
+| Design requirement | `bt` today |
+| --- | --- |
+| Canonical event union with IDs (`runId`/`evalId`/`caseId`) | Legacy events joined by name; evaluator name (progress) and experiment name (start/summary) don't even match |
+| Method-based reporters + manager | One monolithic `EvalUi` |
+| `--reporter` selection, default sets, stdout claiming | Mode flags (`--jsonl`) baked into `EvalUi` |
+| Shared `Terminal` facade | `MultiProgress` owned privately by `EvalUi` |
+| `case:start`/`case:end`, `eval:progress` totals, `case:delta` | Bar-kind progress events only |
+| `onRunEnd` veto → exit code | No reporter influence on exit |
+| Devserver as bridge/collector reporters + wire negotiation | Three ad-hoc closures, one hardcoded vocabulary |
+| structured `error` scope, `console` attribution | Flat message/stack, no scope |
+
+## Phase 1 — Reporter core (behavior-preserving refactor)
+
+**Goal:** the design's machinery exists inside `bt`; output is byte-identical.
+
+1. **Canonical types** in a new `src/eval/reporter.rs` (or module split of `eval.rs`): `EvalReporterEvent` union and payload structs exactly as the design specifies — `EvalRun{run_id, evaluator_count, protocol_version}`, `EvalInfo{run_id, eval_id, name, experiment}`, `EvalCaseInfo`/`EvalCaseResult` (status `completed|errored|skipped`), `EvalEnd`, `EvalRunEnd`, `ReporterError{scope, ...}`, `ConsoleEvent`, `ProgressEvent{eval_id, total_cases}`, `CaseDelta`. Serde-ready from day one (camelCase wire names).
+2. **`EvalReporter` trait**: default no-op methods for every lifecycle event including `on_case_start`/`on_case_end`/`on_case_delta` (uncalled until Phase 3), `on_error`, `on_run_end(&mut self, ...) -> Option<bool>`, plus `finish()` for terminal cleanup.
+3. **`ReporterManager`**: serialized dispatch (trivially — the event loop is single-threaded), per-reporter failure isolation (a reporter error is logged once to stderr, never fatal), guaranteed exactly-once `run:end` (synthesized with `status: errored` if the stream dies; `Drop` safety), veto aggregation returned from `finish()`, and interest advertisement (`wants_case_delta()`).
+4. **`Terminal` facade** wrapping `MultiProgress`: `println` (persistent line, suspends live region), `live_region()` (bars/spinners), `is_interactive()`. Handed to reporters via `EvalReporterContext{terminal, profile, output_file}` — no mode flags in the context.
+5. **Legacy adapter**: a translation layer from today's runner SSE into canonical events, so reporters are written against the final protocol from the start:
+   - `processing` → `run:start` (synthesized `run_id`)
+   - `start` → `eval:start` (`eval_id` = experiment name for now)
+   - `summary` → `eval:end` with `status: completed`, summary attached
+   - progress `increment` → **synthesized `case:end`** `{eval_id: bar name, case_id: synthetic, status: completed}` — the SDKs tick increments exactly once per finished case, so this is honest; scores/duration absent
+   - progress `set_total` → `eval:progress{total_cases}`
+   - progress `start`/`stop` → dropped (reporters create bars on first sight of an unknown `eval_id`)
+   - `error` → `error` with run-level scope; `console` → `console`; `done` → `run:end`
+   - **Resilience rule (permanent, not transitional):** reporters tolerate events for `eval_id`s they haven't seen `eval:start` for. This absorbs the legacy evaluator-name vs experiment-name mismatch without pretending to fix it.
+6. **`FancyReporter`** replaces `EvalUi`: bars positioned by counting `case:end`, totals from `eval:progress`, console echo/suppression, deferred-error footnote, api-key hint. Constructed with `summaries: bool` (default true). JSONL rendering moves out (Phase 2 wires it back via reporter selection); until then the `--jsonl` flag constructs the interim equivalent set internally.
+7. `run_eval_attempt` goes through the manager. Devserver closures untouched.
+
+**Acceptance:** default, `--jsonl`, `--list`, `--verbose` output byte-identical (golden tests against a scripted fake runner); exit codes unchanged; existing `handle_sse_event`/summary-formatting tests pass.
+
+## Phase 2 — Selection, built-ins, exit-code veto
+
+**Goal:** the design's user-facing surface.
+
+1. **Flags** per repo config policy: `--reporter` repeatable + `BRAINTRUST_EVAL_REPORTER` (comma-separated), `--output-file` + `BRAINTRUST_EVAL_OUTPUT_FILE`. Explicit selection replaces the default set.
+2. **Built-ins:** `fancy`, `verbose` (inline errors + stacks, forwarded stderr, per-case lines once case events carry names), `jsonl`, `silent`, `events` (NDJSON canonical stream — nearly free once serde exists). `dot`, `junit`, `github-actions` are declared but **refuse to run** with an actionable error until Phase 3 delivers real case fidelity ("requires per-case results; upgrade braintrust to ≥ X / not yet supported by this runner").
+3. **Default sets** exactly per the design table: `[fancy]`; `--verbose` → `[verbose]`; `--jsonl` → `[jsonl, fancy(summaries: false)]` — stderr keeps bars and footnotes, no summary table, stdout is pure JSONL. Reporter construction options exist internally only (no CLI syntax).
+4. **stdout claiming:** manager errors at startup if two stdout-claiming reporters are installed without file routing.
+5. **Exit-code veto:** `run_eval_attempt` maps any `Some(false)` from `on_run_end` to command failure, alongside (never replacing) the existing exit-status/error-message logic.
+6. Behavior change shipped here, called out in release notes: under `--jsonl`, user `print()` output echoes to stderr instead of interleaving into stdout.
+
+**Acceptance:** `--reporter=silent`/`events` work; `--jsonl` stdout is parseable JSONL under a noisy eval; `--verbose` matches today; vetoes fail the command.
+
+## Phase 3 — Canonical wire protocol in the runner scripts
+
+**Goal:** the runners emit canonical events natively; the legacy adapter becomes a fallback.
+
+1. Runners emit the canonical union over SSE: `run:start` (with `protocolVersion`, `runId`), `eval:start` (emitter-assigned `evalId`, resolving the name-mismatch problem at the source), `eval:progress`, scoped `error`, `console`, `eval:end`, `run:end`. `bt` ships both ends, so this is one atomic change — `bt`'s decoder accepts canonical events first and falls back to the legacy adapter per event name.
+2. **Case events at the best fidelity the installed SDK allows**, feature-detected by the runners (`hasattr` / `typeof`):
+   - New SDK hooks available → real `case:start`/`case:end` with `caseId` = root span ID, status, duration, scores; `case:delta` forwarded from the SDK `stream` plumbing **only when `bt` signals a subscribed reporter** (env var or handshake — interest advertisement crossing the process boundary).
+   - Old SDK → increment-derived synthetic `case:end` (Phase 1 behavior), no deltas. `dot`/`junit`/`github-actions` detect the degraded stream and refuse with the upgrade message.
+3. Dependencies/watch-mode events stay outside the reporter protocol (run-mode machinery, per the design's "what is not a reporter").
+
+**Acceptance:** with a current SDK, `--reporter=dot` renders real per-case status and `junit` writes a valid file (and vetoes on write failure); with an old SDK pinned, both fail actionably and `fancy` still renders bars from synthetic case ends.
+
+## Phase 4 — Devserver as reporters + wire negotiation
+
+**Goal:** the devserver stops being a parallel event-handling path.
+
+1. Replace the three closures with reporters on the same manager: a manifest-stdout collector, a summary/error collector, and an HTTP bridge.
+2. The bridge implements the design's wire-compatibility contract: legacy vocabulary (`start`/`progress`/`summary`/`error`/`done`) by default, canonical events when the client sends `x-bt-stream-fmt`. The browser UI keeps working unchanged; it upgrades whenever the app adopts the canonical format.
+3. Devserver installs its fixed reporter set; `--reporter` is ignored there (documented, not an error).
+
+**Acceptance:** devserver HTTP responses byte-compatible for legacy clients; canonical stream available behind the header; no rendering side effects on the devserver's own stdout/stderr.
+
+## Sequencing and sizing
+
+| Phase | PRs | Depends on |
+| --- | --- | --- |
+| 1 | 2 (types+manager+adapter; FancyReporter+cutover) | — |
+| 2 | 2 (flags+defaults+claiming; veto+events+silent) | 1 |
+| 3 | 2–3 (canonical emission; case events + feature detection; delta subscription) | 1; SDK hooks for full fidelity |
+| 4 | 1–2 | 1 (not 2/3) |
+
+Phases 2 and 4 are independent of each other and of 3; only Phase 3's full-fidelity half waits on SDK releases (JS/Python already have most of the plumbing — `stream` callbacks, per-case root spans, per-case increments). Everything before that point is `bt`-internal with zero coordination.
+
+## Risks and pinned decisions
+
+- **Output-ordering regressions in Phase 1** are the main risk (verbose inline errors interleaved with console lines; footnote after bars clear). Mitigation: golden-output tests with a scripted fake runner before the cutover PR, and `on_error` firing immediately (never deferred to `run:end`) to preserve verbose ordering.
+- **The `Terminal` facade must be the only path to the screen** — a reporter holding its own `eprintln!` reintroduces bar tearing. Enforce by construction: reporters get `&Terminal`, not stdio.
+- **ESM-retry stderr buffering stays where it is** (`drive_eval_runner` + `report_buffered_stderr`): it exists so a failed first attempt's stderr survives the retry, which is run-mode machinery, not rendering. `--verbose` keeps its residual global meaning there.
+- **`--list` remains a run mode**, untouched by reporter selection.
+- **Synthetic case ends are honest but unlabeled** — they carry no scores or duration, and reporters must not invent them. `verbose` per-case lines and `dot` glyph fidelity are gated on real case events, not faked from increments.

bt-reporter-test-plan.md

@@ -0,0 +1,103 @@
+# Plan: Characterization Tests Before the Reporter Refactor
+
+Phase 0 of `bt-reporter-compliance-plan.md`. Everything here lands **before** any reporter code changes, on today's `main`, and must be green there. The refactor's acceptance criterion — byte-identical output — is only meaningful if the bytes are pinned first.
+
+## What exists today (and the gap)
+
+- `tests/eval_fixtures.rs` runs real runtimes (tsx/bun/deno/python) against real eval files and asserts **exit codes** (`expect_success`) and watch behavior — not output content.
+- `tests/eval_dev_server.rs` covers devserver endpoints; wire-level byte assertions need extending.
+- ~80 unit tests in `src/eval.rs` cover SSE parsing and summary formatting.
+- `assert_cmd` + `predicates` are already dev-dependencies. No snapshot crate (plain golden files are fine; `insta` optional).
+
+The gap: **nothing asserts what `bt eval` prints**, on which stream, in what order.
+
+## The fake runner
+
+The mechanism that makes all of this cheap and deterministic:
+
+- `bt` hands runners an SSE callback endpoint via `BT_EVAL_SSE_SOCK` (Unix socket) / `BT_EVAL_SSE_ADDR` (TCP, Windows) — `src/eval.rs:908`.
+- `--runner` / `BT_EVAL_RUNNER` (`src/eval.rs:267`) substitutes the runtime binary.
+
+A fake runner is a small Python script that ignores its argv (the materialized runner script and eval files), connects to the socket from the env var, replays a frame script, and exits with a scripted code. Frame scripts are JSONL data files, one directive per line:
+
+```jsonl
+{"event": "processing", "data": {"evaluators": 2}}
+{"event": "start", "data": {"projectName": "test-project", "experimentName": "exp-1"}}
+{"event": "progress", "data": {"id": "1", "object_type": "task", "format": "code", "output_type": "completion", "name": "exp-1", "event": "start", "data": "{\"type\":\"eval_progress\",\"kind\":\"start\",\"total\":10}"}}
+{"event": "summary", "data": {"projectName": "test-project", "experimentName": "exp-1", "scores": {}}}
+{"event": "done", "data": ""}
+{"exit": 0}
+```
+
+One script interprets many scenario files — the corpus is data, not code. In Phase 3 of the compliance plan, the same corpus becomes the canonical-protocol conformance fixtures (new frame files, same harness).
+
+**Runner-kind nuance (matters for two behaviors):** `bt` selects console policy and ESM-retry eligibility by runner kind — `should_retry_esm` requires the tsx runner (`src/eval.rs:1049`), and `ConsolePolicy::BufferStderr` applies only when retry is allowed. So the harness needs two spawn modes:
+
+1. `--runner <path-to-fake>` → custom runner kind, `ConsolePolicy::Forward`.
+2. A fake binary **named `tsx`** on a test-controlled `PATH` → tsx runner kind, `ConsolePolicy::BufferStderr`, retry-eligible.
+
+Console-routing and retry scenarios must run under mode 2; everything else uses mode 1.
+
+## Golden-output harness
+
+A test helper in a new `tests/eval_golden.rs`:
+
+```text
+run_scenario(frames_file, flags, spawn_mode) -> { stdout, stderr, exit_code }
+```
+
+- Spawns the real `bt` binary via `assert_cmd`, captures stdout and stderr **separately**.
+- Compares each against checked-in golden files: `tests/golden/eval/<scenario>--<mode>.stdout` / `.stderr`, plus expected exit code in the scenario manifest.
+- `UPDATE_GOLDENS=1` regenerates.
+- Captures are non-TTY, which is a feature: `indicatif` hides live bars on a non-terminal stderr, so only persistent lines appear — deterministic by construction. The TTY animation branch is deliberately untested (see Non-goals).
+- Scenario payloads carry all displayed values (names, scores, durations, URLs), so no timestamp/path normalization should be needed; scenarios must avoid embedding temp paths in error messages.
+
+## Scenario corpus
+
+| # | Scenario (frame script) | Modes | What it pins |
+| --- | --- | --- | --- |
+| 1 | Happy path: processing → start → bar progress (start/set_total/increments/stop) → summary → done | default, `--jsonl`, `--verbose` | Persistent lines, summary table vs JSONL line, stdout/stderr separation |
+| 2 | Summary with comparison fields; run with `--profile test-profile` | default, `--jsonl` | `compare_command` enrichment incl. profile flag, table + JSONL forms |
+| 3 | Summary without comparison/metrics | default, `--jsonl` | Minimal-summary rendering |
+| 4 | Errors mid-run: 3 distinct + 1 duplicate + enough to exceed `MAX_DEFERRED_EVAL_ERRORS` | default | Deferred footnote wording, dedup, cap |
+| 5 | Same error frames | `--verbose` | Inline error + stack rendering, **ordering relative to console events** |
+| 6 | Api-key error message | default, `--verbose` | Hint text, hint placement (deferred vs inline) |
+| 7 | Console events: stdout + stderr lines interleaved | default, `--jsonl`, `--list`, `--verbose` | Full routing matrix: stdout forwarding, stderr suppression count |
+| 8 | Same as 7, spawn mode 2 (fake tsx) | default, `--verbose` | `BufferStderr` policy path + `report_buffered_stderr` output |
+| 9 | Crash: frames end with no `done`, exit 3 | default | Footnote still prints, exit code propagates |
+| 10 | `error` events but exit 0; and the empty stream | default | Exit-code independence from rendering |
+| 11 | Unknown SSE event names interleaved with known ones | default | Forward-compat: unknown events ignored silently (Phase 3 dual-emission depends on this) |
+| 12 | Two evaluators interleaved (distinct names, overlapping progress) | default, `--jsonl` | Bar keying by name, per-evaluator summary ordering |
+| 13 | `set_total` below current position; total under `EVAL_MIN_DETERMINATE_TOTAL` | default | Clamping and spinner-vs-bar selection (final output only; live rendering is non-TTY-hidden) |
+| 14 | ESM interop error text on stderr + nonzero exit, spawn mode 2 | default, `--verbose` | Retry actually re-runs, first attempt's buffered stderr replays, "Suppressed N stderr line(s)" comes from the right source |
+
+Frame scripts avoid timing dependence: order frames so the `set_total` smoothing force-paths decide, never `Instant` elapsed intervals.
+
+## Devserver byte-contract tests
+
+Extend `tests/eval_dev_server.rs`, using the same fake runner:
+
+- **`/eval` stream mode**: snapshot the exact SSE frame sequence — event names, JSON key casing, frame framing — for a happy path and an error path. This is the browser UI's wire contract; compliance-plan Phase 4 must leave these snapshots unchanged (legacy format) while adding the negotiated canonical format alongside.
+- **`/eval` non-stream mode**: snapshot the JSON response body; assert the error-event → HTTP-status mapping (including the fallback to 500 and "Eval runner exited with an error").
+- **`/list`**: response shape snapshot.
+
+## Unit tests to add in `src/eval.rs`
+
+Small, on logic that survives the refactor:
+
+- `record_deferred_error`: trims, dedups, respects `MAX_DEFERRED_EVAL_ERRORS`.
+- Progress-total helpers: `should_apply_total_update` force paths and position-clamping (`ensure_total_not_below_position`, `maybe_apply_pending_total` with `force=true`); skip the elapsed-interval branches.
+- `handle_sse_event`: malformed JSON payloads dropped without send; unknown event names produce no event (belt to scenario 11's braces).
+- `EvalUi`: `finish()` idempotent; `Drop` calls `finish()` when not already finished.
+
+## Non-goals
+
+- **TTY animation rendering** — nondeterministic, small code branch, gated cleanly on `is_terminal()`; accept untested.
+- **Time-interval total smoothing** — `Instant`-dependent; scenarios are constructed so it never decides an assertion.
+- **Real-runtime coverage** — `tests/eval_fixtures.rs` already covers spawn/bundling/watch paths with real tsx/bun/deno/python; don't duplicate it. The golden harness runs in milliseconds with no runtime dependency; the two suites are complementary.
+
+## Exit criteria and sizing
+
+- All goldens green on `main` before the Phase 1 branch is cut.
+- Compliance-plan Phase 1's cutover PR must show **zero golden diffs**. Phase 2's deliberate `--jsonl` console change lands as an explicit golden update in the same PR — the diff is the release note.
+- Sizing: fake runner + harness ~1 day, scenario corpus 1–2 days, devserver snapshots ~0.5 day, unit tests ~0.5 day. One PR (or two: harness, then corpus).