diff --git a/.prettierignore b/.prettierignore new file mode 100644 index 0000000..82acf02 --- /dev/null +++ b/.prettierignore @@ -0,0 +1,4 @@ +tests/golden/ +bt-reporter-compliance-plan.md +bt-reporter-test-plan.md +eval-reporter-design.md diff --git a/bt-reporter-compliance-plan.md b/bt-reporter-compliance-plan.md new file mode 100644 index 0000000..9513846 --- /dev/null +++ b/bt-reporter-compliance-plan.md @@ -0,0 +1,103 @@ +# Plan: Make `bt` Compliant with the Eval Reporter Design + +Target: `eval-reporter-design.md`. This plan covers the `bt` side only — the reporter core, built-in reporters, the runner-script wire protocol, and the devserver. SDK-side work (emission hooks for full case fidelity) is a dependency of the final phase, not of this plan. Supersedes `eval-reporters-plan.md`. + +**Phase 0 — characterization tests — is specified separately in `bt-reporter-test-plan.md` and must be green on `main` before Phase 1 begins.** + +## Where `bt` is today + +- `src/eval.rs` parses runner SSE (`handle_sse_event`, ~line 2737) into a legacy `EvalEvent` enum (~2602): `Processing`, `Start`, `Summary`, `Progress` (bar kinds `start`/`increment`/`set_total`/`stop`), `Console`, `Error`, `Dependencies`, `Done`. +- `EvalUi` (~2797) is a monolithic renderer: progress bars via `MultiProgress`, summary tables, console echo/suppression, deferred errors, JSONL mode, all keyed off constructor flags. +- `run_eval_attempt` (~725) wires `EvalUi` directly into `drive_eval_runner` (~964). +- The devserver endpoints drive `drive_eval_runner` with three ad-hoc closures (~1550, ~1679, ~1745): manifest-stdout collection, SSE re-encoding for the browser UI, summary/error collection. +- The runner scripts (`scripts/eval-runner.py`, `scripts/eval-runner-impl.ts`) ship inside the `bt` binary, so **both ends of the SSE protocol change atomically in one `bt` release**. The only cross-release boundary is runner ↔ installed SDK. +- Exit/retry logic is independent of rendering: `drive_eval_runner` collects `error_messages` and buffered stderr itself (~980–1026), and the ESM-retry path consumes those. Reporter work cannot break exit codes. + +## Compliance gaps, mapped to the design + +| Design requirement | `bt` today | +| --- | --- | +| Canonical event union with IDs (`runId`/`evalId`/`caseId`) | Legacy events joined by name; evaluator name (progress) and experiment name (start/summary) don't even match | +| Method-based reporters + manager | One monolithic `EvalUi` | +| `--reporter` selection, default sets, stdout claiming | Mode flags (`--jsonl`) baked into `EvalUi` | +| Shared `Terminal` facade | `MultiProgress` owned privately by `EvalUi` | +| `case:start`/`case:end`, `eval:progress` totals, `case:delta` | Bar-kind progress events only | +| `onRunEnd` veto → exit code | No reporter influence on exit | +| Devserver as bridge/collector reporters + wire negotiation | Three ad-hoc closures, one hardcoded vocabulary | +| structured `error` scope, `console` attribution | Flat message/stack, no scope | + +## Phase 1 — Reporter core (behavior-preserving refactor) + +**Goal:** the design's machinery exists inside `bt`; output is byte-identical. + +1. **Canonical types** in a new `src/eval/reporter.rs` (or module split of `eval.rs`): `EvalReporterEvent` union and payload structs exactly as the design specifies — `EvalRun{run_id, evaluator_count, protocol_version}`, `EvalInfo{run_id, eval_id, name, experiment}`, `EvalCaseInfo`/`EvalCaseResult` (status `completed|errored|skipped`), `EvalEnd`, `EvalRunEnd`, `ReporterError{scope, ...}`, `ConsoleEvent`, `ProgressEvent{eval_id, total_cases}`, `CaseDelta`. Serde-ready from day one (camelCase wire names). +2. **`EvalReporter` trait**: default no-op methods for every lifecycle event including `on_case_start`/`on_case_end`/`on_case_delta` (uncalled until Phase 3), `on_error`, `on_run_end(&mut self, ...) -> Option`, plus `finish()` for terminal cleanup. +3. **`ReporterManager`**: serialized dispatch (trivially — the event loop is single-threaded), per-reporter failure isolation (a reporter error is logged once to stderr, never fatal), guaranteed exactly-once `run:end` (synthesized with `status: errored` if the stream dies; `Drop` safety), veto aggregation returned from `finish()`, and interest advertisement (`wants_case_delta()`). +4. **`Terminal` facade** wrapping `MultiProgress`: `println` (persistent line, suspends live region), `live_region()` (bars/spinners), `is_interactive()`. Handed to reporters via `EvalReporterContext{terminal, profile, output_file}` — no mode flags in the context. +5. **Legacy adapter**: a translation layer from today's runner SSE into canonical events, so reporters are written against the final protocol from the start: + - `processing` → `run:start` (synthesized `run_id`) + - `start` → `eval:start` (`eval_id` = experiment name for now) + - `summary` → `eval:end` with `status: completed`, summary attached + - progress `increment` → **synthesized `case:end`** `{eval_id: bar name, case_id: synthetic, status: completed}` — the SDKs tick increments exactly once per finished case, so this is honest; scores/duration absent + - progress `set_total` → `eval:progress{total_cases}` + - progress `start`/`stop` → dropped (reporters create bars on first sight of an unknown `eval_id`) + - `error` → `error` with run-level scope; `console` → `console`; `done` → `run:end` + - **Resilience rule (permanent, not transitional):** reporters tolerate events for `eval_id`s they haven't seen `eval:start` for. This absorbs the legacy evaluator-name vs experiment-name mismatch without pretending to fix it. +6. **`FancyReporter`** replaces `EvalUi`: bars positioned by counting `case:end`, totals from `eval:progress`, console echo/suppression, deferred-error footnote, api-key hint. Constructed with `summaries: bool` (default true). JSONL rendering moves out (Phase 2 wires it back via reporter selection); until then the `--jsonl` flag constructs the interim equivalent set internally. +7. `run_eval_attempt` goes through the manager. Devserver closures untouched. + +**Acceptance:** default, `--jsonl`, `--list`, `--verbose` output byte-identical (golden tests against a scripted fake runner); exit codes unchanged; existing `handle_sse_event`/summary-formatting tests pass. + +## Phase 2 — Selection, built-ins, exit-code veto + +**Goal:** the design's user-facing surface. + +1. **Flags** per repo config policy: `--reporter` repeatable + `BRAINTRUST_EVAL_REPORTER` (comma-separated), `--output-file` + `BRAINTRUST_EVAL_OUTPUT_FILE`. Explicit selection replaces the default set. +2. **Built-ins:** `fancy`, `verbose` (inline errors + stacks, forwarded stderr, per-case lines once case events carry names), `jsonl`, `silent`, `events` (NDJSON canonical stream — nearly free once serde exists). `dot`, `junit`, `github-actions` are declared but **refuse to run** with an actionable error until Phase 3 delivers real case fidelity ("requires per-case results; upgrade braintrust to ≥ X / not yet supported by this runner"). +3. **Default sets** exactly per the design table: `[fancy]`; `--verbose` → `[verbose]`; `--jsonl` → `[jsonl, fancy(summaries: false)]` — stderr keeps bars and footnotes, no summary table, stdout is pure JSONL. Reporter construction options exist internally only (no CLI syntax). +4. **stdout claiming:** manager errors at startup if two stdout-claiming reporters are installed without file routing. +5. **Exit-code veto:** `run_eval_attempt` maps any `Some(false)` from `on_run_end` to command failure, alongside (never replacing) the existing exit-status/error-message logic. +6. Behavior change shipped here, called out in release notes: under `--jsonl`, user `print()` output echoes to stderr instead of interleaving into stdout. + +**Acceptance:** `--reporter=silent`/`events` work; `--jsonl` stdout is parseable JSONL under a noisy eval; `--verbose` matches today; vetoes fail the command. + +## Phase 3 — Canonical wire protocol in the runner scripts + +**Goal:** the runners emit canonical events natively; the legacy adapter becomes a fallback. + +1. Runners emit the canonical union over SSE: `run:start` (with `protocolVersion`, `runId`), `eval:start` (emitter-assigned `evalId`, resolving the name-mismatch problem at the source), `eval:progress`, scoped `error`, `console`, `eval:end`, `run:end`. `bt` ships both ends, so this is one atomic change — `bt`'s decoder accepts canonical events first and falls back to the legacy adapter per event name. +2. **Case events at the best fidelity the installed SDK allows**, feature-detected by the runners (`hasattr` / `typeof`): + - New SDK hooks available → real `case:start`/`case:end` with `caseId` = root span ID, status, duration, scores; `case:delta` forwarded from the SDK `stream` plumbing **only when `bt` signals a subscribed reporter** (env var or handshake — interest advertisement crossing the process boundary). + - Old SDK → increment-derived synthetic `case:end` (Phase 1 behavior), no deltas. `dot`/`junit`/`github-actions` detect the degraded stream and refuse with the upgrade message. +3. Dependencies/watch-mode events stay outside the reporter protocol (run-mode machinery, per the design's "what is not a reporter"). + +**Acceptance:** with a current SDK, `--reporter=dot` renders real per-case status and `junit` writes a valid file (and vetoes on write failure); with an old SDK pinned, both fail actionably and `fancy` still renders bars from synthetic case ends. + +## Phase 4 — Devserver as reporters + wire negotiation + +**Goal:** the devserver stops being a parallel event-handling path. + +1. Replace the three closures with reporters on the same manager: a manifest-stdout collector, a summary/error collector, and an HTTP bridge. +2. The bridge implements the design's wire-compatibility contract: legacy vocabulary (`start`/`progress`/`summary`/`error`/`done`) by default, canonical events when the client sends `x-bt-stream-fmt`. The browser UI keeps working unchanged; it upgrades whenever the app adopts the canonical format. +3. Devserver installs its fixed reporter set; `--reporter` is ignored there (documented, not an error). + +**Acceptance:** devserver HTTP responses byte-compatible for legacy clients; canonical stream available behind the header; no rendering side effects on the devserver's own stdout/stderr. + +## Sequencing and sizing + +| Phase | PRs | Depends on | +| --- | --- | --- | +| 1 | 2 (types+manager+adapter; FancyReporter+cutover) | — | +| 2 | 2 (flags+defaults+claiming; veto+events+silent) | 1 | +| 3 | 2–3 (canonical emission; case events + feature detection; delta subscription) | 1; SDK hooks for full fidelity | +| 4 | 1–2 | 1 (not 2/3) | + +Phases 2 and 4 are independent of each other and of 3; only Phase 3's full-fidelity half waits on SDK releases (JS/Python already have most of the plumbing — `stream` callbacks, per-case root spans, per-case increments). Everything before that point is `bt`-internal with zero coordination. + +## Risks and pinned decisions + +- **Output-ordering regressions in Phase 1** are the main risk (verbose inline errors interleaved with console lines; footnote after bars clear). Mitigation: golden-output tests with a scripted fake runner before the cutover PR, and `on_error` firing immediately (never deferred to `run:end`) to preserve verbose ordering. +- **The `Terminal` facade must be the only path to the screen** — a reporter holding its own `eprintln!` reintroduces bar tearing. Enforce by construction: reporters get `&Terminal`, not stdio. +- **ESM-retry stderr buffering stays where it is** (`drive_eval_runner` + `report_buffered_stderr`): it exists so a failed first attempt's stderr survives the retry, which is run-mode machinery, not rendering. `--verbose` keeps its residual global meaning there. +- **`--list` remains a run mode**, untouched by reporter selection. +- **Synthetic case ends are honest but unlabeled** — they carry no scores or duration, and reporters must not invent them. `verbose` per-case lines and `dot` glyph fidelity are gated on real case events, not faked from increments. diff --git a/bt-reporter-test-plan.md b/bt-reporter-test-plan.md new file mode 100644 index 0000000..fb599d5 --- /dev/null +++ b/bt-reporter-test-plan.md @@ -0,0 +1,103 @@ +# Plan: Characterization Tests Before the Reporter Refactor + +Phase 0 of `bt-reporter-compliance-plan.md`. Everything here lands **before** any reporter code changes, on today's `main`, and must be green there. The refactor's acceptance criterion — byte-identical output — is only meaningful if the bytes are pinned first. + +## What exists today (and the gap) + +- `tests/eval_fixtures.rs` runs real runtimes (tsx/bun/deno/python) against real eval files and asserts **exit codes** (`expect_success`) and watch behavior — not output content. +- `tests/eval_dev_server.rs` covers devserver endpoints; wire-level byte assertions need extending. +- ~80 unit tests in `src/eval.rs` cover SSE parsing and summary formatting. +- `assert_cmd` + `predicates` are already dev-dependencies. No snapshot crate (plain golden files are fine; `insta` optional). + +The gap: **nothing asserts what `bt eval` prints**, on which stream, in what order. + +## The fake runner + +The mechanism that makes all of this cheap and deterministic: + +- `bt` hands runners an SSE callback endpoint via `BT_EVAL_SSE_SOCK` (Unix socket) / `BT_EVAL_SSE_ADDR` (TCP, Windows) — `src/eval.rs:908`. +- `--runner` / `BT_EVAL_RUNNER` (`src/eval.rs:267`) substitutes the runtime binary. + +A fake runner is a small Python script that ignores its argv (the materialized runner script and eval files), connects to the socket from the env var, replays a frame script, and exits with a scripted code. Frame scripts are JSONL data files, one directive per line: + +```jsonl +{"event": "processing", "data": {"evaluators": 2}} +{"event": "start", "data": {"projectName": "test-project", "experimentName": "exp-1"}} +{"event": "progress", "data": {"id": "1", "object_type": "task", "format": "code", "output_type": "completion", "name": "exp-1", "event": "start", "data": "{\"type\":\"eval_progress\",\"kind\":\"start\",\"total\":10}"}} +{"event": "summary", "data": {"projectName": "test-project", "experimentName": "exp-1", "scores": {}}} +{"event": "done", "data": ""} +{"exit": 0} +``` + +One script interprets many scenario files — the corpus is data, not code. In Phase 3 of the compliance plan, the same corpus becomes the canonical-protocol conformance fixtures (new frame files, same harness). + +**Runner-kind nuance (matters for two behaviors):** `bt` selects console policy and ESM-retry eligibility by runner kind — `should_retry_esm` requires the tsx runner (`src/eval.rs:1049`), and `ConsolePolicy::BufferStderr` applies only when retry is allowed. So the harness needs two spawn modes: + +1. `--runner ` → custom runner kind, `ConsolePolicy::Forward`. +2. A fake binary **named `tsx`** on a test-controlled `PATH` → tsx runner kind, `ConsolePolicy::BufferStderr`, retry-eligible. + +Console-routing and retry scenarios must run under mode 2; everything else uses mode 1. + +## Golden-output harness + +A test helper in a new `tests/eval_golden.rs`: + +```text +run_scenario(frames_file, flags, spawn_mode) -> { stdout, stderr, exit_code } +``` + +- Spawns the real `bt` binary via `assert_cmd`, captures stdout and stderr **separately**. +- Compares each against checked-in golden files: `tests/golden/eval/--.stdout` / `.stderr`, plus expected exit code in the scenario manifest. +- `UPDATE_GOLDENS=1` regenerates. +- Captures are non-TTY, which is a feature: `indicatif` hides live bars on a non-terminal stderr, so only persistent lines appear — deterministic by construction. The TTY animation branch is deliberately untested (see Non-goals). +- Scenario payloads carry all displayed values (names, scores, durations, URLs), so no timestamp/path normalization should be needed; scenarios must avoid embedding temp paths in error messages. + +## Scenario corpus + +| # | Scenario (frame script) | Modes | What it pins | +| --- | --- | --- | --- | +| 1 | Happy path: processing → start → bar progress (start/set_total/increments/stop) → summary → done | default, `--jsonl`, `--verbose` | Persistent lines, summary table vs JSONL line, stdout/stderr separation | +| 2 | Summary with comparison fields; run with `--profile test-profile` | default, `--jsonl` | `compare_command` enrichment incl. profile flag, table + JSONL forms | +| 3 | Summary without comparison/metrics | default, `--jsonl` | Minimal-summary rendering | +| 4 | Errors mid-run: 3 distinct + 1 duplicate + enough to exceed `MAX_DEFERRED_EVAL_ERRORS` | default | Deferred footnote wording, dedup, cap | +| 5 | Same error frames | `--verbose` | Inline error + stack rendering, **ordering relative to console events** | +| 6 | Api-key error message | default, `--verbose` | Hint text, hint placement (deferred vs inline) | +| 7 | Console events: stdout + stderr lines interleaved | default, `--jsonl`, `--list`, `--verbose` | Full routing matrix: stdout forwarding, stderr suppression count | +| 8 | Same as 7, spawn mode 2 (fake tsx) | default, `--verbose` | `BufferStderr` policy path + `report_buffered_stderr` output | +| 9 | Crash: frames end with no `done`, exit 3 | default | Footnote still prints, exit code propagates | +| 10 | `error` events but exit 0; and the empty stream | default | Exit-code independence from rendering | +| 11 | Unknown SSE event names interleaved with known ones | default | Forward-compat: unknown events ignored silently (Phase 3 dual-emission depends on this) | +| 12 | Two evaluators interleaved (distinct names, overlapping progress) | default, `--jsonl` | Bar keying by name, per-evaluator summary ordering | +| 13 | `set_total` below current position; total under `EVAL_MIN_DETERMINATE_TOTAL` | default | Clamping and spinner-vs-bar selection (final output only; live rendering is non-TTY-hidden) | +| 14 | ESM interop error text on stderr + nonzero exit, spawn mode 2 | default, `--verbose` | Retry actually re-runs, first attempt's buffered stderr replays, "Suppressed N stderr line(s)" comes from the right source | + +Frame scripts avoid timing dependence: order frames so the `set_total` smoothing force-paths decide, never `Instant` elapsed intervals. + +## Devserver byte-contract tests + +Extend `tests/eval_dev_server.rs`, using the same fake runner: + +- **`/eval` stream mode**: snapshot the exact SSE frame sequence — event names, JSON key casing, frame framing — for a happy path and an error path. This is the browser UI's wire contract; compliance-plan Phase 4 must leave these snapshots unchanged (legacy format) while adding the negotiated canonical format alongside. +- **`/eval` non-stream mode**: snapshot the JSON response body; assert the error-event → HTTP-status mapping (including the fallback to 500 and "Eval runner exited with an error"). +- **`/list`**: response shape snapshot. + +## Unit tests to add in `src/eval.rs` + +Small, on logic that survives the refactor: + +- `record_deferred_error`: trims, dedups, respects `MAX_DEFERRED_EVAL_ERRORS`. +- Progress-total helpers: `should_apply_total_update` force paths and position-clamping (`ensure_total_not_below_position`, `maybe_apply_pending_total` with `force=true`); skip the elapsed-interval branches. +- `handle_sse_event`: malformed JSON payloads dropped without send; unknown event names produce no event (belt to scenario 11's braces). +- `EvalUi`: `finish()` idempotent; `Drop` calls `finish()` when not already finished. + +## Non-goals + +- **TTY animation rendering** — nondeterministic, small code branch, gated cleanly on `is_terminal()`; accept untested. +- **Time-interval total smoothing** — `Instant`-dependent; scenarios are constructed so it never decides an assertion. +- **Real-runtime coverage** — `tests/eval_fixtures.rs` already covers spawn/bundling/watch paths with real tsx/bun/deno/python; don't duplicate it. The golden harness runs in milliseconds with no runtime dependency; the two suites are complementary. + +## Exit criteria and sizing + +- All goldens green on `main` before the Phase 1 branch is cut. +- Compliance-plan Phase 1's cutover PR must show **zero golden diffs**. Phase 2's deliberate `--jsonl` console change lands as an explicit golden update in the same PR — the diff is the release note. +- Sizing: fake runner + harness ~1 day, scenario corpus 1–2 days, devserver snapshots ~0.5 day, unit tests ~0.5 day. One PR (or two: harness, then corpus). diff --git a/eval-reporter-design.md b/eval-reporter-design.md new file mode 100644 index 0000000..b5eff19 --- /dev/null +++ b/eval-reporter-design.md @@ -0,0 +1,472 @@ +# Eval Reporter System — Target Design + +This document describes the ideal end state of the eval reporter system, designed from first principles. It is a target design, not a migration plan. + +## Design Principles + +1. **One protocol, every host.** A single canonical event stream describes an eval run. The same events are consumed by reporters running inside the SDKs (JS, Python, Go, Java, Ruby), inside the `bt` CLI, and by remote-eval hosts (SDK dev servers serving the playground, `bt devserver`). Every transport — terminal rendering, playground SSE, `bt`'s runner SSE, devserver HTTP — is a serialization of the same canonical events; no host defines its own event vocabulary. +2. **Reporters render; they never decide.** Reporters are pure consumers of the event stream. What runs, in what order, with what concurrency, and what exit code results are decided outside the reporter system (with one narrow exception: `onRunEnd` may veto success — see Exit Codes). +3. **Three concerns, never entangled:** + - **Run mode** — what the runner does (execute, list, watch, sample). Not a reporter concern. + - **Output format** — how results are rendered. This is exactly what a reporter is. + - **Verbosity** — how much a given rendering shows. Expressed by *choosing a different reporter* (`fancy` vs `verbose`), not by flags that mutate reporter behavior. +4. **stdout is machine-owned; stderr is human-owned.** Machine-readable output (JSONL, event streams) goes to stdout. Everything decorative — progress bars, summary tables, echoed user console output, error footnotes — goes to stderr. A pipeline consuming stdout must never see a stray `print()` from user eval code. +5. **Events carry identity, not just names.** Every scoped event carries emitter-assigned IDs (`run_id`, `eval_id`, `case_id`). Names are display metadata. Reporters correlate events by ID; they never join on names. +6. **Reporter failures are non-fatal.** An eval run represents real time and LLM spend. A rendering bug is logged, never fatal — except an artifact reporter's explicit veto at `onRunEnd`. +7. **Big data stays in-process.** `input` / `output` / `expected` never appear on lifecycle events. Reporters that need full case data run inside the SDK process, where they have direct access. The single sanctioned exception is the `case:delta` side channel — live task-output streaming for hosts that render output as it generates (the playground) — and it is emitted only when a subscribed reporter asks for it. + +## The Event Protocol + +### Lifecycle + +```text +run:start + eval:start (one per evaluator; evaluators may interleave) + case:start (one per case; cases may interleave) + case:end + eval:end (terminal per-eval event) +run:end (terminal event, exactly once) +``` + +Side channels, valid at any point between `run:start` and `run:end`: + +```text +error immediate error surfacing; scope carried in payload +console echoed user stdout/stderr; attribution carried in payload +eval:progress announces or revises an eval's expected case total (see notes) +case:delta streaming task output for a live case (opt-in; see notes) +``` + +### Event schema + +```ts +type EvalReporterEvent = + | { type: "run:start"; run: EvalRun } + | { type: "eval:start"; eval: EvalInfo } + | { type: "case:start"; case: EvalCaseInfo } + | { type: "case:end"; case: EvalCaseResult } + | { type: "eval:end"; eval: EvalEnd } + | { type: "run:end"; run: EvalRunEnd } + | { type: "error"; error: ReporterError } + | { type: "console"; log: ConsoleEvent } + | { type: "eval:progress"; progress: ProgressEvent } + | { type: "case:delta"; delta: CaseDelta }; + +interface EvalRun { + runId: string; + evaluatorCount: number; +} + +interface EvalInfo { + runId: string; + evalId: string; // emitter-assigned, unique within the run + name: string; // display name (evaluator name) + experiment?: ExperimentInfo; // project/experiment names, IDs, URLs +} + +interface EvalCaseInfo { + evalId: string; + caseId: string; // emitter-assigned, unique within the eval + index: number; + name?: string; +} + +interface EvalCaseResult extends EvalCaseInfo { + status: "completed" | "errored" | "skipped"; + durationMs: number; + scores: Record; + error?: { message: string; stack?: string }; +} + +interface EvalEnd { + evalId: string; + status: "completed" | "errored"; + durationMs: number; + caseCounts: { completed: number; errored: number; skipped: number }; + summary?: ExperimentSummary; // scores, metrics, comparison, URLs + errors: ReporterError[]; // errors scoped to this eval +} + +interface EvalRunEnd { + runId: string; + status: "completed" | "errored"; + durationMs: number; + errors: ReporterError[]; // errors not attributable to any eval +} + +interface ReporterError { + scope: { runId: string; evalId?: string; caseId?: string }; + message: string; + stack?: string; + status?: number; // HTTP status when the error came from the API +} + +interface ConsoleEvent { + stream: "stdout" | "stderr"; + message: string; + evalId?: string; // when attributable +} + +interface ProgressEvent { + evalId: string; + totalCases: number; // best current estimate; may be revised upward mid-run +} + +interface CaseDelta { + evalId: string; + caseId: string; + kind: "text" | "json" | "reasoning"; + data: string; +} +``` + +Notes on deliberate choices: + +- **A run is one invocation.** In `bt eval` and devservers — hosts with discovery — a run contains many evals. In the in-process SDKs (Go, Java, Ruby today; JS/Python programmatic use), one API call = one eval = one experiment, and a single-eval run is the normal case: the manager synthesizes `run:start`/`run:end` around it. No SDK needs a multi-eval "session" concept for the protocol to hold. +- **`eval:end` is the only terminal per-eval event.** There are no separate `summary` or per-eval `error` lifecycle events; the summary and scoped errors ride on `eval:end`. Side-channel `error` events exist purely for *immediate* rendering. +- **`eval:end.summary` is optional by design, not by accident.** Some SDKs compute summaries locally; others (Go, Java) fetch them from the Braintrust API after traces flush. A host may emit `eval:end` without a summary, or block on the fetch first — both are conforming. Reporters must render sensibly when `summary` is absent. +- **`console` is a host capability, not an SDK obligation.** Console events exist where a host owns a process boundary and can capture streams (`bt` wrapping a runner subprocess, devservers). In-process SDK libraries do not hijack an application's stdout/stderr, so reporters running there simply never receive `console` events — which the all-optional interface tolerates. +- **`eval:progress` carries only what lifecycle events cannot.** With `case:start`/`case:end` as first-class events, completion is derivable: a bar's position is the count of `case:end` events for that eval, and start/stop bracket at `eval:start`/`eval:end`. The single quantitative fact reporters cannot derive is the *expected total* — datasets are lazy iterators, so totals are discovered and revised mid-run. `eval:progress` announces exactly that and nothing else. A reporter that has seen no `eval:progress` event renders indeterminate progress (a spinner). There is deliberately no `increment` event — that would duplicate `case:end`. +- **`case:delta` streams live task output, and it is opt-in.** Live-rendering hosts need output as it is generated — the playground shows tokens streaming per case; a `verbose` terminal reporter may too. This is the one sanctioned exception to keeping big data off the wire, and it is bounded: emitters produce `case:delta` only when an installed reporter implements `onCaseDelta` (the manager advertises interest), and `case:end` never carries final `input`/`output`/`expected` regardless. +- **`caseId` is the case's root span ID.** Every case already gets a root span; using its ID as the case identity links every case event to its trace and permalink for free — a reporter can render a "view trace" link for an errored case with no extra lookup. +- **Status is `completed | errored | skipped` — there is no `failed`.** Evals do not assert; scores are continuous and are carried separately. `errored` means an exception in the task or a scorer. Score-threshold CI gating (fail the run when a score is below X) is a separate feature — a `--fail-under`-style flag that maps scores to exit codes — never a per-case status. +- **`input` / `output` / `expected` are not in any payload.** See Principle 7. +- **The schema is versioned.** `run:start` carries a `protocolVersion` so consumers can detect emitters ahead of or behind them. Unknown event types are ignored. + +## The Reporter Interface + +The lifecycle interface, identical in shape across all three hosts: + +```ts +interface EvalReporter { + onInit?(ctx: EvalReporterContext): Awaitable; + + onRunStart?(run: EvalRun): Awaitable; + onEvalStart?(evalInfo: EvalInfo): Awaitable; + onCaseStart?(caseInfo: EvalCaseInfo): Awaitable; + onCaseEnd?(caseResult: EvalCaseResult): Awaitable; + onEvalEnd?(evalResult: EvalEnd): Awaitable; + onRunEnd?(runResult: EvalRunEnd): Awaitable; + + onError?(error: ReporterError): Awaitable; + onConsole?(log: ConsoleEvent): Awaitable; + onProgress?(progress: ProgressEvent): Awaitable; + onCaseDelta?(delta: CaseDelta): Awaitable; +} +``` + +The TypeScript shape above is the reference, but the contract is defined language-neutrally: + +- **Every method is optional** — a reporter implements only what it renders. How "optional" is expressed follows each language's idiom: optional methods in TS, default no-op implementations in Python, a trait with default methods plus a `finish()` cleanup hook in Rust, interfaces with default methods in Java (already the house idiom there — see `Scorer`), duck typing via `respond_to?` in Ruby, and in Go — the one language with no optional interface methods — an embeddable `NoopReporter` base struct. +- **Methods are synchronous by default.** `Awaitable` is a JS-ism; async hosts may await reporter methods, sync hosts call them directly. A reporter must not assume it can block the event loop or the eval workers — dispatch happens on the manager's cadence (see below). + +### The context + +```ts +interface EvalReporterContext { + terminal: Terminal; // shared output facade — see Output Contract + profile?: string; // active config profile, for rendering correct command hints + outputFile?: string; // resolved output path for artifact reporters +} +``` + +The context is small on purpose. It does **not** carry mode flags (`jsonl`, `list`, `verbose`): output format is expressed by *which* reporters are installed, not by flags reporters must each interpret. If a concern seems to need a context flag, it is usually either a run mode (not a reporter concern) or a missing reporter variant. + +### The error contract + +`onError` fires immediately when an error occurs, so interactive reporters can render it in real time ("your API key is invalid" must not wait for the end of the run). The same errors appear aggregated on `eval:end` (scoped) or `run:end` (unscoped). Summarizing reporters simply don't implement `onError`; interactive ones render it and skip the aggregates. Nothing is lost by ignoring either half. + +## The Reporter Manager + +Each host runs one manager. Its responsibilities: + +- **Dispatch** every event, in arrival order, to every installed reporter. +- **Serialize dispatch.** Eval execution is concurrent in most hosts (goroutine worker pools in Go, thread pools in Ruby and Python, interleaved async in JS), but the manager delivers events to reporters **one at a time, in emission order** — reporters never need internal locking. Cross-case and cross-eval events interleave (that's real concurrency), but per-scope ordering is guaranteed: `case:start` before that case's `case:end`, every case event inside its eval's `eval:start`/`eval:end` bracket. Each host funnels events through whatever serializer is idiomatic (a channel in Go, a queue/mutex in Ruby, a synchronized dispatcher in Java). +- **Isolate failures**: a reporter method that throws is caught; the failure is reported once to stderr and the run continues. (A reporter that throws persistently may be disabled for the rest of the run.) +- **Guarantee termination**: `onRunEnd` fires exactly once, even when the producer crashes or the process is interrupted — the manager synthesizes a `run:end` with `status: "errored"` if the stream ends without one. +- **Advertise interest**: expensive side channels are produced only when someone is listening. If no installed reporter implements `onCaseDelta`, the emitter skips delta generation entirely. +- **Aggregate the exit verdict** — see below. + +### Exit codes + +The process exit code is decided by the run outcome (runner exit status, run-level errors), not by reporters — with one exception: `onRunEnd` may return `false` to veto success. The manager aggregates these verdicts and the host maps "any veto" to a non-zero exit. This exists for artifact reporters: if `junit` cannot write the file CI depends on, the command must fail even though the evals succeeded. No other reporter method can influence exit status. + +## Output Contract + +### Streams + +- **stdout** — machine output only. At most one installed reporter may claim stdout (e.g. `jsonl`). Selecting two stdout-claiming reporters without routing one to a file is an error at startup, not silent interleaving. +- **stderr** — everything human: progress, tables, echoed user console output, error footnotes. +- **User console output is always decoration.** A `print()` in eval code is echoed to stderr regardless of mode. It never appears on stdout, so machine streams are always parseable. (Reporters may choose to suppress, count, or annotate echoed output — that's rendering policy.) + +### The shared terminal + +Multiple reporters render to one screen, so raw writes are forbidden. The context provides a `Terminal` facade owning the live region (progress bars/spinners) and coordinated line output: + +```ts +interface Terminal { + println(line: string): void; // persistent line above the live region + liveRegion(): LiveRegion; // progress bars; cleared before final output + isInteractive(): boolean; // TTY + animations enabled + not quiet +} +``` + +Reporters that render progress do so through the live region; reporters that print lines do so through `println`, which suspends the live region to avoid tearing. Cleanup order is a manager contract: live regions are cleared before end-of-run output (footnotes, summaries) prints. Non-TTY output is the same reporter adapting via `isInteractive()` (no animations, plain lines) — not a different reporter. + +## Built-in Reporters + +| Reporter | Stream | What it renders | +| --- | --- | --- | +| `fancy` *(default)* | stderr | Progress bars, experiment summary tables, deferred error footnote, suppressed-stderr count. Adapts to non-TTY. | +| `verbose` | stderr | Everything `fancy` shows, plus one line per case as it completes, inline errors with stacks, and full echoed stderr. Sibling of `fancy`, not a modifier of it. | +| `dot` | stderr | One character per case (`.` completed, `E` errored, `s` skipped), then the summary. | +| `jsonl` | **stdout** | One JSON object per `eval:end` summary. Claims stdout. | +| `events` | **stdout** | The raw canonical event stream as NDJSON, for tooling. Claims stdout. | +| `junit` | file | JUnit XML: suite per eval, testcase per case, `errored` cases as failures. Requires `--output-file`. May veto success via `onRunEnd`. | +| `github-actions` | stderr | Workflow annotations (`::error` etc.) for errored cases and run errors. | +| `silent` | — | Nothing except fatal errors. | + +Custom reporters are an SDK feature: registered in eval code or SDK config, they run inside the runner process with in-process access to full case data (`input`/`output`/`expected`). `bt --reporter` selects built-ins only; it never loads user code. + +## Selection and Configuration + +Per repo configuration policy, every knob is a `clap` flag with a corresponding env var: + +```bash +bt eval . # fancy (default) +bt eval . --reporter=verbose +bt eval . --reporter=dot --reporter=junit --output-file=results.xml +bt eval . --reporter=jsonl > summaries.jsonl +BRAINTRUST_EVAL_REPORTER=github-actions bt eval . +``` + +- `--reporter` (repeatable) / `BRAINTRUST_EVAL_REPORTER` (comma-separated). Explicit selection **replaces** the default set. +- `--output-file` / `BRAINTRUST_EVAL_OUTPUT_FILE`. Applies to the single file-producing reporter; `--output-file junit=path` disambiguates if there is ever more than one. +- Sugar aliases, kept forever, no deprecations: + - `--jsonl` ≡ install `jsonl` plus `fancy` with summary tables disabled (see default sets below). + - `--verbose` ≡ replace the default `fancy` with `verbose` (when no explicit `--reporter` is given). `--verbose` additionally retains its global meaning for diagnostics that live outside the reporter system (e.g. buffered-stderr reporting on retry paths). + +### Default reporter sets in `bt` + +| Invocation | Installed reporters | +| --- | --- | +| `bt eval` | `fancy` | +| `bt eval --verbose` | `verbose` | +| `bt eval --jsonl` | `jsonl` + `fancy({ summaries: false })` | +| `bt eval --reporter=…` | exactly the named reporters, nothing implicit | +| `bt devserver` | fixed bridge/collector set; reporter flags ignored | + +`fancy` alone reproduces the full default terminal experience — progress bars, persistent status lines, console echo and stderr suppression, the deferred-error footnote. There are no always-on companion reporters: console and error policy are part of what distinguishes `fancy` from `verbose`, so they must travel with the rendering reporter. + +Built-in reporters may take construction options (precedent: Vitest's default reporter accepts `summary: false`). The `--jsonl` sugar uses this to preserve today's experience exactly: stdout carries only JSON lines, and stderr keeps the bars and footnotes but **no summary table** — the summary has exactly one home. Options have no CLI syntax initially; they exist for sugar mappings and host defaults. An explicit `--reporter=jsonl --reporter=fancy` composes literally and does show tables — explicit selection means you get what you asked for. + +One deliberate change under `--jsonl`: user `print()` output is echoed to stderr, not stdout (see Output Contract), so stdout is always parseable JSONL. + +### What is *not* a reporter + +- **`--list`** — a run mode: the runner enumerates evaluators and executes nothing. Its output is produced from the discovery result directly (optionally formatted as JSON), not by echoing runner stdout. Vitest models this the same way: `vitest list` is a command, not a reporter. +- **`--filter`, `--first`, `--sample`, `--terminate-on-failure`** — run modes and execution policy. +- **`--profile`** — context data reporters use to render correct command hints. +- **Score-threshold gating** (`--fail-under`-style) — a mapping from scores to exit codes, evaluated by the host after `run:end`; reporters render its verdict but do not compute it. + +## Architecture: Where Reporters Run + +```text +┌───────────────────────────── SDK process ─────────────────────────────┐ +│ eval execution ──emits──▶ in-process ReporterManager │ +│ installed reporters vary by host: │ +│ ├── terminal + custom reporters (standalone use, full data) │ +│ ├── playground SSE bridge (braintrust eval --dev) │ +│ └── bt bridge reporter ──▶ SSE ──▶ bt (below) │ +└───────────────────────────────────────────────────────────────────────┘ + │ + ┌─────────────┴──────────────┐ + ▼ ▼ + ┌──── bt eval (terminal) ────┐ ┌──── bt devserver ────┐ + │ ReporterManager │ │ ReporterManager │ + │ ├── fancy / verbose / │ │ ├── HTTP bridge │ + │ │ dot / jsonl / ... │ │ └── collectors │ + │ └── junit (artifact) │ │ (--reporter ignored;│ + └────────────────────────────┘ │ browser UI renders)│ + └──────────────────────┘ +``` + +- **SDK standalone** (`braintrust eval`, programmatic `Eval()`): the in-process manager dispatches directly to reporters; progress rendering is itself a reporter, not hardcoded in eval execution. +- **SDK dev server / remote evals** (`braintrust eval --dev`): the playground connects to the SDK's dev server over HTTP. Same manager, one reporter: an SSE bridge encoding canonical events for the app — including `case:delta`, so the playground renders task output live. No terminal reporters are installed, so nothing prints to the server process's stdout as a side effect. +- **`bt eval`**: the runner's bridge reporter serializes canonical events over SSE; `bt`'s manager deserializes and dispatches to terminal reporters. The bridge is just another reporter — the SDK doesn't know or care that `bt` is listening. +- **`bt devserver`**: same manager, different reporter set — an HTTP re-encoding bridge and collectors. `--reporter` has no effect; the browser UI is the display. + +**Remote execution is below the protocol.** A scorer (or task) that executes on Braintrust via `function/invoke` is wrapped as an ordinary scorer before the run starts; its scores and errors flow through normal case handling. No event distinguishes local from remote execution — reporters cannot tell, and must not care. + +## Wire Compatibility + +A bridge reporter owns its wire format, and format negotiation is part of the bridge's design — not a temporary migration shim. + +- **The playground bridge serves two formats from one event stream.** Clients request a format per connection via the `x-bt-stream-fmt` header (already an allowed header on the dev servers). Without it, the bridge translates canonical events into the legacy playground vocabulary; with it, the bridge emits canonical events directly. The translation is mechanical and lossy only in ways the legacy format already was: + + | Canonical | Legacy SSE | + | --- | --- | + | `eval:start` | `start` (experiment metadata) | + | `case:delta` | `progress` with `text_delta` / `json_delta` payloads | + | `error` | `error` | + | `eval:end` (summary) | `summary` | + | `run:end` | `done` | + | `run:start`, `case:start`, `case:end`, `eval:progress`, `console` | dropped (no legacy equivalent) | + +- **Consumers detect capability, not version guesswork.** `run:start` carries `protocolVersion`; unknown event types are ignored by all consumers. A client that wants per-case lifecycle, scoped errors, or run totals opts into the canonical format and gets them; a client that never upgrades keeps working indefinitely. +- **The data plane is untouched.** Experiment data still flows through spans/logs; summaries are still computed server-side; `function/invoke` is unchanged. The reporter protocol is presentation-plane only — the sole consumer that ever notices a format change is one that subscribes to a bridge's stream. + +The runner scripts ship inside the `bt` binary, so the SSE encoding never skews across versions. The only skew boundary is runner ↔ installed SDK, handled by feature detection with graceful fallback (old SDK: no case events; case-dependent reporters fail with a clear "upgrade braintrust to ≥ X" message). + +## Portability Across SDKs + +The design was checked against all five SDKs. The lifecycle interface is idiomatic in each; the per-SDK work is additive (new value types and emission points), never breaking: + +| SDK | Fits today | Work required to adopt | +| --- | --- | --- | +| JavaScript | Richest starting point: a named-reporter registry (`Reporter`, `reportEval`/`reportRun`), a `ProgressReporter` interface, per-case `reportProgress` → `stream` plumbing keyed by root span ID, and a dev server that already drives the same `Eval()` engine via injected callbacks. | Adapt legacy `Reporter`/`reportRun` to the lifecycle interface behind adapters; replace the ad-hoc `stream`/`onStart` callback injection with the manager + bridge reporters; emit `case:start`/`case:end` (case identity and per-case ticks already exist); move `BarProgressReporter`'s hardcoded increments into a reporter; route `reportRun`'s exit-code logic through `onRunEnd`. | +| Python | Mirror of JS: `ReporterDef` (`report_eval`/`report_run`), `stream`/`on_start` callbacks into `EvalAsync`, per-case root spans, and a dev server sharing the same `run_evaluator` engine. | Same shape as JS: legacy-reporter adapters, manager + bridge reporters in place of callback injection, `case:start`/`case:end` emission points, and stop firing `default_reporter` as a stdout side effect in dev-server mode. | +| Go | Generic `Evaluator[I,R].Run` with clean seams at exactly the callback boundaries; `context.Context` plumbed throughout. | Introduce a per-case result type (case data currently escapes only as OTel spans); serialize dispatch from the worker pool through a channel; `NoopReporter` embed for optionality. | +| Java | `Eval.builder()` + default-method interfaces are the house idioms; Devserver already emits `progress`/`summary`/`done`/`error` over SSE. | Introduce per-case/summary value types (`EvalResult` carries only URLs today); unify the duplicated `Eval` vs `Devserver` execution paths behind one event-emitting executor — the reporter manager is the tool that collapses them. | +| Ruby | Closest fit among the newer SDKs: an `on_progress` per-case callback, summary view models (`ExperimentSummary`, `ScorerStats`), and an SSE devserver already exist. | Structure errors (currently collected as strings, discarding the exception and case identity); add the missing emission points (`run/eval/case:start`); serialize dispatch from worker threads. | + +Recurring themes the protocol already accommodates by design: single-eval runs are the norm in-process (see "A run is one invocation"), summaries may be server-fetched (`eval:end.summary` optional), console capture doesn't exist in-process (`console` is a host capability), and concurrent execution is hidden behind the manager's serialized dispatch. + +## Example Reporters + +Illustrative TypeScript sketches, not production code. Each shows which lifecycle methods a real reporter of that kind needs — and, as importantly, which it can ignore. + +### `fancy` (default) + +Progress bars driven by case lifecycle plus `eval:progress` totals; errors deferred to a footnote. All output through the shared `Terminal`. + +```ts +class FancyReporter implements EvalReporter { + private terminal!: Terminal; + private bars = new Map(); // evalId → bar + private deferred: ReporterError[] = []; + + onInit(ctx: EvalReporterContext) { + this.terminal = ctx.terminal; + } + onEvalStart(e: EvalInfo) { + this.bars.set(e.evalId, this.terminal.liveRegion().addBar(e.name)); + } + onCaseEnd(c: EvalCaseResult) { + this.bars.get(c.evalId)?.increment(); // position = case:end count + } + onProgress(p: ProgressEvent) { + this.bars.get(p.evalId)?.setTotal(p.totalCases); // spinner until first total + } + onError(err: ReporterError) { + this.deferred.push(err); // render later; verbose sibling prints inline instead + } + onEvalEnd(e: EvalEnd) { + this.bars.get(e.evalId)?.finish(); + if (e.summary) this.terminal.println(formatSummaryTable(e.summary)); + } + onRunEnd(r: EvalRunEnd) { + for (const err of this.deferred) { + this.terminal.println(` - ${err.message}`); + } + } +} +``` + +### `jsonl` + +The machine reporter: one JSON line per completed eval, nothing else. Claims stdout, so the manager rejects a second stdout-claiming reporter at startup. + +```ts +class JsonlReporter implements EvalReporter { + onEvalEnd(e: EvalEnd) { + if (e.summary) { + process.stdout.write(JSON.stringify(e.summary) + "\n"); + } + } + // No other methods. Progress, console, and errors are stderr concerns + // handled by other reporters; stdout stays pure. +} +``` + +### Remote evals (playground SSE bridge) + +The dev server's only reporter. Serializes events onto the HTTP response — canonical when the client asked for it via `x-bt-stream-fmt`, legacy translation otherwise. Implementing `onCaseDelta` is what subscribes the run to output streaming. + +```ts +class PlaygroundBridgeReporter implements EvalReporter { + constructor( + private sse: SSEWriter, + private format: "canonical" | "legacy", + ) {} + + onEvalStart(e: EvalInfo) { + this.format === "canonical" + ? this.sse.event("eval:start", e) + : this.sse.event("start", toLegacyStart(e.experiment)); + } + onCaseDelta(d: CaseDelta) { // presence of this method enables delta emission + this.format === "canonical" + ? this.sse.event("case:delta", d) + : this.sse.event("progress", toLegacyProgress(d)); // text_delta / json_delta + } + onCaseEnd(c: EvalCaseResult) { + if (this.format === "canonical") this.sse.event("case:end", c); + // legacy: dropped — the old vocabulary has no per-case lifecycle + } + onError(err: ReporterError) { + this.sse.event("error", this.format === "canonical" ? err : { message: err.message }); + } + onEvalEnd(e: EvalEnd) { + this.format === "canonical" + ? this.sse.event("eval:end", e) + : e.summary && this.sse.event("summary", toLegacySummary(e.summary)); + } + onRunEnd(r: EvalRunEnd) { + if (this.format === "canonical") this.sse.event("run:end", r); + this.sse.event("done", ""); + this.sse.close(); + } +} +``` + +### `github-actions` + +Emits [workflow commands](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-commands#setting-an-error-message) so errored cases and run errors become annotations on the workflow run. Note the mandated escaping, and the `onRunEnd` veto if annotation output failed. + +```ts +class GitHubActionsReporter implements EvalReporter { + private terminal!: Terminal; + + onInit(ctx: EvalReporterContext) { + this.terminal = ctx.terminal; + } + onCaseEnd(c: EvalCaseResult) { + if (c.status === "errored" && c.error) { + this.annotate("error", `case ${c.name ?? c.index} errored`, c.error.message); + } + } + onError(err: ReporterError) { + this.annotate("error", "eval run error", err.message); + } + onEvalEnd(e: EvalEnd) { + if (e.summary) { + const scores = Object.entries(e.summary.scores) + .map(([name, s]) => `${name}=${s.score.toFixed(2)}`) + .join(" "); + this.annotate("notice", e.summary.experimentName, scores); + } + } + + // ::error title={title}::{message} — data must be escaped per the docs: + // message: % → %25, \r → %0D, \n → %0A; properties additionally : → %3A, , → %2C + private annotate(kind: "error" | "notice", title: string, message: string) { + const prop = (s: string) => + s.replace(/%/g, "%25").replace(/\r/g, "%0D").replace(/\n/g, "%0A") + .replace(/:/g, "%3A").replace(/,/g, "%2C"); + const msg = (s: string) => + s.replace(/%/g, "%25").replace(/\r/g, "%0D").replace(/\n/g, "%0A"); + this.terminal.println(`::${kind} title=${prop(title)}::${msg(message)}`); + } +} +``` diff --git a/scripts/eval-runner-impl.ts b/scripts/eval-runner-impl.ts index c17ed03..ddb46ee 100644 --- a/scripts/eval-runner-impl.ts +++ b/scripts/eval-runner-impl.ts @@ -17,10 +17,13 @@ type MatrixAxis = { }; type EvalResult = { - results: Array<{ error?: unknown }>; + results: Array & { error?: unknown }>; summary: unknown; }; +const REPORTER_PROTOCOL_VERSION = 1; +const REPORTER_RUN_ID = `run-${process.pid}-${Date.now()}`; + type ProgressReporter = { start: (name: string, total: number) => void; stop: (name: string) => void; @@ -1349,21 +1352,12 @@ function sendEvalProgress( kind: "start" | "increment" | "set_total" | "stop", total?: number, ) { - if (!sse) { + if (!sse || kind === "increment" || kind === "stop" || total === undefined) { return; } - sse.send("progress", { - id: `eval-progress:${evaluatorName}`, - object_type: "task", - format: "global", - output_type: "any", - name: evaluatorName, - event: "progress", - data: JSON.stringify({ - type: "eval_progress", - kind, - ...(total !== undefined ? { total } : {}), - }), + sse.send("eval:progress", { + evalId: evaluatorName, + totalCases: total, }); } @@ -1991,7 +1985,7 @@ function sendEvalError(sse: SseWriter | null, err: unknown, status?: number) { ...(status !== undefined ? { status } : {}), }; if (sse) { - sse.send("error", payload); + sse.send("error", { scope: { runId: REPORTER_RUN_ID }, ...payload }); } else { console.error(payload.message); } @@ -2257,6 +2251,17 @@ async function createEvalRunner(config: RunnerConfig): Promise { const noSendLogs = shouldDisableSendLogs(); const parseParent = loadBraintrustUtilParseParent(); const getState = extractGlobalStateGetter(braintrust); + let runStarted = false; + const ensureRunStart = (evaluatorCount: number) => { + if (sse && !runStarted) { + runStarted = true; + sse.send("run:start", { + runId: REPORTER_RUN_ID, + evaluatorCount, + protocolVersion: REPORTER_PROTOCOL_VERSION, + }); + } + }; const makeEvalOptions = ( evaluatorName: string, @@ -2276,10 +2281,33 @@ async function createEvalRunner(config: RunnerConfig): Promise { }, progress: createEvalProgressReporter(sse, evaluatorName), stream: (data: unknown) => { - sse.send("progress", data); + if (!envFlag("BT_EVAL_REPORTER_CASE_DELTA") || !isObject(data)) { + return; + } + const event = + typeof data.event === "string" ? data.event : "text_delta"; + sse.send("case:delta", { + evalId: evaluatorName, + caseId: String(data.id ?? data.caseId ?? "unknown-case"), + kind: event.includes("json") + ? "json" + : event.includes("reason") + ? "reasoning" + : "text", + data: + typeof data.data === "string" + ? data.data + : JSON.stringify(data.data ?? data), + }); }, onStart: (metadata: unknown) => { - sse.send("start", metadata); + const experiment = isObject(metadata) ? metadata : {}; + sse.send("eval:start", { + runId: REPORTER_RUN_ID, + evalId: evaluatorName, + name: evaluatorName, + experiment, + }); }, }; } @@ -2297,6 +2325,7 @@ async function createEvalRunner(config: RunnerConfig): Promise { paramsOverride?: Record, ) => { globalThis._lazy_load = false; + ensureRunStart(1); const evaluatorName = getEvaluatorName(evaluator, projectName); // Only inject CLI params when the evaluator declares a parameters schema. // Drop any --param keys the evaluator doesn't declare so a single command @@ -2333,7 +2362,52 @@ async function createEvalRunner(config: RunnerConfig): Promise { ); } if (sse) { - sse.send("summary", summary); + result.results.forEach((caseResult, index) => { + const realCaseId = + caseResult.rootSpanId ?? caseResult.root_span_id ?? caseResult.spanId; + const caseId = realCaseId + ? String(realCaseId) + : `synthetic-${evaluatorName}-${index}`; + const name = + typeof caseResult.name === "string" ? caseResult.name : undefined; + sse.send("case:start", { + evalId: evaluatorName, + caseId, + index, + ...(name ? { name } : {}), + }); + const error = caseResult.error; + const rawScores = isObject(caseResult.scores) ? caseResult.scores : {}; + const scores = Object.fromEntries( + Object.entries(rawScores).filter( + (entry): entry is [string, number] => typeof entry[1] === "number", + ), + ); + sse.send("case:end", { + evalId: evaluatorName, + caseId, + index, + ...(name ? { name } : {}), + status: error === undefined ? "completed" : "errored", + durationMs: Number( + caseResult.durationMs ?? caseResult.duration_ms ?? 0, + ), + scores, + ...(error === undefined ? {} : { error: serializeError(error) }), + }); + }); + sse.send("eval:end", { + evalId: evaluatorName, + status: failingResults.length === 0 ? "completed" : "errored", + durationMs: 0, + caseCounts: { + completed: result.results.length - failingResults.length, + errored: failingResults.length, + skipped: 0, + }, + summary, + errors: [], + }); } else if (config.jsonl) { console.log(JSON.stringify(summary)); } @@ -2341,9 +2415,7 @@ async function createEvalRunner(config: RunnerConfig): Promise { }; const runRegisteredEvals = async (evaluators: EvaluatorEntry[]) => { - if (sse) { - sse.send("processing", { evaluators: evaluators.length }); - } + ensureRunStart(evaluators.length); const reporters = getReporters(); const runEntry = async (entry: EvaluatorEntry): Promise => { try { @@ -2367,7 +2439,10 @@ async function createEvalRunner(config: RunnerConfig): Promise { return true; } catch (err) { if (sse) { - sse.send("error", serializeError(err)); + sse.send("error", { + scope: { runId: REPORTER_RUN_ID }, + ...serializeError(err), + }); } else { console.error(err); } @@ -2394,7 +2469,12 @@ async function createEvalRunner(config: RunnerConfig): Promise { const finish = (ok: boolean) => { if (sse) { sse.send("dependencies", { files: collectDependencyFiles() }); - sse.send("done", ""); + sse.send("run:end", { + runId: REPORTER_RUN_ID, + status: ok ? "completed" : "errored", + durationMs: 0, + errors: [], + }); sse.close(); } if (!ok) { @@ -2453,7 +2533,7 @@ export async function main() { await runner.login({}); } catch (err) { if (runner.sse) { - runner.sse.send("error", serializeError(err)); + sendEvalError(runner.sse, err); } else { console.error(err); } @@ -2510,7 +2590,7 @@ export async function main() { const message = "--matrix-param is not supported for eval files that export btEvalMain. Remove the btEvalMain export or drop --matrix-param."; if (runner.sse) { - runner.sse.send("error", serializeError(new Error(message))); + sendEvalError(runner.sse, new Error(message)); } else { console.error(message); } @@ -2524,7 +2604,7 @@ export async function main() { } catch (err) { ok = false; if (runner.sse) { - runner.sse.send("error", serializeError(err)); + sendEvalError(runner.sse, err); } else { console.error(err); } @@ -2540,7 +2620,7 @@ export async function main() { : names.map((n) => ` - ${n}`).join("\n"); const message = `--matrix-param is not supported when running multiple evals.\nMatched evals:\n${listed}\nUse --filter to select exactly one eval.`; if (runner.sse) { - runner.sse.send("error", serializeError(new Error(message))); + sendEvalError(runner.sse, new Error(message)); } else { console.error(message); } diff --git a/scripts/eval-runner.py b/scripts/eval-runner.py index 977b7bb..7d29d01 100755 --- a/scripts/eval-runner.py +++ b/scripts/eval-runner.py @@ -1,4 +1,6 @@ #!/usr/bin/env python3 +from __future__ import annotations + import argparse import asyncio import builtins @@ -63,6 +65,8 @@ def bt_iscoroutinefunction(f): "/venv/", ) _DATASET_TOTAL_CACHE: dict[str, int] = {} +REPORTER_PROTOCOL_VERSION = 1 +REPORTER_RUN_ID = f"run-{os.getpid()}-{int(__import__('time').time() * 1000)}" @dataclass(frozen=True) @@ -370,22 +374,9 @@ def format_summary(summary: dict[str, Any], config: RunnerConfig) -> dict[str, A def send_eval_progress(sse: SseWriter | None, evaluator_name: str, kind: str, total: int | None = None) -> None: - if not sse: + if not sse or kind in {"increment", "stop"} or total is None: return - payload = { - "id": f"eval-progress:{evaluator_name}", - "object_type": "task", - "format": "global", - "output_type": "any", - "name": evaluator_name, - "event": "progress", - "data": json.dumps({ - "type": "eval_progress", - "kind": kind, - **({"total": total} if total is not None else {}), - }), - } - sse.send("progress", payload) + sse.send("eval:progress", {"evalId": evaluator_name, "totalCases": total}) def create_progress_reporter(sse: SseWriter | None, evaluator_name: str) -> Callable[[str, int | None], None] | None: @@ -403,7 +394,7 @@ def serialize_error( stack: str | None = None, status: int | None = None, ) -> dict[str, Any]: - data = {"message": message} + data = {"scope": {"runId": REPORTER_RUN_ID}, "message": message} if stack: data["stack"] = stack if status is not None: @@ -411,6 +402,66 @@ def serialize_error( return data +def send_canonical_eval_result( + sse: SseWriter, + evaluator_name: str, + result: Any, + summary: dict[str, Any], +) -> None: + counts = {"completed": 0, "errored": 0, "skipped": 0} + for index, row in enumerate(result.results): + real_case_id = ( + getattr(row, "root_span_id", None) + or getattr(row, "rootSpanId", None) + or getattr(row, "span_id", None) + ) + case_id = str(real_case_id) if real_case_id else f"synthetic-{evaluator_name}-{index}" + name = getattr(row, "name", None) + case_info = { + "evalId": evaluator_name, + "caseId": case_id, + "index": index, + **({"name": name} if isinstance(name, str) else {}), + } + sse.send("case:start", case_info) + error = getattr(row, "error", None) + status = "errored" if error else "completed" + counts[status] += 1 + raw_scores = getattr(row, "scores", None) + scores = { + str(key): float(value) + for key, value in (raw_scores.items() if isinstance(raw_scores, dict) else []) + if isinstance(value, (int, float)) + } + error_payload = None + if error: + error_payload = { + "message": str(error), + **({"stack": row.exc_info} if getattr(row, "exc_info", None) else {}), + } + sse.send( + "case:end", + { + **case_info, + "status": status, + "durationMs": int(getattr(row, "duration_ms", 0) or 0), + "scores": scores, + **({"error": error_payload} if error_payload else {}), + }, + ) + sse.send( + "eval:end", + { + "evalId": evaluator_name, + "status": "errored" if counts["errored"] else "completed", + "durationMs": 0, + "caseCounts": counts, + "summary": summary, + "errors": [], + }, + ) + + def infer_eval_error_status(message: str) -> int: text = message.lower() if "not found" in text: @@ -777,14 +828,19 @@ def send_experiment_start( try: summary = experiment.summarize(summarize_scores=False) sse.send( - "start", + "eval:start", { - "projectName": getattr(summary, "project_name", None), - "experimentName": getattr(summary, "experiment_name", None), - "projectId": getattr(summary, "project_id", None), - "experimentId": getattr(summary, "experiment_id", None), - "projectUrl": getattr(summary, "project_url", None), - "experimentUrl": getattr(summary, "experiment_url", None), + "runId": REPORTER_RUN_ID, + "evalId": evaluator.eval_name, + "name": evaluator.eval_name, + "experiment": { + "projectName": getattr(summary, "project_name", None), + "experimentName": getattr(summary, "experiment_name", None), + "projectId": getattr(summary, "project_id", None), + "experimentId": getattr(summary, "experiment_id", None), + "projectUrl": getattr(summary, "project_url", None), + "experimentUrl": getattr(summary, "experiment_url", None), + }, }, ) return @@ -795,7 +851,15 @@ def send_experiment_start( evaluator, "eval_name", None ) if experiment_name: - sse.send("start", {"experimentName": experiment_name}) + sse.send( + "eval:start", + { + "runId": REPORTER_RUN_ID, + "evalId": evaluator.eval_name, + "name": evaluator.eval_name, + "experiment": {"experimentName": experiment_name}, + }, + ) def run_evaluator_progress_mode() -> str: @@ -1117,8 +1181,20 @@ async def run_evaluator_task( kwargs = {} if progress_cb and progress_mode == "progress": kwargs["progress"] = progress_cb - if sse and supports_stream: - kwargs["stream"] = lambda event: sse.send("progress", event if isinstance(event, dict) else event.__dict__) + if sse and supports_stream and env_flag("BT_EVAL_REPORTER_CASE_DELTA"): + def stream_event(event): + payload = event if isinstance(event, dict) else event.__dict__ + event_name = str(payload.get("event", "text_delta")) + sse.send( + "case:delta", + { + "evalId": evaluator.eval_name, + "caseId": str(payload.get("id") or payload.get("caseId") or "unknown-case"), + "kind": "json" if "json" in event_name else "reasoning" if "reason" in event_name else "text", + "data": payload.get("data") if isinstance(payload.get("data"), str) else json.dumps(payload.get("data", payload)), + }, + ) + kwargs["stream"] = stream_event if parent: with parent_context(parent): @@ -1234,7 +1310,7 @@ async def run_requested_eval( return False if sse: - sse.send("summary", format_summary(result.summary.as_dict(), config)) + send_canonical_eval_result(sse, evaluator.eval_name, result, format_summary(result.summary.as_dict(), config)) elif config.jsonl: print(json.dumps(format_summary(result.summary.as_dict(), config))) else: @@ -1266,6 +1342,15 @@ async def run_once( return True evaluators = filter_evaluators(evaluators, config.filters) + if sse and not config.list_only and config.dev_mode != "list": + sse.send( + "run:start", + { + "runId": REPORTER_RUN_ID, + "evaluatorCount": len(evaluators), + "protocolVersion": REPORTER_PROTOCOL_VERSION, + }, + ) if config.dev_mode == "list": print(json.dumps(build_eval_definitions(evaluators))) return True @@ -1320,9 +1405,6 @@ async def run_once( expanded.append(cloned_instance) evaluators = expanded - if sse: - sse.send("processing", {"evaluators": len(evaluators)}) - progress_mode = run_evaluator_progress_mode() async def run_single_evaluator( @@ -1391,7 +1473,12 @@ async def run_single_evaluator( continue if sse: - sse.send("summary", format_summary(result.summary.as_dict(), config)) + send_canonical_eval_result( + sse, + evaluator_instance.evaluator.eval_name, + result, + format_summary(result.summary.as_dict(), config), + ) elif config.jsonl: print(json.dumps(format_summary(result.summary.as_dict(), config))) else: @@ -1454,7 +1541,15 @@ def main(argv: list[str] | None = None) -> int: if sse: sse.send("dependencies", {"files": collect_dependency_files(cwd, files)}) - sse.send("done", {"success": success}) + sse.send( + "run:end", + { + "runId": REPORTER_RUN_ID, + "status": "completed" if success else "errored", + "durationMs": 0, + "errors": [], + }, + ) return 0 if success else 1 finally: if sse: diff --git a/src/eval.rs b/src/eval.rs index ebc4f4d..082c776 100644 --- a/src/eval.rs +++ b/src/eval.rs @@ -4,7 +4,7 @@ use std::io::IsTerminal; use std::path::{Path, PathBuf}; use std::process::{ExitStatus, Stdio}; use std::sync::atomic::{AtomicBool, Ordering}; -use std::sync::Arc; +use std::sync::{Arc, Mutex}; use std::time::{Duration, SystemTime}; use actix_web::dev::Service; @@ -36,6 +36,12 @@ use crate::ui::{ SummaryMetricRow, SummaryTableOptions, }; +mod reporter; +use reporter::{ + decode_canonical_sse_event, CaseStatus, ConsoleStream, EvalReporter, EvalReporterEvent, + EvalStatus, LegacyEventAdapter, ReporterManager, +}; + const MAX_NAME_LENGTH: usize = 40; const WATCH_POLL_INTERVAL: Duration = Duration::from_millis(500); const MAIN_ORIGIN: &str = "https://www.braintrust.dev"; @@ -250,6 +256,18 @@ pub enum EvalLanguage { Python, } +#[derive(Debug, Copy, Clone, Eq, PartialEq, ValueEnum)] +pub enum EvalReporterName { + Fancy, + Verbose, + Jsonl, + Silent, + Events, + Dot, + Junit, + GithubActions, +} + #[derive(Debug, Clone, Args)] #[command(after_help = "\ Examples: @@ -295,6 +313,24 @@ pub struct EvalArgs { )] pub jsonl: bool, + /// Reporter used to render eval results. Repeat to compose reporters. Ignored with --dev. + #[arg( + long = "reporter", + env = "BRAINTRUST_EVAL_REPORTER", + value_enum, + value_delimiter = ',', + value_name = "REPORTER" + )] + pub reporters: Vec, + + /// Output path for artifact reporters such as junit. Ignored with --dev. + #[arg( + long = "output-file", + env = "BRAINTRUST_EVAL_OUTPUT_FILE", + value_name = "PATH" + )] + pub output_file: Option, + /// Stop after the first failing evaluator. #[arg( long, @@ -436,6 +472,8 @@ enum EvalSamplingMode { #[derive(Debug, Clone)] struct EvalRunOptions { jsonl: bool, + reporters: Vec, + output_file: Option, terminate_on_failure: bool, num_workers: Option, list: bool, @@ -487,6 +525,8 @@ pub async fn run(base: BaseArgs, args: EvalArgs) -> Result<()> { let options = EvalRunOptions { jsonl: args.jsonl, + reporters: args.reporters, + output_file: args.output_file, terminate_on_failure: args.terminate_on_failure, num_workers: args.num_workers, list: args.list, @@ -619,6 +659,7 @@ struct EvalPlan<'a> { struct EvalAttemptOutput { status: ExitStatus, + reporter_vetoed: bool, dependency_files: Vec, error_messages: Vec, stderr_lines: Vec, @@ -703,6 +744,9 @@ async fn run_eval_files_once( if let Some(message) = missing_vite_node_retry_message(&output) { anyhow::bail!(message); } + if output.reporter_vetoed { + anyhow::bail!("an eval reporter vetoed the successful run"); + } let dependencies = if collect_dependencies { let mut dependencies = @@ -722,6 +766,64 @@ async fn run_eval_files_once( }) } +fn build_eval_reporters( + options: &EvalRunOptions, + profile: Option, +) -> Result>> { + let mut reporters: Vec> = Vec::new(); + if options.reporters.is_empty() { + if options.jsonl { + reporters.push(Box::new(JsonlReporter::default())); + reporters.push(Box::new(FancyReporter::new(false, false, false, profile))); + } else { + reporters.push(Box::new(FancyReporter::new( + true, + options.list, + options.verbose, + profile, + ))); + } + } else { + for reporter in &options.reporters { + match reporter { + EvalReporterName::Fancy => reporters.push(Box::new(FancyReporter::new( + true, + options.list, + false, + profile.clone(), + ))), + EvalReporterName::Verbose => reporters.push(Box::new(FancyReporter::new( + true, + options.list, + true, + profile.clone(), + ))), + EvalReporterName::Jsonl => reporters.push(Box::new(JsonlReporter::default())), + EvalReporterName::Silent => reporters.push(Box::new(SilentReporter)), + EvalReporterName::Events => reporters.push(Box::new(EventsReporter)), + EvalReporterName::Dot => reporters.push(Box::new(DotReporter::new())), + EvalReporterName::Junit => { + if options.output_file.is_none() { + anyhow::bail!("--reporter=junit requires --output-file "); + } + reporters.push(Box::new(JunitReporter::new())); + } + EvalReporterName::GithubActions => { + reporters.push(Box::new(GithubActionsReporter { + terminal: None, + degraded: false, + })); + } + } + } + } + + if options.output_file.is_some() && !options.reporters.contains(&EvalReporterName::Junit) { + anyhow::bail!("--output-file requires --reporter=junit"); + } + Ok(reporters) +} + async fn run_eval_attempt( base: &BaseArgs, plan: &EvalPlan<'_>, @@ -731,6 +833,19 @@ async fn run_eval_attempt( js_mode: JsMode, console_policy: ConsolePolicy, ) -> Result { + let selected_reporters = build_eval_reporters(options, base.profile.clone())?; + let mut reporters = ReporterManager::new( + selected_reporters, + base.profile.clone(), + options.output_file.clone(), + )?; + let mut adapter = LegacyEventAdapter::new(reporters.run_id().to_string()); + let mut runner_env = extra_env.to_vec(); + if reporters.wants_case_delta() { + // Process-internal protocol negotiation; this is not user configuration. + runner_env.push(("BT_EVAL_REPORTER_CASE_DELTA".to_string(), "1".to_string())); + } + let spawned = spawn_eval_runner( base, plan.language, @@ -738,22 +853,28 @@ async fn run_eval_attempt( plan.files, no_send_logs, options, - extra_env, + &runner_env, js_mode, ) .await?; - let mut ui = EvalUi::new( - options.jsonl, - options.list, - options.verbose, - base.profile.clone(), - ); - let output = - drive_eval_runner(spawned.process, console_policy, |event| ui.handle(event)).await?; - ui.finish(); + let output = drive_eval_runner(spawned.process, console_policy, |event| match event { + EvalEvent::Reporter(event) => reporters.dispatch(&event), + legacy => { + if let Some(event) = adapter.translate(&legacy) { + reporters.dispatch(&event); + } + } + }) + .await?; + let reporter_vetoed = reporters.finish(if output.status.success() { + EvalStatus::Completed + } else { + EvalStatus::Errored + }); Ok(EvalAttemptOutput { status: output.status, + reporter_vetoed, dependency_files: output.dependency_files, error_messages: output.error_messages, stderr_lines: output.stderr_lines, @@ -987,6 +1108,22 @@ where &sse_connected, "eval runner process exited without a status", |event| match event { + EvalEvent::Reporter(EvalReporterEvent::Error { ref error }) => { + error_messages.push(error.message.clone()); + if let Some(stack) = error.stack.as_ref() { + error_messages.push(stack.clone()); + } + on_event(event); + } + EvalEvent::Reporter(EvalReporterEvent::Console { ref log }) => { + if log.stream == ConsoleStream::Stderr + && matches!(console_policy, ConsolePolicy::BufferStderr) + { + stderr_lines.push(log.message.clone()); + } else { + on_event(event); + } + } EvalEvent::Dependencies { files } => { dependency_files.extend(files.clone()); on_event(EvalEvent::Dependencies { files }); @@ -1380,6 +1517,8 @@ fn make_dev_mode_env( ("BRAINTRUST_ORG_NAME".to_string(), auth.org_name.clone()), ("BRAINTRUST_APP_URL".to_string(), state.app_url.clone()), ("BT_EVAL_DEV_MODE".to_string(), dev_mode.to_string()), + // Process-internal reporter interest negotiation, not user configuration. + ("BT_EVAL_REPORTER_CASE_DELTA".to_string(), "1".to_string()), ]; if let Some(api_url) = auth.api_url.as_ref() { env.push(("BRAINTRUST_API_URL".to_string(), api_url.clone())); @@ -1396,14 +1535,17 @@ fn serialize_sse_event(event: &str, data: &str) -> String { format!("event: {event}\ndata: {data}\n\n") } +#[cfg(test)] fn is_eval_progress_payload(progress: &SseProgressEventData) -> bool { serde_json::from_str::(&progress.data) .map(|payload| payload.kind_type == "eval_progress") .unwrap_or(false) } +#[cfg(test)] fn encode_eval_event_for_http(event: &EvalEvent) -> Option { match event { + EvalEvent::Reporter(_) => None, EvalEvent::Processing(payload) => serde_json::to_string(payload) .ok() .map(|data| serialize_sse_event("processing", &data)), @@ -1499,6 +1641,21 @@ fn apply_cors_headers( } } +fn dispatch_reporter_event( + manager: &mut ReporterManager, + adapter: &mut LegacyEventAdapter, + event: EvalEvent, +) { + match event { + EvalEvent::Reporter(event) => manager.dispatch(&event), + legacy => { + if let Some(event) = adapter.translate(&legacy) { + manager.dispatch(&event); + } + } + } +} + async fn dev_server_list(state: web::Data, req: HttpRequest) -> HttpResponse { let auth = match authenticate_dev_request(&req, &state).await { Ok(auth) => auth, @@ -1544,36 +1701,42 @@ async fn dev_server_list(state: web::Data, req: HttpRequest) -> } }; - let mut stdout_lines = Vec::new(); - let mut errors: Vec<(String, Option)> = Vec::new(); - let output = - match drive_eval_runner( - spawned.process, - ConsolePolicy::Forward, - |event| match event { - EvalEvent::Console { stream, message } if stream == "stdout" => { - stdout_lines.push(message); - } - EvalEvent::Error { - message, - stack: _, - status, - } => errors.push((message, status)), - _ => {} - }, - ) - .await - { - Ok(output) => output, - Err(err) => { - return json_error_response( - actix_web::http::StatusCode::INTERNAL_SERVER_ERROR, - &format!("{err:#}"), - ); - } - }; + let collected = Arc::new(Mutex::new(DevCollectorState::default())); + let collector: Box = Box::new(DevCollectorReporter { + state: Arc::clone(&collected), + }); + let mut manager = match ReporterManager::new(vec![collector], state.base.profile.clone(), None) + { + Ok(manager) => manager, + Err(err) => { + return json_error_response( + actix_web::http::StatusCode::INTERNAL_SERVER_ERROR, + &format!("{err:#}"), + ); + } + }; + let mut adapter = LegacyEventAdapter::new(manager.run_id().to_string()); + let output = match drive_eval_runner(spawned.process, ConsolePolicy::Forward, |event| { + dispatch_reporter_event(&mut manager, &mut adapter, event); + }) + .await + { + Ok(output) => output, + Err(err) => { + return json_error_response( + actix_web::http::StatusCode::INTERNAL_SERVER_ERROR, + &format!("{err:#}"), + ); + } + }; + manager.finish(if output.status.success() { + EvalStatus::Completed + } else { + EvalStatus::Errored + }); + let collected = collected.lock().unwrap(); - if let Some((message, status)) = errors.first() { + if let Some((message, status)) = collected.errors.first() { let status = status .and_then(|status| actix_web::http::StatusCode::from_u16(status).ok()) .unwrap_or(actix_web::http::StatusCode::INTERNAL_SERVER_ERROR); @@ -1587,14 +1750,14 @@ async fn dev_server_list(state: web::Data, req: HttpRequest) -> } let mut parsed_manifest: Option = None; - for line in stdout_lines.iter().rev() { + for line in collected.stdout_lines.iter().rev() { if let Ok(value) = serde_json::from_str::(line) { parsed_manifest = Some(value); break; } } if parsed_manifest.is_none() { - let joined = stdout_lines.join("\n"); + let joined = collected.stdout_lines.join("\n"); if let Ok(value) = serde_json::from_str::(&joined) { parsed_manifest = Some(value); } @@ -1631,6 +1794,11 @@ async fn dev_server_eval( return response; } let stream_requested = eval_request.stream.unwrap_or(false); + let stream_format = if req.headers().contains_key("x-bt-stream-fmt") { + DevStreamFormat::Canonical + } else { + DevStreamFormat::Legacy + }; let extra_env = match make_dev_mode_env(&auth, &state, Some(&eval_request), "eval") { Ok(extra_env) => extra_env, Err(err) => { @@ -1674,57 +1842,71 @@ async fn dev_server_eval( if stream_requested { let (tx, rx) = mpsc::unbounded_channel::(); tokio::spawn(async move { - let mut saw_error = false; - let mut stderr_lines: Vec = Vec::new(); + let collected = Arc::new(Mutex::new(DevCollectorState::default())); + let bridge: Box = Box::new(HttpBridgeReporter { + tx, + format: stream_format, + pending_run_end: None, + }); + let collector: Box = Box::new(DevCollectorReporter { + state: Arc::clone(&collected), + }); + let mut manager = match ReporterManager::new(vec![bridge, collector], None, None) { + Ok(manager) => manager, + Err(_) => return, + }; + let mut adapter = LegacyEventAdapter::new(manager.run_id().to_string()); let output = drive_eval_runner(spawned.process, ConsolePolicy::Forward, |event| { - if matches!(event, EvalEvent::Error { .. }) { - saw_error = true; - } - if matches!(event, EvalEvent::Done) { - return; - } - if let EvalEvent::Console { - ref stream, - ref message, - } = event - { - for line in message.lines() { - let _ = tx.send(format!(": [{stream}] {line}\n")); - } - if stream == "stderr" { - stderr_lines.push(message.clone()); - } - return; - } - if let Some(encoded) = encode_eval_event_for_http(&event) { - let _ = tx.send(encoded); - } + dispatch_reporter_event(&mut manager, &mut adapter, event); }) .await; - match output { + let status = match output { Ok(output) => { - if !output.status.success() && !saw_error { + let state = collected.lock().unwrap(); + if !output.status.success() && state.errors.is_empty() { let mut detail = format!("Eval runner exited with {}.", output.status); - for line in stderr_lines.iter() { + for line in &state.stderr_lines { detail.push('\n'); detail.push_str(line); } - let error = - serialize_sse_event("error", &json!({ "message": detail }).to_string()); - let _ = tx.send(error); + drop(state); + manager.dispatch(&EvalReporterEvent::Error { + error: reporter::ReporterError { + scope: reporter::ErrorScope { + run_id: manager.run_id().to_string(), + eval_id: None, + case_id: None, + }, + message: detail, + stack: None, + status: None, + }, + }); + } + if output.status.success() { + EvalStatus::Completed + } else { + EvalStatus::Errored } } Err(err) => { - let error = serialize_sse_event( - "error", - &json!({ "message": format!("{err:#}") }).to_string(), - ); - let _ = tx.send(error); + manager.dispatch(&EvalReporterEvent::Error { + error: reporter::ReporterError { + scope: reporter::ErrorScope { + run_id: manager.run_id().to_string(), + eval_id: None, + case_id: None, + }, + message: format!("{err:#}"), + stack: None, + status: None, + }, + }); + EvalStatus::Errored } - } - - let _ = tx.send(serialize_sse_event("done", "")); + }; + manager.finish(status); }); let response_stream = stream::unfold(rx, |mut rx| async { @@ -1739,40 +1921,48 @@ async fn dev_server_eval( .streaming(response_stream); } - let mut summary: Option = None; - let mut errors: Vec<(String, Option)> = Vec::new(); - let output = - match drive_eval_runner( - spawned.process, - ConsolePolicy::Forward, - |event| match event { - EvalEvent::Summary(current) => summary = Some(current), - EvalEvent::Error { - message, - stack: _, - status, - } => errors.push((message, status)), - _ => {} - }, - ) - .await - { - Ok(output) => output, - Err(err) => { - return json_error_response( - actix_web::http::StatusCode::INTERNAL_SERVER_ERROR, - &format!("{err:#}"), - ); - } - }; + let collected = Arc::new(Mutex::new(DevCollectorState::default())); + let collector: Box = Box::new(DevCollectorReporter { + state: Arc::clone(&collected), + }); + let mut manager = match ReporterManager::new(vec![collector], state.base.profile.clone(), None) + { + Ok(manager) => manager, + Err(err) => { + return json_error_response( + actix_web::http::StatusCode::INTERNAL_SERVER_ERROR, + &format!("{err:#}"), + ); + } + }; + let mut adapter = LegacyEventAdapter::new(manager.run_id().to_string()); + let output = match drive_eval_runner(spawned.process, ConsolePolicy::Forward, |event| { + dispatch_reporter_event(&mut manager, &mut adapter, event); + }) + .await + { + Ok(output) => output, + Err(err) => { + return json_error_response( + actix_web::http::StatusCode::INTERNAL_SERVER_ERROR, + &format!("{err:#}"), + ); + } + }; + manager.finish(if output.status.success() { + EvalStatus::Completed + } else { + EvalStatus::Errored + }); + let collected = collected.lock().unwrap(); - if let Some((message, status)) = errors.first() { + if let Some((message, status)) = collected.errors.first() { let status = status .and_then(|status| actix_web::http::StatusCode::from_u16(status).ok()) .unwrap_or(actix_web::http::StatusCode::INTERNAL_SERVER_ERROR); return json_error_response(status, message); } - if let Some(summary) = summary { + if let Some(summary) = collected.summary.as_ref() { return HttpResponse::Ok().json(summary); } if !output.status.success() { @@ -2600,6 +2790,7 @@ fn materialize_runner_script(cache_dir: &Path, file_name: &str, source: &str) -> #[derive(Debug)] enum EvalEvent { + Reporter(EvalReporterEvent), Processing(ProcessingEventData), Start(ExperimentStart), Summary(ExperimentSummary), @@ -2625,7 +2816,7 @@ struct ProcessingEventData { evaluators: usize, } -#[derive(Debug, Deserialize, Serialize, Default)] +#[derive(Debug, Clone, Deserialize, Serialize, Default)] #[serde(rename_all = "camelCase")] struct ExperimentStart { #[serde(default, alias = "project_name")] @@ -2643,7 +2834,7 @@ struct ExperimentStart { } #[allow(dead_code)] -#[derive(Debug, Deserialize, Serialize)] +#[derive(Debug, Clone, Deserialize, Serialize)] #[serde(rename_all = "camelCase")] struct ExperimentSummary { project_name: String, @@ -2671,7 +2862,7 @@ struct ExperimentSummary { compare_more: Option, } -#[derive(Debug, Deserialize, Serialize)] +#[derive(Debug, Clone, Deserialize, Serialize)] struct ScoreSummary { name: String, score: f64, @@ -2689,7 +2880,7 @@ struct EvalErrorPayload { status: Option, } -#[derive(Debug, Deserialize, Serialize)] +#[derive(Debug, Clone, Deserialize, Serialize)] struct MetricSummary { name: String, metric: f64, @@ -2703,7 +2894,7 @@ struct MetricSummary { } #[allow(dead_code)] -#[derive(Debug, Deserialize, Serialize)] +#[derive(Debug, Clone, Deserialize, Serialize)] struct SseProgressEventData { id: String, object_type: String, @@ -2736,6 +2927,12 @@ struct SseDependenciesEventData { fn handle_sse_event(event: Option, data: String, tx: &mpsc::UnboundedSender) { let event_name = event.unwrap_or_default(); + if event_name.contains(':') { + if let Some(event) = decode_canonical_sse_event(&event_name, &data) { + let _ = tx.send(EvalEvent::Reporter(event)); + } + return; + } match event_name.as_str() { "processing" => { if let Ok(payload) = serde_json::from_str::(&data) { @@ -2794,16 +2991,570 @@ fn handle_sse_event(event: Option, data: String, tx: &mpsc::UnboundedSen } } -struct EvalUi { - progress: MultiProgress, +#[derive(Default)] +struct DevCollectorState { + stdout_lines: Vec, + stderr_lines: Vec, + summary: Option, + errors: Vec<(String, Option)>, +} + +struct DevCollectorReporter { + state: Arc>, +} + +impl EvalReporter for DevCollectorReporter { + fn name(&self) -> &'static str { + "dev-collector" + } + + fn on_console(&mut self, log: &reporter::ConsoleEvent) -> Result<()> { + let mut state = self.state.lock().unwrap(); + match log.stream { + ConsoleStream::Stdout => state.stdout_lines.push(log.message.clone()), + ConsoleStream::Stderr => state.stderr_lines.push(log.message.clone()), + } + Ok(()) + } + + fn on_eval_end(&mut self, eval: &reporter::EvalEnd) -> Result<()> { + if let Some(summary) = eval.summary.clone() { + self.state.lock().unwrap().summary = Some(summary); + } + Ok(()) + } + + fn on_error(&mut self, error: &reporter::ReporterError) -> Result<()> { + self.state + .lock() + .unwrap() + .errors + .push((error.message.clone(), error.status)); + Ok(()) + } +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum DevStreamFormat { + Legacy, + Canonical, +} + +struct HttpBridgeReporter { + tx: mpsc::UnboundedSender, + format: DevStreamFormat, + pending_run_end: Option, +} + +impl HttpBridgeReporter { + fn send(&self, event: &str, payload: &T) { + if let Ok(data) = serde_json::to_string(payload) { + let _ = self.tx.send(serialize_sse_event(event, &data)); + } + } +} + +impl EvalReporter for HttpBridgeReporter { + fn name(&self) -> &'static str { + "http-bridge" + } + + fn wants_case_delta(&self) -> bool { + true + } + + fn on_run_start(&mut self, run: &reporter::EvalRun) -> Result<()> { + match self.format { + DevStreamFormat::Canonical => self.send("run:start", run), + DevStreamFormat::Legacy => { + self.send( + "processing", + &ProcessingEventData { + evaluators: run.evaluator_count, + }, + ); + } + } + Ok(()) + } + + fn on_eval_start(&mut self, eval: &reporter::EvalInfo) -> Result<()> { + match self.format { + DevStreamFormat::Canonical => self.send("eval:start", eval), + DevStreamFormat::Legacy => { + if let Some(experiment) = eval.experiment.as_ref() { + self.send("start", experiment); + } + } + } + Ok(()) + } + + fn on_case_start(&mut self, case: &reporter::EvalCaseInfo) -> Result<()> { + if self.format == DevStreamFormat::Canonical { + self.send("case:start", case); + } + Ok(()) + } + + fn on_case_end(&mut self, case: &reporter::EvalCaseResult) -> Result<()> { + if self.format == DevStreamFormat::Canonical { + self.send("case:end", case); + } + Ok(()) + } + + fn on_eval_end(&mut self, eval: &reporter::EvalEnd) -> Result<()> { + match self.format { + DevStreamFormat::Canonical => self.send("eval:end", eval), + DevStreamFormat::Legacy => { + if let Some(summary) = eval.summary.as_ref() { + self.send("summary", summary); + } + } + } + Ok(()) + } + + fn on_error(&mut self, error: &reporter::ReporterError) -> Result<()> { + match self.format { + DevStreamFormat::Canonical => self.send("error", error), + DevStreamFormat::Legacy => self.send( + "error", + &json!({ + "message": error.message, + "stack": error.stack, + "status": error.status, + }), + ), + } + Ok(()) + } + + fn on_console(&mut self, log: &reporter::ConsoleEvent) -> Result<()> { + if self.format == DevStreamFormat::Canonical { + self.send("console", log); + } else { + for line in log.message.lines() { + let stream = match log.stream { + ConsoleStream::Stdout => "stdout", + ConsoleStream::Stderr => "stderr", + }; + let _ = self.tx.send(format!(": [{stream}] {line}\n")); + } + } + Ok(()) + } + + fn on_progress(&mut self, progress: &reporter::ProgressEvent) -> Result<()> { + if self.format == DevStreamFormat::Canonical { + self.send("eval:progress", progress); + } + Ok(()) + } + + fn on_case_delta(&mut self, delta: &reporter::CaseDelta) -> Result<()> { + match self.format { + DevStreamFormat::Canonical => self.send("case:delta", delta), + DevStreamFormat::Legacy => { + if let Some(progress) = delta.legacy_progress.as_ref() { + self.send("progress", progress); + } else { + let event = match delta.kind { + reporter::DeltaKind::Text => "text_delta", + reporter::DeltaKind::Json => "json_delta", + reporter::DeltaKind::Reasoning => "reasoning_delta", + }; + self.send( + "progress", + &SseProgressEventData { + id: delta.case_id.clone(), + object_type: "task".to_string(), + origin: None, + format: "code".to_string(), + output_type: "completion".to_string(), + name: delta.eval_id.clone(), + event: event.to_string(), + data: delta.data.clone(), + }, + ); + } + } + } + Ok(()) + } + + fn on_run_end(&mut self, run: &reporter::EvalRunEnd) -> Result> { + self.pending_run_end = Some(run.clone()); + Ok(None) + } + + fn finish(&mut self) -> Result<()> { + if let Some(run) = self.pending_run_end.take() { + match self.format { + DevStreamFormat::Canonical => self.send("run:end", &run), + DevStreamFormat::Legacy => { + let _ = self.tx.send(serialize_sse_event("done", "")); + } + } + } + Ok(()) + } +} + +struct SilentReporter; + +impl EvalReporter for SilentReporter { + fn name(&self) -> &'static str { + "silent" + } +} + +struct EventsReporter; + +impl EvalReporter for EventsReporter { + fn name(&self) -> &'static str { + "events" + } + + fn claims_stdout(&self) -> bool { + true + } + + fn on_event(&mut self, event: &EvalReporterEvent) -> Result<()> { + println!("{}", serde_json::to_string(event)?); + Ok(()) + } +} + +struct DotReporter { + terminal: Option, + glyphs: HashMap, + degraded: bool, +} + +impl DotReporter { + fn new() -> Self { + Self { + terminal: None, + glyphs: HashMap::new(), + degraded: false, + } + } +} + +impl EvalReporter for DotReporter { + fn name(&self) -> &'static str { + "dot" + } + + fn on_init(&mut self, ctx: &reporter::EvalReporterContext) -> Result<()> { + self.terminal = Some(ctx.terminal.clone()); + Ok(()) + } + + fn on_run_start(&mut self, run: &reporter::EvalRun) -> Result<()> { + self.degraded = run.protocol_version < reporter::REPORTER_PROTOCOL_VERSION; + Ok(()) + } + + fn on_case_end(&mut self, case: &reporter::EvalCaseResult) -> Result<()> { + if case.info.synthetic { + self.degraded = true; + return Ok(()); + } + let glyph = match case.status { + CaseStatus::Completed => '.', + CaseStatus::Errored => 'E', + CaseStatus::Skipped => 's', + }; + self.glyphs + .entry(case.info.eval_id.clone()) + .or_default() + .push(glyph); + Ok(()) + } + + fn on_eval_end(&mut self, eval: &reporter::EvalEnd) -> Result<()> { + if self.degraded { + return Ok(()); + } + if let Some(glyphs) = self.glyphs.remove(&eval.eval_id) { + if !glyphs.is_empty() { + self.terminal.as_ref().unwrap().println(glyphs); + } + } + if let Some(summary) = eval.summary.as_ref() { + self.terminal + .as_ref() + .unwrap() + .multiline(format_experiment_summary(summary)); + } + Ok(()) + } + + fn on_run_end(&mut self, _run: &reporter::EvalRunEnd) -> Result> { + if self.degraded { + self.terminal.as_ref().unwrap().println( + "Reporter 'dot' requires real per-case results; upgrade the installed braintrust SDK.", + ); + return Ok(Some(false)); + } + Ok(None) + } +} + +struct JunitReporter { + terminal: Option, + output_file: Option, + eval_names: HashMap, + cases: HashMap>, + degraded: bool, +} + +impl JunitReporter { + fn new() -> Self { + Self { + terminal: None, + output_file: None, + eval_names: HashMap::new(), + cases: HashMap::new(), + degraded: false, + } + } +} + +impl EvalReporter for JunitReporter { + fn name(&self) -> &'static str { + "junit" + } + + fn on_init(&mut self, ctx: &reporter::EvalReporterContext) -> Result<()> { + self.terminal = Some(ctx.terminal.clone()); + self.output_file = ctx.output_file.clone(); + Ok(()) + } + + fn on_run_start(&mut self, run: &reporter::EvalRun) -> Result<()> { + self.degraded = run.protocol_version < reporter::REPORTER_PROTOCOL_VERSION; + Ok(()) + } + + fn on_eval_start(&mut self, eval: &reporter::EvalInfo) -> Result<()> { + self.eval_names + .insert(eval.eval_id.clone(), eval.name.clone()); + Ok(()) + } + + fn on_case_end(&mut self, case: &reporter::EvalCaseResult) -> Result<()> { + if case.info.synthetic { + self.degraded = true; + } else { + self.cases + .entry(case.info.eval_id.clone()) + .or_default() + .push(case.clone()); + } + Ok(()) + } + + fn on_run_end(&mut self, _run: &reporter::EvalRunEnd) -> Result> { + let terminal = self.terminal.as_ref().unwrap(); + if self.degraded { + terminal.println( + "Reporter 'junit' requires real per-case results; upgrade the installed braintrust SDK.", + ); + return Ok(Some(false)); + } + let Some(path) = self.output_file.as_ref() else { + terminal.println("Reporter 'junit' requires --output-file ."); + return Ok(Some(false)); + }; + let mut xml = String::from("\n\n"); + let mut eval_ids: Vec<&String> = self.cases.keys().collect(); + eval_ids.sort(); + for eval_id in eval_ids { + let cases = &self.cases[eval_id]; + let failures = cases + .iter() + .filter(|case| case.status == CaseStatus::Errored) + .count(); + let suite_name = self.eval_names.get(eval_id).unwrap_or(eval_id); + xml.push_str(&format!( + " \n", + xml_escape(suite_name), + cases.len(), + failures + )); + for case in cases { + let name = case + .info + .name + .clone() + .unwrap_or_else(|| case.info.index.to_string()); + xml.push_str(&format!( + " ", + xml_escape(&name), + case.duration_ms as f64 / 1000.0 + )); + match case.status { + CaseStatus::Errored => { + let error = case.error.as_ref(); + xml.push_str(&format!( + "{}", + xml_escape( + error + .map(|error| error.message.as_str()) + .unwrap_or("case errored") + ), + xml_escape( + error.and_then(|error| error.stack.as_deref()).unwrap_or("") + ) + )); + } + CaseStatus::Skipped => xml.push_str(""), + CaseStatus::Completed => {} + } + xml.push_str("\n"); + } + xml.push_str(" \n"); + } + xml.push_str("\n"); + if let Err(error) = std::fs::write(path, xml) { + terminal.println(format!( + "Reporter 'junit' could not write {}: {error}", + path.display() + )); + return Ok(Some(false)); + } + Ok(None) + } +} + +fn xml_escape(value: &str) -> String { + value + .replace('&', "&") + .replace('<', "<") + .replace('>', ">") + .replace('"', """) + .replace('\'', "'") +} + +struct GithubActionsReporter { + terminal: Option, + degraded: bool, +} + +impl GithubActionsReporter { + fn annotate(&self, kind: &str, title: &str, message: &str) { + let property = |value: &str| github_escape(value).replace(':', "%3A").replace(',', "%2C"); + self.terminal.as_ref().unwrap().println(format!( + "::{kind} title={}::{}", + property(title), + github_escape(message) + )); + } +} + +impl EvalReporter for GithubActionsReporter { + fn name(&self) -> &'static str { + "github-actions" + } + + fn on_init(&mut self, ctx: &reporter::EvalReporterContext) -> Result<()> { + self.terminal = Some(ctx.terminal.clone()); + Ok(()) + } + + fn on_run_start(&mut self, run: &reporter::EvalRun) -> Result<()> { + self.degraded = run.protocol_version < reporter::REPORTER_PROTOCOL_VERSION; + Ok(()) + } + + fn on_case_end(&mut self, case: &reporter::EvalCaseResult) -> Result<()> { + if case.info.synthetic { + self.degraded = true; + } else if case.status == CaseStatus::Errored { + self.annotate( + "error", + &format!( + "case {} errored", + case.info + .name + .clone() + .unwrap_or_else(|| case.info.index.to_string()) + ), + case.error + .as_ref() + .map(|error| error.message.as_str()) + .unwrap_or("case errored"), + ); + } + Ok(()) + } + + fn on_error(&mut self, error: &reporter::ReporterError) -> Result<()> { + self.annotate("error", "eval run error", &error.message); + Ok(()) + } + + fn on_run_end(&mut self, _run: &reporter::EvalRunEnd) -> Result> { + if self.degraded { + self.terminal.as_ref().unwrap().println( + "Reporter 'github-actions' requires real per-case results; upgrade the installed braintrust SDK.", + ); + return Ok(Some(false)); + } + Ok(None) + } +} + +fn github_escape(value: &str) -> String { + value + .replace('%', "%25") + .replace('\r', "%0D") + .replace('\n', "%0A") +} + +#[derive(Default)] +struct JsonlReporter { + profile: Option, +} + +impl EvalReporter for JsonlReporter { + fn name(&self) -> &'static str { + "jsonl" + } + + fn claims_stdout(&self) -> bool { + true + } + + fn on_init(&mut self, ctx: &reporter::EvalReporterContext) -> Result<()> { + self.profile = ctx.profile.clone(); + Ok(()) + } + + fn on_eval_end(&mut self, eval: &reporter::EvalEnd) -> Result<()> { + if let Some(summary) = eval.summary.clone() { + let summary = enrich_experiment_summary(summary, self.profile.as_deref()); + println!("{}", serde_json::to_string(&summary)?); + } + Ok(()) + } +} + +struct FancyReporter { + progress: Arc, bars: HashMap, bar_style: ProgressStyle, spinner_style: ProgressStyle, - jsonl: bool, - list: bool, + summaries: bool, + stdout_console: bool, verbose: bool, deferred_errors: Vec, suppressed_stderr_lines: usize, + interactive: bool, finished: bool, profile: Option, } @@ -2814,15 +3565,15 @@ struct EvalBarState { last_total_update: Option, } -impl EvalUi { - fn new(jsonl: bool, list: bool, verbose: bool, profile: Option) -> Self { +impl FancyReporter { + fn new(summaries: bool, stdout_console: bool, verbose: bool, profile: Option) -> Self { let draw_target = if std::io::stderr().is_terminal() && animations_enabled() && !is_quiet() { ProgressDrawTarget::stderr_with_hz(10) } else { ProgressDrawTarget::stderr() }; - let progress = MultiProgress::with_draw_target(draw_target); + let progress = Arc::new(MultiProgress::with_draw_target(draw_target)); let bar_style = ProgressStyle::with_template("{bar:10.blue} {msg} {percent}% {pos}/{len} {eta}") .unwrap(); @@ -2835,11 +3586,12 @@ impl EvalUi { bars: HashMap::new(), bar_style, spinner_style, - jsonl, - list, + summaries, + stdout_console, verbose, deferred_errors: Vec::new(), suppressed_stderr_lines: 0, + interactive: std::io::stderr().is_terminal() && animations_enabled() && !is_quiet(), finished: false, profile, } @@ -2860,6 +3612,7 @@ impl EvalUi { fn handle(&mut self, event: EvalEvent) { match event { + EvalEvent::Reporter(_) => {} EvalEvent::Processing(payload) => { self.print_persistent_line(format_processing_line(payload.evaluators)); } @@ -2869,12 +3622,8 @@ impl EvalUi { } } EvalEvent::Summary(summary) => { - let summary = enrich_experiment_summary(summary, self.profile.as_deref()); - if self.jsonl { - if let Ok(line) = serde_json::to_string(&summary) { - println!("{line}"); - } - } else { + if self.summaries { + let summary = enrich_experiment_summary(summary, self.profile.as_deref()); let rendered = format_experiment_summary(&summary); self.print_persistent_multiline(rendered); } @@ -2884,7 +3633,7 @@ impl EvalUi { } EvalEvent::Dependencies { .. } => {} EvalEvent::Console { stream, message } => { - if stream == "stdout" && (self.list || self.jsonl) { + if stream == "stdout" && self.stdout_console { println!("{message}"); } else if stream == "stderr" && !self.verbose { self.suppressed_stderr_lines += 1; @@ -2935,7 +3684,7 @@ impl EvalUi { } else { let bar = self.progress.add(ProgressBar::new_spinner()); bar.set_style(self.spinner_style.clone()); - if std::io::stderr().is_terminal() && animations_enabled() && !is_quiet() { + if self.interactive { bar.enable_steady_tick(Duration::from_millis(80)); } bar @@ -2943,7 +3692,7 @@ impl EvalUi { } else { let bar = self.progress.add(ProgressBar::new_spinner()); bar.set_style(self.spinner_style.clone()); - if std::io::stderr().is_terminal() && animations_enabled() && !is_quiet() { + if self.interactive { bar.enable_steady_tick(Duration::from_millis(80)); } bar @@ -3121,7 +3870,144 @@ impl EvalUi { } } -impl Drop for EvalUi { +impl EvalReporter for FancyReporter { + fn name(&self) -> &'static str { + if self.verbose { + "verbose" + } else { + "fancy" + } + } + + fn wants_case_delta(&self) -> bool { + self.verbose + } + + fn on_init(&mut self, ctx: &reporter::EvalReporterContext) -> Result<()> { + self.progress = ctx.terminal.live_region(); + self.interactive = ctx.terminal.is_interactive(); + self.profile = ctx.profile.clone(); + Ok(()) + } + + fn on_run_start(&mut self, run: &reporter::EvalRun) -> Result<()> { + self.handle(EvalEvent::Processing(ProcessingEventData { + evaluators: run.evaluator_count, + })); + Ok(()) + } + + fn on_eval_start(&mut self, eval: &reporter::EvalInfo) -> Result<()> { + if let Some(experiment) = eval.experiment.clone() { + self.handle(EvalEvent::Start(experiment)); + } + Ok(()) + } + + fn on_case_end(&mut self, case: &reporter::EvalCaseResult) -> Result<()> { + if !self.bars.contains_key(&case.info.eval_id) { + self.handle_progress(eval_progress_event_for_reporter( + &case.info.eval_id, + "start", + None, + )); + } + self.handle_progress(eval_progress_event_for_reporter( + &case.info.eval_id, + "increment", + None, + )); + if self.verbose && !case.info.synthetic { + let name = case + .info + .name + .clone() + .unwrap_or_else(|| format!("case {}", case.info.index)); + let status = match case.status { + CaseStatus::Completed => "completed", + CaseStatus::Errored => "errored", + CaseStatus::Skipped => "skipped", + }; + self.print_persistent_line(format!("{name}: {status}")); + } + Ok(()) + } + + fn on_eval_end(&mut self, eval: &reporter::EvalEnd) -> Result<()> { + if let Some(summary) = eval.summary.clone() { + self.handle(EvalEvent::Summary(summary)); + } + Ok(()) + } + + fn on_run_end(&mut self, _run: &reporter::EvalRunEnd) -> Result> { + FancyReporter::finish(self); + Ok(None) + } + + fn on_error(&mut self, error: &reporter::ReporterError) -> Result<()> { + self.handle(EvalEvent::Error { + message: error.message.clone(), + stack: error.stack.clone(), + status: error.status, + }); + Ok(()) + } + + fn on_console(&mut self, log: &reporter::ConsoleEvent) -> Result<()> { + self.handle(EvalEvent::Console { + stream: match log.stream { + ConsoleStream::Stdout => "stdout", + ConsoleStream::Stderr => "stderr", + } + .to_string(), + message: log.message.clone(), + }); + Ok(()) + } + + fn on_progress(&mut self, progress: &reporter::ProgressEvent) -> Result<()> { + let kind = if self.bars.contains_key(&progress.eval_id) { + "set_total" + } else { + "start" + }; + self.handle_progress(eval_progress_event_for_reporter( + &progress.eval_id, + kind, + Some(progress.total_cases), + )); + Ok(()) + } + + fn finish(&mut self) -> Result<()> { + FancyReporter::finish(self); + Ok(()) + } +} + +fn eval_progress_event_for_reporter( + eval_id: &str, + kind: &str, + total: Option, +) -> SseProgressEventData { + let mut data = json!({"type": "eval_progress", "kind": kind}); + if let Some(total) = total { + data["total"] = json!(total); + } + SseProgressEventData { + id: eval_id.to_string(), + object_type: "task".to_string(), + origin: None, + format: "code".to_string(), + output_type: "completion".to_string(), + name: eval_id.to_string(), + event: "progress".to_string(), + data: data.to_string(), + } +} + +impl Drop for FancyReporter { fn drop(&mut self) { self.finish(); } @@ -3516,6 +4402,61 @@ mod tests { } } + #[test] + fn junit_reporter_writes_real_case_results() { + let temp = tempfile::tempdir().unwrap(); + let output = temp.path().join("results.xml"); + let mut reporter = JunitReporter::new(); + reporter + .on_init(&reporter::EvalReporterContext { + terminal: reporter::Terminal::new(), + profile: None, + output_file: Some(output.clone()), + }) + .unwrap(); + reporter + .on_eval_start(&reporter::EvalInfo { + run_id: "run-test".into(), + eval_id: "eval-test".into(), + name: "test suite".into(), + experiment: None, + }) + .unwrap(); + reporter + .on_case_end(&reporter::EvalCaseResult { + info: reporter::EvalCaseInfo { + eval_id: "eval-test".into(), + case_id: "span-test".into(), + index: 0, + name: Some("case ".into()), + synthetic: false, + }, + status: CaseStatus::Errored, + duration_ms: 125, + scores: HashMap::new(), + error: Some(reporter::CaseError { + message: "expected & actual differ".into(), + stack: Some("stack".into()), + }), + }) + .unwrap(); + assert_eq!( + reporter + .on_run_end(&reporter::EvalRunEnd { + run_id: "run-test".into(), + status: EvalStatus::Errored, + duration_ms: 125, + errors: Vec::new(), + }) + .unwrap(), + None + ); + let xml = std::fs::read_to_string(output).unwrap(); + assert!(xml.contains("")); + assert!(xml.contains("case <one>")); + assert!(xml.contains("expected & actual differ")); + } + #[test] fn join_app_url_normalizes_slashes() { let joined = @@ -4145,6 +5086,7 @@ mod tests { fn missing_vite_node_retry_message_is_user_facing() { let output = EvalAttemptOutput { status: success_status(), + reporter_vetoed: false, dependency_files: Vec::new(), error_messages: Vec::new(), stderr_lines: vec!["sh: vite-node: command not found".to_string()], @@ -4161,6 +5103,7 @@ mod tests { fn missing_vite_node_retry_message_uses_exit_code_127_fallback() { let output = EvalAttemptOutput { status: exit_status(127), + reporter_vetoed: false, dependency_files: Vec::new(), error_messages: Vec::new(), stderr_lines: Vec::new(), @@ -4214,9 +5157,66 @@ mod tests { assert!(encoded.contains("json_delta")); } + #[test] + fn eval_ui_record_deferred_error_trims_deduplicates_and_caps() { + let mut ui = FancyReporter::new(false, false, false, None); + ui.record_deferred_error(" repeated error ".to_string()); + ui.record_deferred_error("repeated error".to_string()); + ui.record_deferred_error(" ".to_string()); + for index in 0..MAX_DEFERRED_EVAL_ERRORS + 2 { + ui.record_deferred_error(format!("error {index}")); + } + + assert_eq!(ui.deferred_errors.len(), MAX_DEFERRED_EVAL_ERRORS); + assert_eq!(ui.deferred_errors[0], "repeated error"); + assert_eq!(ui.deferred_errors[1], "error 0"); + ui.finish(); + } + + #[test] + fn eval_ui_total_helpers_apply_force_paths_and_clamp_position() { + let progress = MultiProgress::with_draw_target(ProgressDrawTarget::hidden()); + let bar = progress.add(ProgressBar::new(2)); + let style = ProgressStyle::default_bar(); + bar.set_position(3); + let mut state = EvalBarState { + bar, + pending_total: Some(1), + last_total_update: Some(std::time::Instant::now()), + }; + + FancyReporter::ensure_total_not_below_position(&mut state, &style); + assert_eq!(state.bar.length(), Some(3)); + assert_eq!(state.pending_total, Some(3)); + + state.pending_total = Some(4); + FancyReporter::maybe_apply_pending_total(&mut state, &style, true); + assert_eq!(state.bar.length(), Some(4)); + assert_eq!(state.pending_total, None); + + let spinner = progress.add(ProgressBar::new_spinner()); + let spinner_state = EvalBarState { + bar: spinner, + pending_total: None, + last_total_update: None, + }; + assert!(FancyReporter::should_apply_total_update(&spinner_state, 2)); + } + + #[test] + fn eval_ui_finish_is_idempotent_and_drop_finishes() { + let mut ui = FancyReporter::new(false, false, false, None); + ui.finish(); + ui.finish(); + assert!(ui.finished); + + let dropped = FancyReporter::new(false, false, false, None); + drop(dropped); + } + #[test] fn eval_ui_preserves_spinner_increments_before_set_total() { - let mut ui = EvalUi::new(false, false, false, None); + let mut ui = FancyReporter::new(false, false, false, None); let eval_name = "My evaluation"; ui.handle_progress(eval_progress_event(eval_name, "start", None)); @@ -4236,7 +5236,7 @@ mod tests { #[test] fn eval_ui_never_sets_total_below_position() { - let mut ui = EvalUi::new(false, false, false, None); + let mut ui = FancyReporter::new(false, false, false, None); let eval_name = "My evaluation"; ui.handle_progress(eval_progress_event(eval_name, "start", Some(1))); @@ -4260,7 +5260,7 @@ mod tests { #[test] fn eval_ui_keeps_spinner_until_total_exceeds_one() { - let mut ui = EvalUi::new(false, false, false, None); + let mut ui = FancyReporter::new(false, false, false, None); let eval_name = "My evaluation"; ui.handle_progress(eval_progress_event(eval_name, "start", None)); @@ -4321,6 +5321,19 @@ mod tests { assert_eq!(rendered, "short "); } + #[test] + fn handle_sse_event_drops_malformed_and_unknown_events() { + let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel(); + handle_sse_event(Some("processing".to_string()), "not-json".to_string(), &tx); + handle_sse_event( + Some("future:event".to_string()), + r#"{"valid":"but unknown"}"#.to_string(), + &tx, + ); + + assert!(rx.try_recv().is_err()); + } + #[test] fn handle_sse_event_parses_processing_and_start_payloads() { let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel(); @@ -4830,6 +5843,8 @@ mod tests { .unwrap_or_else(|poisoned| poisoned.into_inner()); let keys = [ "BT_EVAL_JSONL", + "BRAINTRUST_EVAL_REPORTER", + "BRAINTRUST_EVAL_OUTPUT_FILE", "BT_EVAL_TERMINATE_ON_FAILURE", "BT_EVAL_NUM_WORKERS", "BT_EVAL_LIST", @@ -4846,6 +5861,8 @@ mod tests { let previous: Vec<(&str, Option)> = keys.iter().map(|key| (*key, clear_env_var(key))).collect(); set_env_var("BT_EVAL_JSONL", "true"); + set_env_var("BRAINTRUST_EVAL_REPORTER", "fancy,events"); + set_env_var("BRAINTRUST_EVAL_OUTPUT_FILE", "results.xml"); set_env_var("BT_EVAL_TERMINATE_ON_FAILURE", "1"); set_env_var("BT_EVAL_NUM_WORKERS", "4"); set_env_var("BT_EVAL_LIST", "yes"); @@ -4859,6 +5876,11 @@ mod tests { let parsed = EvalArgsHarness::try_parse_from(["bt", "sample.eval.ts"]) .expect("env vars should parse into eval args"); assert!(parsed.eval.jsonl); + assert_eq!( + parsed.eval.reporters, + vec![EvalReporterName::Fancy, EvalReporterName::Events] + ); + assert_eq!(parsed.eval.output_file, Some(PathBuf::from("results.xml"))); assert!(parsed.eval.terminate_on_failure); assert_eq!(parsed.eval.num_workers, Some(4)); assert!(parsed.eval.list); diff --git a/src/eval/reporter.rs b/src/eval/reporter.rs new file mode 100644 index 0000000..e7784fb --- /dev/null +++ b/src/eval/reporter.rs @@ -0,0 +1,704 @@ +use std::collections::HashSet; +use std::io::IsTerminal; +use std::sync::Arc; + +use anyhow::Result; +use indicatif::{MultiProgress, ProgressDrawTarget}; +use serde::{Deserialize, Serialize}; + +use super::{ + animations_enabled, is_quiet, EvalEvent, EvalProgressData, ExperimentStart, ExperimentSummary, +}; + +pub(super) const REPORTER_PROTOCOL_VERSION: u32 = 1; + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(super) struct EvalRun { + pub run_id: String, + pub evaluator_count: usize, + pub protocol_version: u32, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(super) struct EvalInfo { + pub run_id: String, + pub eval_id: String, + pub name: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub experiment: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(super) struct EvalCaseInfo { + pub eval_id: String, + pub case_id: String, + pub index: usize, + #[serde(skip_serializing_if = "Option::is_none")] + pub name: Option, + #[serde(skip, default)] + pub synthetic: bool, +} + +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "lowercase")] +pub(super) enum CaseStatus { + Completed, + Errored, + Skipped, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(super) struct CaseError { + pub message: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub stack: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(super) struct EvalCaseResult { + #[serde(flatten)] + pub info: EvalCaseInfo, + pub status: CaseStatus, + pub duration_ms: u64, + pub scores: std::collections::HashMap, + #[serde(skip_serializing_if = "Option::is_none")] + pub error: Option, +} + +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "lowercase")] +pub(super) enum EvalStatus { + Completed, + Errored, +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(super) struct CaseCounts { + pub completed: usize, + pub errored: usize, + pub skipped: usize, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(super) struct EvalEnd { + pub eval_id: String, + pub status: EvalStatus, + pub duration_ms: u64, + pub case_counts: CaseCounts, + #[serde(skip_serializing_if = "Option::is_none")] + pub summary: Option, + pub errors: Vec, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(super) struct EvalRunEnd { + pub run_id: String, + pub status: EvalStatus, + pub duration_ms: u64, + pub errors: Vec, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(super) struct ErrorScope { + pub run_id: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub eval_id: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub case_id: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(super) struct ReporterError { + pub scope: ErrorScope, + pub message: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub stack: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub status: Option, +} + +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "lowercase")] +pub(super) enum ConsoleStream { + Stdout, + Stderr, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(super) struct ConsoleEvent { + pub stream: ConsoleStream, + pub message: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub eval_id: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(super) struct ProgressEvent { + pub eval_id: String, + pub total_cases: u64, +} + +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "lowercase")] +pub(super) enum DeltaKind { + Text, + Json, + Reasoning, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(super) struct CaseDelta { + pub eval_id: String, + pub case_id: String, + pub kind: DeltaKind, + pub data: String, + #[serde(skip, default)] + pub legacy_progress: Option, +} + +// Canonical payloads intentionally vary from tiny progress updates to full summaries. +#[allow(clippy::large_enum_variant)] +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(tag = "type")] +pub(super) enum EvalReporterEvent { + #[serde(rename = "run:start")] + RunStart { run: EvalRun }, + #[serde(rename = "eval:start")] + EvalStart { eval: EvalInfo }, + #[serde(rename = "case:start")] + CaseStart { case: EvalCaseInfo }, + #[serde(rename = "case:end")] + CaseEnd { case: EvalCaseResult }, + #[serde(rename = "eval:end")] + EvalEnd { eval: EvalEnd }, + #[serde(rename = "run:end")] + RunEnd { run: EvalRunEnd }, + #[serde(rename = "error")] + Error { error: ReporterError }, + #[serde(rename = "console")] + Console { log: ConsoleEvent }, + #[serde(rename = "eval:progress")] + Progress { progress: ProgressEvent }, + #[serde(rename = "case:delta")] + CaseDelta { delta: CaseDelta }, +} + +#[derive(Clone)] +pub(super) struct Terminal { + progress: Arc, + interactive: bool, +} + +impl Terminal { + pub fn new() -> Self { + let interactive = std::io::stderr().is_terminal() && animations_enabled() && !is_quiet(); + let target = if interactive { + ProgressDrawTarget::stderr_with_hz(10) + } else { + ProgressDrawTarget::stderr() + }; + Self { + progress: Arc::new(MultiProgress::with_draw_target(target)), + interactive, + } + } + + pub fn println(&self, line: impl AsRef) { + let line = line.as_ref(); + self.progress.suspend(|| eprintln!("{line}")); + } + + pub fn multiline(&self, text: impl AsRef) { + let text = text.as_ref(); + self.progress.suspend(|| { + for line in text.lines() { + eprintln!("{line}"); + } + }); + } + + pub fn live_region(&self) -> Arc { + Arc::clone(&self.progress) + } + + pub fn is_interactive(&self) -> bool { + self.interactive + } + + pub fn clear(&self) { + let _ = self.progress.clear(); + } +} + +pub(super) struct EvalReporterContext { + pub terminal: Terminal, + pub profile: Option, + pub output_file: Option, +} + +pub(super) trait EvalReporter: Send { + fn name(&self) -> &'static str { + "reporter" + } + fn claims_stdout(&self) -> bool { + false + } + fn wants_case_delta(&self) -> bool { + false + } + fn on_event(&mut self, _event: &EvalReporterEvent) -> Result<()> { + Ok(()) + } + fn on_init(&mut self, _ctx: &EvalReporterContext) -> Result<()> { + Ok(()) + } + fn on_run_start(&mut self, _run: &EvalRun) -> Result<()> { + Ok(()) + } + fn on_eval_start(&mut self, _eval: &EvalInfo) -> Result<()> { + Ok(()) + } + fn on_case_start(&mut self, _case: &EvalCaseInfo) -> Result<()> { + Ok(()) + } + fn on_case_end(&mut self, _case: &EvalCaseResult) -> Result<()> { + Ok(()) + } + fn on_eval_end(&mut self, _eval: &EvalEnd) -> Result<()> { + Ok(()) + } + fn on_run_end(&mut self, _run: &EvalRunEnd) -> Result> { + Ok(None) + } + fn on_error(&mut self, _error: &ReporterError) -> Result<()> { + Ok(()) + } + fn on_console(&mut self, _log: &ConsoleEvent) -> Result<()> { + Ok(()) + } + fn on_progress(&mut self, _progress: &ProgressEvent) -> Result<()> { + Ok(()) + } + fn on_case_delta(&mut self, _delta: &CaseDelta) -> Result<()> { + Ok(()) + } + fn finish(&mut self) -> Result<()> { + Ok(()) + } +} + +pub(super) struct ReporterManager { + reporters: Vec>, + terminal: Terminal, + failed_reporters: HashSet, + run_id: String, + run_ended: bool, + finished: bool, + vetoed: bool, + run_errors: Vec, +} + +impl ReporterManager { + pub fn new( + reporters: Vec>, + profile: Option, + output_file: Option, + ) -> Result { + let terminal = Terminal::new(); + let context = EvalReporterContext { + terminal: terminal.clone(), + profile, + output_file, + }; + let stdout_reporters: Vec<&str> = reporters + .iter() + .filter(|reporter| reporter.claims_stdout()) + .map(|reporter| reporter.name()) + .collect(); + if stdout_reporters.len() > 1 { + anyhow::bail!( + "reporters {} all claim stdout; select only one machine-output reporter", + stdout_reporters.join(", ") + ); + } + let mut manager = Self { + reporters, + terminal, + failed_reporters: HashSet::new(), + // Legacy streams carry no run identity. A stable synthetic ID keeps + // machine event output deterministic while remaining unique within this run. + run_id: "legacy-run".to_string(), + run_ended: false, + finished: false, + vetoed: false, + run_errors: Vec::new(), + }; + for index in 0..manager.reporters.len() { + let result = manager.reporters[index].on_init(&context); + manager.record_failure(index, result); + } + Ok(manager) + } + + pub fn run_id(&self) -> &str { + &self.run_id + } + + pub fn wants_case_delta(&self) -> bool { + self.reporters.iter().enumerate().any(|(index, reporter)| { + !self.failed_reporters.contains(&index) && reporter.wants_case_delta() + }) + } + + pub fn dispatch(&mut self, event: &EvalReporterEvent) { + if self.finished { + return; + } + if let EvalReporterEvent::Error { error } = event { + if error.scope.eval_id.is_none() { + self.run_errors.push(error.clone()); + } + } + if matches!(event, EvalReporterEvent::RunEnd { .. }) { + if self.run_ended { + return; + } + self.run_ended = true; + } + + for index in 0..self.reporters.len() { + if self.failed_reporters.contains(&index) { + continue; + } + let result = self.reporters[index].on_event(event); + if result.is_err() { + self.record_failure(index, result); + continue; + } + let result = match event { + EvalReporterEvent::RunStart { run } => self.reporters[index].on_run_start(run), + EvalReporterEvent::EvalStart { eval } => self.reporters[index].on_eval_start(eval), + EvalReporterEvent::CaseStart { case } => self.reporters[index].on_case_start(case), + EvalReporterEvent::CaseEnd { case } => self.reporters[index].on_case_end(case), + EvalReporterEvent::EvalEnd { eval } => self.reporters[index].on_eval_end(eval), + EvalReporterEvent::RunEnd { run } => match self.reporters[index].on_run_end(run) { + Ok(Some(false)) => { + self.vetoed = true; + Ok(()) + } + Ok(_) => Ok(()), + Err(error) => Err(error), + }, + EvalReporterEvent::Error { error } => self.reporters[index].on_error(error), + EvalReporterEvent::Console { log } => self.reporters[index].on_console(log), + EvalReporterEvent::Progress { progress } => { + self.reporters[index].on_progress(progress) + } + EvalReporterEvent::CaseDelta { delta } => { + self.reporters[index].on_case_delta(delta) + } + }; + self.record_failure(index, result); + } + } + + fn record_failure(&mut self, index: usize, result: Result<()>) { + if let Err(error) = result { + if self.failed_reporters.insert(index) { + self.terminal.println(format!( + "Reporter '{}' failed: {error:#}", + self.reporters[index].name() + )); + } + } + } + + pub fn finish(&mut self, status: EvalStatus) -> bool { + if self.finished { + return self.vetoed; + } + if !self.run_ended { + self.dispatch(&EvalReporterEvent::RunEnd { + run: EvalRunEnd { + run_id: self.run_id.clone(), + status, + duration_ms: 0, + errors: self.run_errors.clone(), + }, + }); + } + self.terminal.clear(); + for index in 0..self.reporters.len() { + if self.failed_reporters.contains(&index) { + continue; + } + let result = self.reporters[index].finish(); + self.record_failure(index, result); + } + self.finished = true; + self.vetoed + } +} + +impl Drop for ReporterManager { + fn drop(&mut self) { + self.finish(EvalStatus::Errored); + } +} + +pub(super) struct LegacyEventAdapter { + run_id: String, + next_case: usize, +} + +pub(super) fn decode_canonical_sse_event( + event_name: &str, + data: &str, +) -> Option { + if let Ok(event) = serde_json::from_str::(data) { + return Some(mark_synthetic_case(event)); + } + let event = match event_name { + "run:start" => serde_json::from_str(data) + .ok() + .map(|run| EvalReporterEvent::RunStart { run }), + "eval:start" => serde_json::from_str(data) + .ok() + .map(|eval| EvalReporterEvent::EvalStart { eval }), + "case:start" => serde_json::from_str(data) + .ok() + .map(|case| EvalReporterEvent::CaseStart { case }), + "case:end" => serde_json::from_str(data) + .ok() + .map(|case| EvalReporterEvent::CaseEnd { case }), + "eval:end" => serde_json::from_str(data) + .ok() + .map(|eval| EvalReporterEvent::EvalEnd { eval }), + "run:end" => serde_json::from_str(data) + .ok() + .map(|run| EvalReporterEvent::RunEnd { run }), + "error" => serde_json::from_str(data) + .ok() + .map(|error| EvalReporterEvent::Error { error }), + "console" => serde_json::from_str(data) + .ok() + .map(|log| EvalReporterEvent::Console { log }), + "eval:progress" => serde_json::from_str(data) + .ok() + .map(|progress| EvalReporterEvent::Progress { progress }), + "case:delta" => serde_json::from_str(data) + .ok() + .map(|delta| EvalReporterEvent::CaseDelta { delta }), + _ => None, + }?; + Some(mark_synthetic_case(event)) +} + +fn mark_synthetic_case(mut event: EvalReporterEvent) -> EvalReporterEvent { + match &mut event { + EvalReporterEvent::CaseStart { case } if case.case_id.starts_with("synthetic-") => { + case.synthetic = true; + } + EvalReporterEvent::CaseEnd { case } if case.info.case_id.starts_with("synthetic-") => { + case.info.synthetic = true; + } + _ => {} + } + event +} + +impl LegacyEventAdapter { + pub fn new(run_id: String) -> Self { + Self { + run_id, + next_case: 0, + } + } + + pub fn translate(&mut self, event: &EvalEvent) -> Option { + match event { + EvalEvent::Reporter(event) => Some(event.clone()), + EvalEvent::Processing(payload) => Some(EvalReporterEvent::RunStart { + run: EvalRun { + run_id: self.run_id.clone(), + evaluator_count: payload.evaluators, + protocol_version: 0, + }, + }), + EvalEvent::Start(start) => { + let name = start + .experiment_name + .clone() + .unwrap_or_else(|| "evaluation".to_string()); + Some(EvalReporterEvent::EvalStart { + eval: EvalInfo { + run_id: self.run_id.clone(), + eval_id: name.clone(), + name, + experiment: Some(start.clone()), + }, + }) + } + EvalEvent::Summary(summary) => Some(EvalReporterEvent::EvalEnd { + eval: EvalEnd { + eval_id: summary.experiment_name.clone(), + status: EvalStatus::Completed, + duration_ms: 0, + case_counts: CaseCounts::default(), + summary: Some(summary.clone()), + errors: Vec::new(), + }, + }), + EvalEvent::Progress(progress) => { + let Ok(payload) = serde_json::from_str::(&progress.data) else { + let kind = if progress.event.contains("json") { + DeltaKind::Json + } else if progress.event.contains("reason") { + DeltaKind::Reasoning + } else { + DeltaKind::Text + }; + return Some(EvalReporterEvent::CaseDelta { + delta: CaseDelta { + eval_id: progress.name.clone(), + case_id: progress.id.clone(), + kind, + data: progress.data.clone(), + legacy_progress: Some(progress.clone()), + }, + }); + }; + if payload.kind_type != "eval_progress" { + return None; + } + match payload.kind.as_str() { + "increment" => { + self.next_case += 1; + Some(EvalReporterEvent::CaseEnd { + case: EvalCaseResult { + info: EvalCaseInfo { + eval_id: progress.name.clone(), + case_id: format!("synthetic-{}", self.next_case), + index: self.next_case - 1, + name: None, + synthetic: true, + }, + status: CaseStatus::Completed, + duration_ms: 0, + scores: Default::default(), + error: None, + }, + }) + } + "start" | "set_total" => { + payload + .total + .map(|total_cases| EvalReporterEvent::Progress { + progress: ProgressEvent { + eval_id: progress.name.clone(), + total_cases, + }, + }) + } + "stop" => None, + _ => None, + } + } + EvalEvent::Error { + message, + stack, + status, + } => Some(EvalReporterEvent::Error { + error: ReporterError { + scope: ErrorScope { + run_id: self.run_id.clone(), + eval_id: None, + case_id: None, + }, + message: message.clone(), + stack: stack.clone(), + status: *status, + }, + }), + EvalEvent::Console { stream, message } => Some(EvalReporterEvent::Console { + log: ConsoleEvent { + stream: if stream == "stderr" { + ConsoleStream::Stderr + } else { + ConsoleStream::Stdout + }, + message: message.clone(), + eval_id: None, + }, + }), + EvalEvent::Done => Some(EvalReporterEvent::RunEnd { + run: EvalRunEnd { + run_id: self.run_id.clone(), + status: EvalStatus::Completed, + duration_ms: 0, + errors: Vec::new(), + }, + }), + EvalEvent::Dependencies { .. } => None, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + struct RecordingReporter { + run_ends: usize, + veto: bool, + } + + impl EvalReporter for RecordingReporter { + fn on_run_end(&mut self, _run: &EvalRunEnd) -> Result> { + self.run_ends += 1; + Ok((self.veto).then_some(false)) + } + } + + #[test] + fn canonical_event_serializes_with_camel_case() { + let event = EvalReporterEvent::RunStart { + run: EvalRun { + run_id: "run-test".into(), + evaluator_count: 2, + protocol_version: REPORTER_PROTOCOL_VERSION, + }, + }; + assert_eq!( + serde_json::to_value(event).unwrap(), + serde_json::json!({ + "type": "run:start", + "run": {"runId":"run-test", "evaluatorCount":2, "protocolVersion":1} + }) + ); + } + + #[test] + fn manager_synthesizes_run_end_and_finishes_once() { + let reporter = Box::new(RecordingReporter { + run_ends: 0, + veto: true, + }); + let mut manager = ReporterManager::new(vec![reporter], None, None).unwrap(); + assert!(manager.finish(EvalStatus::Errored)); + assert!(manager.finish(EvalStatus::Completed)); + } +} diff --git a/tests/eval_dev_server.rs b/tests/eval_dev_server.rs index 227221d..b91530f 100644 --- a/tests/eval_dev_server.rs +++ b/tests/eval_dev_server.rs @@ -255,8 +255,18 @@ fn curl_get(url: &str, headers: &[(&str, &str)]) -> String { } fn curl_post(url: &str, headers: &[(&str, &str)], body: &str) -> String { + let (status, body) = curl_post_with_status(url, headers, body); + assert_eq!(status, 200, "curl POST {url} returned {status}: {body}"); + body +} + +fn curl_post_with_status(url: &str, headers: &[(&str, &str)], body: &str) -> (u16, String) { + let output_file = tempfile::NamedTempFile::new().expect("create curl output file"); let mut cmd = Command::new("curl"); - cmd.args(["-s", "--max-time", "60", "-X", "POST", "-d", body, url]); + cmd.args(["-s", "--max-time", "60", "-X", "POST", "-d", body]) + .arg("-o") + .arg(output_file.path()) + .args(["-w", "%{http_code}", url]); for (key, value) in headers { cmd.arg("-H").arg(format!("{key}: {value}")); } @@ -266,7 +276,27 @@ fn curl_post(url: &str, headers: &[(&str, &str)], body: &str) -> String { "curl POST {url} failed: {}", String::from_utf8_lossy(&output.stderr) ); - String::from_utf8_lossy(&output.stdout).to_string() + let status = String::from_utf8_lossy(&output.stdout) + .parse() + .expect("parse curl status"); + let body = std::fs::read_to_string(output_file.path()).expect("read curl response body"); + (status, body) +} + +fn assert_devserver_golden(name: &str, actual: &str) { + let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("tests/golden/eval") + .join(format!("devserver--{name}")); + if std::env::var_os("UPDATE_GOLDENS").is_some() { + std::fs::write(&path, actual).expect("update devserver golden"); + } + let expected = std::fs::read_to_string(&path).unwrap_or_else(|_| { + panic!( + "missing {}; run UPDATE_GOLDENS=1 cargo test --test eval_dev_server", + path.display() + ) + }); + assert_eq!(actual, expected, "devserver response differed for {name}"); } fn ensure_python_env(fixtures_py_root: &std::path::Path) -> Option { @@ -344,6 +374,145 @@ fn bt_binary_path(root: &std::path::Path) -> PathBuf { } } +#[test] +fn eval_dev_server_fake_runner_byte_contracts() { + let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")); + let fake_runner = root.join("tests/fixtures/eval/fake_runner.py"); + let frames = root.join("tests/fixtures/eval/scenarios/devserver.jsonl"); + let fixture_dir = tempfile::tempdir().expect("create fake eval directory"); + std::fs::write( + fixture_dir.path().join("fixture.eval.ts"), + "// Scripted by fake_runner.py.\n", + ) + .unwrap(); + + let (mock_auth_port, _mock_handle) = start_mock_auth_server(); + let dev_port = free_port(); + let bt_path = bt_binary_path(&root); + let mut child = Command::new(&bt_path) + .args([ + "eval", + "--dev", + "--dev-port", + &dev_port.to_string(), + "--no-send-logs", + "--runner", + ]) + .arg(&fake_runner) + .arg("fixture.eval.ts") + .current_dir(fixture_dir.path()) + .env( + "BRAINTRUST_APP_URL", + format!("http://127.0.0.1:{mock_auth_port}"), + ) + .env("BRAINTRUST_API_KEY", "test-key") + .env("BT_TEST_FRAME_SCRIPT", &frames) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .expect("spawn fake eval devserver"); + + let output = Arc::new(Mutex::new(String::new())); + let mut threads = Vec::new(); + if let Some(stdout) = child.stdout.take() { + threads.push(spawn_output_collector(stdout, Arc::clone(&output))); + } + if let Some(stderr) = child.stderr.take() { + threads.push(spawn_output_collector(stderr, Arc::clone(&output))); + } + wait_for_output( + &mut child, + &output, + "Starting eval dev server", + Duration::from_secs(30), + ); + thread::sleep(Duration::from_millis(200)); + + let base_url = format!("http://127.0.0.1:{dev_port}"); + let auth_headers = [ + ("x-bt-auth-token", "test-key"), + ("x-bt-org-name", "test-org"), + ]; + assert_devserver_golden( + "list.json", + &curl_get(&format!("{base_url}/list"), &auth_headers), + ); + + let post_headers = [ + ("x-bt-auth-token", "test-key"), + ("x-bt-org-name", "test-org"), + ("Content-Type", "application/json"), + ]; + let request = |name: &str, stream: bool| { + serde_json::json!({"name": name, "data": {"data": []}, "stream": stream}).to_string() + }; + + let stream_body = curl_post( + &format!("{base_url}/eval"), + &post_headers, + &request("test-eval", true), + ); + assert_devserver_golden("stream-happy.sse", &stream_body); + + let error_stream = curl_post( + &format!("{base_url}/eval"), + &post_headers, + &request("error-eval", true), + ); + assert_devserver_golden("stream-error.sse", &error_stream); + + let canonical_headers = [ + ("x-bt-auth-token", "test-key"), + ("x-bt-org-name", "test-org"), + ("Content-Type", "application/json"), + ("x-bt-stream-fmt", "canonical"), + ]; + let canonical_stream = curl_post( + &format!("{base_url}/eval"), + &canonical_headers, + &request("canonical-eval", true), + ); + assert_devserver_golden("stream-canonical.sse", &canonical_stream); + + let (status, response) = curl_post_with_status( + &format!("{base_url}/eval"), + &post_headers, + &request("test-eval", false), + ); + assert_eq!(status, 200); + assert_devserver_golden("response-happy.json", &response); + + let (status, response) = curl_post_with_status( + &format!("{base_url}/eval"), + &post_headers, + &request("canonical-eval", false), + ); + assert_eq!(status, 200); + assert_devserver_golden("response-canonical.json", &response); + + let (status, response) = curl_post_with_status( + &format!("{base_url}/eval"), + &post_headers, + &request("error-eval", false), + ); + assert_eq!(status, 429); + assert_devserver_golden("response-error.json", &response); + + let (status, response) = curl_post_with_status( + &format!("{base_url}/eval"), + &post_headers, + &request("fallback-eval", false), + ); + assert_eq!(status, 500); + assert_eq!(response, r#"{"error":"Eval runner exited with an error."}"#); + + let _ = child.kill(); + let _ = child.wait(); + for handle in threads { + let _ = handle.join(); + } +} + #[test] fn eval_dev_server_streams_python_events() { let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")); diff --git a/tests/eval_golden.rs b/tests/eval_golden.rs new file mode 100644 index 0000000..401b229 --- /dev/null +++ b/tests/eval_golden.rs @@ -0,0 +1,432 @@ +use std::ffi::OsString; +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::Command; + +use assert_cmd::cargo::CommandCargoExt; + +#[derive(Clone, Copy)] +enum SpawnMode { + Custom, + Tsx, +} + +struct Scenario { + name: &'static str, + mode: &'static str, + flags: &'static [&'static str], + spawn_mode: SpawnMode, + expected_code: i32, + profile: bool, +} + +fn prepend_path(dir: &Path) -> OsString { + let mut paths = vec![dir.to_path_buf()]; + paths.extend(std::env::split_paths( + &std::env::var_os("PATH").unwrap_or_default(), + )); + std::env::join_paths(paths).expect("construct test PATH") +} + +#[cfg(unix)] +fn make_executable(path: &Path) { + use std::os::unix::fs::PermissionsExt; + let mut permissions = fs::metadata(path).unwrap().permissions(); + permissions.set_mode(0o755); + fs::set_permissions(path, permissions).unwrap(); +} + +#[cfg(not(unix))] +fn make_executable(_path: &Path) {} + +fn run_scenario(scenario: &Scenario) -> (Vec, Vec, i32) { + let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")); + let fixture_root = root.join("tests/fixtures/eval"); + let runner = fixture_root.join("fake_runner.py"); + let frames = fixture_root + .join("scenarios") + .join(format!("{}.jsonl", scenario.name)); + let temp = tempfile::tempdir().expect("create scenario temp dir"); + let eval_file = temp.path().join("fixture.eval.ts"); + fs::write(&eval_file, "// The scripted runner ignores this file.\n").unwrap(); + + let attempt_file = temp.path().join("attempt"); + let config_dir = temp.path().join("config/bt"); + fs::create_dir_all(&config_dir).unwrap(); + fs::write( + config_dir.join("auth.json"), + r#"{"profiles":{"test-profile":{"auth_kind":"api_key","api_url":"https://api.example.test","app_url":"https://example.test","org_name":"test-org"}}}"#, + ) + .unwrap(); + fs::write( + config_dir.join("secrets.json"), + r#"{"secrets":{"test-profile":"test-api-key"}}"#, + ) + .unwrap(); + + let mut command = Command::cargo_bin("bt").expect("locate bt binary"); + command.args(["eval", "--no-color", "--no-send-logs"]); + if scenario.profile { + command.args(["--profile", "test-profile"]); + } + if scenario.flags.contains(&"--verbose") { + command.arg("--verbose"); + } + for flag in scenario.flags { + if *flag != "--verbose" { + command.arg(flag); + } + } + + match scenario.spawn_mode { + SpawnMode::Custom => { + command.arg("--runner").arg(&runner); + } + SpawnMode::Tsx => { + for binary in ["tsx", "vite-node"] { + let destination = temp.path().join(binary); + fs::copy(&runner, &destination).unwrap(); + make_executable(&destination); + } + command.env("PATH", prepend_path(temp.path())); + } + } + + let output = command + .arg(&eval_file) + .env("BT_TEST_FRAME_SCRIPT", frames) + .env("BT_TEST_ATTEMPT_FILE", attempt_file) + .env("XDG_CONFIG_HOME", temp.path().join("config")) + .env_remove("BRAINTRUST_API_KEY") + .env("NO_COLOR", "1") + .output() + .expect("run scripted eval"); + ( + output.stdout, + output.stderr, + output.status.code().unwrap_or(-1), + ) +} + +fn assert_golden(scenario: &Scenario) { + let (stdout, stderr, code) = run_scenario(scenario); + let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")); + let golden_root = root.join("tests/golden/eval"); + let stem = format!("{}--{}", scenario.name, scenario.mode); + let stdout_path = golden_root.join(format!("{stem}.stdout")); + let stderr_path = golden_root.join(format!("{stem}.stderr")); + + if std::env::var_os("UPDATE_GOLDENS").is_some() { + fs::create_dir_all(&golden_root).unwrap(); + fs::write(&stdout_path, &stdout).unwrap(); + fs::write(&stderr_path, &stderr).unwrap(); + } + + assert_eq!(code, scenario.expected_code, "exit code for {stem}"); + assert_eq!( + stdout, + fs::read(&stdout_path).unwrap_or_else(|_| panic!( + "missing {}; run UPDATE_GOLDENS=1 cargo test --test eval_golden", + stdout_path.display() + )), + "stdout differed for {stem}" + ); + assert_eq!( + stderr, + fs::read(&stderr_path).unwrap_or_else(|_| panic!( + "missing {}; run UPDATE_GOLDENS=1 cargo test --test eval_golden", + stderr_path.display() + )), + "stderr differed for {stem}" + ); +} + +#[test] +fn eval_output_matches_characterization_goldens() { + let scenarios = [ + Scenario { + name: "happy", + mode: "default", + flags: &[], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "happy", + mode: "jsonl", + flags: &["--jsonl"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "happy", + mode: "verbose", + flags: &["--verbose"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "comparison", + mode: "default", + flags: &[], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: true, + }, + Scenario { + name: "comparison", + mode: "jsonl", + flags: &["--jsonl"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: true, + }, + Scenario { + name: "minimal-summary", + mode: "default", + flags: &[], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "minimal-summary", + mode: "jsonl", + flags: &["--jsonl"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "errors", + mode: "default", + flags: &[], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "errors", + mode: "verbose", + flags: &["--verbose"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "api-key-error", + mode: "default", + flags: &[], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "api-key-error", + mode: "verbose", + flags: &["--verbose"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "console", + mode: "default", + flags: &[], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "console", + mode: "jsonl", + flags: &["--jsonl"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "console", + mode: "list", + flags: &["--list"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "console", + mode: "verbose", + flags: &["--verbose"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "console", + mode: "silent", + flags: &["--reporter=silent"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "console", + mode: "events", + flags: &["--reporter=events"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "console", + mode: "two-stdout-reporters", + flags: &["--reporter=events", "--reporter=jsonl"], + spawn_mode: SpawnMode::Custom, + expected_code: 1, + profile: false, + }, + Scenario { + name: "console", + mode: "unsupported-dot", + flags: &["--reporter=dot"], + spawn_mode: SpawnMode::Custom, + expected_code: 1, + profile: false, + }, + Scenario { + name: "console", + mode: "tsx-default", + flags: &[], + spawn_mode: SpawnMode::Tsx, + expected_code: 0, + profile: false, + }, + Scenario { + name: "console", + mode: "tsx-verbose", + flags: &["--verbose"], + spawn_mode: SpawnMode::Tsx, + expected_code: 0, + profile: false, + }, + Scenario { + name: "crash", + mode: "default", + flags: &[], + spawn_mode: SpawnMode::Custom, + expected_code: 1, + profile: false, + }, + Scenario { + name: "error-exit-zero", + mode: "default", + flags: &[], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "empty", + mode: "default", + flags: &[], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "unknown", + mode: "default", + flags: &[], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "interleaved", + mode: "default", + flags: &[], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "interleaved", + mode: "jsonl", + flags: &["--jsonl"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "progress-edge", + mode: "default", + flags: &[], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "canonical-real", + mode: "dot", + flags: &["--reporter=dot"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "canonical-real", + mode: "github-actions", + flags: &["--reporter=github-actions"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "canonical-real", + mode: "verbose", + flags: &["--reporter=verbose"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "canonical-real", + mode: "events", + flags: &["--reporter=events"], + spawn_mode: SpawnMode::Custom, + expected_code: 0, + profile: false, + }, + Scenario { + name: "canonical-synthetic", + mode: "dot", + flags: &["--reporter=dot"], + spawn_mode: SpawnMode::Custom, + expected_code: 1, + profile: false, + }, + Scenario { + name: "esm-retry", + mode: "tsx-default", + flags: &[], + spawn_mode: SpawnMode::Tsx, + expected_code: 0, + profile: false, + }, + Scenario { + name: "esm-retry", + mode: "tsx-verbose", + flags: &["--verbose"], + spawn_mode: SpawnMode::Tsx, + expected_code: 0, + profile: false, + }, + ]; + + for scenario in scenarios { + assert_golden(&scenario); + } +} diff --git a/tests/fixtures/eval/fake_runner.py b/tests/fixtures/eval/fake_runner.py new file mode 100755 index 0000000..7f87642 --- /dev/null +++ b/tests/fixtures/eval/fake_runner.py @@ -0,0 +1,85 @@ +#!/usr/bin/env python3 +"""Replay deterministic eval SSE fixtures for CLI integration tests.""" + +import json +import os +import socket +import sys +import time +from pathlib import Path + + +def attempt_number() -> int: + state_file = os.environ.get("BT_TEST_ATTEMPT_FILE") + if not state_file: + return 1 + path = Path(state_file) + try: + attempt = int(path.read_text()) + 1 + except (FileNotFoundError, ValueError): + attempt = 1 + path.write_text(str(attempt)) + return attempt + + +def connect(): + unix_path = os.environ.get("BT_EVAL_SSE_SOCK") + if unix_path: + client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + client.connect(unix_path) + return client + host, port = os.environ["BT_EVAL_SSE_ADDR"].rsplit(":", 1) + return socket.create_connection((host, int(port))) + + +def main() -> int: + fixture = Path(os.environ["BT_TEST_FRAME_SCRIPT"]) + attempt = attempt_number() + directives = [ + json.loads(line) + for line in fixture.read_text().splitlines() + if line.strip() and not line.lstrip().startswith("#") + ] + mode = os.environ.get("BT_EVAL_DEV_MODE") + request_name = None + if os.environ.get("BT_EVAL_DEV_REQUEST_JSON"): + request_name = json.loads(os.environ["BT_EVAL_DEV_REQUEST_JSON"]).get("name") + selected = [ + d + for d in directives + if d.get("attempt", attempt) == attempt + and d.get("mode", mode) == mode + and d.get("request", request_name) == request_name + ] + + with connect() as client: + stream = client.makefile("w", encoding="utf-8", newline="\n") + exit_code = 0 + for directive in selected: + if "exit" in directive: + exit_code = int(directive["exit"]) + continue + if "process_stdout" in directive: + print(directive["process_stdout"], flush=True) + continue + if "process_stderr" in directive: + print(directive["process_stderr"], file=sys.stderr, flush=True) + continue + event = directive.get("event") + if event is None: + continue + data = directive.get("data", "") + encoded = data if isinstance(data, str) else json.dumps(data, separators=(",", ":")) + stream.write(f"event: {event}\n") + for line in encoded.splitlines() or [""]: + stream.write(f"data: {line}\n") + stream.write("\n") + stream.flush() + # Keep the process alive briefly after closing SSE so the CLI drains every + # queued frame before observing process exit. + time.sleep(0.05) + return exit_code + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/fixtures/eval/scenarios/api-key-error.jsonl b/tests/fixtures/eval/scenarios/api-key-error.jsonl new file mode 100644 index 0000000..4bed804 --- /dev/null +++ b/tests/fixtures/eval/scenarios/api-key-error.jsonl @@ -0,0 +1,4 @@ +{"event":"processing","data":{"evaluators":1}} +{"event":"error","data":{"message":"Please specify an api key to continue","stack":"auth stack"}} +{"event":"done","data":""} +{"exit":0} diff --git a/tests/fixtures/eval/scenarios/canonical-real.jsonl b/tests/fixtures/eval/scenarios/canonical-real.jsonl new file mode 100644 index 0000000..5079906 --- /dev/null +++ b/tests/fixtures/eval/scenarios/canonical-real.jsonl @@ -0,0 +1,12 @@ +{"event":"run:start","data":{"runId":"run-canonical","evaluatorCount":1,"protocolVersion":1}} +{"event":"eval:start","data":{"runId":"run-canonical","evalId":"eval-canonical","name":"canonical-eval","experiment":{"projectName":"test-project","experimentName":"canonical-exp","experimentUrl":"https://example.test/experiment/canonical-exp"}}} +{"event":"eval:progress","data":{"evalId":"eval-canonical","totalCases":3}} +{"event":"case:start","data":{"evalId":"eval-canonical","caseId":"span-case-1","index":0,"name":"first case"}} +{"event":"case:end","data":{"evalId":"eval-canonical","caseId":"span-case-1","index":0,"name":"first case","status":"completed","durationMs":120,"scores":{"Accuracy":1.0}}} +{"event":"case:start","data":{"evalId":"eval-canonical","caseId":"span-case-2","index":1,"name":"second case"}} +{"event":"case:end","data":{"evalId":"eval-canonical","caseId":"span-case-2","index":1,"name":"second case","status":"errored","durationMs":250,"scores":{},"error":{"message":"case failed","stack":"case stack"}}} +{"event":"case:start","data":{"evalId":"eval-canonical","caseId":"span-case-3","index":2,"name":"third case"}} +{"event":"case:end","data":{"evalId":"eval-canonical","caseId":"span-case-3","index":2,"name":"third case","status":"skipped","durationMs":0,"scores":{}}} +{"event":"eval:end","data":{"evalId":"eval-canonical","status":"errored","durationMs":370,"caseCounts":{"completed":1,"errored":1,"skipped":1},"summary":{"projectName":"test-project","experimentName":"canonical-exp","projectId":"project-test-id","experimentId":"canonical-exp-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/canonical-exp","scores":{},"metrics":null},"errors":[]}} +{"event":"run:end","data":{"runId":"run-canonical","status":"errored","durationMs":400,"errors":[]}} +{"exit":0} diff --git a/tests/fixtures/eval/scenarios/canonical-synthetic.jsonl b/tests/fixtures/eval/scenarios/canonical-synthetic.jsonl new file mode 100644 index 0000000..1defb17 --- /dev/null +++ b/tests/fixtures/eval/scenarios/canonical-synthetic.jsonl @@ -0,0 +1,4 @@ +{"event":"run:start","data":{"runId":"run-old","evaluatorCount":1,"protocolVersion":1}} +{"event":"case:end","data":{"evalId":"eval-old","caseId":"synthetic-eval-old-0","index":0,"status":"completed","durationMs":0,"scores":{}}} +{"event":"run:end","data":{"runId":"run-old","status":"completed","durationMs":0,"errors":[]}} +{"exit":0} diff --git a/tests/fixtures/eval/scenarios/comparison.jsonl b/tests/fixtures/eval/scenarios/comparison.jsonl new file mode 100644 index 0000000..ddd1468 --- /dev/null +++ b/tests/fixtures/eval/scenarios/comparison.jsonl @@ -0,0 +1,5 @@ +{"event":"processing","data":{"evaluators":1}} +{"event":"start","data":{"projectName":"test-project","experimentName":"challenger","projectId":"project-test-id","experimentId":"challenger-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/challenger"}} +{"event":"summary","data":{"projectName":"test-project","experimentName":"challenger","projectId":"project-test-id","experimentId":"challenger-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/challenger","scores":{"Accuracy":{"name":"Accuracy","score":0.875,"diff":0.125,"improvements":3,"regressions":1}},"metrics":{"duration":{"name":"duration","metric":1.25,"unit":"s","diff":-0.2,"improvements":2,"regressions":0}},"comparisonExperimentName":"baseline"}} +{"event":"done","data":""} +{"exit":0} diff --git a/tests/fixtures/eval/scenarios/console.jsonl b/tests/fixtures/eval/scenarios/console.jsonl new file mode 100644 index 0000000..74088e4 --- /dev/null +++ b/tests/fixtures/eval/scenarios/console.jsonl @@ -0,0 +1,7 @@ +{"event":"processing","data":{"evaluators":1}} +{"event":"console","data":{"stream":"stdout","message":"user stdout one"}} +{"event":"console","data":{"stream":"stderr","message":"user stderr one"}} +{"event":"console","data":{"stream":"stdout","message":"user stdout two"}} +{"event":"console","data":{"stream":"stderr","message":"user stderr two"}} +{"event":"done","data":""} +{"exit":0} diff --git a/tests/fixtures/eval/scenarios/crash.jsonl b/tests/fixtures/eval/scenarios/crash.jsonl new file mode 100644 index 0000000..ad3853f --- /dev/null +++ b/tests/fixtures/eval/scenarios/crash.jsonl @@ -0,0 +1,3 @@ +{"event":"processing","data":{"evaluators":1}} +{"event":"error","data":{"message":"error before crash","stack":"crash stack"}} +{"exit":3} diff --git a/tests/fixtures/eval/scenarios/devserver.jsonl b/tests/fixtures/eval/scenarios/devserver.jsonl new file mode 100644 index 0000000..a8790f0 --- /dev/null +++ b/tests/fixtures/eval/scenarios/devserver.jsonl @@ -0,0 +1,20 @@ +{"mode":"list","process_stdout":"{\"test-eval\":{\"name\":\"test-eval\",\"parameters\":[]}}"} +{"mode":"list","exit":0} +{"mode":"eval","request":"test-eval","event":"processing","data":{"evaluators":1}} +{"mode":"eval","request":"test-eval","event":"start","data":{"projectName":"test-project","experimentName":"test-experiment","projectId":"project-test-id","experimentId":"experiment-test-id"}} +{"mode":"eval","request":"test-eval","event":"progress","data":{"id":"case-test-id","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"test-eval","event":"json_delta","data":"{\"answer\":\"ok\"}"}} +{"mode":"eval","request":"test-eval","event":"summary","data":{"projectName":"test-project","experimentName":"test-experiment","projectId":"project-test-id","experimentId":"experiment-test-id","projectUrl":"https://example.test/project","experimentUrl":"https://example.test/experiment","comparisonExperimentName":null,"scores":{},"metrics":null}} +{"mode":"eval","request":"test-eval","event":"done","data":""} +{"mode":"eval","request":"test-eval","exit":0} +{"mode":"eval","request":"error-eval","event":"error","data":{"message":"scripted request failed","stack":"scripted stack","status":429}} +{"mode":"eval","request":"error-eval","exit":0} +{"mode":"eval","request":"fallback-eval","process_stderr":"scripted runner crash"} +{"mode":"eval","request":"fallback-eval","exit":3} +{"mode":"eval","request":"canonical-eval","event":"run:start","data":{"runId":"run-dev-canonical","evaluatorCount":1,"protocolVersion":1}} +{"mode":"eval","request":"canonical-eval","event":"eval:start","data":{"runId":"run-dev-canonical","evalId":"eval-dev-canonical","name":"canonical-eval","experiment":{"projectName":"test-project","experimentName":"canonical-experiment","projectId":"project-test-id","experimentId":"canonical-experiment-id"}}} +{"mode":"eval","request":"canonical-eval","event":"case:delta","data":{"evalId":"eval-dev-canonical","caseId":"span-dev-case","kind":"json","data":"{\"answer\":\"ok\"}"}} +{"mode":"eval","request":"canonical-eval","event":"case:start","data":{"evalId":"eval-dev-canonical","caseId":"span-dev-case","index":0,"name":"dev case"}} +{"mode":"eval","request":"canonical-eval","event":"case:end","data":{"evalId":"eval-dev-canonical","caseId":"span-dev-case","index":0,"name":"dev case","status":"completed","durationMs":10,"scores":{}}} +{"mode":"eval","request":"canonical-eval","event":"eval:end","data":{"evalId":"eval-dev-canonical","status":"completed","durationMs":10,"caseCounts":{"completed":1,"errored":0,"skipped":0},"summary":{"projectName":"test-project","experimentName":"canonical-experiment","projectId":"project-test-id","experimentId":"canonical-experiment-id","projectUrl":"https://example.test/project","experimentUrl":"https://example.test/experiment","comparisonExperimentName":null,"scores":{},"metrics":null},"errors":[]}} +{"mode":"eval","request":"canonical-eval","event":"run:end","data":{"runId":"run-dev-canonical","status":"completed","durationMs":10,"errors":[]}} +{"mode":"eval","request":"canonical-eval","exit":0} diff --git a/tests/fixtures/eval/scenarios/empty.jsonl b/tests/fixtures/eval/scenarios/empty.jsonl new file mode 100644 index 0000000..72a4f25 --- /dev/null +++ b/tests/fixtures/eval/scenarios/empty.jsonl @@ -0,0 +1 @@ +{"exit":0} diff --git a/tests/fixtures/eval/scenarios/error-exit-zero.jsonl b/tests/fixtures/eval/scenarios/error-exit-zero.jsonl new file mode 100644 index 0000000..f8f8823 --- /dev/null +++ b/tests/fixtures/eval/scenarios/error-exit-zero.jsonl @@ -0,0 +1,3 @@ +{"event":"error","data":{"message":"rendered but successful"}} +{"event":"done","data":""} +{"exit":0} diff --git a/tests/fixtures/eval/scenarios/errors.jsonl b/tests/fixtures/eval/scenarios/errors.jsonl new file mode 100644 index 0000000..753b688 --- /dev/null +++ b/tests/fixtures/eval/scenarios/errors.jsonl @@ -0,0 +1,17 @@ +{"event":"processing","data":{"evaluators":1}} +{"event":"start","data":{"projectName":"test-project","experimentName":"exp-1","projectId":"project-test-id","experimentId":"exp-1-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/exp-1"}} +{"event":"console","data":{"stream":"stdout","message":"before errors"}} +{"event":"error","data":{"message":" evaluator error 1 ","stack":"stack line 1\nstack detail 1","status":422}} +{"event":"error","data":{"message":" evaluator error 2 ","stack":"stack line 2\nstack detail 2","status":null}} +{"event":"error","data":{"message":"evaluator error 1","stack":"duplicate stack"}} +{"event":"error","data":{"message":" evaluator error 3 ","stack":"stack line 3\nstack detail 3","status":null}} +{"event":"error","data":{"message":" evaluator error 4 ","stack":"stack line 4\nstack detail 4","status":null}} +{"event":"console","data":{"stream":"stderr","message":"between errors"}} +{"event":"error","data":{"message":" evaluator error 5 ","stack":"stack line 5\nstack detail 5","status":null}} +{"event":"error","data":{"message":" evaluator error 6 ","stack":"stack line 6\nstack detail 6","status":null}} +{"event":"error","data":{"message":" evaluator error 7 ","stack":"stack line 7\nstack detail 7","status":null}} +{"event":"error","data":{"message":" evaluator error 8 ","stack":"stack line 8\nstack detail 8","status":null}} +{"event":"error","data":{"message":" evaluator error 9 ","stack":"stack line 9\nstack detail 9","status":null}} +{"event":"error","data":{"message":" evaluator error 10 ","stack":"stack line 10\nstack detail 10","status":null}} +{"event":"done","data":""} +{"exit":0} diff --git a/tests/fixtures/eval/scenarios/esm-retry.jsonl b/tests/fixtures/eval/scenarios/esm-retry.jsonl new file mode 100644 index 0000000..e9ccc4b --- /dev/null +++ b/tests/fixtures/eval/scenarios/esm-retry.jsonl @@ -0,0 +1,7 @@ +{"attempt":1,"event":"processing","data":{"evaluators":1}} +{"attempt":1,"event":"console","data":{"stream":"stderr","message":"Error [ERR_REQUIRE_ESM]: require() of ES Module"}} +{"attempt":1,"exit":3} +{"attempt":2,"event":"processing","data":{"evaluators":1}} +{"attempt":2,"event":"start","data":{"projectName":"test-project","experimentName":"esm-success"}} +{"attempt":2,"event":"done","data":""} +{"attempt":2,"exit":0} diff --git a/tests/fixtures/eval/scenarios/happy.jsonl b/tests/fixtures/eval/scenarios/happy.jsonl new file mode 100644 index 0000000..7a19fdf --- /dev/null +++ b/tests/fixtures/eval/scenarios/happy.jsonl @@ -0,0 +1,10 @@ +{"event":"processing","data":{"evaluators":1}} +{"event":"start","data":{"projectName":"test-project","experimentName":"exp-1","projectId":"project-test-id","experimentId":"exp-1-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/exp-1"}} +{"event":"progress","data":{"id":"p1","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"eval-one","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"start\",\"total\":3}"}} +{"event":"progress","data":{"id":"p1","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"eval-one","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"set_total\",\"total\":4}"}} +{"event":"progress","data":{"id":"p1","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"eval-one","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"increment\"}"}} +{"event":"progress","data":{"id":"p1","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"eval-one","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"increment\"}"}} +{"event":"progress","data":{"id":"p1","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"eval-one","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"stop\"}"}} +{"event":"summary","data":{"projectName":"test-project","experimentName":"exp-1","projectId":"project-test-id","experimentId":"exp-1-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/exp-1","scores":{"Accuracy":{"name":"Accuracy","score":0.875,"diff":0.125,"improvements":3,"regressions":1}},"metrics":{"duration":{"name":"duration","metric":1.25,"unit":"s","diff":-0.2,"improvements":2,"regressions":0}}}} +{"event":"done","data":""} +{"exit":0} diff --git a/tests/fixtures/eval/scenarios/interleaved.jsonl b/tests/fixtures/eval/scenarios/interleaved.jsonl new file mode 100644 index 0000000..93c6683 --- /dev/null +++ b/tests/fixtures/eval/scenarios/interleaved.jsonl @@ -0,0 +1,14 @@ +{"event":"processing","data":{"evaluators":2}} +{"event":"start","data":{"projectName":"test-project","experimentName":"exp-a","projectId":"project-test-id","experimentId":"exp-a-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/exp-a"}} +{"event":"start","data":{"projectName":"test-project","experimentName":"exp-b","projectId":"project-test-id","experimentId":"exp-b-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/exp-b"}} +{"event":"progress","data":{"id":"a","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"eval-a","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"start\",\"total\":2}"}} +{"event":"progress","data":{"id":"b","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"eval-b","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"start\",\"total\":2}"}} +{"event":"progress","data":{"id":"a","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"eval-a","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"increment\"}"}} +{"event":"progress","data":{"id":"b","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"eval-b","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"increment\"}"}} +{"event":"progress","data":{"id":"a","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"eval-a","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"stop\"}"}} +{"event":"summary","data":{"projectName":"test-project","experimentName":"exp-a","projectId":"project-test-id","experimentId":"exp-a-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/exp-a","scores":{"Accuracy":{"name":"Accuracy","score":0.875,"diff":0.125,"improvements":3,"regressions":1}},"metrics":{"duration":{"name":"duration","metric":1.25,"unit":"s","diff":-0.2,"improvements":2,"regressions":0}}}} +{"event":"progress","data":{"id":"b","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"eval-b","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"increment\"}"}} +{"event":"progress","data":{"id":"b","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"eval-b","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"stop\"}"}} +{"event":"summary","data":{"projectName":"test-project","experimentName":"exp-b","projectId":"project-test-id","experimentId":"exp-b-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/exp-b","scores":{"Accuracy":{"name":"Accuracy","score":0.875,"diff":0.125,"improvements":3,"regressions":1}},"metrics":{"duration":{"name":"duration","metric":1.25,"unit":"s","diff":-0.2,"improvements":2,"regressions":0}}}} +{"event":"done","data":""} +{"exit":0} diff --git a/tests/fixtures/eval/scenarios/minimal-summary.jsonl b/tests/fixtures/eval/scenarios/minimal-summary.jsonl new file mode 100644 index 0000000..a77bd24 --- /dev/null +++ b/tests/fixtures/eval/scenarios/minimal-summary.jsonl @@ -0,0 +1,5 @@ +{"event":"processing","data":{"evaluators":1}} +{"event":"start","data":{"projectName":"test-project","experimentName":"minimal","projectId":"project-test-id","experimentId":"minimal-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/minimal"}} +{"event":"summary","data":{"projectName":"test-project","experimentName":"minimal","projectId":null,"experimentId":null,"projectUrl":null,"experimentUrl":null,"scores":{},"metrics":null}} +{"event":"done","data":""} +{"exit":0} diff --git a/tests/fixtures/eval/scenarios/progress-edge.jsonl b/tests/fixtures/eval/scenarios/progress-edge.jsonl new file mode 100644 index 0000000..c5c0b5a --- /dev/null +++ b/tests/fixtures/eval/scenarios/progress-edge.jsonl @@ -0,0 +1,8 @@ +{"event":"processing","data":{"evaluators":1}} +{"event":"progress","data":{"id":"p1","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"edge","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"start\",\"total\":1}"}} +{"event":"progress","data":{"id":"p1","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"edge","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"increment\"}"}} +{"event":"progress","data":{"id":"p1","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"edge","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"increment\"}"}} +{"event":"progress","data":{"id":"p1","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"edge","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"set_total\",\"total\":1}"}} +{"event":"progress","data":{"id":"p1","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"edge","event":"progress","data":"{\"type\":\"eval_progress\",\"kind\":\"stop\"}"}} +{"event":"done","data":""} +{"exit":0} diff --git a/tests/fixtures/eval/scenarios/unknown.jsonl b/tests/fixtures/eval/scenarios/unknown.jsonl new file mode 100644 index 0000000..6424548 --- /dev/null +++ b/tests/fixtures/eval/scenarios/unknown.jsonl @@ -0,0 +1,7 @@ +{"event":"processing","data":{"evaluators":1}} +{"event":"future:event","data":{"future":true}} +{"event":"start","data":{"projectName":"test-project","experimentName":"exp-1","projectId":"project-test-id","experimentId":"exp-1-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/exp-1"}} +{"event":"another-unknown","data":"ignored"} +{"event":"summary","data":{"projectName":"test-project","experimentName":"exp-1","projectId":"project-test-id","experimentId":"exp-1-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/exp-1","scores":{"Accuracy":{"name":"Accuracy","score":0.875,"diff":0.125,"improvements":3,"regressions":1}},"metrics":{"duration":{"name":"duration","metric":1.25,"unit":"s","diff":-0.2,"improvements":2,"regressions":0}}}} +{"event":"done","data":""} +{"exit":0} diff --git a/tests/golden/eval/api-key-error--default.stderr b/tests/golden/eval/api-key-error--default.stderr new file mode 100644 index 0000000..e58d22f --- /dev/null +++ b/tests/golden/eval/api-key-error--default.stderr @@ -0,0 +1,3 @@ +Processing 1 evaluator... +Please specify an api key to continue +auth stack diff --git a/tests/golden/eval/api-key-error--default.stdout b/tests/golden/eval/api-key-error--default.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/api-key-error--verbose.stderr b/tests/golden/eval/api-key-error--verbose.stderr new file mode 100644 index 0000000..e58d22f --- /dev/null +++ b/tests/golden/eval/api-key-error--verbose.stderr @@ -0,0 +1,3 @@ +Processing 1 evaluator... +Please specify an api key to continue +auth stack diff --git a/tests/golden/eval/api-key-error--verbose.stdout b/tests/golden/eval/api-key-error--verbose.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/canonical-real--dot.stderr b/tests/golden/eval/canonical-real--dot.stderr new file mode 100644 index 0000000..58ad670 --- /dev/null +++ b/tests/golden/eval/canonical-real--dot.stderr @@ -0,0 +1,5 @@ +.Es + +╭ Experiment summary ──────────────────────────────────────────╮ +│ See results at https://example.test/experiment/canonical-exp │ +╰──────────────────────────────────────────────────────────────╯ diff --git a/tests/golden/eval/canonical-real--dot.stdout b/tests/golden/eval/canonical-real--dot.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/canonical-real--events.stderr b/tests/golden/eval/canonical-real--events.stderr new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/canonical-real--events.stdout b/tests/golden/eval/canonical-real--events.stdout new file mode 100644 index 0000000..6dde9b0 --- /dev/null +++ b/tests/golden/eval/canonical-real--events.stdout @@ -0,0 +1,11 @@ +{"type":"run:start","run":{"runId":"run-canonical","evaluatorCount":1,"protocolVersion":1}} +{"type":"eval:start","eval":{"runId":"run-canonical","evalId":"eval-canonical","name":"canonical-eval","experiment":{"projectName":"test-project","experimentName":"canonical-exp","projectId":null,"experimentId":null,"projectUrl":null,"experimentUrl":"https://example.test/experiment/canonical-exp"}}} +{"type":"eval:progress","progress":{"evalId":"eval-canonical","totalCases":3}} +{"type":"case:start","case":{"evalId":"eval-canonical","caseId":"span-case-1","index":0,"name":"first case"}} +{"type":"case:end","case":{"evalId":"eval-canonical","caseId":"span-case-1","index":0,"name":"first case","status":"completed","durationMs":120,"scores":{"Accuracy":1.0}}} +{"type":"case:start","case":{"evalId":"eval-canonical","caseId":"span-case-2","index":1,"name":"second case"}} +{"type":"case:end","case":{"evalId":"eval-canonical","caseId":"span-case-2","index":1,"name":"second case","status":"errored","durationMs":250,"scores":{},"error":{"message":"case failed","stack":"case stack"}}} +{"type":"case:start","case":{"evalId":"eval-canonical","caseId":"span-case-3","index":2,"name":"third case"}} +{"type":"case:end","case":{"evalId":"eval-canonical","caseId":"span-case-3","index":2,"name":"third case","status":"skipped","durationMs":0,"scores":{}}} +{"type":"eval:end","eval":{"evalId":"eval-canonical","status":"errored","durationMs":370,"caseCounts":{"completed":1,"errored":1,"skipped":1},"summary":{"projectName":"test-project","experimentName":"canonical-exp","projectId":"project-test-id","experimentId":"canonical-exp-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/canonical-exp","comparisonExperimentName":null,"scores":{},"metrics":null,"runMode":null,"isFinal":null,"runLabel":null,"sampleCount":null,"sampleSeed":null},"errors":[]}} +{"type":"run:end","run":{"runId":"run-canonical","status":"errored","durationMs":400,"errors":[]}} diff --git a/tests/golden/eval/canonical-real--github-actions.stderr b/tests/golden/eval/canonical-real--github-actions.stderr new file mode 100644 index 0000000..215be44 --- /dev/null +++ b/tests/golden/eval/canonical-real--github-actions.stderr @@ -0,0 +1 @@ +::error title=case second case errored::case failed diff --git a/tests/golden/eval/canonical-real--github-actions.stdout b/tests/golden/eval/canonical-real--github-actions.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/canonical-real--verbose.stderr b/tests/golden/eval/canonical-real--verbose.stderr new file mode 100644 index 0000000..8214d47 --- /dev/null +++ b/tests/golden/eval/canonical-real--verbose.stderr @@ -0,0 +1,9 @@ +Processing 1 evaluator... +▶ Experiment canonical-exp is running at https://example.test/experiment/canonical-exp +first case: completed +second case: errored +third case: skipped + +╭ Experiment summary ──────────────────────────────────────────╮ +│ See results at https://example.test/experiment/canonical-exp │ +╰──────────────────────────────────────────────────────────────╯ diff --git a/tests/golden/eval/canonical-real--verbose.stdout b/tests/golden/eval/canonical-real--verbose.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/canonical-synthetic--dot.stderr b/tests/golden/eval/canonical-synthetic--dot.stderr new file mode 100644 index 0000000..5e06cd1 --- /dev/null +++ b/tests/golden/eval/canonical-synthetic--dot.stderr @@ -0,0 +1,3 @@ +Reporter 'dot' requires real per-case results; upgrade the installed braintrust SDK. +error: an eval reporter vetoed the successful run +If this seems like a bug, file an issue at https://github.com/braintrustdata/bt/issues/new and include `bt --version`, `bt status --json`, and the command you ran. diff --git a/tests/golden/eval/canonical-synthetic--dot.stdout b/tests/golden/eval/canonical-synthetic--dot.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/comparison--default.stderr b/tests/golden/eval/comparison--default.stderr new file mode 100644 index 0000000..cdfb1cf --- /dev/null +++ b/tests/golden/eval/comparison--default.stderr @@ -0,0 +1,21 @@ +Processing 1 evaluator... +▶ Experiment challenger is running at https://example.test/experiment/challenger + +╭ Experiment summary ────────────────────────────────────────────────────────────────────────────────╮ +│ baseline (baseline) ← challenger (comparison) │ +│ │ +│ experiments: │ +│ baseline (baseline) = baseline │ +│ challenger (comparison) = challenger │ +│ │ +│ counts: ↑n/↓m = improvements/regressions │ +│ │ +│ Scores and metrics baseline (baseline) challenger (comparison) │ +│ ◯ Accuracy 75.00% 87.50% (+12.50%; ↑3/↓1) │ +│ ◯ duration 1.45s 1.25s (-0.20s; ↑2)  │ +│ │ +│ See results at https://example.test/experiment/challenger │ +│ │ +│ compare_command: bt experiments compare --profile test-profile -p test-project baseline challenger │ +│ compare_more: append more experiment names at the end; max 7 comparisons │ +╰────────────────────────────────────────────────────────────────────────────────────────────────────╯ diff --git a/tests/golden/eval/comparison--default.stdout b/tests/golden/eval/comparison--default.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/comparison--jsonl.stderr b/tests/golden/eval/comparison--jsonl.stderr new file mode 100644 index 0000000..2b59fb1 --- /dev/null +++ b/tests/golden/eval/comparison--jsonl.stderr @@ -0,0 +1,2 @@ +Processing 1 evaluator... +▶ Experiment challenger is running at https://example.test/experiment/challenger diff --git a/tests/golden/eval/comparison--jsonl.stdout b/tests/golden/eval/comparison--jsonl.stdout new file mode 100644 index 0000000..70cfb10 --- /dev/null +++ b/tests/golden/eval/comparison--jsonl.stdout @@ -0,0 +1 @@ +{"projectName":"test-project","experimentName":"challenger","projectId":"project-test-id","experimentId":"challenger-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/challenger","comparisonExperimentName":"baseline","scores":{"Accuracy":{"name":"Accuracy","score":0.875,"diff":0.125,"improvements":3,"regressions":1}},"metrics":{"duration":{"name":"duration","metric":1.25,"unit":"s","diff":-0.2,"improvements":2,"regressions":0}},"runMode":null,"isFinal":null,"runLabel":null,"sampleCount":null,"sampleSeed":null,"compareCommand":"bt experiments compare --profile test-profile -p test-project baseline challenger","compareMore":"append more experiment names at the end; max 7 comparisons"} diff --git a/tests/golden/eval/console--default.stderr b/tests/golden/eval/console--default.stderr new file mode 100644 index 0000000..2fce5cb --- /dev/null +++ b/tests/golden/eval/console--default.stderr @@ -0,0 +1 @@ +Processing 1 evaluator... diff --git a/tests/golden/eval/console--default.stdout b/tests/golden/eval/console--default.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/console--events.stderr b/tests/golden/eval/console--events.stderr new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/console--events.stdout b/tests/golden/eval/console--events.stdout new file mode 100644 index 0000000..5cc3c53 --- /dev/null +++ b/tests/golden/eval/console--events.stdout @@ -0,0 +1,6 @@ +{"type":"run:start","run":{"runId":"legacy-run","evaluatorCount":1,"protocolVersion":0}} +{"type":"console","log":{"stream":"stdout","message":"user stdout one"}} +{"type":"console","log":{"stream":"stderr","message":"user stderr one"}} +{"type":"console","log":{"stream":"stdout","message":"user stdout two"}} +{"type":"console","log":{"stream":"stderr","message":"user stderr two"}} +{"type":"run:end","run":{"runId":"legacy-run","status":"completed","durationMs":0,"errors":[]}} diff --git a/tests/golden/eval/console--jsonl.stderr b/tests/golden/eval/console--jsonl.stderr new file mode 100644 index 0000000..312d582 --- /dev/null +++ b/tests/golden/eval/console--jsonl.stderr @@ -0,0 +1,3 @@ +Processing 1 evaluator... + +Suppressed 2 stderr line(s). Re-run with `bt eval --verbose ...` to inspect details. diff --git a/tests/golden/eval/console--jsonl.stdout b/tests/golden/eval/console--jsonl.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/console--list.stderr b/tests/golden/eval/console--list.stderr new file mode 100644 index 0000000..2fce5cb --- /dev/null +++ b/tests/golden/eval/console--list.stderr @@ -0,0 +1 @@ +Processing 1 evaluator... diff --git a/tests/golden/eval/console--list.stdout b/tests/golden/eval/console--list.stdout new file mode 100644 index 0000000..56c2e7a --- /dev/null +++ b/tests/golden/eval/console--list.stdout @@ -0,0 +1,2 @@ +user stdout one +user stdout two diff --git a/tests/golden/eval/console--silent.stderr b/tests/golden/eval/console--silent.stderr new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/console--silent.stdout b/tests/golden/eval/console--silent.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/console--tsx-default.stderr b/tests/golden/eval/console--tsx-default.stderr new file mode 100644 index 0000000..381b58a --- /dev/null +++ b/tests/golden/eval/console--tsx-default.stderr @@ -0,0 +1,3 @@ +Processing 1 evaluator... +user stderr one +user stderr two diff --git a/tests/golden/eval/console--tsx-default.stdout b/tests/golden/eval/console--tsx-default.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/console--tsx-verbose.stderr b/tests/golden/eval/console--tsx-verbose.stderr new file mode 100644 index 0000000..381b58a --- /dev/null +++ b/tests/golden/eval/console--tsx-verbose.stderr @@ -0,0 +1,3 @@ +Processing 1 evaluator... +user stderr one +user stderr two diff --git a/tests/golden/eval/console--tsx-verbose.stdout b/tests/golden/eval/console--tsx-verbose.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/console--two-stdout-reporters.stderr b/tests/golden/eval/console--two-stdout-reporters.stderr new file mode 100644 index 0000000..0d4fe82 --- /dev/null +++ b/tests/golden/eval/console--two-stdout-reporters.stderr @@ -0,0 +1,2 @@ +error: reporters events, jsonl all claim stdout; select only one machine-output reporter +If this seems like a bug, file an issue at https://github.com/braintrustdata/bt/issues/new and include `bt --version`, `bt status --json`, and the command you ran. diff --git a/tests/golden/eval/console--two-stdout-reporters.stdout b/tests/golden/eval/console--two-stdout-reporters.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/console--unsupported-dot.stderr b/tests/golden/eval/console--unsupported-dot.stderr new file mode 100644 index 0000000..5e06cd1 --- /dev/null +++ b/tests/golden/eval/console--unsupported-dot.stderr @@ -0,0 +1,3 @@ +Reporter 'dot' requires real per-case results; upgrade the installed braintrust SDK. +error: an eval reporter vetoed the successful run +If this seems like a bug, file an issue at https://github.com/braintrustdata/bt/issues/new and include `bt --version`, `bt status --json`, and the command you ran. diff --git a/tests/golden/eval/console--unsupported-dot.stdout b/tests/golden/eval/console--unsupported-dot.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/console--verbose.stderr b/tests/golden/eval/console--verbose.stderr new file mode 100644 index 0000000..2fce5cb --- /dev/null +++ b/tests/golden/eval/console--verbose.stderr @@ -0,0 +1 @@ +Processing 1 evaluator... diff --git a/tests/golden/eval/console--verbose.stdout b/tests/golden/eval/console--verbose.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/crash--default.stderr b/tests/golden/eval/crash--default.stderr new file mode 100644 index 0000000..4174b11 --- /dev/null +++ b/tests/golden/eval/crash--default.stderr @@ -0,0 +1,5 @@ +Processing 1 evaluator... +error before crash +crash stack +error: eval runner exited with status exit status: 3 +If this seems like a bug, file an issue at https://github.com/braintrustdata/bt/issues/new and include `bt --version`, `bt status --json`, and the command you ran. diff --git a/tests/golden/eval/crash--default.stdout b/tests/golden/eval/crash--default.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/devserver--list.json b/tests/golden/eval/devserver--list.json new file mode 100644 index 0000000..6929652 --- /dev/null +++ b/tests/golden/eval/devserver--list.json @@ -0,0 +1 @@ +{"test-eval":{"name":"test-eval","parameters":[]}} \ No newline at end of file diff --git a/tests/golden/eval/devserver--response-canonical.json b/tests/golden/eval/devserver--response-canonical.json new file mode 100644 index 0000000..36390e2 --- /dev/null +++ b/tests/golden/eval/devserver--response-canonical.json @@ -0,0 +1 @@ +{"projectName":"test-project","experimentName":"canonical-experiment","projectId":"project-test-id","experimentId":"canonical-experiment-id","projectUrl":"https://example.test/project","experimentUrl":"https://example.test/experiment","comparisonExperimentName":null,"scores":{},"metrics":null,"runMode":null,"isFinal":null,"runLabel":null,"sampleCount":null,"sampleSeed":null} \ No newline at end of file diff --git a/tests/golden/eval/devserver--response-error.json b/tests/golden/eval/devserver--response-error.json new file mode 100644 index 0000000..70d6159 --- /dev/null +++ b/tests/golden/eval/devserver--response-error.json @@ -0,0 +1 @@ +{"error":"scripted request failed"} \ No newline at end of file diff --git a/tests/golden/eval/devserver--response-happy.json b/tests/golden/eval/devserver--response-happy.json new file mode 100644 index 0000000..a31249a --- /dev/null +++ b/tests/golden/eval/devserver--response-happy.json @@ -0,0 +1 @@ +{"projectName":"test-project","experimentName":"test-experiment","projectId":"project-test-id","experimentId":"experiment-test-id","projectUrl":"https://example.test/project","experimentUrl":"https://example.test/experiment","comparisonExperimentName":null,"scores":{},"metrics":null,"runMode":null,"isFinal":null,"runLabel":null,"sampleCount":null,"sampleSeed":null} \ No newline at end of file diff --git a/tests/golden/eval/devserver--stream-canonical.sse b/tests/golden/eval/devserver--stream-canonical.sse new file mode 100644 index 0000000..a2fbac1 --- /dev/null +++ b/tests/golden/eval/devserver--stream-canonical.sse @@ -0,0 +1,21 @@ +event: run:start +data: {"runId":"run-dev-canonical","evaluatorCount":1,"protocolVersion":1} + +event: eval:start +data: {"runId":"run-dev-canonical","evalId":"eval-dev-canonical","name":"canonical-eval","experiment":{"projectName":"test-project","experimentName":"canonical-experiment","projectId":"project-test-id","experimentId":"canonical-experiment-id","projectUrl":null,"experimentUrl":null}} + +event: case:delta +data: {"evalId":"eval-dev-canonical","caseId":"span-dev-case","kind":"json","data":"{\"answer\":\"ok\"}"} + +event: case:start +data: {"evalId":"eval-dev-canonical","caseId":"span-dev-case","index":0,"name":"dev case"} + +event: case:end +data: {"evalId":"eval-dev-canonical","caseId":"span-dev-case","index":0,"name":"dev case","status":"completed","durationMs":10,"scores":{}} + +event: eval:end +data: {"evalId":"eval-dev-canonical","status":"completed","durationMs":10,"caseCounts":{"completed":1,"errored":0,"skipped":0},"summary":{"projectName":"test-project","experimentName":"canonical-experiment","projectId":"project-test-id","experimentId":"canonical-experiment-id","projectUrl":"https://example.test/project","experimentUrl":"https://example.test/experiment","comparisonExperimentName":null,"scores":{},"metrics":null,"runMode":null,"isFinal":null,"runLabel":null,"sampleCount":null,"sampleSeed":null},"errors":[]} + +event: run:end +data: {"runId":"run-dev-canonical","status":"completed","durationMs":10,"errors":[]} + diff --git a/tests/golden/eval/devserver--stream-error.sse b/tests/golden/eval/devserver--stream-error.sse new file mode 100644 index 0000000..969f2b0 --- /dev/null +++ b/tests/golden/eval/devserver--stream-error.sse @@ -0,0 +1,6 @@ +event: error +data: {"message":"scripted request failed","stack":"scripted stack","status":429} + +event: done +data: + diff --git a/tests/golden/eval/devserver--stream-happy.sse b/tests/golden/eval/devserver--stream-happy.sse new file mode 100644 index 0000000..10da03d --- /dev/null +++ b/tests/golden/eval/devserver--stream-happy.sse @@ -0,0 +1,15 @@ +event: processing +data: {"evaluators":1} + +event: start +data: {"projectName":"test-project","experimentName":"test-experiment","projectId":"project-test-id","experimentId":"experiment-test-id","projectUrl":null,"experimentUrl":null} + +event: progress +data: {"id":"case-test-id","object_type":"task","origin":null,"format":"code","output_type":"completion","name":"test-eval","event":"json_delta","data":"{\"answer\":\"ok\"}"} + +event: summary +data: {"projectName":"test-project","experimentName":"test-experiment","projectId":"project-test-id","experimentId":"experiment-test-id","projectUrl":"https://example.test/project","experimentUrl":"https://example.test/experiment","comparisonExperimentName":null,"scores":{},"metrics":null,"runMode":null,"isFinal":null,"runLabel":null,"sampleCount":null,"sampleSeed":null} + +event: done +data: + diff --git a/tests/golden/eval/empty--default.stderr b/tests/golden/eval/empty--default.stderr new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/empty--default.stdout b/tests/golden/eval/empty--default.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/error-exit-zero--default.stderr b/tests/golden/eval/error-exit-zero--default.stderr new file mode 100644 index 0000000..b5d5069 --- /dev/null +++ b/tests/golden/eval/error-exit-zero--default.stderr @@ -0,0 +1 @@ +rendered but successful diff --git a/tests/golden/eval/error-exit-zero--default.stdout b/tests/golden/eval/error-exit-zero--default.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/errors--default.stderr b/tests/golden/eval/errors--default.stderr new file mode 100644 index 0000000..78db4ae --- /dev/null +++ b/tests/golden/eval/errors--default.stderr @@ -0,0 +1,34 @@ +Processing 1 evaluator... +▶ Experiment exp-1 is running at https://example.test/experiment/exp-1 + evaluator error 1  +stack line 1 +stack detail 1 + evaluator error 2  +stack line 2 +stack detail 2 +evaluator error 1 +duplicate stack + evaluator error 3  +stack line 3 +stack detail 3 + evaluator error 4  +stack line 4 +stack detail 4 + evaluator error 5  +stack line 5 +stack detail 5 + evaluator error 6  +stack line 6 +stack detail 6 + evaluator error 7  +stack line 7 +stack detail 7 + evaluator error 8  +stack line 8 +stack detail 8 + evaluator error 9  +stack line 9 +stack detail 9 + evaluator error 10  +stack line 10 +stack detail 10 diff --git a/tests/golden/eval/errors--default.stdout b/tests/golden/eval/errors--default.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/errors--verbose.stderr b/tests/golden/eval/errors--verbose.stderr new file mode 100644 index 0000000..78db4ae --- /dev/null +++ b/tests/golden/eval/errors--verbose.stderr @@ -0,0 +1,34 @@ +Processing 1 evaluator... +▶ Experiment exp-1 is running at https://example.test/experiment/exp-1 + evaluator error 1  +stack line 1 +stack detail 1 + evaluator error 2  +stack line 2 +stack detail 2 +evaluator error 1 +duplicate stack + evaluator error 3  +stack line 3 +stack detail 3 + evaluator error 4  +stack line 4 +stack detail 4 + evaluator error 5  +stack line 5 +stack detail 5 + evaluator error 6  +stack line 6 +stack detail 6 + evaluator error 7  +stack line 7 +stack detail 7 + evaluator error 8  +stack line 8 +stack detail 8 + evaluator error 9  +stack line 9 +stack detail 9 + evaluator error 10  +stack line 10 +stack detail 10 diff --git a/tests/golden/eval/errors--verbose.stdout b/tests/golden/eval/errors--verbose.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/esm-retry--tsx-default.stderr b/tests/golden/eval/esm-retry--tsx-default.stderr new file mode 100644 index 0000000..88813cb --- /dev/null +++ b/tests/golden/eval/esm-retry--tsx-default.stderr @@ -0,0 +1,4 @@ +Processing 1 evaluator... +Eval failed with ESM/CJS interop error. Retrying in ESM mode... +Processing 1 evaluator... +▶ Experiment esm-success is running at locally diff --git a/tests/golden/eval/esm-retry--tsx-default.stdout b/tests/golden/eval/esm-retry--tsx-default.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/esm-retry--tsx-verbose.stderr b/tests/golden/eval/esm-retry--tsx-verbose.stderr new file mode 100644 index 0000000..88813cb --- /dev/null +++ b/tests/golden/eval/esm-retry--tsx-verbose.stderr @@ -0,0 +1,4 @@ +Processing 1 evaluator... +Eval failed with ESM/CJS interop error. Retrying in ESM mode... +Processing 1 evaluator... +▶ Experiment esm-success is running at locally diff --git a/tests/golden/eval/esm-retry--tsx-verbose.stdout b/tests/golden/eval/esm-retry--tsx-verbose.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/happy--default.stderr b/tests/golden/eval/happy--default.stderr new file mode 100644 index 0000000..e624ae3 --- /dev/null +++ b/tests/golden/eval/happy--default.stderr @@ -0,0 +1,10 @@ +Processing 1 evaluator... +▶ Experiment exp-1 is running at https://example.test/experiment/exp-1 + +╭ Experiment summary ──────────────────────────────────╮ +│ Scores and metrics exp-1  │ +│ ◯ Accuracy 87.50% │ +│ ◯ duration 1.25s  │ +│ │ +│ See results at https://example.test/experiment/exp-1 │ +╰──────────────────────────────────────────────────────╯ diff --git a/tests/golden/eval/happy--default.stdout b/tests/golden/eval/happy--default.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/happy--jsonl.stderr b/tests/golden/eval/happy--jsonl.stderr new file mode 100644 index 0000000..2bb957c --- /dev/null +++ b/tests/golden/eval/happy--jsonl.stderr @@ -0,0 +1,2 @@ +Processing 1 evaluator... +▶ Experiment exp-1 is running at https://example.test/experiment/exp-1 diff --git a/tests/golden/eval/happy--jsonl.stdout b/tests/golden/eval/happy--jsonl.stdout new file mode 100644 index 0000000..6692fe3 --- /dev/null +++ b/tests/golden/eval/happy--jsonl.stdout @@ -0,0 +1 @@ +{"projectName":"test-project","experimentName":"exp-1","projectId":"project-test-id","experimentId":"exp-1-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/exp-1","comparisonExperimentName":null,"scores":{"Accuracy":{"name":"Accuracy","score":0.875,"diff":0.125,"improvements":3,"regressions":1}},"metrics":{"duration":{"name":"duration","metric":1.25,"unit":"s","diff":-0.2,"improvements":2,"regressions":0}},"runMode":null,"isFinal":null,"runLabel":null,"sampleCount":null,"sampleSeed":null} diff --git a/tests/golden/eval/happy--verbose.stderr b/tests/golden/eval/happy--verbose.stderr new file mode 100644 index 0000000..e624ae3 --- /dev/null +++ b/tests/golden/eval/happy--verbose.stderr @@ -0,0 +1,10 @@ +Processing 1 evaluator... +▶ Experiment exp-1 is running at https://example.test/experiment/exp-1 + +╭ Experiment summary ──────────────────────────────────╮ +│ Scores and metrics exp-1  │ +│ ◯ Accuracy 87.50% │ +│ ◯ duration 1.25s  │ +│ │ +│ See results at https://example.test/experiment/exp-1 │ +╰──────────────────────────────────────────────────────╯ diff --git a/tests/golden/eval/happy--verbose.stdout b/tests/golden/eval/happy--verbose.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/interleaved--default.stderr b/tests/golden/eval/interleaved--default.stderr new file mode 100644 index 0000000..010b4da --- /dev/null +++ b/tests/golden/eval/interleaved--default.stderr @@ -0,0 +1,19 @@ +Processing 2 evaluators... +▶ Experiment exp-a is running at https://example.test/experiment/exp-a +▶ Experiment exp-b is running at https://example.test/experiment/exp-b + +╭ Experiment summary ──────────────────────────────────╮ +│ Scores and metrics exp-a  │ +│ ◯ Accuracy 87.50% │ +│ ◯ duration 1.25s  │ +│ │ +│ See results at https://example.test/experiment/exp-a │ +╰──────────────────────────────────────────────────────╯ + +╭ Experiment summary ──────────────────────────────────╮ +│ Scores and metrics exp-b  │ +│ ◯ Accuracy 87.50% │ +│ ◯ duration 1.25s  │ +│ │ +│ See results at https://example.test/experiment/exp-b │ +╰──────────────────────────────────────────────────────╯ diff --git a/tests/golden/eval/interleaved--default.stdout b/tests/golden/eval/interleaved--default.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/interleaved--jsonl.stderr b/tests/golden/eval/interleaved--jsonl.stderr new file mode 100644 index 0000000..125b01e --- /dev/null +++ b/tests/golden/eval/interleaved--jsonl.stderr @@ -0,0 +1,3 @@ +Processing 2 evaluators... +▶ Experiment exp-a is running at https://example.test/experiment/exp-a +▶ Experiment exp-b is running at https://example.test/experiment/exp-b diff --git a/tests/golden/eval/interleaved--jsonl.stdout b/tests/golden/eval/interleaved--jsonl.stdout new file mode 100644 index 0000000..10c793f --- /dev/null +++ b/tests/golden/eval/interleaved--jsonl.stdout @@ -0,0 +1,2 @@ +{"projectName":"test-project","experimentName":"exp-a","projectId":"project-test-id","experimentId":"exp-a-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/exp-a","comparisonExperimentName":null,"scores":{"Accuracy":{"name":"Accuracy","score":0.875,"diff":0.125,"improvements":3,"regressions":1}},"metrics":{"duration":{"name":"duration","metric":1.25,"unit":"s","diff":-0.2,"improvements":2,"regressions":0}},"runMode":null,"isFinal":null,"runLabel":null,"sampleCount":null,"sampleSeed":null} +{"projectName":"test-project","experimentName":"exp-b","projectId":"project-test-id","experimentId":"exp-b-id","projectUrl":"https://example.test/project/test-project","experimentUrl":"https://example.test/experiment/exp-b","comparisonExperimentName":null,"scores":{"Accuracy":{"name":"Accuracy","score":0.875,"diff":0.125,"improvements":3,"regressions":1}},"metrics":{"duration":{"name":"duration","metric":1.25,"unit":"s","diff":-0.2,"improvements":2,"regressions":0}},"runMode":null,"isFinal":null,"runLabel":null,"sampleCount":null,"sampleSeed":null} diff --git a/tests/golden/eval/minimal-summary--default.stderr b/tests/golden/eval/minimal-summary--default.stderr new file mode 100644 index 0000000..29519fe --- /dev/null +++ b/tests/golden/eval/minimal-summary--default.stderr @@ -0,0 +1,5 @@ +Processing 1 evaluator... +▶ Experiment minimal is running at https://example.test/experiment/minimal + +╭ Experiment summary ╮ +╰──╯ diff --git a/tests/golden/eval/minimal-summary--default.stdout b/tests/golden/eval/minimal-summary--default.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/minimal-summary--jsonl.stderr b/tests/golden/eval/minimal-summary--jsonl.stderr new file mode 100644 index 0000000..2bdc03d --- /dev/null +++ b/tests/golden/eval/minimal-summary--jsonl.stderr @@ -0,0 +1,2 @@ +Processing 1 evaluator... +▶ Experiment minimal is running at https://example.test/experiment/minimal diff --git a/tests/golden/eval/minimal-summary--jsonl.stdout b/tests/golden/eval/minimal-summary--jsonl.stdout new file mode 100644 index 0000000..29c3333 --- /dev/null +++ b/tests/golden/eval/minimal-summary--jsonl.stdout @@ -0,0 +1 @@ +{"projectName":"test-project","experimentName":"minimal","projectId":null,"experimentId":null,"projectUrl":null,"experimentUrl":null,"comparisonExperimentName":null,"scores":{},"metrics":null,"runMode":null,"isFinal":null,"runLabel":null,"sampleCount":null,"sampleSeed":null} diff --git a/tests/golden/eval/progress-edge--default.stderr b/tests/golden/eval/progress-edge--default.stderr new file mode 100644 index 0000000..2fce5cb --- /dev/null +++ b/tests/golden/eval/progress-edge--default.stderr @@ -0,0 +1 @@ +Processing 1 evaluator... diff --git a/tests/golden/eval/progress-edge--default.stdout b/tests/golden/eval/progress-edge--default.stdout new file mode 100644 index 0000000..e69de29 diff --git a/tests/golden/eval/unknown--default.stderr b/tests/golden/eval/unknown--default.stderr new file mode 100644 index 0000000..e624ae3 --- /dev/null +++ b/tests/golden/eval/unknown--default.stderr @@ -0,0 +1,10 @@ +Processing 1 evaluator... +▶ Experiment exp-1 is running at https://example.test/experiment/exp-1 + +╭ Experiment summary ──────────────────────────────────╮ +│ Scores and metrics exp-1  │ +│ ◯ Accuracy 87.50% │ +│ ◯ duration 1.25s  │ +│ │ +│ See results at https://example.test/experiment/exp-1 │ +╰──────────────────────────────────────────────────────╯ diff --git a/tests/golden/eval/unknown--default.stdout b/tests/golden/eval/unknown--default.stdout new file mode 100644 index 0000000..e69de29