diff --git a/README.md b/README.md index b301d93..d0ba9b2 100644 --- a/README.md +++ b/README.md @@ -362,6 +362,7 @@ Deeper references for specific topics live under [`docs/`](docs/): - [`test_cases_deep_dive.md`](docs/test_cases_deep_dive.md) — Deeper examples and patterns for test-case authoring. - [`history_and_metadata.md`](docs/history_and_metadata.md) — Multi-turn test cases with prior conversation history loaded from a companion `.meta.yaml`. - [`debugging_tests.md`](docs/debugging_tests.md) — How to debug a failing test (VSCode launch configs and friends). +- [`analyzing_results.md`](docs/analyzing_results.md) — Use `tb agg` to compute pass@1, pass@20, and pass^20 from a `tb infer` result JSONL. **Fixtures and judges** - [`fixtures.md`](docs/fixtures.md) — How fixtures are wired up (dependency injection via `conftest.yaml` and scenario overrides). diff --git a/docs/analyzing_results.md b/docs/analyzing_results.md new file mode 100644 index 0000000..0f99e97 --- /dev/null +++ b/docs/analyzing_results.md @@ -0,0 +1,19 @@ +# Analyzing inference results + +## Result file + +Run `tb infer` as described in the [ThinkingBox README](../README.md) to +produce a batch result JSONL file. Each line represents one task attempt. To +report pass@20 and pass^20, run 20 attempts per task. + +## Compute the reported metrics + +```bash +uv run tb agg output_thinkingbox_bench_v1.jsonl +``` + +For a JSONL containing 20 attempts for every task, the aggregate output +includes the paper's pass@1, pass@20, and pass^20 values. pass@20 measures +whether at least one of 20 attempts succeeds; pass^20 estimates whether all 20 +attempts succeed. These metrics are omitted if tasks have unequal attempt +counts.