From 41f10c5a273d20d0b4aae9a886a68cf63eacdf43 Mon Sep 17 00:00:00 2001 From: Liang-Chun Tsai Date: Mon, 17 Aug 2026 16:24:20 -0700 Subject: [PATCH] Document ThinkingBox-Bench result aggregation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- releases/thinkingbox_bench_v1/README.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/releases/thinkingbox_bench_v1/README.md b/releases/thinkingbox_bench_v1/README.md index b1f5aad..f7bfb93 100644 --- a/releases/thinkingbox_bench_v1/README.md +++ b/releases/thinkingbox_bench_v1/README.md @@ -140,12 +140,21 @@ uv run tb infer -c config/config_o4mini.yaml \ --output output_thinkingbox_bench_v1.jsonl ``` -Aggregate pass rates: +## Analyze the results + +The `tb infer` command writes one JSON object per trial to +`output_thinkingbox_bench_v1.jsonl`. Use `tb agg` to compute the aggregate +metrics: ```bash uv run tb agg output_thinkingbox_bench_v1.jsonl ``` +For a JSONL containing 20 attempts for every task, the aggregate output includes +pass@1, pass@20, and pass^20. pass@20 measures whether at least one of 20 +attempts succeeds; pass^20 estimates whether all 20 attempts succeed. These +metrics are omitted if tasks have unequal attempt counts. + Press Ctrl+C in the background-services terminal to stop Typesense and the MCP Session Proxy.