[pdw207]

Hypothesis: Opus 5.5 reviewing (critic stays glm-5.3-flash) closes the recall gap to Copilot (9/13 final-push reals, 5% nits) without adding wrong findings.

Read against main's baseline in the Eval results comment and against Copilot in evals/bench/copilot/report.md (#187). Five eval-full runs planned; roughly $30 per run at Opus pricing.

🤖 Generated with Claude Code

[github-actions]

Eval results

e3ae5be-full against main on anthropic/claude-opus-5.5 (critic z-ai/glm-5.3-flash), Thingie 0.11.6

Headline	main	e3ae5be-full	Change
Real recall	7/13	6/13	-0.08 worse
Nit rate	16%	13%	-3 better
Wrong rate	0%	0%	same
Minutes	29.6	4.7	-24.9 better
Dollars	9.76	7.19	-2.57 better

Full report

Quality

	main	e3ae5be-full
Real recall	7/13	6/13
Caught at the first push	8/21	11/21
Reals held back	0	0

Noise

	main	e3ae5be-full
Threads	64	45
Nits	10	6
Wrong	0	0
Unlabelled	46	33
Threads per 100 changed lines	1.6	1.2
Duplicate threads on one bug	1	0

Calibration

	main	e3ae5be-full
Highs that are real	6/19	5/11
Mediums that are nits	7/40	5/33
Confidence 3 posted	0	0

Latency

	main	e3ae5be-full
Seconds	1778	284
Unchecked (critic timed out)	3	0
Critic timeouts	3	0

Cost

	main	e3ae5be-full
Dollars	9.76	7.19
Input tokens	1603560	1615083
Output tokens	1297086	144079

Cases

Case	Ref	Real	Nit	Wrong	Threads	Seconds	Dollars
attach-drafts-to-lists	final-push	1/3	1	0	10	57	1.69
attach-drafts-to-lists	first-push	3/5	1	0	12	46	1.63
draft-tasks-api	final-push	2/3	3	0	14	64	1.74
draft-tasks-api	first-push	3/5	4	0	13	66	1.64
draft-tasks-model	final-push	3/3	1	0	9	56	1.51
draft-tasks-model	first-push	2/4	2	0	10	52	1.44
paged-connectors	final-push	0/2	0	0	6	57	1.43
paged-connectors	first-push	1/4	1	0	9	40	1.45
readme-typo	final-push	0/0	0	0	0	4	0.02
route-discovery	final-push	0/2	1	0	6	46	0.8
route-discovery	first-push	2/3	1	0	10	47	0.82

run file: tmp/bench/run.json

Run

• Tier: full (6 cases, first and final push each)
• Run time: 3 min wall clock, cases in parallel
• Logs and run file: https://github.com/bonusly/thingie/actions/runs/41097388039
• Re-run: remove and add the eval-fast or eval-full label

[pdw207]

Five eval-full runs of Opus 5.5 reviewing (critic glm-5.3-flash), mean (min to max), against main's baseline and Copilot (evals/bench/copilot/run.json):

	main (glm-5.3)	Copilot	Opus 5.5 x5
Real recall, final push	7/13	9/13	5.6/13 (5 to 6)
Caught at first push	8/21	17/21	8.6/21 (5 to 12)
Nit rate	16%	5%	17.8% (13 to 21)
Wrong	0	0	0
Threads	64	57	46
Minutes	29.6	43.1	4.8
Dollars	9.76	n/a	7.16

Missed in all five runs, caught by Copilot: Retry after task creation makes a second task (draft-tasks-api), Sync advances last_synced_at even when the page cap cut the walk short (paged-connectors), and both route-discovery reals. Opus catches Update can overwrite a draft that was sent in the meantime every run, which Copilot misses. No labelled real was held back by post-process.

Screened out on recall: below main and Copilot, though about 6x faster and cheaper than main.

🤖 Generated with Claude Code

[review-thingie]

✅ No new changes recommended

Files reviewed (3)

• evals/bench/config.toml
• evals/bench/lib/bench.rb
• evals/bench/variants/opus-cli.toml

Change risk: routine (review need 0.32)

File	Rating	Review need	Security	User impact	Performance	New dependencies	Maintainability
evals/bench/lib/bench.rb	routine	0.32	0.06	0.17	0.13	0.07	0.13
evals/bench/config.toml	trivial	0.16	0.05	0.10	0.33	0.36	0.13
evals/bench/variants/opus-cli.toml	trivial	0.03	0.03	0.06	0.09	0.08	0.09

Escalations: none

Thingie details

• Thingie version: 0.11.6
• Review model: openai/gpt-6-sol
• Decision model: jev-latest
• Base: 037b165
• Reviewed at: 2026-10-30 17:40 UTC

1 finding not posted as review comments

Medium · CLI sees future changes during first-push runs · evals/bench/lib/bench.rb:217

What's wrong: The CLI client reads repository files from config.root, which is checked out at the final push. For a first-push run, it can see changes that have not happened yet.

When it happens: A case has separate first and final pushes, and the CLI reads a file changed only in the final push.

Fix: Run the CLI against a checkout at the push being evaluated.

[pdw207]

One local run through the Claude Code CLI (--variant opus-cli, Opus 5.5 as reviewer and critic, 3 calls at a time, bddcadd):

	Copilot	Opus via OpenRouter x5	Opus via CLI x1
Real recall, final push	9/13	5.6/13	5/13
Caught at first push	17/21	8.6/21	10/21
Nit rate	5%	17.8%	21%
Wrong	0	0	0
Threads	57	46	24
Highs that are real	11/35	5.2/13.4	4/6
Minutes	43.1	4.8	20.1
Dollars (CLI-reported, on plan)	n/a	7.16	26.01

It is the first Opus run to catch Sync advances last_synced_at even when the page cap cut the walk short, but it missed Note comment failure is swallowed, which every OpenRouter run caught. The Opus critic cut hard (28 first-pass findings to 3 posted on attach-drafts-to-lists final push). The matcher also ran on Opus here rather than glm-5.3, so the judge differs from the other columns.

🤖 Generated with Claude Code