Feat/referee self inconsistency floor

[armaanvgrewal]

Closes #417

Referee self-inconsistency floor

Summary

This PR addresses the referee self-inconsistency floor experiment and hardens the supporting infrastructure used to run it.

The experiment now:

• measures self-inconsistency over explicitly declared answers only,
• tracks undeclared responses separately,
• uses a shared declaration-detection utility in the extraction layer,
• and applies a default Gemini output-token cap to prevent runaway generations from stalling runs.

Changes

Declaration-aware self-inconsistency accounting

Added declaration-aware answer tracking to the referee self-inconsistency experiment.

Per-row outputs now include:

• declared_1
• declared_2

Summary outputs now include:

• declared_pairs
• undeclared_pairs
• undeclared_draws

temp0_self_inconsistency_rate is now computed over declared pairs only.

Shared declaration-detection utility

Added a shared utility in benchmaxxing.extract:

declared_mcq_choice(...)

This makes it possible to distinguish:

• explicit answer declarations,

from

• parser fallback behavior.

The existing parsing behavior remains unchanged.

Gemini output-token limit

Updated GeminiBackend to apply a default:

max_output_tokens = 16384

This limit is merged into existing decoding settings and may still be overridden when needed.

Tests

Added and updated test coverage for:

• declaration detection,
• declared vs undeclared accounting,
• declared-only inconsistency metrics,
• Gemini default output-token behavior.

Validation

Targeted validation

159 passed
1 skipped

including:

• gateway tests,
• extraction tests,
• parse regression tests,
• MedQA parser compatibility tests,
• referee self-inconsistency tests.

End-to-end validation

A full 40-case self-inconsistency run completed successfully and produced:

{
  "n": 40,
  "declared_pairs": 39,
  "undeclared_pairs": 1,
  "undeclared_draws": 2,
  "stable_cases": 39,
  "unstable_cases": 0,
  "temp0_self_inconsistency_rate": 0.0
}

The run completed successfully and in reasonable time under the updated configuration and exercised the new declaration-aware reporting.

Commits

• Track declared answers separately from parser fallbacks
• Limit Gemini output tokens by default

[armaanvgrewal]

@sebasmos @Agastya191 @duckyquang @azzy-chemE @felipeocampoos @PeterLi-jpg

Review requested, thanks!

[sebasmos]

Read 336f78d by hand: extract.py, gateway.py, referee_self_inconsistency.py, and referee_threshold._Cache, which the runner imports and the diff does not show. Full suite keyless on the PR head: 1221 passed, 8 skipped. Ruff clean.

What holds up. declared_mcq_choice mirrors parse_mcq_choice's Heuristic 1 plus the single-letter reply exactly, and parse_mcq_choice itself is untouched, so no committed number moves. On the 38 unique degenerate completions in experiments/referee/results/call_cache.jsonl it flags exactly the 2 conditional declarations and leaves 36 undeclared, which is the audit's count. _Cache.complete keys on (model, temperature, draw, prompt), so draw 1 and draw 2 are two real calls and a 0.0 rate cannot come from a shared cache entry. _Cache builds GeminiBackend with no default_decoding, so the 16384 cap reaches this lane, and config = {**default_decoding, **decoding} does forward it to generate_content. Prompt and temperature are unchanged.

One blocker. The 40-case run in the description is not in the repo: no referee_self_inconsistency.jsonl, no referee_self_inconsistency_summary.json, no referee_self_inconsistency_cache.jsonl. The sibling lane commits all three (referee_threshold.jsonl, its summary, and referee_threshold_requery_cache.jsonl force-added past the *_cache.jsonl ignore, since MedQA is public). Without the rows nobody can check 39/1/2/0 or replay it keylessly, and the results directory is not ignored for the rows or the summary. Please commit the three files.

Two smaller asks. "cache_bypassed": true in the summary is a hardcoded literal and is false: the draws are cached under distinct keys, which is what makes a keyless replay possible. Drop it (test_summary_contains_metadata asserts it, so that test goes with it); new_api_calls_this_run already says what happened. And the PR says Closes #417 while items 4 and 5 are proposed as follow-ups. That split is fine, but please open the two issues before this merges and link them here, so they are not lost when #417 closes.

Test gap, small. The description says the cap "may still be overridden". It can, I checked both default_decoding={"max_output_tokens": 100} at construction and decoding={"max_output_tokens": 7} per call, but no test covers either; one line in test_gateway.py would pin it.

One note, not an ask. declared_mcq_choice does not run is_abstention first, so a refusal that also matches a declaration pattern would come back declared=True with an empty legacy answer. Zero of the 6,134 committed referee rows hit it, so it is a comment, not a defect.