Pin and persist LLM judge models

[Copilot]

BC-Bench relied on separate implicit model defaults for code-review matching and LMChecklist scoring, without recording judge provenance. This could mix results produced by different judge models.

• Configuration

  • Define both judge models in shared configuration:

    judges:
      code-review:
        model: "gpt-5.3-codex"
      lm-checklist:
        model: "gpt-41-2025-04-14"

  • Load these pins centrally without fallback defaults.

• Evaluation

  • Preserve the existing Copilot subprocess for code-review matching.
  • Pass the exact LMChecklist model to bc-eval through JUDGE_MODEL.

• Provenance

  • Record judge_model in evaluation results, bc-eval metadata, summaries, and leaderboard aggregates.
  • Include judge_model in aggregation identity to prevent combining runs judged by different models.

Original prompt

Implement. Add anything unclear or concerns in the comments or PR description

[Chronological Review: 1. User asked: “In BC-Bench, we are using LLM as judge 2 places at least. 1. for code review agent 2. for LMChecklist. Review how are we pinning the AI model used there, and if there are better ways.” 2. BC-Bench inspection found: - Code-review comment matching uses a fixed Copilot CLI model. - LMChecklist scoring is delegated to the external `bc-eval` package. 3. Initial `bc-eval --help` commands hung. The user supplied authoritative source at `C:\depot\BC-LanguageModelTools`, which was then inspected directly. 4. `bc-eval` source confirmed LMChecklist defaults to `gpt-41`, which CAPI resolves to `gpt-41-2025-04-14`. 5. Result schemas were inspected and found not to persist judge-model provenance. 6. Initial recommendation included replacing the code-review subprocess. User explicitly rejected that: “Leave teh code-review Copilot agent subprocess, it is only used for comment matching, should be fine.” 7. User proposed pinning judge models in results and shared `config.yaml`. The shared configuration and result/summary schemas were inspected. 8. An initial richer judge-profile design was proposed. User simplified it: “Not sure if we need properties like profile-version, isn't a model enough? Backend is also not really needed.” 9. Final design converged on a model-only configuration and `judge_model` result field. No implementation has been performed.]

[Intent Mapping:

• Initial review request: “Review how are we pinning the AI model used there, and if there are better ways.”
• Source guidance: “the source code is avaiable on disk: C:\depot\BC-LanguageModelTools”.
• Preserve current code-review architecture: “Leave teh code-review Copilot agent subprocess”.
• Centralize pins: “pin those versions somehow, both in results and in the config.yaml where we keep all the configuration.”
• Avoid unnecessary schema: “isn't a model enough?” and “Backend is also not really needed”.
• Current expected direction: add only model identifiers to configuration/results, without profile/backend fields.]

[Technical Inventory:

• BC-Bench: Python 3.13, Typer, Pydantic, PyYAML, GitHub Actions.
• Code-review judge:
  • Copilot CLI subprocess.
  • Model selector gpt-5.3-codex.
  • Copilot CLI package pinned to @github/copilot@1.0.75.
  • Used only as semantic gate after structural file/line matching.
• LMChecklist:
  • Built-in bc-eval scorer using autoevals.OpenAIScorer.
  • bc-eval[capi]==0.3.14 in workflows.
  • Default environment model gpt-41.
  • CAPI mapping resolves to dated model gpt-41-2025-04-14.
  • Temperature 0 and JSON response format.
• Judge-based categories using LMChecklist:
  • nl2al
  • extensibility-request-implement
  • extensibility-request-triage
• Existing shared configuration:
  • src/bcbench/agent/shared/config.yaml
  • Currently contains prompts, instructions, skills, agents, plugins and MCP configuration.
• Results:
  • Candidate model is stored as result.model.
  • No judge_model currently exists.
  • Summaries aggregate by agent, candidate model, experiment and benchmark version.
• Existing code-review calibration:
  • Human-labelled dataset/judge_calibration.jsonl.
  • Live test is opt-in via BCBENCH_RUN_JUDGE_CALIBRATION.
  • Threshold is accuracy ≥ 0.8.]

[Code Archaeology:

• src/bcbench/config.py
  • JudgeConfig currently contains code_review_model.
  • JudgeConfig.default() hardcodes code_review_model="gpt-5.3-codex".
  • This hardcode should eventually be replaced by YAML loading.
• src/bcbench/evaluate/codereview_judge.py
  • _build_judge_prompt() builds semantic matching prompt.
  • _parse_judge_results() parses verdict JSON.
  • judge_expected_and_ignored() combines expected and ignored pairs in one call.
  • judge_verdicts() invokes Copilot CLI with f"--model={model}".
  • The subprocess architecture is explicitly retained.
• src/bcbench/evaluate/codereview.py
  • Performs structural matching, calls judge_expected_and_ignored(), then builds CodeReviewResult.
• src/bcbench/evaluate/codereview_judge_calibration.py
  • Loads calibration cases and runs the same judge model.
• src/bcbench/commands/evaluate.py
  • Exposes judge-calibration; its model defaults to _config.judge.code_review_model.
• src/bcbench/cli_options.py
  • CopilotModel literal includes gpt-5.3-codex.
• .github/actions/install-eval-clis/action.yml
  • Pins GitHub Copilot CLI to @github/copilot@1.0.75.
• .github/workflows/copilot-evaluation.yml
  • Supplies token/permissions needed by the code-review judge.
• .github/workflows/claude-evaluation.yml
  • Also installs/uses Copilot CLI because Claude code-review runs still use Copilot for matching.
• .github/workflows/summarize-results.yml
  • Installs bc-eval[capi]==0.3.14.
  • Invokes bceval metrics calculate --use-capi.
  • Does not currently set JUDGE_MODEL.
  • Its `in...

Created from Copilot CLI via the copilot delegate command.

[Copilot]

Pull request overview

Pins LLM judge models centrally and preserves judge provenance through evaluation exports, summaries, and leaderboard aggregation.

Changes:

• Adds shared judge-model configuration and workflow propagation.
• Persists judge models in results, summaries, and aggregates.
• Separates leaderboard runs by judge model and adds coverage.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_type_exhaustiveness.py	Tests category judge mapping.
tests/test_result_writer.py	Tests export metadata.
tests/test_result_hierarchy.py	Updates judge summary fixture.
tests/test_evaluation_summary.py	Tests judge-aware aggregation.
tests/test_evaluation_factories.py	Tests provenance propagation.
tests/test_codereview.py	Updates result fixture.
tests/test_category_command.py	Tests workflow outputs.
src/bcbench/types.py	Maps categories to judges.
src/bcbench/results/summary.py	Adds summary provenance.
src/bcbench/results/leaderboard.py	Adds aggregate provenance.
src/bcbench/results/codereview.py	Marks reviews judge-scored.
src/bcbench/results/bceval_export.py	Exports provenance metadata.
src/bcbench/results/base.py	Defines judge-scored results.
src/bcbench/config.py	Loads judge configuration.
src/bcbench/commands/result.py	Groups by judge-aware identity.
src/bcbench/commands/category.py	Emits judge workflow output.
src/bcbench/agent/shared/config.yaml	Pins judge models.
docs/_data/test-generation.json	Refreshes leaderboard data.
docs/_data/code-review.json	Adds historical provenance fields.
.github/workflows/summarize-results.yml	Passes JUDGE_MODEL to bc-eval.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/bcbench/results/summary.py:196

• This silently labels the whole summary with the first result's judge even when the input artifacts contain different judge_model values. The resulting leaderboard then treats all instances as one judge run, defeating the provenance guarantee. Reject non-judge results or any judge-model mismatch before constructing the summary.

        first_result = results[0]
        assert isinstance(first_result, JudgeScoredEvaluationResult)
        return {**super()._base_fields(results, run_id), "judge_model": first_result.judge_model}

[Wenjie Fan (gggdttt)]

Only one thing I'd actually ask for: JUDGE_MODEL is emitted for code-review (which has no lm_checklist evaluator) and as an empty string for the unjudged categories. See the two comments on summarize-results.yml and category.py — same fix covers both.

The rest are a nit and a question. I retracted my comment on test-generation.json, it was wrong.

Extending combination_key with judge_model is the best part of this PR — runs judged by different models won't get silently merged anymore.

[Wenjie Fan (gggdttt)]

One more question, on the docs side.