Skip to content

Add PR review performance telemetry and dashboards - #807

Open
dayland wants to merge 1 commit into
mainfrom
perf-eval-engine-metrics
Open

Add PR review performance telemetry and dashboards#807
dayland wants to merge 1 commit into
mainfrom
perf-eval-engine-metrics

Conversation

@dayland

@dayland dayland commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

What & why

Extends the BC PR Review category added in #774 with performance telemetry, experiment provenance, and dashboard visibility. This is enablement only: it does not add or publish a new experiment result set.

Changes

  • Collect Copilot token usage, API-call count, estimated AI credits, and elapsed time from the engine transcript or structured run metrics.
  • Record BCQuality knowledge articles retained and pruned by the review filter.
  • Aggregate performance fields only for Code Review, preserving unrelated category result and leaderboard schemas.
  • Add Quality and Performance tabs to the Code Review experiment leaderboard, including engine/BCQuality revisions and self-contained labels for legacy runs.
  • Make BC-ALAgents source selection symmetric with BCQuality through repo, ref, and local_path configuration plus matching CLI overrides.
  • Resolve and record exact BC-ALAgents and BCQuality commits for reproducibility.
  • Let BC-Bench perform authenticated source checkout instead of hardcoding BC-ALAgents main in the workflow.

Experiment configuration

A private BC-Bench branch can independently select either source under pr_review in src/bcbench/agent/shared/config.yaml:

pr_review:
  engine:
    repo: microsoft/BC-ALAgents
    ref: main
    local_path: null
  bcquality:
    repo: null
    ref: null
    local_path: null

Branch names, tags, and commit SHAs are supported. Local checkout overrides are available for unpushed development.

Validation

  • Ruff format and lint
  • Changed-file type checks
  • 762 tests passed, 2 skipped
  • Jekyll dashboard build

AB#645219

Collect BC-ALAgents usage and knowledge metrics, expose Code Review performance dashboards, and make engine and BCQuality experiment sources independently configurable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants