Skip to content

feat(benchmark): support inherited and extended effort - #572

Merged
zzwong merged 4 commits into
mainfrom
feat/571-benchmark-effort-controls
Aug 20, 2026
Merged

feat(benchmark): support inherited and extended effort#572
zzwong merged 4 commits into
mainfrom
feat/571-benchmark-effort-controls

Conversation

@zzwong

@zzwong zzwong commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Benchmark candidates can now preserve each reviewer's configured effort instead of forcing one effort across the entire reviewer cohort. The review and benchmark surfaces also support extended effort levels for runtimes that declare them, including xhigh and max through Pi RPC.

What changed

  • Makes stages.reviewers.effort optional for benchmark runs and omits --reviewer-effort when it is inherited.
  • Records reviewer effort provenance as inherited or override in doctor and benchmark artifacts.
  • Adds runtime-aware effort validation before adapter startup.
  • Extends effort ordering and Pi RPC passthrough through xhigh and max while keeping other built-in runtimes capped at high.
  • Preserves profile max_effort ceilings for inherited reviewer effort; explicit effort overrides keep their existing precedence.
  • Documents the benchmark schema, CLI flags, runtime limits, and effort-resolution behavior.

Validation

  • make check

Closes #571

@monit-reviewer monit-reviewer left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated PR Review

Reviewed commit: 45209af0f937
Profile: pi-codex-monit-reviewer - Posting as: monit-reviewer

Summary

Reviewer Findings
go:implementation-tests 1
policies:conventions 0
structure:repo-health 0
architecture:solid-reviewer-agnostic 3
harness-engineering:repo-health 0
security:code-auditor 0
go:implementation-tests (1 finding)

Major - internal/stagemodel/resolver.go:69

When a benchmark omits reviewers.effort but supplies the required exact reviewers.model, this model-override branch uses each agent's DefaultEffort without applying that agent tier's max_effort ceiling. The new inherited-effort contract therefore exceeds configured ceilings for the common exact-model recipe, while model-tier recipes remain capped. Preserve explicit effort override precedence, but when effort is inherited pass/apply the agent's tier ceiling before validation; add a regression test for an exact reviewer-model candidate with omitted effort and a capped agent tier.

architecture:solid-reviewer-agnostic (3 findings)

Minor - internal/config/config.go:644

U-L2: this new exported validation boundary returns only untyped fmt.Errorf values for invalid, unknown-runtime, and unsupported-effort cases. Callers cannot classify these failures programmatically; an inherited unsupported agent effort reaches cmdruntime.MapRunError as an unrecognized generic error, while command-specific preflights manually classify equivalent failures. Introduce typed errors or sentinels such as ErrUnsupportedEffort and wrap them here so all entry points can map the same contract consistently.

Minor - internal/cmd/benchmarkcmd/run.go:535

U-L1: summarizeCandidates unconditionally records effort_source: "inherited", but it is also used by benchmark select, where the reviewer stage may be omitted entirely. Those selector artifacts now claim reviewer-effort inheritance for a stage that does not exist or run. Make the field omittable and populate it only when a reviewer recipe is present, with a selector-only artifact test covering an omitted reviewer stage.

Major - internal/stagemodel/resolver.go:103

U-S3/U-L1: runtime capability validation is not actually on the documented single resolution path for tier-based reviewers. pipeline.resolveReviewerRuntime calls the resolver without ReviewerEffortOverride, then assigns that override directly to ResolvedEffort, bypassing this check. An unsupported effort can therefore pass through the core pipeline whenever a caller did not duplicate the CLI/suite preflight. Thread the reviewer override into ResolveStageModel before it returns the final effort, remove the post-resolution assignment, and add a tier-based reviewer test for an unsupported runtime.

Reviewer Coverage

  • go:implementation-tests — complete (broad); inspected 17 assigned files (21 inspected across reviewers): internal/agents/agents_test.go, internal/benchmark/suite.go, internal/benchmark/suite_test.go, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/benchmarkcmd_test.go, internal/cmd/benchmarkcmd/executor_test.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/reviewcmd/reviewcmd.go, internal/cmd/reviewcmd/reviewcmd_test.go, internal/config/config.go, internal/config/config_effort_test.go, internal/config/config_max_effort_test.go, internal/llmadapters/pi_rpc_test.go, internal/modelprefs/modelprefs.go, internal/modelprefs/modelprefs_effort_test.go, internal/stagemodel/resolver.go, internal/stagemodel/resolver_test.go; skipped: none; constraints: Read-only review tools; tests were inspected but not executed.
  • policies:conventions — complete (broad); inspected 8 assigned files (21 inspected across reviewers): BENCHMARKING.md, README.md, docs/architecture.md, docs/init-config-surface.md, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/reviewcmd/reviewcmd.go, internal/config/config.go; skipped: none; constraints: Read-only review; I did not run the stated validation command. Shared Open CLI Collective source-of-truth documentation was not present in the review context, so I did not infer requirements from it.
  • structure:repo-health — complete (broad); inspected 7 assigned files (21 inspected across reviewers): internal/benchmark/suite.go, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/reviewcmd/reviewcmd.go, internal/config/config.go, internal/modelprefs/modelprefs.go, internal/stagemodel/resolver.go; skipped: none; constraints: Read-only review; inspected assigned source and relevant diff/test hunks but did not execute validation.
  • architecture:solid-reviewer-agnostic — complete (broad); inspected 7 assigned files (21 inspected across reviewers): internal/benchmark/suite.go, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/reviewcmd/reviewcmd.go, internal/config/config.go, internal/modelprefs/modelprefs.go, internal/stagemodel/resolver.go; skipped: none; constraints: Review was limited to the pinned diff and repository reads; available CR tools cannot execute make check. The external harness-engineering PRINCIPLES.md path is outside the disposable repository and unavailable through allowed CR tools.
  • harness-engineering:repo-health — complete (broad); inspected 9 assigned files (21 inspected across reviewers): BENCHMARKING.md, README.md, docs/architecture.md, docs/init-config-surface.md, internal/benchmark/suite.go, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/config/config.go, internal/modelprefs/modelprefs.go, internal/stagemodel/resolver.go; skipped: none; constraints: Static diff and repository inspection only; no test execution tool was available.
  • security:code-auditor — complete (broad); inspected 6 assigned files (21 inspected across reviewers): internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/reviewcmd/reviewcmd.go, internal/config/config.go, internal/modelprefs/modelprefs.go, internal/stagemodel/resolver.go; skipped: none; constraints: I inspected the pinned diff and source but did not execute tests or runtime commands. Review was limited to the six assigned changed files; tests and adapter implementations were not audited.
Inspected files (21)
  • BENCHMARKING.md
  • README.md
  • docs/architecture.md
  • docs/init-config-surface.md
  • internal/agents/agents_test.go
  • internal/benchmark/suite.go
  • internal/benchmark/suite_test.go
  • internal/cmd/benchmarkcmd/benchmarkcmd.go
  • internal/cmd/benchmarkcmd/benchmarkcmd_test.go
  • internal/cmd/benchmarkcmd/executor_test.go
  • internal/cmd/benchmarkcmd/run.go
  • internal/cmd/reviewcmd/reviewcmd.go
  • internal/cmd/reviewcmd/reviewcmd_test.go
  • internal/config/config.go
  • internal/config/config_effort_test.go
  • internal/config/config_max_effort_test.go
  • internal/llmadapters/pi_rpc_test.go
  • internal/modelprefs/modelprefs.go
  • internal/modelprefs/modelprefs_effort_test.go
  • internal/stagemodel/resolver.go
  • internal/stagemodel/resolver_test.go

0 PR discussion threads considered. 0 summarized; 0 resolved.


Completed in 4m 18s | $0.27 | openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna | cr dev
Field Value
Model openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna
Reviewers go:implementation-tests, policies:conventions, structure:repo-health, architecture:solid-reviewer-agnostic, harness-engineering:repo-health, security:code-auditor
Engine pi_rpc · openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna
Reviewed by cr · monit-reviewer
Duration 4m 18s wall · 9m 03s compute
Cost $0.27
Tokens 26.3k in / 5.7k out

Per-workstream usage

  • orchestrator-selection — openai-codex/gpt-5.6-terra
    • In: 5.9k
    • Out: 846
    • Cache read: 2.8k
    • Cache create: 0
    • Cost: $0.03
    • Duration: 17s
  • go:implementation-tests — openai-codex/gpt-5.6-terra
    • In: 2.5k
    • Out: 1.1k
    • Cache read: 107.0k
    • Cache create: 0
    • Cost: $0.05
    • Duration: 1m 06s
  • policies:conventions — openai-codex/gpt-5.6-terra
    • In: 1.7k
    • Out: 225
    • Cache read: 32.3k
    • Cache create: 0
    • Cost: $0.02
    • Duration: 1m 04s
  • structure:repo-health — openai-codex/gpt-5.6-terra
    • In: 8.7k
    • Out: 746
    • Cache read: 22.0k
    • Cache create: 0
    • Cost: $0.04
    • Duration: 56s
  • architecture:solid-reviewer-agnostic — openai-codex/gpt-5.6-sol
    • In: 1.5k
    • Out: 2.1k
    • Cache read: 70.1k
    • Cache create: 0
    • Cost: $0.11
    • Duration: 3m 46s
  • harness-engineering:repo-health — openai-codex/gpt-5.6-terra
    • In: 1.3k
    • Out: 273
    • Cache read: 25.1k
    • Cache create: 0
    • Cost: $0.01
    • Duration: 1m 11s
  • security:code-auditor — openai-codex/gpt-5.6-luna
    • In: 1.1k
    • Out: 207
    • Cache read: 14.8k
    • Cache create: 0
    • Cost: $0.00
    • Duration: 35s
  • orchestrator-rollup — openai-codex/gpt-5.6-terra
    • In: 3.5k
    • Out: 197
    • Cache read: 2.8k
    • Cache create: 0
    • Cost: $0.01
    • Duration: 5s

Comment thread internal/stagemodel/resolver.go
Comment thread internal/config/config.go
Comment thread internal/stagemodel/resolver.go
Comment thread internal/cmd/benchmarkcmd/run.go Outdated
monit-reviewer
monit-reviewer previously approved these changes Aug 19, 2026

@monit-reviewer monit-reviewer left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated PR Review

Reviewed commit: 928b03839e1b
Profile: pi-codex-monit-reviewer - Posting as: monit-reviewer

Summary

Reviewer Findings
go:implementation-tests 0
policies:conventions 0
structure:repo-health 1
architecture:solid-reviewer-agnostic 0
harness-engineering:repo-health 0
security:code-auditor 0
structure:repo-health (1 finding)

Minor - internal/cmd/benchmarkcmd/benchmarkcmd.go:219

benchmark doctor unconditionally emits reviewers.effort_source: "inherited", including selector-only candidates where no reviewer recipe exists. That contradicts the run artifact behavior (which intentionally omits the field without a reviewer recipe) and turns absence of a stage into misleading provenance. Gate this assignment on candidate.Stages.ReviewersConfigured() (and use omitempty or an equivalent representation) so both artifact surfaces share the same contract; add selector-only doctor JSON coverage.

Reviewer Coverage

  • go:implementation-tests — complete (constrained); inspected 21 assigned files (25 inspected across reviewers): internal/agents/agents_test.go, internal/benchmark/suite.go, internal/benchmark/suite_test.go, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/benchmarkcmd_test.go, internal/cmd/benchmarkcmd/executor_test.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/cmderr/cmderr.go, internal/cmd/cmdruntime/cmdruntime_test.go, internal/cmd/reviewcmd/reviewcmd.go, internal/cmd/reviewcmd/reviewcmd_test.go, internal/config/config.go, internal/config/config_effort_test.go, internal/config/config_max_effort_test.go, internal/llmadapters/pi_rpc_test.go, internal/modelprefs/modelprefs.go, internal/modelprefs/modelprefs_effort_test.go, internal/pipeline/pipeline.go, internal/pipeline/pipeline_test.go, internal/stagemodel/resolver.go, internal/stagemodel/resolver_test.go; skipped: none; constraints: Reviewed the pinned diff and assigned Go implementation/test files with read-only repository tools; no test commands were available to run.
  • policies:conventions — complete (constrained); inspected 8 assigned files (25 inspected across reviewers): BENCHMARKING.md, README.md, docs/architecture.md, docs/init-config-surface.md, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/reviewcmd/reviewcmd.go, internal/config/config.go; skipped: none; constraints: Shared Open CLI Collective standards were referenced but their source content was not available in the review context.
  • structure:repo-health — complete (constrained); inspected 7 assigned files (25 inspected across reviewers): internal/benchmark/suite.go, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/reviewcmd/reviewcmd.go, internal/config/config.go, internal/modelprefs/modelprefs.go, internal/stagemodel/resolver.go; skipped: none; constraints: Review limited to the seven assigned changed implementation files; tests were inspected only through the pinned diff and not executed.
  • architecture:solid-reviewer-agnostic — complete (constrained); inspected 7 assigned files (25 inspected across reviewers): internal/benchmark/suite.go, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/reviewcmd/reviewcmd.go, internal/config/config.go, internal/modelprefs/modelprefs.go, internal/stagemodel/resolver.go; skipped: none; constraints: Review was limited to the seven assigned changed files and repository context available through read-only CR tools. Test execution was unavailable; I inspected the added and updated tests but did not run make check. The external harness-engineering principles file was outside the disposable repository and inaccessible through the permitted CR tools.
  • harness-engineering:repo-health — complete (constrained); inspected 9 assigned files (25 inspected across reviewers): BENCHMARKING.md, README.md, docs/architecture.md, docs/init-config-surface.md, internal/benchmark/suite.go, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/config/config.go, internal/modelprefs/modelprefs.go, internal/stagemodel/resolver.go; skipped: none; constraints: Read-only review; tests were not executed.
  • security:code-auditor — complete (constrained); inspected 6 assigned files (25 inspected across reviewers): internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/reviewcmd/reviewcmd.go, internal/config/config.go, internal/modelprefs/modelprefs.go, internal/stagemodel/resolver.go; skipped: none; constraints: Review limited to the six assigned changed files; runtime tests were not executed.
Inspected files (25)
  • BENCHMARKING.md
  • README.md
  • docs/architecture.md
  • docs/init-config-surface.md
  • internal/agents/agents_test.go
  • internal/benchmark/suite.go
  • internal/benchmark/suite_test.go
  • internal/cmd/benchmarkcmd/benchmarkcmd.go
  • internal/cmd/benchmarkcmd/benchmarkcmd_test.go
  • internal/cmd/benchmarkcmd/executor_test.go
  • internal/cmd/benchmarkcmd/run.go
  • internal/cmd/cmderr/cmderr.go
  • internal/cmd/cmdruntime/cmdruntime_test.go
  • internal/cmd/reviewcmd/reviewcmd.go
  • internal/cmd/reviewcmd/reviewcmd_test.go
  • internal/config/config.go
  • internal/config/config_effort_test.go
  • internal/config/config_max_effort_test.go
  • internal/llmadapters/pi_rpc_test.go
  • internal/modelprefs/modelprefs.go
  • internal/modelprefs/modelprefs_effort_test.go
  • internal/pipeline/pipeline.go
  • internal/pipeline/pipeline_test.go
  • internal/stagemodel/resolver.go
  • internal/stagemodel/resolver_test.go

0 PR discussion threads considered. 0 summarized; 0 resolved.


Completed in 2m 01s | $0.28 | openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna | cr dev
Field Value
Model openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna
Reviewers go:implementation-tests, policies:conventions, structure:repo-health, architecture:solid-reviewer-agnostic, harness-engineering:repo-health, security:code-auditor
Engine pi_rpc · openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna
Reviewed by cr · monit-reviewer
Duration 2m 01s wall · 5m 43s compute
Cost $0.28
Tokens 29.8k in / 4.7k out

Per-workstream usage

  • go:implementation-tests — openai-codex/gpt-5.6-terra
    • In: 997
    • Out: 922
    • Cache read: 25.1k
    • Cache create: 0
    • Cost: $0.02
    • Duration: 1m 04s
  • policies:conventions — openai-codex/gpt-5.6-terra
    • In: 2.5k
    • Out: 463
    • Cache read: 18.9k
    • Cache create: 0
    • Cost: $0.02
    • Duration: 37s
  • structure:repo-health — openai-codex/gpt-5.6-terra
    • In: 1.0k
    • Out: 576
    • Cache read: 24.1k
    • Cache create: 0
    • Cost: $0.02
    • Duration: 43s
  • architecture:solid-reviewer-agnostic — openai-codex/gpt-5.6-sol
    • In: 17.5k
    • Out: 2.1k
    • Cache read: 63.0k
    • Cache create: 0
    • Cost: $0.18
    • Duration: 1m 43s
  • harness-engineering:repo-health — openai-codex/gpt-5.6-terra
    • In: 3.5k
    • Out: 350
    • Cache read: 44.5k
    • Cache create: 0
    • Cost: $0.03
    • Duration: 55s
  • security:code-auditor — openai-codex/gpt-5.6-luna
    • In: 1.4k
    • Out: 137
    • Cache read: 8.7k
    • Cache create: 0
    • Cost: $0.00
    • Duration: 35s
  • orchestrator-rollup — openai-codex/gpt-5.6-terra
    • In: 2.8k
    • Out: 70
    • Cache read: 2.8k
    • Cache create: 0
    • Cost: $0.01
    • Duration: 3s

Comment thread internal/cmd/benchmarkcmd/benchmarkcmd.go Outdated

@monit-reviewer monit-reviewer left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated PR Review

Reviewed commit: 8d2679e91f1d
Profile: pi-codex-monit-reviewer - Posting as: monit-reviewer

Summary

Reviewer Findings
go:implementation-tests 0
policies:conventions 0
structure:repo-health 0
architecture:solid-reviewer-agnostic 1
harness-engineering:repo-health 1
security:code-auditor 0
architecture:solid-reviewer-agnostic (1 finding)

Minor - internal/modelprefs/modelprefs.go:12

U-O1: The effort domain is still duplicated as literal value lists across suite.go, reviewcmd.go, and config.go. This extension required updating each site, so another effort level can make accepted values and diagnostics diverge. Let modelprefs own the ordered values and parsing/formatting (for example, AllEfforts/ParseEffort), and derive validation messages from that canonical definition while keeping runtime-capability checks in config.

harness-engineering:repo-health (1 finding)

Minor - internal/config/config.go:543

MaximumEffort is now a required behavioral field on every runtime spec, but nothing mechanically enforces that a newly added spec sets it to a valid value. Its zero value ranks below low, so a future runtime omitted from this field will reject every non-empty effort at resolution time. Add a table-driven config test over LLMRuntimeSpecs() that requires MaximumEffort.Valid() (and preferably validate this invariant where specs are declared).

Reviewer Coverage

  • go:implementation-tests — complete (constrained); inspected 21 assigned files (25 inspected across reviewers): internal/agents/agents_test.go, internal/benchmark/suite.go, internal/benchmark/suite_test.go, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/benchmarkcmd_test.go, internal/cmd/benchmarkcmd/executor_test.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/cmderr/cmderr.go, internal/cmd/cmdruntime/cmdruntime_test.go, internal/cmd/reviewcmd/reviewcmd.go, internal/cmd/reviewcmd/reviewcmd_test.go, internal/config/config.go, internal/config/config_effort_test.go, internal/config/config_max_effort_test.go, internal/llmadapters/pi_rpc_test.go, internal/modelprefs/modelprefs.go, internal/modelprefs/modelprefs_effort_test.go, internal/pipeline/pipeline.go, internal/pipeline/pipeline_test.go, internal/stagemodel/resolver.go, internal/stagemodel/resolver_test.go; skipped: none; constraints: Reviewed Go implementation and behavioral tests through the pinned diff and read-only repository tools; tests were not executed.
  • policies:conventions — complete (constrained); inspected 8 assigned files (25 inspected across reviewers): BENCHMARKING.md, README.md, docs/architecture.md, docs/init-config-surface.md, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/reviewcmd/reviewcmd.go, internal/config/config.go; skipped: none; constraints: Review was limited to the eight assigned changed files; shared CLI standards were not present in the provided review context.
  • structure:repo-health — complete (constrained); inspected 7 assigned files (25 inspected across reviewers): internal/benchmark/suite.go, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/reviewcmd/reviewcmd.go, internal/config/config.go, internal/modelprefs/modelprefs.go, internal/stagemodel/resolver.go; skipped: none; constraints: Review scope was limited to the seven assigned changed source files; I did not inspect unassigned tests, documentation, or pipeline implementation.
  • architecture:solid-reviewer-agnostic — complete (constrained); inspected 7 assigned files (25 inspected across reviewers): internal/benchmark/suite.go, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/reviewcmd/reviewcmd.go, internal/config/config.go, internal/modelprefs/modelprefs.go, internal/stagemodel/resolver.go; skipped: none; constraints: Read-only CR tools do not support command execution, so I did not independently run make check. Review scope was restricted to the seven assigned files; unassigned changes were considered only through the pinned diff. The external SignalFT PRINCIPLES.md path was not accessible through the repository-scoped CR tools.
  • harness-engineering:repo-health — complete (constrained); inspected 9 assigned files (25 inspected across reviewers): BENCHMARKING.md, README.md, docs/architecture.md, docs/init-config-surface.md, internal/benchmark/suite.go, internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/config/config.go, internal/modelprefs/modelprefs.go, internal/stagemodel/resolver.go; skipped: none; constraints: Review limited to assigned changed files and static diff/source inspection; test execution was unavailable.
  • security:code-auditor — complete (constrained); inspected 6 assigned files (25 inspected across reviewers): internal/cmd/benchmarkcmd/benchmarkcmd.go, internal/cmd/benchmarkcmd/run.go, internal/cmd/reviewcmd/reviewcmd.go, internal/config/config.go, internal/modelprefs/modelprefs.go, internal/stagemodel/resolver.go; skipped: none; constraints: Inspected only the six assigned changed production files; tests and unrelated callers were not reviewed. No runtime tests were executed; conclusions are based on static inspection.
Inspected files (25)
  • BENCHMARKING.md
  • README.md
  • docs/architecture.md
  • docs/init-config-surface.md
  • internal/agents/agents_test.go
  • internal/benchmark/suite.go
  • internal/benchmark/suite_test.go
  • internal/cmd/benchmarkcmd/benchmarkcmd.go
  • internal/cmd/benchmarkcmd/benchmarkcmd_test.go
  • internal/cmd/benchmarkcmd/executor_test.go
  • internal/cmd/benchmarkcmd/run.go
  • internal/cmd/cmderr/cmderr.go
  • internal/cmd/cmdruntime/cmdruntime_test.go
  • internal/cmd/reviewcmd/reviewcmd.go
  • internal/cmd/reviewcmd/reviewcmd_test.go
  • internal/config/config.go
  • internal/config/config_effort_test.go
  • internal/config/config_max_effort_test.go
  • internal/llmadapters/pi_rpc_test.go
  • internal/modelprefs/modelprefs.go
  • internal/modelprefs/modelprefs_effort_test.go
  • internal/pipeline/pipeline.go
  • internal/pipeline/pipeline_test.go
  • internal/stagemodel/resolver.go
  • internal/stagemodel/resolver_test.go

0 PR discussion threads considered. 0 summarized; 0 resolved.


Completed in 2m 28s | $0.22 | openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna | cr dev
Field Value
Model openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna
Reviewers go:implementation-tests, policies:conventions, structure:repo-health, architecture:solid-reviewer-agnostic, harness-engineering:repo-health, security:code-auditor
Engine pi_rpc · openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna
Reviewed by cr · monit-reviewer
Duration 2m 28s wall · 7m 19s compute
Cost $0.22
Tokens 16.4k in / 5.5k out

Per-workstream usage

  • go:implementation-tests — openai-codex/gpt-5.6-terra
    • In: 4.3k
    • Out: 602
    • Cache read: 29.2k
    • Cache create: 0
    • Cost: $0.03
    • Duration: 1m 11s
  • policies:conventions — openai-codex/gpt-5.6-terra
    • In: 2.4k
    • Out: 218
    • Cache read: 10.8k
    • Cache create: 0
    • Cost: $0.01
    • Duration: 56s
  • structure:repo-health — openai-codex/gpt-5.6-terra
    • In: 2.3k
    • Out: 464
    • Cache read: 25.1k
    • Cache create: 0
    • Cost: $0.02
    • Duration: 1m 41s
  • architecture:solid-reviewer-agnostic — openai-codex/gpt-5.6-sol
    • In: 831
    • Out: 2.5k
    • Cache read: 70.1k
    • Cache create: 0
    • Cost: $0.11
    • Duration: 2m 07s
  • harness-engineering:repo-health — openai-codex/gpt-5.6-terra
    • In: 2.8k
    • Out: 1.3k
    • Cache read: 24.1k
    • Cache create: 0
    • Cost: $0.03
    • Duration: 48s
  • security:code-auditor — openai-codex/gpt-5.6-luna
    • In: 743
    • Out: 231
    • Cache read: 9.7k
    • Cache create: 0
    • Cost: $0.00
    • Duration: 29s
  • orchestrator-rollup — openai-codex/gpt-5.6-terra
    • In: 3.1k
    • Out: 114
    • Cache read: 2.8k
    • Cache create: 0
    • Cost: $0.01
    • Duration: 4s

Comment thread internal/modelprefs/modelprefs.go
Comment thread internal/config/config.go
@zzwong
zzwong marked this pull request as ready for review August 19, 2026 23:43
@zzwong
zzwong merged commit fe02f5d into main Aug 20, 2026
10 checks passed
@zzwong
zzwong deleted the feat/571-benchmark-effort-controls branch August 20, 2026 01:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support inherited reviewer effort and extended Pi effort in benchmarks

2 participants