feat(benchmark): support inherited and extended effort - #572
Conversation
monit-reviewer
left a comment
There was a problem hiding this comment.
Automated PR Review
Reviewed commit: 45209af0f937
Profile: pi-codex-monit-reviewer - Posting as: monit-reviewer
Summary
| Reviewer | Findings |
|---|---|
| go:implementation-tests | 1 |
| policies:conventions | 0 |
| structure:repo-health | 0 |
| architecture:solid-reviewer-agnostic | 3 |
| harness-engineering:repo-health | 0 |
| security:code-auditor | 0 |
go:implementation-tests (1 finding)
Major - internal/stagemodel/resolver.go:69
When a benchmark omits
reviewers.effortbut supplies the required exactreviewers.model, this model-override branch uses each agent'sDefaultEffortwithout applying that agent tier'smax_effortceiling. The new inherited-effort contract therefore exceeds configured ceilings for the common exact-model recipe, while model-tier recipes remain capped. Preserve explicit effort override precedence, but when effort is inherited pass/apply the agent's tier ceiling before validation; add a regression test for an exact reviewer-model candidate with omitted effort and a capped agent tier.
architecture:solid-reviewer-agnostic (3 findings)
Minor - internal/config/config.go:644
U-L2: this new exported validation boundary returns only untyped
fmt.Errorfvalues for invalid, unknown-runtime, and unsupported-effort cases. Callers cannot classify these failures programmatically; an inherited unsupported agent effort reachescmdruntime.MapRunErroras an unrecognized generic error, while command-specific preflights manually classify equivalent failures. Introduce typed errors or sentinels such asErrUnsupportedEffortand wrap them here so all entry points can map the same contract consistently.
Minor - internal/cmd/benchmarkcmd/run.go:535
U-L1:
summarizeCandidatesunconditionally recordseffort_source: "inherited", but it is also used bybenchmark select, where the reviewer stage may be omitted entirely. Those selector artifacts now claim reviewer-effort inheritance for a stage that does not exist or run. Make the field omittable and populate it only when a reviewer recipe is present, with a selector-only artifact test covering an omitted reviewer stage.
Major - internal/stagemodel/resolver.go:103
U-S3/U-L1: runtime capability validation is not actually on the documented single resolution path for tier-based reviewers.
pipeline.resolveReviewerRuntimecalls the resolver withoutReviewerEffortOverride, then assigns that override directly toResolvedEffort, bypassing this check. An unsupported effort can therefore pass through the core pipeline whenever a caller did not duplicate the CLI/suite preflight. Thread the reviewer override intoResolveStageModelbefore it returns the final effort, remove the post-resolution assignment, and add a tier-based reviewer test for an unsupported runtime.
Reviewer Coverage
go:implementation-tests— complete (broad); inspected 17 assigned files (21 inspected across reviewers):internal/agents/agents_test.go,internal/benchmark/suite.go,internal/benchmark/suite_test.go,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/benchmarkcmd_test.go,internal/cmd/benchmarkcmd/executor_test.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/reviewcmd/reviewcmd.go,internal/cmd/reviewcmd/reviewcmd_test.go,internal/config/config.go,internal/config/config_effort_test.go,internal/config/config_max_effort_test.go,internal/llmadapters/pi_rpc_test.go,internal/modelprefs/modelprefs.go,internal/modelprefs/modelprefs_effort_test.go,internal/stagemodel/resolver.go,internal/stagemodel/resolver_test.go; skipped: none; constraints: Read-only review tools; tests were inspected but not executed.policies:conventions— complete (broad); inspected 8 assigned files (21 inspected across reviewers):BENCHMARKING.md,README.md,docs/architecture.md,docs/init-config-surface.md,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/reviewcmd/reviewcmd.go,internal/config/config.go; skipped: none; constraints: Read-only review; I did not run the stated validation command. Shared Open CLI Collective source-of-truth documentation was not present in the review context, so I did not infer requirements from it.structure:repo-health— complete (broad); inspected 7 assigned files (21 inspected across reviewers):internal/benchmark/suite.go,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/reviewcmd/reviewcmd.go,internal/config/config.go,internal/modelprefs/modelprefs.go,internal/stagemodel/resolver.go; skipped: none; constraints: Read-only review; inspected assigned source and relevant diff/test hunks but did not execute validation.architecture:solid-reviewer-agnostic— complete (broad); inspected 7 assigned files (21 inspected across reviewers):internal/benchmark/suite.go,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/reviewcmd/reviewcmd.go,internal/config/config.go,internal/modelprefs/modelprefs.go,internal/stagemodel/resolver.go; skipped: none; constraints: Review was limited to the pinned diff and repository reads; available CR tools cannot executemake check. The external harness-engineeringPRINCIPLES.mdpath is outside the disposable repository and unavailable through allowed CR tools.harness-engineering:repo-health— complete (broad); inspected 9 assigned files (21 inspected across reviewers):BENCHMARKING.md,README.md,docs/architecture.md,docs/init-config-surface.md,internal/benchmark/suite.go,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/config/config.go,internal/modelprefs/modelprefs.go,internal/stagemodel/resolver.go; skipped: none; constraints: Static diff and repository inspection only; no test execution tool was available.security:code-auditor— complete (broad); inspected 6 assigned files (21 inspected across reviewers):internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/reviewcmd/reviewcmd.go,internal/config/config.go,internal/modelprefs/modelprefs.go,internal/stagemodel/resolver.go; skipped: none; constraints: I inspected the pinned diff and source but did not execute tests or runtime commands. Review was limited to the six assigned changed files; tests and adapter implementations were not audited.
Inspected files (21)
BENCHMARKING.mdREADME.mddocs/architecture.mddocs/init-config-surface.mdinternal/agents/agents_test.gointernal/benchmark/suite.gointernal/benchmark/suite_test.gointernal/cmd/benchmarkcmd/benchmarkcmd.gointernal/cmd/benchmarkcmd/benchmarkcmd_test.gointernal/cmd/benchmarkcmd/executor_test.gointernal/cmd/benchmarkcmd/run.gointernal/cmd/reviewcmd/reviewcmd.gointernal/cmd/reviewcmd/reviewcmd_test.gointernal/config/config.gointernal/config/config_effort_test.gointernal/config/config_max_effort_test.gointernal/llmadapters/pi_rpc_test.gointernal/modelprefs/modelprefs.gointernal/modelprefs/modelprefs_effort_test.gointernal/stagemodel/resolver.gointernal/stagemodel/resolver_test.go
0 PR discussion threads considered. 0 summarized; 0 resolved.
Completed in 4m 18s | $0.27 | openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna | cr dev
| Field | Value |
|---|---|
| Model | openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna |
| Reviewers | go:implementation-tests, policies:conventions, structure:repo-health, architecture:solid-reviewer-agnostic, harness-engineering:repo-health, security:code-auditor |
| Engine | pi_rpc · openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna |
| Reviewed by | cr · monit-reviewer |
| Duration | 4m 18s wall · 9m 03s compute |
| Cost | $0.27 |
| Tokens | 26.3k in / 5.7k out |
Per-workstream usage
orchestrator-selection— openai-codex/gpt-5.6-terra- In: 5.9k
- Out: 846
- Cache read: 2.8k
- Cache create: 0
- Cost: $0.03
- Duration: 17s
go:implementation-tests— openai-codex/gpt-5.6-terra- In: 2.5k
- Out: 1.1k
- Cache read: 107.0k
- Cache create: 0
- Cost: $0.05
- Duration: 1m 06s
policies:conventions— openai-codex/gpt-5.6-terra- In: 1.7k
- Out: 225
- Cache read: 32.3k
- Cache create: 0
- Cost: $0.02
- Duration: 1m 04s
structure:repo-health— openai-codex/gpt-5.6-terra- In: 8.7k
- Out: 746
- Cache read: 22.0k
- Cache create: 0
- Cost: $0.04
- Duration: 56s
architecture:solid-reviewer-agnostic— openai-codex/gpt-5.6-sol- In: 1.5k
- Out: 2.1k
- Cache read: 70.1k
- Cache create: 0
- Cost: $0.11
- Duration: 3m 46s
harness-engineering:repo-health— openai-codex/gpt-5.6-terra- In: 1.3k
- Out: 273
- Cache read: 25.1k
- Cache create: 0
- Cost: $0.01
- Duration: 1m 11s
security:code-auditor— openai-codex/gpt-5.6-luna- In: 1.1k
- Out: 207
- Cache read: 14.8k
- Cache create: 0
- Cost: $0.00
- Duration: 35s
orchestrator-rollup— openai-codex/gpt-5.6-terra- In: 3.5k
- Out: 197
- Cache read: 2.8k
- Cache create: 0
- Cost: $0.01
- Duration: 5s
monit-reviewer
left a comment
There was a problem hiding this comment.
Automated PR Review
Reviewed commit: 928b03839e1b
Profile: pi-codex-monit-reviewer - Posting as: monit-reviewer
Summary
| Reviewer | Findings |
|---|---|
| go:implementation-tests | 0 |
| policies:conventions | 0 |
| structure:repo-health | 1 |
| architecture:solid-reviewer-agnostic | 0 |
| harness-engineering:repo-health | 0 |
| security:code-auditor | 0 |
structure:repo-health (1 finding)
Minor - internal/cmd/benchmarkcmd/benchmarkcmd.go:219
benchmark doctorunconditionally emitsreviewers.effort_source: "inherited", including selector-only candidates where no reviewer recipe exists. That contradicts the run artifact behavior (which intentionally omits the field without a reviewer recipe) and turns absence of a stage into misleading provenance. Gate this assignment oncandidate.Stages.ReviewersConfigured()(and useomitemptyor an equivalent representation) so both artifact surfaces share the same contract; add selector-only doctor JSON coverage.
Reviewer Coverage
go:implementation-tests— complete (constrained); inspected 21 assigned files (25 inspected across reviewers):internal/agents/agents_test.go,internal/benchmark/suite.go,internal/benchmark/suite_test.go,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/benchmarkcmd_test.go,internal/cmd/benchmarkcmd/executor_test.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/cmderr/cmderr.go,internal/cmd/cmdruntime/cmdruntime_test.go,internal/cmd/reviewcmd/reviewcmd.go,internal/cmd/reviewcmd/reviewcmd_test.go,internal/config/config.go,internal/config/config_effort_test.go,internal/config/config_max_effort_test.go,internal/llmadapters/pi_rpc_test.go,internal/modelprefs/modelprefs.go,internal/modelprefs/modelprefs_effort_test.go,internal/pipeline/pipeline.go,internal/pipeline/pipeline_test.go,internal/stagemodel/resolver.go,internal/stagemodel/resolver_test.go; skipped: none; constraints: Reviewed the pinned diff and assigned Go implementation/test files with read-only repository tools; no test commands were available to run.policies:conventions— complete (constrained); inspected 8 assigned files (25 inspected across reviewers):BENCHMARKING.md,README.md,docs/architecture.md,docs/init-config-surface.md,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/reviewcmd/reviewcmd.go,internal/config/config.go; skipped: none; constraints: Shared Open CLI Collective standards were referenced but their source content was not available in the review context.structure:repo-health— complete (constrained); inspected 7 assigned files (25 inspected across reviewers):internal/benchmark/suite.go,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/reviewcmd/reviewcmd.go,internal/config/config.go,internal/modelprefs/modelprefs.go,internal/stagemodel/resolver.go; skipped: none; constraints: Review limited to the seven assigned changed implementation files; tests were inspected only through the pinned diff and not executed.architecture:solid-reviewer-agnostic— complete (constrained); inspected 7 assigned files (25 inspected across reviewers):internal/benchmark/suite.go,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/reviewcmd/reviewcmd.go,internal/config/config.go,internal/modelprefs/modelprefs.go,internal/stagemodel/resolver.go; skipped: none; constraints: Review was limited to the seven assigned changed files and repository context available through read-only CR tools. Test execution was unavailable; I inspected the added and updated tests but did not runmake check. The external harness-engineering principles file was outside the disposable repository and inaccessible through the permitted CR tools.harness-engineering:repo-health— complete (constrained); inspected 9 assigned files (25 inspected across reviewers):BENCHMARKING.md,README.md,docs/architecture.md,docs/init-config-surface.md,internal/benchmark/suite.go,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/config/config.go,internal/modelprefs/modelprefs.go,internal/stagemodel/resolver.go; skipped: none; constraints: Read-only review; tests were not executed.security:code-auditor— complete (constrained); inspected 6 assigned files (25 inspected across reviewers):internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/reviewcmd/reviewcmd.go,internal/config/config.go,internal/modelprefs/modelprefs.go,internal/stagemodel/resolver.go; skipped: none; constraints: Review limited to the six assigned changed files; runtime tests were not executed.
Inspected files (25)
BENCHMARKING.mdREADME.mddocs/architecture.mddocs/init-config-surface.mdinternal/agents/agents_test.gointernal/benchmark/suite.gointernal/benchmark/suite_test.gointernal/cmd/benchmarkcmd/benchmarkcmd.gointernal/cmd/benchmarkcmd/benchmarkcmd_test.gointernal/cmd/benchmarkcmd/executor_test.gointernal/cmd/benchmarkcmd/run.gointernal/cmd/cmderr/cmderr.gointernal/cmd/cmdruntime/cmdruntime_test.gointernal/cmd/reviewcmd/reviewcmd.gointernal/cmd/reviewcmd/reviewcmd_test.gointernal/config/config.gointernal/config/config_effort_test.gointernal/config/config_max_effort_test.gointernal/llmadapters/pi_rpc_test.gointernal/modelprefs/modelprefs.gointernal/modelprefs/modelprefs_effort_test.gointernal/pipeline/pipeline.gointernal/pipeline/pipeline_test.gointernal/stagemodel/resolver.gointernal/stagemodel/resolver_test.go
0 PR discussion threads considered. 0 summarized; 0 resolved.
Completed in 2m 01s | $0.28 | openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna | cr dev
| Field | Value |
|---|---|
| Model | openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna |
| Reviewers | go:implementation-tests, policies:conventions, structure:repo-health, architecture:solid-reviewer-agnostic, harness-engineering:repo-health, security:code-auditor |
| Engine | pi_rpc · openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna |
| Reviewed by | cr · monit-reviewer |
| Duration | 2m 01s wall · 5m 43s compute |
| Cost | $0.28 |
| Tokens | 29.8k in / 4.7k out |
Per-workstream usage
go:implementation-tests— openai-codex/gpt-5.6-terra- In: 997
- Out: 922
- Cache read: 25.1k
- Cache create: 0
- Cost: $0.02
- Duration: 1m 04s
policies:conventions— openai-codex/gpt-5.6-terra- In: 2.5k
- Out: 463
- Cache read: 18.9k
- Cache create: 0
- Cost: $0.02
- Duration: 37s
structure:repo-health— openai-codex/gpt-5.6-terra- In: 1.0k
- Out: 576
- Cache read: 24.1k
- Cache create: 0
- Cost: $0.02
- Duration: 43s
architecture:solid-reviewer-agnostic— openai-codex/gpt-5.6-sol- In: 17.5k
- Out: 2.1k
- Cache read: 63.0k
- Cache create: 0
- Cost: $0.18
- Duration: 1m 43s
harness-engineering:repo-health— openai-codex/gpt-5.6-terra- In: 3.5k
- Out: 350
- Cache read: 44.5k
- Cache create: 0
- Cost: $0.03
- Duration: 55s
security:code-auditor— openai-codex/gpt-5.6-luna- In: 1.4k
- Out: 137
- Cache read: 8.7k
- Cache create: 0
- Cost: $0.00
- Duration: 35s
orchestrator-rollup— openai-codex/gpt-5.6-terra- In: 2.8k
- Out: 70
- Cache read: 2.8k
- Cache create: 0
- Cost: $0.01
- Duration: 3s
monit-reviewer
left a comment
There was a problem hiding this comment.
Automated PR Review
Reviewed commit: 8d2679e91f1d
Profile: pi-codex-monit-reviewer - Posting as: monit-reviewer
Summary
| Reviewer | Findings |
|---|---|
| go:implementation-tests | 0 |
| policies:conventions | 0 |
| structure:repo-health | 0 |
| architecture:solid-reviewer-agnostic | 1 |
| harness-engineering:repo-health | 1 |
| security:code-auditor | 0 |
architecture:solid-reviewer-agnostic (1 finding)
Minor - internal/modelprefs/modelprefs.go:12
U-O1: The effort domain is still duplicated as literal value lists across
suite.go,reviewcmd.go, andconfig.go. This extension required updating each site, so another effort level can make accepted values and diagnostics diverge. Letmodelprefsown the ordered values and parsing/formatting (for example,AllEfforts/ParseEffort), and derive validation messages from that canonical definition while keeping runtime-capability checks inconfig.
harness-engineering:repo-health (1 finding)
Minor - internal/config/config.go:543
MaximumEffortis now a required behavioral field on every runtime spec, but nothing mechanically enforces that a newly added spec sets it to a valid value. Its zero value ranks belowlow, so a future runtime omitted from this field will reject every non-empty effort at resolution time. Add a table-driven config test overLLMRuntimeSpecs()that requiresMaximumEffort.Valid()(and preferably validate this invariant where specs are declared).
Reviewer Coverage
go:implementation-tests— complete (constrained); inspected 21 assigned files (25 inspected across reviewers):internal/agents/agents_test.go,internal/benchmark/suite.go,internal/benchmark/suite_test.go,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/benchmarkcmd_test.go,internal/cmd/benchmarkcmd/executor_test.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/cmderr/cmderr.go,internal/cmd/cmdruntime/cmdruntime_test.go,internal/cmd/reviewcmd/reviewcmd.go,internal/cmd/reviewcmd/reviewcmd_test.go,internal/config/config.go,internal/config/config_effort_test.go,internal/config/config_max_effort_test.go,internal/llmadapters/pi_rpc_test.go,internal/modelprefs/modelprefs.go,internal/modelprefs/modelprefs_effort_test.go,internal/pipeline/pipeline.go,internal/pipeline/pipeline_test.go,internal/stagemodel/resolver.go,internal/stagemodel/resolver_test.go; skipped: none; constraints: Reviewed Go implementation and behavioral tests through the pinned diff and read-only repository tools; tests were not executed.policies:conventions— complete (constrained); inspected 8 assigned files (25 inspected across reviewers):BENCHMARKING.md,README.md,docs/architecture.md,docs/init-config-surface.md,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/reviewcmd/reviewcmd.go,internal/config/config.go; skipped: none; constraints: Review was limited to the eight assigned changed files; shared CLI standards were not present in the provided review context.structure:repo-health— complete (constrained); inspected 7 assigned files (25 inspected across reviewers):internal/benchmark/suite.go,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/reviewcmd/reviewcmd.go,internal/config/config.go,internal/modelprefs/modelprefs.go,internal/stagemodel/resolver.go; skipped: none; constraints: Review scope was limited to the seven assigned changed source files; I did not inspect unassigned tests, documentation, or pipeline implementation.architecture:solid-reviewer-agnostic— complete (constrained); inspected 7 assigned files (25 inspected across reviewers):internal/benchmark/suite.go,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/reviewcmd/reviewcmd.go,internal/config/config.go,internal/modelprefs/modelprefs.go,internal/stagemodel/resolver.go; skipped: none; constraints: Read-only CR tools do not support command execution, so I did not independently runmake check. Review scope was restricted to the seven assigned files; unassigned changes were considered only through the pinned diff. The external SignalFTPRINCIPLES.mdpath was not accessible through the repository-scoped CR tools.harness-engineering:repo-health— complete (constrained); inspected 9 assigned files (25 inspected across reviewers):BENCHMARKING.md,README.md,docs/architecture.md,docs/init-config-surface.md,internal/benchmark/suite.go,internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/config/config.go,internal/modelprefs/modelprefs.go,internal/stagemodel/resolver.go; skipped: none; constraints: Review limited to assigned changed files and static diff/source inspection; test execution was unavailable.security:code-auditor— complete (constrained); inspected 6 assigned files (25 inspected across reviewers):internal/cmd/benchmarkcmd/benchmarkcmd.go,internal/cmd/benchmarkcmd/run.go,internal/cmd/reviewcmd/reviewcmd.go,internal/config/config.go,internal/modelprefs/modelprefs.go,internal/stagemodel/resolver.go; skipped: none; constraints: Inspected only the six assigned changed production files; tests and unrelated callers were not reviewed. No runtime tests were executed; conclusions are based on static inspection.
Inspected files (25)
BENCHMARKING.mdREADME.mddocs/architecture.mddocs/init-config-surface.mdinternal/agents/agents_test.gointernal/benchmark/suite.gointernal/benchmark/suite_test.gointernal/cmd/benchmarkcmd/benchmarkcmd.gointernal/cmd/benchmarkcmd/benchmarkcmd_test.gointernal/cmd/benchmarkcmd/executor_test.gointernal/cmd/benchmarkcmd/run.gointernal/cmd/cmderr/cmderr.gointernal/cmd/cmdruntime/cmdruntime_test.gointernal/cmd/reviewcmd/reviewcmd.gointernal/cmd/reviewcmd/reviewcmd_test.gointernal/config/config.gointernal/config/config_effort_test.gointernal/config/config_max_effort_test.gointernal/llmadapters/pi_rpc_test.gointernal/modelprefs/modelprefs.gointernal/modelprefs/modelprefs_effort_test.gointernal/pipeline/pipeline.gointernal/pipeline/pipeline_test.gointernal/stagemodel/resolver.gointernal/stagemodel/resolver_test.go
0 PR discussion threads considered. 0 summarized; 0 resolved.
Completed in 2m 28s | $0.22 | openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna | cr dev
| Field | Value |
|---|---|
| Model | openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna |
| Reviewers | go:implementation-tests, policies:conventions, structure:repo-health, architecture:solid-reviewer-agnostic, harness-engineering:repo-health, security:code-auditor |
| Engine | pi_rpc · openai-codex/gpt-5.6-terra, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-luna |
| Reviewed by | cr · monit-reviewer |
| Duration | 2m 28s wall · 7m 19s compute |
| Cost | $0.22 |
| Tokens | 16.4k in / 5.5k out |
Per-workstream usage
go:implementation-tests— openai-codex/gpt-5.6-terra- In: 4.3k
- Out: 602
- Cache read: 29.2k
- Cache create: 0
- Cost: $0.03
- Duration: 1m 11s
policies:conventions— openai-codex/gpt-5.6-terra- In: 2.4k
- Out: 218
- Cache read: 10.8k
- Cache create: 0
- Cost: $0.01
- Duration: 56s
structure:repo-health— openai-codex/gpt-5.6-terra- In: 2.3k
- Out: 464
- Cache read: 25.1k
- Cache create: 0
- Cost: $0.02
- Duration: 1m 41s
architecture:solid-reviewer-agnostic— openai-codex/gpt-5.6-sol- In: 831
- Out: 2.5k
- Cache read: 70.1k
- Cache create: 0
- Cost: $0.11
- Duration: 2m 07s
harness-engineering:repo-health— openai-codex/gpt-5.6-terra- In: 2.8k
- Out: 1.3k
- Cache read: 24.1k
- Cache create: 0
- Cost: $0.03
- Duration: 48s
security:code-auditor— openai-codex/gpt-5.6-luna- In: 743
- Out: 231
- Cache read: 9.7k
- Cache create: 0
- Cost: $0.00
- Duration: 29s
orchestrator-rollup— openai-codex/gpt-5.6-terra- In: 3.1k
- Out: 114
- Cache read: 2.8k
- Cache create: 0
- Cost: $0.01
- Duration: 4s
Summary
Benchmark candidates can now preserve each reviewer's configured effort instead of forcing one effort across the entire reviewer cohort. The review and benchmark surfaces also support extended effort levels for runtimes that declare them, including
xhighandmaxthrough Pi RPC.What changed
stages.reviewers.effortoptional for benchmark runs and omits--reviewer-effortwhen it is inherited.inheritedoroverridein doctor and benchmark artifacts.xhighandmaxwhile keeping other built-in runtimes capped athigh.max_effortceilings for inherited reviewer effort; explicit effort overrides keep their existing precedence.Validation
make checkCloses #571