feat(benchmark): support inherited and extended effort

BENCHMARKING.md

@@ -152,8 +152,12 @@ Candidate `profile` must reference a configured profile. Candidate PR hosts must
 match the candidate profile's Git host. For the current full-pipeline
 `validate`, `doctor`, and `run` commands, candidates must declare explicit
 selection `model` and `effort`. Reviewer candidates must declare reviewer
-`effort`, `agent_dirs`, and one reviewer model selector: either exact
+`agent_dirs` and one reviewer model selector: either exact
 `stages.reviewers.model` or floor-based `stages.reviewers.model_tier`.
+Reviewer `effort` is optional. When omitted, the benchmark does not pass a
+`--reviewer-effort` override, so each selected agent keeps the effort resolved
+from its catalog entry and the profile runtime's `max_effort` ceiling. When
+present, reviewer `effort` is an explicit override.
 Selector-only `benchmark select` still requires explicit
 `stages.selection.model` and `stages.selection.effort`, but it allows the
 reviewer stage to be omitted. `stages.selection.prompt` is optional, but when
@@ -255,7 +259,7 @@ When set on the candidate, `run` also passes:
 | `stages.selection.prompt` | `--selection-prompt <path>` |
 | `stages.reviewers.model` | `--reviewer-model <model>` |
 | `stages.reviewers.model_tier` | `--reviewer-model-tier <tier>` |
-| `stages.reviewers.effort` | `--reviewer-effort <effort>` |
+| `stages.reviewers.effort` | `--reviewer-effort <effort>` when present; omitted to inherit agent/profile resolution |
 | `stages.reviewers.agent_dirs[]` | `--agents-dir <path>` |
 | `max_agents` | `--max-agents <n>` |
 | `max_concurrency` | `--max-concurrency <n>` |
@@ -267,7 +271,15 @@ When set on the case, `run` also passes:
 | `review_base_sha` | `--review-base-sha <sha>` |
 | `review_head_sha` | `--review-head-sha <sha>` |
 
-Unset fields are omitted. Posting, retry, approval, thread-resolution, session,
+Unset fields are omitted. Benchmark candidate artifacts record reviewer
+`effort_source` as `inherited` or `override`, so reports can distinguish the
+two execution recipes even when the inherited effort is resolved later per
+agent. Effort values are `low`, `medium`, `high`, `xhigh`, or `max`; validation
+rejects levels unsupported by the candidate profile's runtime. Pi RPC supports
+the full range, while the other built-in runtimes currently support through
+`high`.
+
+Posting, retry, approval, thread-resolution, session,
 and live-review flags are never taken from the suite.
 
 `--cr-bin <path>` selects the binary used for child review runs. If omitted,

README.md

@@ -634,7 +634,7 @@ Supported values:
 | `llm.auth` | `subscription`, `api_key` |
 | `llm.adapter` | `claude_cli`, `anthropic_api`, `openai_api`, `pi_rpc`, and `codex_cli` are usable for review. `codex_cli` requires `provider: openai` and `auth: subscription`, and is currently best-effort/beta because Codex does not yet expose an explicit all-tools-disabled flag. |
 | `llm.model_map` keys | `small`, `medium`, `large` |
-| `llm_runtimes.<name>.max_effort` keys | `small`, `medium`, `large`; values `low`, `medium`, `high` |
+| `llm_runtimes.<name>.max_effort` keys | `small`, `medium`, `large`; values `low`, `medium`, `high`, plus `xhigh` and `max` for runtimes that support them (currently `pi_rpc`) |
 | `llm.reviewer_model_tier` | `small`, `medium`, `large` |
 | `review_policy.major_event` | `comment`, `request_changes` |
 | `review_policy.resolve_threads` | `auto`, `never` |
@@ -667,7 +667,8 @@ raise the reviewer baseline without editing shared agent catalogs.
 
 ### Model-Tier Floors and Effort Ceilings
 
-Agent catalogs declare an absolute `effort` (`low`, `medium`, `high`) that
+Agent catalogs declare an absolute `effort` (`low`, `medium`, `high`, `xhigh`,
+or `max`) that
 becomes the provider's reasoning-effort setting. For reviewer resolution,
 `agent.model_tier` and `llm.reviewer_model_tier` are minimum floors for model
 selection. The selected runtime's `max_effort`
@@ -716,18 +717,20 @@ The complete precedence and bypass table is:
 | `--reviewer-model-tier` | Raises the reviewer baseline before the agent floor is applied | Agent effort | Caps at the final resolved tier |
 | Other tier-resolved internal stage | That stage's own tier, then `model_map` | Stage effort | Caps the default at the stage's final tier |
 | `--selection-effort` or `--reviewer-effort` | Normal tier or exact-model selection | Requested effort | Explicit effort wins after the ceiling |
-| `--selection-model` or `--reviewer-model` | Exact requested model ID | Stage/agent effort or explicit effort | Bypassed; exact model overrides intentionally bypass tier resolution and the cap |
+| `--selection-model` | Exact requested model ID | Stage effort or explicit effort | Bypassed; selection exact-model overrides do not carry a reviewer tier |
+| `--reviewer-model` | Exact requested model ID | Agent effort or explicit effort | Inherited agent effort is capped at the effective reviewer tier; explicit effort bypasses the cap |
 | Agent `model_id` | Exact agent model ID | Agent effort | Bypassed; exact model selection intentionally bypasses tier resolution and the cap |
-| `cr benchmark run` stage model/effort overrides | Exact benchmark model when supplied; otherwise normal tier selection | Explicit benchmark effort when supplied | Explicit benchmark overrides bypass the profile ceiling |
+| `cr benchmark run` stage model/effort overrides | Exact benchmark model when supplied; otherwise normal tier selection | Explicit benchmark effort when supplied; inherited agent effort when reviewer effort is omitted | Explicit benchmark overrides bypass the profile ceiling; inherited reviewer effort retains it |
 
 For example, with `agent.model_tier: small`, `effort: high`,
 `llm.reviewer_model_tier: large`, and the selected runtime's
 `max_effort.large: medium`, the reviewer
 runs with the large model at medium effort. Adding `--reviewer-effort high`
-runs that same large model at high effort. Adding
-`--reviewer-model my-provider/model` selects that exact model and keeps high
-effort without applying the tier ceiling. `--selection-effort high` follows
-the same post-ceiling override rule for selection.
+runs that same large model at high effort. Adding only
+`--reviewer-model my-provider/model` selects that exact model while retaining
+the inherited medium ceiling; combining it with `--reviewer-effort high`
+selects the exact model at high effort. `--selection-effort high` follows the
+same post-ceiling override rule for selection.
 
 `cr init` preserves runtime `max_effort` but cannot yet edit it; set
 `llm_runtimes.<name>.max_effort` by hand in `config.yml`.
@@ -1209,11 +1212,11 @@ Review selection and execution flags:
 | `--max-agents <n>` | Set a hard total reviewer limit. Omit the flag or pass `0` to run all applicable repo-local reviewers and all matching `required_on_match` reviewers, plus up to 5 optional shared reviewers. A positive value below that combined required set fails. Negative values are rejected. |
 | `--max-concurrency <n>` | Limit concurrent reviewer agents. Omit the flag or pass `0` for the default limit of 5. Negative values are rejected. |
 | `--selection-model <model>` | Exact provider model ID passthrough for the selection stage only. Bypasses the default medium-tier selection model resolution. Requires `--dry-run` or `--no-post`. |
-| `--selection-effort <effort>` | Override selection-stage effort only with `low`, `medium`, or `high`. Requires `--dry-run` or `--no-post`. |
+| `--selection-effort <effort>` | Override selection-stage effort with `low`, `medium`, `high`, `xhigh`, or `max`, subject to runtime support. Requires `--dry-run` or `--no-post`. |
 | `--selection-prompt <path>` | Load selection-stage instruction text from a file while preserving the structured JSON selection protocol. Requires `--dry-run` or `--no-post`. |
 | `--reviewer-model <model>` | Exact provider model ID passthrough for reviewer stages only. Bypasses reviewer agent `model_tier`, `model_id`, and profile model-map resolution. Available for dry-run, no-post, and live reviews. |
 | `--reviewer-model-tier <tier>` | Override the reviewer baseline tier only with `small`, `medium`, or `large`. This still respects higher agent `model_tier` floors. Requires `--dry-run` or `--no-post`. |
-| `--reviewer-effort <effort>` | Override reviewer-stage effort only with `low`, `medium`, or `high`. Available for dry-run, no-post, and live reviews. |
+| `--reviewer-effort <effort>` | Override reviewer-stage effort with `low`, `medium`, `high`, `xhigh`, or `max`, subject to runtime support. Available for dry-run, no-post, and live reviews. |
 | `--review-base-sha <sha>` | Review this base commit SHA instead of the PR's current base SHA. Requires `--review-head-sha` and `--dry-run` or `--no-post`. |
 | `--review-head-sha <sha>` | Review this head commit SHA instead of the PR's current head SHA. Requires `--review-base-sha` and `--dry-run` or `--no-post`. |
 | `--session <name>` | Override the PR's default orchestrator session with a named live-review session. Reviewer cohorts remain PR-scoped. Not allowed with `--dry-run`, `--no-post`, or `--retry-posts`. |
@@ -1341,8 +1344,9 @@ and `stages.selection.effort` map to `--selection-model` and
 `--selection-effort`; optional `stages.selection.prompt` maps to
 `--selection-prompt`. `stages.reviewers.model` remains the exact-model bypass,
 while `stages.reviewers.model_tier` maps to `--reviewer-model-tier`.
-`stages.reviewers.effort` and `stages.reviewers.agent_dirs[]` map to
-`--reviewer-effort` and repeated `--agents-dir` flags. Case YAML fields
+When present, `stages.reviewers.effort` maps to `--reviewer-effort`; when
+omitted, reviewer effort is inherited from normal agent/profile resolution.
+`stages.reviewers.agent_dirs[]` maps to repeated `--agents-dir` flags. Case YAML fields
 `review_base_sha` and `review_head_sha` pin the exact base/head commit pair
 reviewed by the dry-run child command. Optional `stages.synthesis` metadata is
 reserved for future benchmark support and does not change `benchmark run`

docs/architecture.md

@@ -65,11 +65,13 @@ it ultimately resolves, while `--selection-effort` and `--reviewer-effort` win
 after the ceiling. Other tier-resolved internal stages use their own stage
 tier before applying `max_effort` at that final tier.
 
-An explicit `ModelOverride` returns with its requested effort or default effort
-while intentionally bypassing tier resolution and the `max_effort` cap.
-`--selection-model`, `--reviewer-model`, and agent `model_id` use this
-exact-model path. Benchmark stage model and effort overrides are explicit
-runtime inputs and retain the same ceiling bypass.
+An explicit `ModelOverride` bypasses model-map resolution. Explicit effort
+still bypasses `max_effort`; inherited reviewer effort is capped when the
+request carries an effective reviewer tier. `--reviewer-model` uses that tier,
+including benchmark reviewer model overrides. `--selection-model` and agent
+`model_id` do not carry one and remain uncapped. Runtime capability validation
+happens after stage resolution, before adapter startup, so unsupported extended
+effort levels fail without opening a model session.
 
 Reviewer `agent.model_id` is an exact provider-specific model override. It must
 still enter runtime execution through `stagemodel.ResolveStageModel` as a model

docs/init-config-surface.md

@@ -187,7 +187,8 @@ The canonical ceiling path is `llm_runtimes.<name>.max_effort`, and
 `profiles.<name>.llm_runtime` selects that runtime. The legacy
 `profiles.<name>.llm.max_effort` path is compatibility/projection only, not the
 canonical storage location. The map accepts `small`, `medium`, and `large`
-tier keys with `low`, `medium`, or `high` ceiling values. Interactive and
+tier keys with `low`, `medium`, or `high` ceiling values for all runtimes;
+`pi_rpc` also accepts `xhigh` and `max`. Interactive and
 non-interactive `cr init` must preserve an existing map, including when the
 profile or selected LLM runtime is staged and saved; init does not edit or
 remove it. Configure it by editing `config.yml` directly. Model-map JSON-row
@@ -196,9 +197,10 @@ parity and init editing for this field are out of scope.
 At review time, reviewer floors apply only to reviewer resolution. Other
 tier-resolved internal stages use their own stage tier, and default effort is
 capped only after that final tier is resolved. Explicit `--selection-effort`
-and `--reviewer-effort` values win after the cap. Exact
-`--selection-model`, `--reviewer-model`, agent `model_id`, and benchmark stage
-model/effort overrides intentionally bypass tier resolution and the cap.
+and `--reviewer-effort` values win after the cap. Exact model overrides bypass
+model-map resolution. `--selection-model` and agent `model_id` remain uncapped;
+`--reviewer-model` retains the effective reviewer-tier cap when effort is
+inherited, including benchmark reviewer model overrides.
 
 ## Scripted Install Ownership
 

internal/agents/agents_test.go

@@ -107,8 +107,8 @@ func TestLoadRejectsInvalidAgentModelMetadata(t *testing.T) {
 		},
 		{
 			name:  "invalid effort",
-			index: "name: reviewer\ndescription: desc\nmodel_tier: medium\neffort: xhigh\n",
-			want:  `effort "xhigh" is invalid`,
+			index: "name: reviewer\ndescription: desc\nmodel_tier: medium\neffort: ultra\n",
+			want:  `effort "ultra" is invalid`,
 		},
 		{
 			name:  "legacy model field",

internal/benchmark/suite.go

@@ -61,6 +61,12 @@ type CandidateStages struct {
 	synthesisSet bool
 }
 
+// ReviewersConfigured reports whether a reviewer-stage recipe was present or
+// was constructed programmatically with reviewer values.
+func (s CandidateStages) ReviewersConfigured() bool {
+	return s.reviewersSet || s.Reviewers.Model != "" || s.Reviewers.ModelTier != "" || s.Reviewers.Effort != "" || s.Reviewers.AgentDirs != nil
+}
+
 // SelectionStage configures the benchmark selection/orchestration phase.
 type SelectionStage struct {
 	Model  string `yaml:"model,omitempty" json:"model,omitempty"`
@@ -403,7 +409,8 @@ func validateCandidates(candidates []Candidate, cfg config.File) error {
 		if candidate.Profile == "" {
 			return fmt.Errorf("%w: candidate %q profile is required", ErrInvalid, candidate.ID)
 		}
-		if _, ok := cfg.Profiles[candidate.Profile]; !ok {
+		profile, ok := cfg.Profiles[candidate.Profile]
+		if !ok {
 			return fmt.Errorf("%w: candidate %q references unknown profile %q", ErrInvalid, candidate.ID, candidate.Profile)
 		}
 		if !candidate.stagesSet {
@@ -418,6 +425,9 @@ func validateCandidates(candidates []Candidate, cfg config.File) error {
 		if err := validateSynthesisStageStructural(candidate.ID, candidate.Stages); err != nil {
 			return err
 		}
+		if err := validateCandidateEfforts(candidate, profile); err != nil {
+			return err
+		}
 		if candidate.MaxAgents < 0 {
 			return fmt.Errorf("%w: candidate %q max_agents must be non-negative", ErrInvalid, candidate.ID)
 		}
@@ -428,6 +438,26 @@ func validateCandidates(candidates []Candidate, cfg config.File) error {
 	return nil
 }
 
+func validateCandidateEfforts(candidate Candidate, profile config.Profile) error {
+	stages := []struct {
+		name   string
+		effort string
+	}{
+		{name: "selection", effort: candidate.Stages.Selection.Effort},
+		{name: "reviewers", effort: candidate.Stages.Reviewers.Effort},
+		{name: "synthesis", effort: candidate.Stages.Synthesis.Effort},
+	}
+	for _, stage := range stages {
+		if strings.TrimSpace(stage.effort) == "" {
+			continue
+		}
+		if err := config.ValidateEffortForRuntime(profile.LLM, stage.effort); err != nil {
+			return fmt.Errorf("%w: candidate %q stages.%s.effort: %w", ErrInvalid, candidate.ID, stage.name, err)
+		}
+	}
+	return nil
+}
+
 func validateSelectionStage(candidateID string, stages CandidateStages) error {
 	if !stages.selectionSet {
 		return fmt.Errorf("%w: candidate %q stages.selection is required", ErrInvalid, candidateID)
@@ -442,7 +472,7 @@ func validateSelectionStage(candidateID string, stages CandidateStages) error {
 		return fmt.Errorf("%w: candidate %q stages.selection.prompt must be non-empty when present", ErrInvalid, candidateID)
 	}
 	if stages.Selection.Effort != "" && !modelprefs.Effort(stages.Selection.Effort).Valid() {
-		return fmt.Errorf("%w: candidate %q stages.selection.effort must be one of low, medium, high", ErrInvalid, candidateID)
+		return fmt.Errorf("%w: candidate %q stages.selection.effort must be one of low, medium, high, xhigh, max", ErrInvalid, candidateID)
 	}
 	return nil
 }
@@ -467,7 +497,7 @@ func validateReviewerStageStructural(candidateID string, stages CandidateStages)
 		return fmt.Errorf("%w: candidate %q stages.reviewers.effort must be non-empty when present", ErrInvalid, candidateID)
 	}
 	if stages.Reviewers.Effort != "" && !modelprefs.Effort(stages.Reviewers.Effort).Valid() {
-		return fmt.Errorf("%w: candidate %q stages.reviewers.effort must be one of low, medium, high", ErrInvalid, candidateID)
+		return fmt.Errorf("%w: candidate %q stages.reviewers.effort must be one of low, medium, high, xhigh, max", ErrInvalid, candidateID)
 	}
 	if !stages.Reviewers.agentDirsSet {
 		return nil
@@ -494,7 +524,7 @@ func validateSynthesisStageStructural(candidateID string, stages CandidateStages
 		return fmt.Errorf("%w: candidate %q stages.synthesis.prompt must be non-empty when present", ErrInvalid, candidateID)
 	}
 	if !modelprefs.Effort(stages.Synthesis.Effort).Valid() {
-		return fmt.Errorf("%w: candidate %q stages.synthesis.effort must be one of low, medium, high", ErrInvalid, candidateID)
+		return fmt.Errorf("%w: candidate %q stages.synthesis.effort must be one of low, medium, high, xhigh, max", ErrInvalid, candidateID)
 	}
 	return nil
 }
@@ -511,9 +541,6 @@ func validateRunCandidates(suite SuiteFile) error {
 		if candidate.Stages.Reviewers.Model == "" && candidate.Stages.Reviewers.ModelTier == "" {
 			return fmt.Errorf("%w: candidate %q stages.reviewers.model or stages.reviewers.model_tier is required for benchmark run", ErrInvalid, candidate.ID)
 		}
-		if candidate.Stages.Reviewers.Effort == "" {
-			return fmt.Errorf("%w: candidate %q stages.reviewers.effort is required for benchmark run", ErrInvalid, candidate.ID)
-		}
 		if !candidate.Stages.Reviewers.agentDirsSet {
 			return fmt.Errorf("%w: candidate %q stages.reviewers.agent_dirs is required for benchmark run", ErrInvalid, candidate.ID)
 		}