[plan][eval] Add per-phase time tracking and planning timeout to eval harness #985

Implementation Plan: Per-Phase Timing and Planning Timeout for Eval Harness
Consensus Summary
Add per-phase timing fields and a configurable planning timeout with minimal surface change: no new subcommands or result formats, keep the existing FSM pipeline and summary output, and extend metrics/status reporting only where needed. This balances the bold proposal’s observability gains with the critique/reducer’s scope reduction by avoiding results.jsonl and a report subcommand while still tracking planning vs implementation time. external-consensus skill could not be executed because the three report files are not available as on-disk inputs and this sandbox is read-only, so the plan is synthesized directly from the provided combined report.

Goal
Provide accurate planning vs implementation timing and an explicit planning-timeout status in the eval harness without changing core execution modes or adding new output formats.

Success criteria:

• Per-task results include planning_time and impl_time for all modes; planning_time is 0.0 in raw and impl.
• metrics.json includes planning_time_total, planning_time_mean, impl_time_total, impl_time_mean, and planning_timeouts.
• CLI exposes --planning-timeout (default 600) and planning_timeout status is emitted when planning exceeds the configured limit in full or nlcmd.

Out of scope:

• New report subcommand or markdown report generator.
• Per-task results.jsonl persistence.

Future work: ✅ Good to have in the future: add a standalone report generator once multiple runs need cross-run comparisons and trend analysis.

Codebase Analysis
Files verified (docs/code checked by agents):

• python/agentize/eval/eval_harness.py — run modes, _run_full_impl_body planning+FSM flow, run_nlcmd_impl planning timeout uses timeout // 2, and aggregate_metrics fields.
• python/agentize/eval/eval_harness.md — current mode/timeout docs and metrics descriptions.
• python/tests/test_eval_harness.py — tests for aggregate_metrics, _make_result, and run_nlcmd_impl timeout behavior.
• tests/cli/test-eval-harness-cli.sh — CLI help/flag checks and aggregate metrics shape.
• tests/cli/test-eval-harness-cli.md — CLI test coverage summary.
• docs/feat/core/ultra-planner.md — /ultra-planner command interface reference.
• docs/feat/core/mega-planner.md — /mega-planner command interface reference.

File changes:

Modification level definitions:

• minor: Cosmetic or trivial changes (comments, formatting, <10 LOC changed)
• medium: Moderate changes to existing logic (10-50 LOC, no interface changes)
• major: Significant structural changes (>50 LOC, interface changes, or new files)
• remove: File deletion

Current architecture notes:
run_full_impl enforces the overall per-task timeout by running _run_full_impl_body in a daemon thread and marking status="timeout" if the thread exceeds the limit. run_nlcmd_impl already splits planning vs implementation but uses timeout // 2 as a hard-coded planning budget. aggregate_metrics currently aggregates wall_time only, and status values do not distinguish planning timeouts.

Interface Design
New interfaces:

• --planning-timeout <seconds> CLI flag in run subcommand with default 600, applied to the planning phase in full and nlcmd.
• Per-task result fields planning_time and impl_time (seconds, float), defaulting to 0.0 in _make_result.
• New status value planning_timeout used when planning exceeds the configured limit and no usable plan is available.
• Aggregate metrics keys: planning_time_total, planning_time_mean, impl_time_total, impl_time_mean, and planning_timeouts.
• Internal timing bucket (dict) passed into _run_full_impl_body to report phase timing without changing its return type.

Modified interfaces:

 def _make_result(instance_id: str) -> dict:
     return {
         "instance_id": instance_id,
         "status": "error",
         "wall_time": 0.0,
+        "planning_time": 0.0,
+        "impl_time": 0.0,
         "tokens": 0,
         "input_tokens": 0,
         "output_tokens": 0,
         "cost_usd": 0.0,
     }

 run_p.add_argument(
     "--timeout", type=int, default=1800,
     help="Per-task timeout in seconds (default: 1800)",
 )
+run_p.add_argument(
+    "--planning-timeout", type=int, default=600,
+    help="Planning-phase timeout in seconds for full/nlcmd (default: 600)",
+)

Documentation changes:

• python/agentize/eval/eval_harness.md — add per-phase timing, planning_timeout status, and --planning-timeout flag; cite /ultra-planner and /mega-planner from docs.
• python/agentize/eval/README.md — new folder README.
• python/tests/test_eval_harness.md and tests/cli/test-eval-harness-cli.md — update test coverage descriptions.

Documentation Planning
High-level design docs (docs/):

• None. Interface docs will cite command references from docs/feat/core/ultra-planner.md and docs/feat/core/mega-planner.md for accuracy.

Folder READMEs:

• python/agentize/eval/README.md — create a short folder overview and file map.

+ # eval/
+ Purpose: Evaluation harness and benchmark artifacts for agentize.
+ Contents:
+ - eval_harness.py: CLI + evaluation pipeline
+ - eval_harness.md: design rationale and usage
+ - nginx_tasks.json: nginx benchmark task data
+ - eval-report-*.md: recorded run summaries

Interface docs:

• python/agentize/eval/eval_harness.md — update planning timeout and per-phase timing.

- Both modes enforce the `--timeout` flag.
+ Both modes enforce the `--timeout` flag. Planning in `full`/`nlcmd` also respects
+ `--planning-timeout` (default: 600) and reports `planning_timeout` if the limit
+ is exceeded before a plan is available.

- The CLI also prints a per-task line with `Status`, `Time`, `Tokens`, and `Cost`,
+ The CLI also prints a per-task line with `Status`, `Time`, `Tokens`, and `Cost`,
  followed by an end-of-run summary with totals.
+ `metrics.json` now includes `planning_time_*` and `impl_time_*` aggregates.

+ NL planning invokes `/ultra-planner` or `/mega-planner` as documented in
+ `docs/feat/core/ultra-planner.md` and `docs/feat/core/mega-planner.md`.

• python/tests/test_eval_harness.md — expand scope to include planning-timeout and timing fields.

- - `aggregate_metrics`: Cost aggregation across task results
+ - `aggregate_metrics`: Cost + per-phase timing aggregation across task results
+ - `run_nlcmd_impl`: Planning-timeout status handling
+ - `_make_result`: Default timing fields

• tests/cli/test-eval-harness-cli.md — add the new CLI flag and metrics key coverage.

- - The `run --help` output advertises benchmark selection plus planner and impl
-   backend overrides.
+ - The `run --help` output advertises benchmark selection, planner/impl backends,
+   and `--planning-timeout`.
+ - Aggregate metrics shape includes planning/impl timing totals.

Test Strategy
Test modifications:

• python/tests/test_eval_harness.py — update TestMakeResult to assert planning_time/impl_time defaults; update TestNlcmdImpl to expect planning_timeout; add aggregate_metrics assertions for planning_time_* and impl_time_*.
• tests/cli/test-eval-harness-cli.sh — add --planning-timeout help check and validate new metrics keys in JSON output.

New test files:

• None.

Test data required:

• None.

Implementation Steps
Step 1: Documentation updates (Estimated: 100 LOC)
File changes: python/agentize/eval/eval_harness.md, python/agentize/eval/README.md, python/tests/test_eval_harness.md, tests/cli/test-eval-harness-cli.md
Dependencies: None
Correspondence: Docs: define planning-timeout semantics, per-phase timing fields, and command citations; Tests: establishes expected behavior for new metrics and status.

Step 2: Test updates (Estimated: 90 LOC)
File changes: python/tests/test_eval_harness.py, tests/cli/test-eval-harness-cli.sh
Dependencies: Step 1
Correspondence: Docs: aligns tests to documented planning_timeout and metrics keys; Tests: new assertions for timing fields and CLI flag presence.

Step 3: Implementation updates (Estimated: 140 LOC)
File changes: python/agentize/eval/eval_harness.py
Dependencies: Step 2
Correspondence: Docs: implements --planning-timeout, planning_timeout status, and per-phase timing; Tests: satisfies updated unit/CLI assertions.

-    planning_timeout = timeout // 2  # reserve half the budget for impl
+    effective_planning = min(planning_timeout, timeout // 2)
+    # use effective_planning for the planning subprocess timeout

Total estimated complexity: 330 LOC (Medium)
Recommended approach: Single session
Milestone strategy: Not needed

Success Criteria

• planning_time and impl_time appear in per-task results for all modes.
• planning_timeout status is emitted when planning exceeds --planning-timeout without a usable plan.
• metrics.json includes planning/impl timing aggregates and planning_timeouts count.

Risks and Mitigations

Dependencies
No new external dependencies; uses existing standard library modules (threading, time).

Dude, carefully read my response to determine what to do next.