feat(reports): report agent wall and grading separately from end-to-end latency

[tmatup]

Short version

The run.md Latency column and the evalboard Duration column show a task's end-to-end time: setup, then the agent's turns, then grading. "Grading" means running the success checkers. For maestro-flow tasks a checker runs a live flow debug on the tenant, so 20-50 s of checker time per task looked like agent time.

task.json already measured setup_ms and grading_ms. run.json dropped them, so no report could split them out. This PR carries them into run.json, adds an agent wall number (the agent's own turns, summed), and shows all three in run.md, the HTML report, and the evalboard grid. Nothing about what is checked changes.

Before / after (real data)

Regenerated from adhoc-2026-10-01_19-24-36 (54 task.json files, flow-v2 and n8n arms) with coder-eval report <copy> --rebuild.

Before, run.md Task Details:

| Task ID | Status | Reliability Score | Latency | Model | Tags |
| n8n-vs-flow-calculator | SUCCESS | 1.000 | 101.3s | claude-sonnet-5-5 | n8n-vs-flow, runtime |

After:

| Task ID | Status | Reliability Score | Latency (end-to-end) | Agent Wall | Grading | Model | Tags |
| n8n-vs-flow-calculator | SUCCESS | 1.000 | 101.3s | 55.7s | 27.7s | claude-sonnet-5-5 | n8n-vs-flow, runtime |
| n8n-vs-flow-feet-inches | SUCCESS | 1.000 | 160.8s | 95.1s | 48.1s | claude-sonnet-5-5 | n8n-vs-flow, runtime |
| n8n-vs-flow-customer-escalation | SUCCESS | 1.000 | 234.6s | 215.8s | 5.5s | claude-sonnet-5-5 | n8n-vs-flow, structural |

The calculator row is the issue's example: flow-v2 run 00 is 101.3 s end-to-end, 55.7 s of agent turns, 13.7 s setup, and 27.7 s grading.

New Summary lines:

- **Avg End-to-end Latency**: 122.2s
- **Avg Agent Wall**: 88.2s
- **Avg Grading**: 23.4s

Per-arm totals over 27 rows each, from the regenerated run.json:

Arm	End-to-end	Agent wall	Grading	Grading per task
flow-v2	4,430 s	2,964 s	1,109 s	41.1 s
n8n	2,171 s	1,800 s	155 s	5.8 s

What changed

• run.json rows (run_record.py): three new keys. Existing keys are unchanged, so current readers keep working.
  • agent_wall_ms: the sum of the positive iterations[].duration_seconds, in ms. It is computed once, in result_metrics.agent_wall_ms. It is None when no turn was timed, never 0.
  • setup_ms and grading_ms: copied from task.json.
• Why a sum and not duration - setup - grading: a detached grade does not carry grading_ms, and coder-eval execute never sets it. On those rows the subtraction has no value, but the sum still does. The evalboard already used the sum (agentSecondsFromRaw), so the two surfaces now use the same rule.
• run.md (reports/markdown.py): Latency is now labeled end-to-end. Task Details gets Agent Wall and Grading columns. Generation Metrics gets Agent Wall. The Summary's Avg Generation Latency was an average of duration, so it is renamed to Avg End-to-end Latency, and two new lines follow it: Avg Agent Wall and Avg Grading.
  • A run.json written before this change has none of the new keys. For those rows, agent wall is derived from the row's iterations and grading shows N/A.
• HTML report (reports/html.py): the task card gets Agent Wall, Setup, and Grading stats next to Total Latency (end-to-end). The variant card gets Total Agent Wall and Total Grading.
• Evalboard (evalboard/): in the run grid, Duration becomes End-to-end, and two new sortable columns are added: Agent and Grading. Each has a header tooltip, and the mobile card shows them too. agentSecondsFromRaw now uses the stored agent_wall_ms when it exists and falls back to the iterations sum for older runs. vs Expected still compares end-to-end time, because the runner derives expected_seconds from end-to-end history.
• Docs: docs/REPORT_SCHEMA.md lists the new keys and gives the calculator example.

Verification

• uv run pytest tests/test_run_record.py tests/test_reports.py tests/test_reports_html.py: passes. New tests cover the row keys, None when no turn was timed, skipping turns with no recorded duration, the new run.md columns, and the fallback for older rows. Report snapshots were regenerated.
• make check, make lint (735 passed) and make docs-budget pass. make test: 6,035 passed. The 10 failures are all in test_judge_litellm.py, because the optional litellm extra is not installed on my machine. None of them touch this change.
• evalboard: tsc --noEmit and next build pass. vitest passes 818 of 821 tests. The other 3 hit the 5 s timeout because the dev box was overloaded (load average 36 on 16 cores). A different set timed out on each run, and all 3 pass when run alone.

Fixes #212

🤖 Generated with Claude Code
Co-Authored-By: Claude

🤖 Generated with Claude Code

https://claude.ai/code/session_016sCMS4Kv57eDUxpmW24z9e

[github-actions]

Claude finished @tmatup's task in 48s —— View job

---

I'll analyze this and get back to you.

[uipreliga]

Review: coder_eval — pr:213 (13 files) axis:1,2,3,4,5,6,7,8

Scope: pr:213 (13 files) axis:1,2,3,4,5,6,7,8 · branch fix/212-agent-wall · f23e0f0 · 2026-10-02T20:49Z · workflow variant

Change class: complex — adds three keys to the cross-repo run.json row contract and a new agent-wall metric that changes how latency is computed and shown in run.md, HTML and the evalboard

The codebase is healthy (9.6/10, no critical or high findings, security 10/10), and no finding changes a task's score or final_status for identical agent output; the real risks are in the new timing split, where a wrong grading_ms contract for detached grades, an A/B report that still compares only end-to-end duration, and a rule copied in three places with drift can make readers draw wrong conclusions about agent speed, so fix the docs and unify the rule before downstream consumers depend on these fields.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	9.8 / 10	0	0	0	2	New totals bypass the existing sum-measured helpers (HTML _sum_measured, evalboard sumMeasured)
2. Type Safety	9.9 / 10	0	0	0	1	New TaskResultSummary.gradingSeconds is both optional and nullable, which weakens a field that the only producer always sets
3. Test Health	9.3 / 10	0	0	1	2	New HTML stat cards (task Agent Wall/Setup/Grading; variant Total Agent Wall/Total Grading) have no value assertion
4. Security	10 / 10	0	0	0	0	—
5. Architecture & Design	9.4 / 10	0	0	1	1	Agent-wall derivation rule implemented in three places (two Python, one TS); the Python copies already differ on non-finite input
6. Error Handling & Resilience	9.9 / 10	0	0	0	1	run.md regeneration takes stored agent_wall_ms / grading_ms as untyped Any; only the legacy branch checks the type
7. API Surface & Maintainability	9.7 / 10	0	0	0	3	HTML Generation Metrics shows end-to-end in 'Xm Ys' and Agent Wall/Setup/Grading in raw seconds, side by side
8. Evaluation Harness Quality	8.8 / 10	0	0	2	2	REPORT_SCHEMA.md says grading_ms is None on a detached grade, but a re-grade records its own grading_ms

Overall Score: 9.6 / 10 · Weakest Axis: Evaluation Harness Quality at 8.8 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 4 · 🔵 12 across 8 axes.

Blockers

None.

Non-blocking, but please consider before merge

1. [Axis 3] New HTML stat cards (task Agent Wall/Setup/Grading; variant Total Agent Wall/Total Grading) have no value assertion (src/coder_eval/reports/html.py:985) — The PR adds new report rows to both HTML renderers, but no test asserts on them. grep -rn "Agent Wall\|Total Grading\|setup_ms" tests/test_reports_html.py returns nothing. The only nearby test is TestGenerationMetricsBuckets.test_the_existing_four_stats_are_unchanged (tests/test_reports_html.py:1618), and it checks only that 'Total Latency', 'Turns', 'Assistant Turns' and 'Avg Turn Latency' are non-empty. Its _stat regex {label}[^<]* also matches the relabelled 'Total Latency (end-to-end)'. So these new lines are not checked: html.py:985-987 agent_wall = format_ms(agent_wall_ms(result)) / setup = format_ms(result.setup_ms) / grading = format_ms(result.grading_ms), rendered at :996-999, and the variant-card aggregation at :1239-1242 walls = [ms for r in eval_results if (ms := agent_wall_ms(r)) is not None] / gradings = [r.grading_ms for r in eval_results if r.grading_ms is not None] / total_grading_fmt = _esc(format_ms(sum(gradings) if gradings else None)). A swap of setup and grading, a dropped is not None filter (sum over None raises TypeError), or a 0 where a dash belongs (CE058) would ship green. Add tests that reuse TestGenerationMetricsBuckets._stat. (1) generate_task_html with setup_ms=13677.6, grading_ms=27656.9 and one timed turn: assert the 'Agent Wall', 'Setup' and 'Grading' values. (2) The same with setup_ms/grading_ms=None: assert the dash, not '0ms'. (3) generate_variant_html with results that mix measured and None grading_ms: assert 'Total Grading' sums only the measured rows and shows the dash when none are measured.
2. [Axis 5] Agent-wall derivation rule implemented in three places (two Python, one TS); the Python copies already differ on non-finite input (src/coder_eval/reports/markdown.py:280) — The PR adds result_metrics.agent_wall_ms as the single writer, but it also re-implements the same summation in the markdown renderer's legacy path. markdown.py:279-282 reads seconds = [ d for t in row.get("iterations") or [] if isinstance(d := t.get("duration_seconds"), (int, float)) and d > 0 ] / return sum(seconds) * 1000.0 if seconds else None. result_metrics.py:112-116 reads if isinstance(t.duration_seconds, (int, float)) and math.isfinite(t.duration_seconds) and t.duration_seconds > 0. The two predicates have already drifted, because only the result_metrics copy has the math.isfinite guard. Python's json.loads accepts NaN/Infinity, so a legacy row with a non-finite turn duration renders differently from a fresh row. A third copy is evalboard/lib/runs.ts:1287-1290 (.filter((d): d is number => typeof d === "number" && d > 0)). The docstring at result_metrics.py and the TS comment both claim the copies agree 'by construction', but nothing enforces that. .claude/notes/reporting.md § 'Read the stored value, do not re-derive it' says: 'The span SELECTION is the shared rule, not a copy of it'. The existing _turn_tool_union_ms precedent keeps the legacy fallback but makes it call the shared selection. Fix: extract a field-level helper in result_metrics, for example sum_positive_turn_ms(durations: Iterable[object]) -> float | None (finite and > 0, None when empty). Have agent_wall_ms and _row_agent_wall_ms both call it. Add a cross-language fixture test that pins the TS agentSecondsFromRaw legacy path to the same cases (including 0.0, negative and missing values). n/a
3. [Axis 8] REPORT_SCHEMA.md says grading_ms is None on a detached grade, but a re-grade records its own grading_ms (docs/REPORT_SCHEMA.md:105) — The doc states | grading_ms|float | None| Copied fromtask.json. None on an ungraded row (coder-eval execute) and on a detached grade. | (line 105). Line 97-98 also says that duration "includes setup_ms ... and grading_ms (every success check)". Both statements are false for a detached grade (evaluate <run_dir>, or run --resume over an executed run). Orchestrator._finalize_result (orchestrator.py:998-999) sets self.result.grading_ms = self.success_checker.grading_ms on every graded pass, and that includes the re-grade. _finalize_regrade_timing (orchestrator.py:956-957) then puts back the PRIOR run's duration_seconds. tests/test_seed_from_prior_result.py:91-98 pins this on purpose: "The cost of THIS pass's grading ... Its sibling duration_seconds is restored from the prior instead". Failure scenario: run coder-eval execute, then evaluate <run_dir>. Each row gets duration = the execute run's end-to-end time, which contains no grading, and a non-None grading_ms from the separate grading pass. run.md, the HTML card and the evalboard then show End-to-end, Agent Wall and Grading side by side, and the documented identity (duration ≈ setup + agent + grading) breaks, because Grading is counted outside the duration. A cross-repo consumer (the eval-runner) that trusts REPORT_SCHEMA.md and computes duration - agent_wall_ms - setup_ms - grading_ms gets a negative residual. The same wrong claim also appears at evalboard/lib/runs.ts:515 ("grading_ms is null on an ungraded or detached grade"), at src/coder_eval/result_metrics.py:104-105 ("A detached grade does not carry grading_ms"), and in the evalboard duration tooltip in task-grid.tsx. Fix: correct the doc to say that a detached grade records its own pass's grading, which is NOT inside duration. Optionally add a flag or a note on re-graded rows (e.g. key off environment_info.grading_duration_seconds) so that readers do not sum the two.
4. [Axis 8] The agent-wall / grading split reaches the HTML variant card but not the A/B experiment report, whose Welch t-test still compares only end-to-end duration (src/coder_eval/reports/html.py:1251) — This PR adds a per-variant <div class="label">Total Agent Wall</div> and Total Grading (html.py:1251-1252, from walls = [ms for r in eval_results if (ms := agent_wall_ms(r)) is not None] at line 1239). The parallel A/B renderer, reports/experiment.py, is unchanged. Its Aggregate Metrics row | Avg Duration (s) runs welch_t_test(series[vid_a].durations, series[vid_b].durations) (experiment.py:208-212), and the per-task | Task | Score | Status | Avg Duration | table (experiment.py:487) also reads only the end-to-end VariantResult.duration_seconds. This is the cross-variant comparison that #212 is about. If one variant produces output that makes a live checker or agent_judge run longer, the variant gets a statistically 'slower' verdict, but the cause is grading time and not agent time. The renderers now disagree on which time they report per variant (Technique 2). Fix: carry agent wall (and grading) on VariantResult / collect_variant_series, and add an Avg Agent Wall (s) row with its own p-value to experiment.md. Or record in a note why the A/B report deliberately compares end-to-end only.

Nits

1. [Axis 1] New totals bypass the existing sum-measured helpers (HTML _sum_measured, evalboard sumMeasured) (src/coder_eval/reports/html.py:1241) — Lines 1239-1242 write the 'sum what was measured, else None' idiom out twice: walls = [ms for r in eval_results if (ms := agent_wall_ms(r)) is not None] / total_agent_fmt = _esc(format_ms(sum(walls) if walls else None)) and the same for gradings. result_metrics.py already has _sum_measured(values: Iterable[float | None]) -> float | None with this exact contract, and it also drops non-finite values. The markdown summary (markdown.py:469-474) builds a similar filtered list again for its averages. Make _sum_measured public (it is the Python twin of the evalboard's sumMeasured) and call it: format_ms(sum_measured(agent_wall_ms(r) for r in eval_results)).
2. [Axis 1] New tests repeat the 12-line RunSummary boilerplate verbatim instead of using a builder (tests/test_reports.py:1550) — test_task_details_split_end_to_end_into_agent_wall_and_grading (line 1550, summary = RunSummary() and test_task_details_legacy_row_derives_agent_wall_and_never_fakes_grading (line 1583, summary = RunSummary() each build a one-row RunSummary field by field. They differ only in the task row and the timestamps. The file already has a single-row builder pattern (_summary_with_notes, line 1246) and 30 RunSummary( constructions. Add a small _one_row_summary(row) helper, or generalize the existing one, and use it in both tests.
3. [Axis 2] New TaskResultSummary.gradingSeconds is both optional and nullable, which weakens a field that the only producer always sets (evalboard/lib/runs.ts:110) — gradingSeconds?: number | null; (line 110) is always set by toTaskRow: gradingSeconds: typeof t.grading_ms === "number" ? t.grading_ms / 1000 : null. The ? therefore only gives two separate absence states (undefined and null), and every consumer must collapse them: fmtTableDuration(t.gradingSeconds ?? null) in task-grid.tsx lines 829 and 1011. It also lets a new TaskResultSummary constructor leave out the field without a compile error. Fix: declare it gradingSeconds: number | null and update the test factories. The same applies to the existing agentSeconds? sibling.
4. [Axis 3] Agent-wall rule (markdown stored-None/mixed-row path and result_metrics.agent_wall_ms non-finite filter) is not unit-tested (src/coder_eval/reports/markdown.py:277) — The docstring states a contract: "Presence is tested with in, not with truthiness, so a stored None (no turn timed) stays None". The code is if "agent_wall_ms" in row: return row["agent_wall_ms"]. The two new tests in tests/test_reports.py cover only a row with a stored number and a legacy row with no key. No test covers a row with agent_wall_ms: None plus timed iterations. The evalboard twin has this test ('a stored null stays null rather than falling back' in evalboard/lib/tests/runs.test.ts), so the two readers are tested unevenly. The Summary averages at :469-474 say they average 'each over only the rows that measured it'. They are tested only with all rows measured or no rows measured, never mixed, so a regression that adds a None row to the denominator would not be caught. Add a markdown test with two rows: one with a stored value and grading_ms, one with agent_wall_ms=None, grading_ms=None and timed iterations. Assert that the second row renders 'N/A' in the Agent Wall and Grading columns, and that 'Avg Agent Wall' / 'Avg Grading' equal the first row's values.
5. [Axis 3] New Agent/Grading sort branches and Agent dash rendering in TaskGrid untested (evalboard/app/runs/[id]/task-grid.tsx:357) — The PR adds two sortable columns. Their comparator branches are case "agent": return (a.agentSeconds ?? -Infinity) - (b.agentSeconds ?? -Infinity); and case "grading": return (a.gradingSeconds ?? -Infinity) - (b.gradingSeconds ?? -Infinity); (task-grid.tsx:357-365), with new DEFAULT_DIR entries agent: "desc" / grading: "desc". The new tests in task-grid.test.tsx check the header order, the tooltips, the cell values and the Grading dash. No test clicks the 'Agent' or 'Grading' header to check the order or where null rows sort. No test checks that a missing agentSeconds renders '—'. Add a test in the style of the existing 'End-to-end' sort test (task-grid.test.tsx:200): three rows with agentSeconds 10 / 50 / undefined. Click /^Agent$/ and assert the 50 row is first and the undefined row is last. Do the same for Grading.
6. [Axis 5] run.md adds its own formatter _fmt_seconds_ms ('N/A') instead of using the shared format_ms ('—') (src/coder_eval/reports/markdown.py:285) — markdown.py:285-287 adds def _fmt_seconds_ms(ms: float | None) -> str: ... return f"{ms / 1000.0:.1f}s" if ms is not None else "N/A". html.py renders the same agent-wall / grading values with the shared format_ms(agent_wall_ms(result)), which gives an em dash for an unmeasured value and 55.66s/850ms for a measured one. In the markdown Generation Metrics table (markdown.py:394-400), the new Agent Wall column shows N/A while the bucket columns next to it use format_ms and show a dash. The format_ms docstring names this drift as the reason for the shared formatter: 'formatting them twice is how one surface comes to print 0ms where the other prints a dash'. Fix: use format_ms for the Agent Wall and Grading cells (or one shared seconds-formatter in durations.py if the .1f s shape is wanted), so that HTML and markdown agree on the unmeasured sentinel. n/a
7. [Axis 6] run.md regeneration takes stored agent_wall_ms / grading_ms as untyped Any; only the legacy branch checks the type (src/coder_eval/reports/markdown.py:278) — _row_agent_wall_ms returns the stored value unchecked (line 278: return row["agent_wall_ms"]). Its legacy branch on line 280 does type-check: isinstance(d := t.get("duration_seconds"), (int, float)) and d > 0. The value then goes, also unchecked, into _fmt_seconds_ms (line 286: f"{ms / 1000.0:.1f}s") and into the sum(agent_walls) / sum(gradings) averages (lines 469-473). grading_ms gets the same treatment on lines 472 and 533. The report/show fallback (RunSummary.model_validate_json → generate_markdown, line ~842) reads task_results as untyped dict[str, Any] from disk. So if a run.json row has a non-numeric value under either key (hand-edited, or written by an external producer), the whole run.md render fails with a TypeError instead of showing N/A for that cell. The PR's own writer only emits float | None, so in practice this is unlikely. That is why this is Low, and the neighbouring t["duration"] accesses already have the same exposure. Fix: apply one numeric check to both branches, e.g. a _measured_ms(v) -> float | None that returns v only when isinstance(v, (int, float)) and math.isfinite(v) (this matches result_metrics._sum_measured), and use it for both agent_wall_ms and grading_ms.
8. [Axis 7] HTML Generation Metrics shows end-to-end in 'Xm Ys' and Agent Wall/Setup/Grading in raw seconds, side by side (src/coder_eval/reports/html.py:1241) — The task card uses total_latency = _esc(_format_duration(result.duration_seconds)) (line 956, minutes form above 60 s) next to agent_wall = format_ms(agent_wall_ms(result)) (line 985, ms/1000:.2f s form). The variant card has the same split: total_latency_fmt = _esc(_format_duration(total_duration)) (line 1235) next to total_agent_fmt = _esc(format_ms(sum(walls) if walls else None)) (line 1241). On a variant total this renders "45m 12s" next to "1834.20s", and the reader must convert units to compare the parts of one number. Fix: format all four values in a card with the same formatter.
9. [Axis 7] The same field has a different name on each surface: 'Agent Wall' in run.md/HTML, 'Agent' in the evalboard (evalboard/app/runs/[id]/task-grid.tsx:410) — { key: "agent", header: "Agent", align: "right" } (line 410) and label="Agent" (line 1006) label the value that run.md (markdown.py line 371, "Agent Wall") and the HTML report (html.py line 996, "Agent Wall") call Agent Wall, and that run.json calls agent_wall_ms. A bare "Agent" column next to "Variant" and "Model" reads like an agent identity, not a duration. Use "Agent Wall" (or "Agent time") on all three surfaces.
10. [Axis 7] Issue-number history comments and rationale prose in comments/docstrings across the diff (src/coder_eval/reports/markdown.py:507) — Near-identical comments appear at markdown.py:507-509 ("# Latency is END-TO-END: setup + the agent's turns + grading. Agent Wall and Grading split it, so a checker that runs a live command is not read as agent time (#212)."), markdown.py:465-467, html.py:982-984 and run_record.py:138-140. CLAUDE.md says "Comments are a last resort" and "what it used to be belongs in git". The column labels and the REPORT_SCHEMA section already say this. run_record.py:140 also cites CE049, a score rule, for a timing value; the timing rule is CE058. Fix: delete the restated comments. Keep at most one Rationale: pointer to a notes section.
11. [Axis 8] Adjacent run.md summary averages use three different row sets, so Avg Agent Wall can exceed Avg End-to-end Latency (src/coder_eval/reports/markdown.py:469) — durations = [t["duration"] for t in summary.task_results if t["duration"] > 0] (line 457) feeds Avg End-to-end Latency (line 468). agent_walls = [ms for t in summary.task_results if (ms := _row_agent_wall_ms(t)) is not None] (line 469) and gradings = [... t.get("grading_ms") ...] (line 472) each average over their own subset. A run that has errored or setup-failed rows (duration > 0, no timed turn, grading None) puts those rows into the end-to-end average only. That pulls the end-to-end average down and leaves the agent average unchanged, so the summary can show Avg Agent Wall > Avg End-to-end, which cannot be true. The three adjacent lines invite the reader to add them up. Fix: either average all three over the same row set (rows that measured all of them), or show the row count next to each average, e.g. Avg Agent Wall: 55.7s (n=8/10).
12. [Axis 8] agent_wall_ms counts the user simulator's generation time on a crashed dialog row as agent time (src/coder_eval/result_metrics.py:115) — agent_wall_ms sums every positive t.duration_seconds for t in result.iterations (lines 112-116), and the docstring calls this "the agent's own turns". In dialog mode, when the agent's turn fails after the simulator has produced a user message, orchestrator.py:2659-2667 appends a standalone TurnRecord(..., messages=[pending_user_turn], duration_seconds=sim_ms / 1000.0 ...). This record holds only the SIMULATOR's generation time. The new metric then reports that time as Agent Wall, in run.json, run.md, the HTML report and the evalboard. Fix: skip iterations that have no agent-side messages (e.g. records whose only message is a UserMessage), or document the exception. The same rule also lives in markdown.py _row_agent_wall_ms and evalboard agentSecondsFromRaw, so keep all three the same.

What's Missing

Parallel paths:

• 🟡 The agent-wall/grading split reaches run.md, html.py and the evalboard grid, but not the A/B experiment report. reports/experiment.py still prints an unlabelled 'Avg Duration (s)' with a Welch p-value, a per-task 'Avg Duration' column and 'Total Duration'. All three are end-to-end, and reports/helpers.py VariantSeries/collect_variant_series carries only end-to-end durations. This is the two-arm comparison that issue #212 is about. (trigger: src/coder_eval/reports/html.py) (restates: Axis 8: The agent-wall / grading split reaches the HTML variant card but not the A/B experiment report)
• 🟡 The agent-wall selection rule (sum of finite, positive turn durations) has three hand-copied implementations: result_metrics.agent_wall_ms, markdown._row_agent_wall_ms and runs.ts agentSecondsFromRaw. Only the first has math.isfinite. No shared helper and no cross-language fixture keeps them aligned, but the docstring and the TS comment both say the copies agree 'by construction'. (trigger: src/coder_eval/reports/markdown.py) (restates: Axis 5: Agent-wall derivation rule implemented in three places (two Python, one TS))
• 🔵 The HTML variant card got 'Total Grading' next to 'Total Agent Wall'. The evalboard run-level metrics (evalboard/app/runs/[id]/run-view.tsx, which already sums agentSeconds with p50/p90) got no grading total or percentile. So the per-run 'how much went to grading' number exists in the static HTML twin but not on the board. (trigger: evalboard/lib/runs.ts)
• 🔵 The dialog-mode crashed-turn path (orchestrator.py:2659-2667) appends a TurnRecord that holds only the simulator's generation time. All three agent-wall copies count it as agent time. None of them excludes simulator-only records, so a fix must be applied to all three at once. (trigger: src/coder_eval/result_metrics.py) (restates: Axis 8: agent_wall_ms counts the user simulator's generation time on a crashed dialog row as agent time)

Tests:

• 🟡 No test checks the new HTML stat values: the task card's Agent Wall/Setup/Grading and the variant card's Total Agent Wall/Total Grading. That includes the dash-not-0ms case and mixed measured/None sums. The new tests and snapshot fixtures cover only markdown, so a setup/grading swap in html.py would still pass. (trigger: src/coder_eval/reports/html.py) (restates: Axis 3: New HTML stat cards (task Agent Wall/Setup/Grading; variant Total Agent Wall/Total Grading) have no value assertion)
• 🔵 Three markdown cases have no test: a row with a stored agent_wall_ms=None plus timed iterations (it must stay N/A and not fall back), a row with a non-finite turn duration, and a mixed row set for the 'Avg Agent Wall'/'Avg Grading' denominators. The evalboard twin does test the stored-null case. (trigger: src/coder_eval/reports/markdown.py) (restates: Axis 3: Agent-wall rule (markdown stored-None/mixed-row path and result_metrics.agent_wall_ms non-finite filter) is not unit-tested)
• 🔵 No test covers the new 'agent' and 'grading' sort comparators in TaskGrid: the click-to-sort order and null rows sorting last via -Infinity. No test covers the '—' rendering for a missing agentSeconds. (trigger: evalboard/app/runs/[id]/task-grid.tsx) (restates: Axis 3: New Agent/Grading sort branches and Agent dash rendering in TaskGrid untested)
• 🔵 No test pins the re-grade path: execute, then evaluate <run_dir> or run --resume. That path writes a non-None grading_ms that is NOT inside the restored duration, which contradicts the new REPORT_SCHEMA identity (duration = setup + agent wall + grading). A run_record/report test over a seeded prior result would have exposed the wrong doc claim. (trigger: docs/REPORT_SCHEMA.md) (restates: Axis 8: REPORT_SCHEMA.md says grading_ms is None on a detached grade, but a re-grade records its own grading_ms)

Downstream consumers:

• 🟡 Four places restate the claim that grading_ms is None on a detached grade and that duration includes grading: REPORT_SCHEMA.md:97-105, runs.ts RawTaskResult comment (~line 515), the result_metrics.agent_wall_ms docstring, and the task-grid 'End-to-end' tooltip. On a re-graded row both claims are false, so a consumer that computes duration - agent_wall - setup - grading gets a negative residual. All four must be corrected together. (trigger: docs/REPORT_SCHEMA.md) (restates: Axis 8: REPORT_SCHEMA.md says grading_ms is None on a detached grade, but a re-grade records its own grading_ms)
• 🔵 The renamed run.md Summary now shows Avg End-to-end Latency, Avg Agent Wall and Avg Grading next to each other, but each averages over a different row set. Errored or setup-failed rows count only in the end-to-end average, so Avg Agent Wall can exceed Avg End-to-end. (trigger: src/coder_eval/reports/markdown.py) (restates: Axis 8: Adjacent run.md summary averages use three different row sets, so Avg Agent Wall can exceed Avg End-to-end Latency)
• 🔵 The evalboard 'vs Expected' column and the runner's expected_seconds both still use end-to-end history. A grading-heavy task (live flow debug checker) is therefore still flagged as slow against its expectation, which is the misreading #212 targets. The PR body states this choice, but neither REPORT_SCHEMA.md nor .claude/notes/reporting.md records it, so the next reader will 'fix' one side only. (trigger: evalboard/app/runs/[id]/task-grid.tsx)

Display & mapping dicts:

• 🔵 The same metric has three labels: 'Agent Wall' in run.md/HTML, 'Agent' in the evalboard grid header and mobile card, and agent_wall_ms in run.json. Unmeasured values also show two sentinels: run.md _fmt_seconds_ms prints 'N/A', while format_ms in HTML and the bucket columns next to it print '—'. Labels and formatters should agree across the three surfaces. (trigger: evalboard/app/runs/[id]/task-grid.tsx) (restates: Axis 7: The same field has a different name on each surface: 'Agent Wall' in run.md/HTML, 'Agent' in the evalboard)
• 🔵 setup_ms is added to run.json and declared on evalboard RawTaskResult, but toTaskRow never maps it, so the board has no Setup value. The HTML task card shows Setup next to Agent Wall/Grading. The board grid therefore cannot show where the rest of End-to-end goes (End-to-end minus Agent minus Grading leaves setup plus unnamed time unexplained), and the field is carried but never read. (trigger: evalboard/lib/runs.ts)

Daily/nightly:

• 🔵 The PR says 'existing keys are unchanged' but does not state the impact on the nightly pipeline or the cross-repo eval-runner / coder_eval_uipath consumers. Three changes need a stated impact: the run.md labels changed ('Latency' -> 'Latency (end-to-end)', 'Avg Generation Latency' -> 'Avg End-to-end Latency'), the evalboard 'Duration' column is now 'End-to-end', and three run.json row keys are new. Re-graded nightly rows (execute + evaluate) will also carry grading_ms outside duration. The PR should say whether any dashboard (e.g. infra/dashboards) or scraper reads these labels or would sum these keys. (trigger: src/coder_eval/run_record.py)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] CE067 no-inline-measured-sum: make result_metrics._sum_measured public as sum_measured, then forbid, in src/coder_eval outside result_metrics.py, any IfExp whose test is a bare Name n, whose body contains a sum(n) call (also inside a BinOp, e.g. sum(n) * 1000.0), and whose orelse is Constant(None). New file tests/lint/rules/ce067_no_inline_measured_sum.py, wired in tests/lint/runner.py. The docstring names the TS twin sumMeasured as a blind spot (evalboard/lib/runs.ts:909 is not reached by a Python AST rule). Prevents: Finding 'New totals bypass the existing sum-measured helpers': html.py:1241-1242 (sum(walls) if walls else None, sum(gradings) if gradings else None) and the markdown averages at markdown.py:469-474. It also catches the legacy copy at markdown.py:282 (sum(seconds) * 1000.0 if seconds else None), which is the core of the 'agent-wall derivation implemented in three places' finding. The duplicate cannot be written without tripping the rule, so the copies cannot drift on math.isfinite again.
• [ce-lint] CE068 reports-do-not-rederive-turn-timing: in src/coder_eval/reports/*.py, forbid the string literal "duration_seconds" as a Subscript key or as a .get(...) argument, and forbid a comprehension over iterations that reads .duration_seconds. A renderer reads the stored agent_wall_ms or calls a result_metrics helper (e.g. sum_positive_turn_ms). It never selects turn spans itself. This puts .claude/notes/reporting.md § 'Read the stored value, do not re-derive it' into code. Wire it in tests/lint/runner.py. Prevents: Finding 'Agent-wall derivation rule implemented in three places' (markdown.py:279-282 _row_agent_wall_ms legacy branch re-selects t.get("duration_seconds") with a predicate that has drifted from result_metrics.py:112-116). It also blocks the next copy of the selection rule. A single selection would have made the 'simulator time counted as agent' fix (result_metrics.py:115) a one-place change.
• [ce-lint] CE069 one-duration-formatter: forbid a FunctionDef in src/coder_eval/reports/*.py whose name matches _?(fmt|format)_?.*(ms|sec|seconds|duration|latency|time) (for example _fmt_seconds_ms or _format_duration). Also forbid an f-string FormattedValue that wraps <expr> / 1000 or / 1000.0. Duration formatting happens only in src/coder_eval/durations.py. Delete before you guard: first fold html.py:313 _format_duration into durations.py, so that one formatter serves every card, then add the rule at zero violations. Prevents: Finding 'run.md adds its own formatter _fmt_seconds_ms (N/A) instead of format_ms (dash)' (markdown.py:285-287). Also finding 'HTML Generation Metrics shows end-to-end in Xm Ys and Agent Wall in raw seconds' (html.py:956 _format_duration next to html.py:985/1241 format_ms). Once one formatter owns the unit choice and the unmeasured sentinel, the unit split and the N/A-versus-dash drift cannot happen.
• [pyright] Change RunSummary.task_results: list[dict[str, Any]] (models/results.py:1213) to list[TaskRow]. TaskRow is a Pydantic model or TypedDict with agent_wall_ms: FiniteFloat | None, grading_ms: FiniteFloat | None and setup_ms: FiniteFloat | None, built from Annotated[float, Field(allow_inf_nan=False)]. pyright then types row["agent_wall_ms"] as float | None in markdown.py, and validation rejects non-numeric or non-finite values when run.json is loaded, not when it is rendered. It is a pyright tightening and not a lint rule, because the defect is the Any hole itself. Prevents: Finding 'run.md regeneration takes stored agent_wall_ms / grading_ms as untyped Any' (markdown.py:278, 286, 469-473, 533). It also removes the +Infinity divergence in finding 'Agent-wall derivation implemented in three places' at the boundary. A hand-edited or external run.json row fails validation with a clear error instead of a TypeError in the middle of the render or an 'infs' cell.
• [ce-lint] CE070 no-optional-nullable-ts-field: a @pytest.mark.lint class in tests/test_custom_lint.py (a whole-tree, non-Python surface). It scans interface/type bodies in evalboard/lib/*.ts with the regex ^\s+\w+\?:\s*[^;]*\|\s*null;. A field is either optional or nullable, never both. Today there are 76 hits (runs.ts 72, overview.ts 3, variants.ts 1). Ship it as a cap rule with the limit set BELOW the current count, or with a frozen baseline list that may only shrink. New fields then fail at once, and the existing ones can be removed over time. Prevents: Finding 'TaskResultSummary.gradingSeconds is both optional and nullable' (evalboard/lib/runs.ts:110). It also covers the sibling agentSeconds? and the ?? null coercions at task-grid.tsx:829/1011 that the double absence state forces.
• [ce-lint] Extend tests/lint/prose_budget.py (make docs-budget) with a no-history-reference check. A comment or docstring under src/coder_eval and tests may not contain an issue or PR reference: (?<![\w/])#\d{2,5}\b (hex colours are excluded by length and by the [0-9a-fA-F]{6} shape), coder_eval#\d+ or PR #\d+. URLs are exempt. There are 5 hits in src today (utils.py:414, orchestrator.py:1846, noop_agent.py:13, enums.py:130, batch.py:737), so it ships as a cap below 5 or with those entries rewritten first. As a second clause, each CEnnn cited in a src comment must be a registered id in tests/lint/runner.py. Checking that the cited rule is the RIGHT one is out of reach; record that as a blind spot. Prevents: Finding 'Issue-number history comments and rationale prose' (markdown.py:507-509 and 465-467, html.py:982-984, run_record.py:138-140, all citing #212, against the CLAUDE.md rule 'what it used to be belongs in git'). The registered-id clause would only catch a typo in the CE049-for-CE058 miscitation at run_record.py:140, not the miscitation itself.
• [ce-lint] CE071 html-stat-label-is-asserted: a @pytest.mark.lint class that extracts every literal stat-card label from src/coder_eval/reports/html.py (the text inside <div class="label">…</div>). It requires each label to appear in a _stat( call or an equivalent assertion in tests/test_reports_html.py. Do the same for every markdown table header cell in reports/markdown.py against tests/test_reports.py. This is a presence check, not a value check. The docstring records that limit as the rule's blind spot. Prevents: Finding 'New HTML stat cards (Agent Wall/Setup/Grading; Total Agent Wall/Total Grading) have no value assertion' (html.py:985-999, 1239-1252). Today a grep 'Agent Wall\|Total Grading' tests/test_reports_html.py returns nothing, and this rule would fail on that.
• [ce-lint] CE072 renderer-metric-parity: a whole-tree @pytest.mark.lint class. For each public result_metrics timing function (agent_wall_ms, plus the grading_ms / setup_ms fields) that is referenced by reports/html.py or reports/markdown.py, require a reference in reports/experiment.py and reports/helpers.py (VariantSeries / collect_variant_series) too. The alternative is an entry in an EXEMPT_FROM_AB = {name: reason} dict in the rule module. Declaring a deliberate end-to-end-only A/B comparison then becomes a recorded decision, not an omission. Prevents: Finding 'The agent-wall / grading split reaches the HTML variant card but not the A/B experiment report' (html.py:1239-1252 against experiment.py:208-212 and 487-501, and helpers.py:66/100). The Welch t-test then compares a duration that includes grading time, and nothing reports it.

Harness improvements (not statically reachable):

• Add a cross-language timing parity fixture, for example tests/fixtures/timing_parity/agent_wall_cases.json. Each case gives the iterations' duration_seconds as 0.0, negative, missing, null or mixed, gives a stored agent_wall_ms as absent, null or a number, and states the expected agent_wall_ms. pytest drives it through result_metrics.agent_wall_ms and the markdown row reader. vitest (evalboard/lib/tests/runs.test.ts) drives it through agentSecondsFromRaw. Both read the same file, so the 'agree by construction' claim becomes a test. Why not static: Equivalence of two implementations in two languages is semantic. No AST rule can prove that a Python predicate and a TS predicate select the same values, so only a shared golden dataset run through both can. Prevents: Finding 'Agent-wall derivation implemented in three places' (the Python and TS copies have already drifted on non-finite input). Finding 'Agent-wall rule markdown stored-None/mixed-row path not unit-tested' (the stored-null and mixed-row cases become fixture rows, so the Python and TS readers are tested evenly).
• Add a detached-grade timing contract test. Run coder-eval execute on a NoOp task, then run evaluate <run_dir>. Assert, against the documented contract in docs/REPORT_SCHEMA.md, that grading_ms is not None, that duration is the prior execute duration (it does not include this pass's grading), and that environment_info.grading_duration_seconds is set. Pair it with a doc-claim assertion that fails if REPORT_SCHEMA.md still says grading_ms is None on a detached grade. Why not static: Whether grading_ms is inside duration on a re-grade depends on the runtime order in _finalize_result / _finalize_regrade_timing across two separate runs. A static check cannot see that interaction or compare it to prose. Prevents: Finding 'REPORT_SCHEMA.md says grading_ms is None on a detached grade, but a re-grade records its own grading_ms' (docs/REPORT_SCHEMA.md:105 and 97-98, runs.ts:515, result_metrics.py:104-105, the task-grid.tsx duration tooltip). The cross-repo eval-runner reads this contract.
• Add a property test (Hypothesis, or a table of mixed rows) over run.md summary generation. For any set of rows, including errored or setup-failed rows with duration > 0 and no timed turn, each rendered average must state its row count (n=k/N) or be computed over the same row set as Avg End-to-end. Also assert per row that agent_wall + setup + grading <= duration, except on re-graded rows. Why not static: The defect is an arithmetic relation between three averages that depends on which rows have which fields at runtime. The code for each average is locally correct, so no AST shape is wrong. Prevents: Finding 'Adjacent run.md summary averages use three different row sets, so Avg Agent Wall can exceed Avg End-to-end' (markdown.py:457, 469-474). Finding 'Agent-wall rule mixed-row path not unit-tested' (the mixed denominator regression).
• Make the evalboard TaskGrid sort test table-driven over the exported COLUMNS / DEFAULT_DIR definitions. For every sortable column key, render rows with a high value, a low value and an undefined value, click the header, and assert the DEFAULT_DIR order with the null row last. Every new sortable column is then covered with no new test code. Why not static: Sort order and null placement are only visible after the React component renders and handles a click. The comparator case branches are free code, not declarative data a lint could check. Prevents: Finding 'New Agent/Grading sort branches and Agent dash rendering in TaskGrid untested' (task-grid.tsx:357-365 and the DEFAULT_DIR entries).
• Add a dialog-mode crash test. The agent's turn fails after the simulator has produced a user message, so orchestrator.py:2659-2667 appends a simulator-only TurnRecord. Assert that agent_wall_ms excludes that record's duration. Also consider a structural fix: give the simulator-only record a distinguishing marker (for example no agent-side messages, or an explicit origin field) that the shared selection helper filters on. Why not static: The wrong value comes from a runtime sequence (a simulator turn followed by an agent crash) in the dialog loop. Statically, the TurnRecord construction and the summation are each valid. Prevents: Finding 'agent_wall_ms counts the user simulator's generation time on a crashed dialog row as agent time' (result_metrics.py:115 and its markdown/TS copies).
• Add a shared one-row RunSummary test builder, for example make_run_summary(rows=..., **overrides) in tests/conftest.py, which generalizes _summary_with_notes (tests/test_reports.py:1246). Migrate the 30 hand-built RunSummary( constructions in tests/test_reports.py to it opportunistically. Why not static: Whether a test should use a builder is a judgment about duplication and readability. A cap on RunSummary( call sites in tests would be noisy and would not tell the author what to do. Prevents: Finding 'New tests repeat the 12-line RunSummary boilerplate verbatim' (tests/test_reports.py:1550 and 1583).
• Add a timing-label SSOT. Declare the display label for each run.json timing field (duration -> 'End-to-end', agent_wall_ms -> 'Agent Wall', setup_ms -> 'Setup', grading_ms -> 'Grading') once in Python. Generate a TS constant into evalboard/lib/ the same way make pricing-mirror does, and use it in task-grid.tsx, markdown.py and html.py. A drift test (modelled on CE065) then diffs the generated file. Why not static: While the labels are free JSX and f-string literals, there is no declared pairing between a field and its label for a rule to check. The mapping must exist first. After that, the drift check is static. Prevents: Finding 'The same field has a different name on each surface: Agent Wall in run.md/HTML, Agent in the evalboard' (task-grid.tsx:410 and 1006 against markdown.py:371 and html.py:996).

Top 5 Priority Actions

1. Correct the grading_ms contract at docs/REPORT_SCHEMA.md:105 (and :97-98, evalboard/lib/runs.ts:515, src/coder_eval/result_metrics.py:104-105, the task-grid.tsx duration tooltip): a detached grade or run --resume re-grade records its own pass's grading_ms (src/coder_eval/orchestrator.py:998-999), which is NOT inside the restored duration, so consumers that compute duration - setup - agent_wall - grading get a negative residual.
2. Carry agent wall and grading on VariantSeries/collect_variant_series (src/coder_eval/reports/helpers.py:66/100) and add an 'Avg Agent Wall (s)' Welch row to src/coder_eval/reports/experiment.py:208-212, because the A/B p-value still compares end-to-end duration, so a variant whose output makes a live checker or agent_judge slower is reported as the 'slower agent'.
3. Exclude simulator-only TurnRecords (only a UserMessage, appended at src/coder_eval/orchestrator.py:2659-2667 on a crashed dialog turn) from agent_wall_ms at src/coder_eval/result_metrics.py:112-116, so user-simulator generation time is not reported as Agent Wall in run.json, run.md, HTML and the evalboard.
4. Replace the three copies of the agent-wall rule (src/coder_eval/reports/markdown.py:279-282, src/coder_eval/result_metrics.py:112-116, evalboard/lib/runs.ts:1287-1290) with one shared helper (for example sum_positive_turn_ms, finite and > 0) plus a numeric guard on stored values (markdown.py:278), and add a cross-language fixture test that pins the TS legacy path to the same cases.
5. Add value assertions for the new HTML stat cards (src/coder_eval/reports/html.py:985-999 task Agent Wall/Setup/Grading, :1239-1252 variant totals) and a mixed measured/None markdown test for the averages at markdown.py:469-474, covering the dash-not-0ms rule (CE058) and the denominator; also make the averages use one row set or show n so Avg Agent Wall cannot exceed Avg End-to-end.

---

Stats: 0 🔴 · 0 🟠 · 4 🟡 · 12 🔵 across 8 axes reviewed.