feat(bench): an effort column and argument, gpt-6 luna and sol at medium and high

[using-system]

Closes #650

First batch of the llms-benchmark update tracked in #650 - more changes will follow on the same issue.

What changes

• An Effort column right after Model in both results tables. Every row measured before this one ran at medium, and says so.
• /launch-llms-benchmark <cli> <vendor/model> [effort] - the effort is an optional third argument, medium by default, passed verbatim to each CLI's flag (--variant for opencode, --effort for claude and copilot): in the smoke runs, the launch lines and the branch names. Model, effort and CLI identify a row.
• The copilot launch line sets COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200. In prompt mode Copilot waits 600 s for background tasks and then cancels them; the first gpt-6-luna run dispatched observe-run in background mode and lost its observation ten minutes in (background_task_wait_timeout, subagent.completed cancelled: true). That run is void (0.069 USD, 35 turns). The earlier copilot rows all ended under ten minutes, so none is invalidated.
• Four rows: openai/gpt-6-luna and openai/gpt-6-sol on copilot, at medium and high.
• Ten more rows: anthropic/claude-opus-5.5 on claude, and openai/gpt-6-luna-pro, openai/gpt-6-sol-pro, x-ai/grok-4.7 and z-ai/glm-5.3-prime on opencode, each at medium (or default) and high; the whole table re-sorted (28 rows).
• opencode's effort is checked, and four existing rows now read default. opencode run accepts any --variant name - an unknown one included (verified with --variant bogus) - records it on every message and sends no reasoning effort. opencode models openrouter --verbose lists each model's variants: the z-ai/glm-5.3* models and deepseek/deepseek-v4.1-flash offer low, high and max only, so their rows measured "at medium" ran at the provider's default effort and now read default (z-ai/glm-5.3-flashx, z-ai/glm-5.3, z-ai/glm-5.3-flash, deepseek/deepseek-v4.1-flash). z-ai/glm-5.3-prime's medium slot ran without --variant and reads default too. The command's opencode preflight now checks the variant, and its claude teardown also clears /tmp/odd-observe-scratch/.
• The effort actually applied was verified on every run: copilot's subagent.configured carries reasoningEffort at the requested level (medium or high) and reasoning tokens rise with it; every opencode assistant message carries the requested variant (none for default); every opencode cost reconciles to the cent at OpenRouter's prices with the doubled tier above 200,000 prompt tokens. The oddyssey column reads 1.13.0 on the maintainer's decision; the runs executed on 1.13.1 (oddyssey-mcp==1.13.1, package from main at fa1bd7f, identical to v1.13.1), whose .apm/ differs from 1.13.0's by setup-local-stack/references/otel-lgtm-env.md only.

Copilot CLI 1.0.88 for every run. Cost at OpenAI's list price (USD per million, input and cache write / cache read / output): gpt-6-luna 0.10 / 0.01 / 0.50, gpt-6-sol 2.00 / 0.20 / 10.00.

Rows and placement

Model	Effort	Row taken from	Confirmed / reported	Total	Cost	Placement
openai/gpt-6-sol	high	run 1	11 / 13	11m49s	1.46 USD	#5, just under gpt-5.6-sol (12/13, 9m32s, 1.41)
openai/gpt-6-sol	medium	run 1	8 / 10	10m23s	1.29 USD	#7, under gemini-3.7-flash (8/9, 10m17s, 1.08)
openai/gpt-6-luna	medium	run 2	11 / 11	25m34s	0.18 USD	#9, above deepseek (16/16 but 29m05s)
openai/gpt-6-luna	high	run 2	10 / 15	25m07s	0.18 USD	#11, under deepseek: same cost and time as medium, a third of its findings do not hold

What the effort changes: luna at high costs and lasts the same as at medium and reports more that does not hold; sol at high confirms three more for 1m30s and 0.17 USD more.

Second runs: both sol second runs were stopped by the command's rule (past run 1's total with the report still a skeleton): medium at 10m53s, 0.885 USD, 35 turns; high just past 11m49s, 1.179 USD. A first launch on the wrong model id (gpt-5.6-luna for gpt-6-luna) was killed before its drive.

On every run: no source file read before the drive, no traffic of its own outside the stored scenario, a replayable verification protocol. One exception to note: luna high run 2 drove the stored script.js with a k6 run it composed itself instead of the packaged replay script, without the OpenTelemetry output - the stored scenario, but no driver series in the store.

Second phase (Opus 5.5 on claude, four models on opencode)

Model	CLI	Effort	Row taken from	Confirmed / reported	Total	Cost	Placement
anthropic/claude-opus-5.5	claude	medium	run 1	17 / 20	12m28s	3.61 USD	#9
anthropic/claude-opus-5.5	claude	high	run 1	19 / 21	17m29s	4.59 USD	#10
x-ai/grok-4.7	opencode	medium	run 2	10 / 14	16m32s	1.99 USD	#13
z-ai/glm-5.3-prime	opencode	high	run 1	12 / 13	17m52s	3.42 USD	#16
x-ai/grok-4.7	opencode	high	run 1	10 / 14	23m54s	2.70 USD	#17
openai/gpt-6-sol-pro	opencode	medium	run 1	11 / 11	10m53s	4.25 USD	#18

Four rows of this phase were measured and then removed on the maintainer's decision: openai/gpt-6-luna-pro medium (11 / 13, 25m46s, 0.59 USD) and high (10 / 11, 70m27s, 1.32 USD), openai/gpt-6-sol-pro high (14 / 15, 19m03s, 7.34 USD), z-ai/glm-5.3-prime default (7 / 8, 23m09s, 4.73 USD); their rulings stay in the phase-two comment. Placement is provisional and discussed separately. Claude Code 2.1.282 (Opus 5.5 at list: 4.00 / 20.00 / 0.20 / 5.00 5m / 8.00 1h USD per million; total_cost_usd matched the transcript reconstruction exactly; observe-run dispatched with model: opus, which resolved to claude-opus-5-5, the only modelUsage key). opencode 1.18.31 from ~/.opencode/bin.

Stopped second runs (the command's rule - past run 1's total with no finished report): Opus 5.5 high 4.34 USD; gpt-6-luna-pro medium 0.37 USD; gpt-6-sol-pro medium 4.62 USD; glm-5.3-prime default 3.14 USD. The machine went into maintenance sleep on battery once, between two runs (00:59-06:40 UTC): no run spanned it, the interrupted preparation was redone on AC power.

The per-finding rulings of this phase are in the PR comment "Rulings - second phase". A new behavior finding shows up in this phase and is graded like any other: some answers quote the tools' cents as euros, and some rank prices out of order - verified in the traces each report cites.

Rulings

openai/gpt-6-sol (copilot, effort high) - run 1 - 11 confirmed / 13 reported

Window 21:34:46Z-21:36:49Z (k6's own 122.1 s, summary mtime 21:36:48). 7 anomaly rows (F2, F6 and F7 each bundle two defects with two fixes) and 7 gaps, of which four restate F6a, F6b, F7a and F7b: 13 items.

• F1 did not hold up: nine /ask arrivals are right, but the ninth is the constant-arrival-rate stage's arrival at t0+120 s, the load generator's boundary - not a defect of the services.
• F2a held up: trace cbc62264… carries 25 detail GETs under one search; SEARCH_DETAIL_FANOUT defaults to 25. Perf.
• F2b held up: mcp profile create_default_context 57.92 % self; catalog.py builds an httpx.Client per GET. Perf.
• F3 held up: 22 out-of-stock WARN lines, 905 POST /orders at 200, 883 created. Behavior.
• F4 held up: GET /stats p50 284 ms / p95 478 ms; stats_scan 300 ms on 7a203459…; api query 81.50 % self. Perf.
• F5 held up: search_products counter 160 against 458 spans, 298 cache-hit lines; the cache-hit branch returns before the increment. Telemetry.
• F6a held up: agent WARN model call failed attempt=1/3 … finish_reason on 979ee388…, ERROR model span, retry, root 200. Behavior.
• F6b held up: gen_ai_client_token_usage_count 20 for 21 chat spans; the errored call carries no usage. Telemetry.
• F7a held up: profile labels carry no service_instance_id. Telemetry.
• F7b held up: memory:alloc_space empty for the three services while the store holds allocation data from other emitters. Telemetry.
• G1 held up: the MCP exports only its outbound http_client_duration, mcp_tool_calls_total and target_info - no server-side status or duration per tool. Telemetry.
• G2 held up: no SQLite child under POST /orders (0 against the sibling GET /orders/{order_ref}). Telemetry.
• G7 did not hold up: the services emit the older HTTP semantic-convention names (http.user_agent, http_target) consistently; the report itself says nothing is absent - a naming to query by, not a gap.
• Not counted: G3 (restates F6a's missing failure count), G4 (restates F6b), G5 (restates F7a), G6 (restates F7b).

By kind, confirmed: Telemetry 6 (F5, F6b, F7a, F7b, G1, G2) / Perf 3 (F2a, F2b, F4) / Behavior 2 (F3, F6a).
Figures: launch 21:32:41Z, end 21:44:30Z - preflight 2m05s, drive 2m03s, observation 7m41s, total 11m49s; 51 turns (root + one observe-run subagent), median model-call latency 5.1 s, max 167 s; Input 3,906,289 (uncached 153, cache read 3,686,733, cache write 219,403) / Output 28,633 (reasoning 6,564) / Cache 3,906,136; cost at list (153+219,403)x2.00/M + 3,686,733x0.20/M + 28,633x10.00/M = 1.462789 USD; 1 premium request, 157.25 AIU; signals 4/4 (15 metrics, 10 traces, 7 logs, 9 profiles); 0 source files read before the drive (first at 21:39:29); no traffic of its own; replayable protocol: yes. Copilot CLI 1.0.88, --effort high, COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200. Scratch under /tmp/oddyssey/-local/.

openai/gpt-6-sol (copilot, effort medium) - run 1 - 8 confirmed / 10 reported

Window 21:10:58Z-21:13:01Z (k6's own 122.7 s, summary mtime 21:13:01). 7 anomalies, 4 gaps of which the second restates F7: 10 items.

• F1 did not hold up: nine POST /ask arrivals and counter 9 are right, but the ninth is the constant-arrival-rate stage's arrival at t0+120 s - the load generator's boundary, not a defect of the services (and not a consent breach: the mission accepts the scenario's calls as authored).
• F2 held up: GET /stats p95 468 ms; trace 8a03164… spends 288 ms in stats_scan; api profile query 79.87 % self. Perf.
• F3 held up: trace 1e585f… carries one list and 25 detail GETs (239 ms) against a 1 ms cache hit; SEARCH_DETAIL_FANOUT defaults to 25. Perf.
• F4 held up: search_products counter 160 against 468 spans, 308 cache-hit + 160 miss log lines; the cache-hit branch returns before the increment. Telemetry.
• F5 held up: 22 out-of-stock WARN lines, all 925 POST /orders answered 200. Behavior.
• F6 held up: agent spans carry gen_ai.input.messages and the other content keys. Telemetry.
• F7 held up: a service_instance_id selector empties the api's profile, dropping it restores 63.72 s. Telemetry.
• G1 held up: memory:alloc_space empty for the three services. Telemetry.
• G3 held up: no SQLite child under POST /orders (the structural query returns 0 against the sibling GET /orders/{order_ref}). Telemetry.
• G4 did not hold up: service-level check outcomes are a workload-contract decision the report itself names as such, not a missing signal.
• Not counted: G2 (restates F7).

By kind, confirmed: Telemetry 5 (F4, F6, F7, G1, G3) / Perf 2 (F2, F3) / Behavior 1 (F5).
Figures: launch 21:09:20Z, end 21:19:43Z - preflight 1m38s, drive 2m03s, observation 6m42s, total 10m23s; 44 turns (root + one observe-run subagent), median model-call latency 5.9 s, max 166 s; Input 3,298,055 (uncached 132, cache read 3,085,729, cache write 212,194) / Output 24,399 (reasoning 2,426) / Cache 3,297,923; cost at list (132+212,194)x2.00/M + 3,085,729x0.20/M + 24,399x10.00/M = 1.285788 USD; 1 premium request, 139.19 AIU; signals 4/4 (13 metrics, 9 traces, 6 logs, 4 profiles); 0 source files read before the drive (first at 21:14:48); no traffic of its own; replayable protocol: yes. Copilot CLI 1.0.88, --effort medium, COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200. Scratch under /tmp/oddyssey/-local/.

openai/gpt-6-luna (copilot, effort medium) - run 2 - 11 confirmed / 11 reported - THE ROW (tie on confirmed with run 1, cheaper)

Window 19:41:12Z-19:43:12Z (k6's own 121.3 s, summary mtime 19:43:13). Mode observe: the root drove the stored scenario itself, then dispatched observe-run on the driven window. 6 anomalies (F3 bundles two defects with two fixes) and 5 gaps of which the first restates F5: 11 items.

• F1 held up: GET /stats p95 ~471 ms; trace 7f8221b4… spends 307 ms in stats_scan. Perf.
• F2 held up: mcp_tool_calls_total{search_products} 160 against 437 tool traces; the cache-hit branch returns before _tool_calls.add. Telemetry.
• F3a held up: trace b38ceb42… carries 25 product-detail spans under one search (SEARCH_DETAIL_FANOUT defaults to 25). Perf.
• F3b held up: mcp profile create_default_context 58.16 % self; mcp/app/catalog.py opens a new httpx.Client per call. Perf.
• F4 held up: agent WARN model call failed attempt=1/3 … finish_reason 'error' on trace 1400ce68…, one ERROR chat span of 2,679 ms, root 200 at 5,364 ms. Behavior.
• F5 held up: no llmbench-mcp span carries the run's User-Agent. Telemetry.
• F6 held up: 22 out-of-stock rejections answered 200 among 865 attempts, 843 created. Behavior.
• G2 held up: no gen_ai_client_operation_duration among the agent's metric names. Telemetry.
• G3 held up: the ERROR chat span of 1400ce68… carries no gen_ai.usage.* attribute while its siblings do; gen_ai_client_token_usage_count = 18 for 19 chat spans. Telemetry.
• G4 held up: profile labels carry no service_instance_id. Telemetry.
• G5 held up: memory:alloc_space empty for the three services. Telemetry.
• Not counted: G1 (restates F5); the gen_ai.provider.name=openai remark is prose flagged suspected, not a finding row.

By kind, confirmed: Telemetry 6 (F2, F5, G2, G3, G4, G5) / Perf 3 (F1, F3a, F3b) / Behavior 2 (F4, F6).
Figures: launch 19:40:09Z, end 20:05:43Z - preflight 1m03s, drive 2m01s, observation 22m30s, total 25m34s; 64 turns (root + one observe-run subagent), median model-call latency 7.6 s, max 112 s; Input 7,695,591 (uncached 192, cache read 7,301,673, cache write 393,726) / Output 137,707 (reasoning 109,687) / Cache 7,695,399; cost at list (192+393,726)x0.10/M + 7,301,673x0.01/M + 137,707x0.50/M = 0.181262 USD; 1 premium request, 19.11 AIU; signals 4/4 (34 metrics, 18 traces, 7 logs, 10 profiles); 0 source files read before the drive (first at 19:51:30); no traffic of its own (one replay launch at 19:40:44 refused its arguments, the 19:41:11 one drove); replayable protocol: yes. Copilot CLI 1.0.88, --effort medium, COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200. Scratch under /tmp/llmbench-observe-*.

openai/gpt-6-luna (copilot, effort medium) - run 1 - 11 confirmed / 12 reported

Window 19:12:49Z-19:14:50Z (k6's own 121.1 s and the summary's mtime agree). 5 anomalies, 8 gaps of which G4 restates F3: 12 items.

• F1 held up: api profile query 82.75 % self of 75.6 s, stats 75.42 % total; GET /stats p99 ~494 ms on the api histogram. Perf.
• F2 held up: SEARCH_DETAIL_FANOUT defaults to 25 in mcp/app/server.py; cold-half search_products p95 220 ms vs warm 2.8 ms, trace 483ae… carries 25 detail GETs. Perf.
• F3 held up: mcp_tool_calls_total{search_products} delta 160 against 443 tools/call search_products roots; the cache-hit branch returns before _tool_calls.add. Telemetry.
• F4 held up: 22 order rejected … out-of-stock WARN lines answered 200; catalog_orders_created_total 855 of 877 attempts. Behavior.
• F5 held up: 8 agent traces carry gen_ai.input.messages on their spans (the report stores the attribute names only). Telemetry.
• G1 held up: no llmbench-mcp span carries http.user_agent or user_agent.original (0 of 443 roots). Telemetry.
• G2 did not hold up: checks_total{condition="nonzero"} is the passed-check count (10,098 = every check), so the outcome is exported; the report read the Rate encoding as missing polarity.
• G3 held up: llmbench-mcp exports http_client_duration_*, mcp_tool_calls_total, target_info only - no server-side duration. Telemetry.
• G5 held up: memory:alloc_space on {service_name="llmbench-api"} is empty while the store holds allocation data for other services. Telemetry.
• G6 held up: the store's profile labels carry no service_instance_id. Telemetry.
• G7 held up: the agent's 18 metric names carry gen_ai_client_token_usage but no gen_ai_client_operation_duration. Telemetry.
• G8 held up: { POST /orders } >> { sqlite } returns 0 of 877 while the sibling GET /orders/{order_ref} returns 866. Telemetry.

By kind, confirmed: Telemetry 8 (F3, F5, G1, G3, G5, G6, G7, G8) / Perf 2 (F1, F2) / Behavior 1 (F4).
Figures: launch 19:10:30Z, end 19:37:44Z - preflight 2m19s, drive 2m01s, observation 22m54s, total 27m14s; 60 turns (root + one observe-run subagent), median model-call latency 10.8 s, max 175 s; Input 8,848,387 (uncached 180, cache read 8,515,129, cache write 333,078) / Output 151,292 (reasoning 114,258) / Cache 8,848,207; cost at list (180+333,078)x0.10/M + 8,515,129x0.01/M + 151,292x0.50/M = 0.194123 USD; 1 premium request, 23.45 AIU; signals 4/4 (31 metrics, 23 traces, 14 logs, 9 profiles); 0 source files read before the drive (five views from 19:26:03, after); no traffic of its own; replayable protocol: yes. Copilot CLI 1.0.88, --effort medium, COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200 (the first attempt, void, was cancelled by the 600 s wait: 0.069 USD, 35 turns).

openai/gpt-6-luna (copilot, effort high) - run 2 - 10 confirmed / 15 reported - THE ROW (more confirmed than run 1's 9)

Window 20:44:41Z-20:46:43Z (k6's own 121.1 s, summary mtime 20:46:42). The run drove the stored script with a k6 run it composed itself (no replay_benchmark.py, no -o opentelemetry) - the stored scenario, so no traffic of its own, but the driver exported nothing. 8 anomaly rows (2 and 8 each bundle two defects with two fixes) and 8 gap bullets, of which the first is filled and two restate rows 5 and 6: 15 items.

• 1 held up: GET /stats p95 477 ms, stats_scan selects every product column and aggregates in Python; api profile query dominant. Perf.
• 2a held up: 25 detail GETs under one search; SEARCH_DETAIL_FANOUT defaults to 25. Perf.
• 2b held up: mcp profile create_default_context 55.82 % self; catalog.py builds an httpx.Client per call. Perf.
• 3 held up: search_products counter 162 against 436 spans; the cache-hit branch returns before the increment. Telemetry.
• 4 did not hold up: four /ask traces per half and a model span dominating is the paid provider's variance, not a defect the services own.
• 5 held up: no llmbench-mcp span carries the run's User-Agent. Telemetry.
• 6 did not hold up: the store had no k6 series because the run launched k6 without its OpenTelemetry output; the stored scenario's replay exports checks_total with its condition label.
• 7 did not hold up: a p99 over eight requests disagreeing between a histogram and a span estimate is a sample-size caveat, not a defect.
• 8a did not hold up: gen_ai.provider.name=openai names the OpenAI-compatible client the SDK speaks through, with server.address=openrouter.ai beside it; nothing is mislabelled.
• 8b held up: agent spans carry the content-bearing GenAI keys. Telemetry.
• G4 held up: memory:alloc_space empty for the three services. Telemetry.
• G5 held up: no service_instance_id among the profile labels. Telemetry.
• G6 held up: no gen_ai_client_operation_duration among the agent's metrics. Telemetry.
• G7 held up: the MCP's only latency instrument is its outbound http_client_duration by method and status; no per-tool or server-side duration. Telemetry.
• G8 did not hold up: GET /products?category and ?category&q share the /products route by design; a route template carries no query string.
• Not counted: G1 (filled, pre-drive emptiness), G2 (restates 5), G3 (restates 6).

By kind, confirmed: Telemetry 7 (3, 5, 8b, G4, G5, G6, G7) / Perf 3 (1, 2a, 2b) / Behavior 0 (the out-of-stock rejections are named and dismissed as the authored contract).
Figures: launch 20:42:37Z, end 21:07:44Z - preflight 2m04s, drive 2m01s, observation 21m01s, total 25m07s; 57 turns (root + one observe-run subagent), median model-call latency 10.9 s, max 183 s; Input 6,643,162 (uncached 171, cache read 6,280,430, cache write 362,561) / Output 165,474 (reasoning 135,967) / Cache 6,642,991; cost at list (171+362,561)x0.10/M + 6,280,430x0.01/M + 165,474x0.50/M = 0.181814 USD; 1 premium request, 27.06 AIU; signals 4/4 (22 metrics, 22 traces, 6 logs, 21 profiles); 0 source files read before the drive (first at 20:54:18); no traffic of its own (the stored script, launched by hand); replayable protocol: yes. Copilot CLI 1.0.88, --effort high, COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200. Scratch under /tmp/llmbench-store-load-*-local/ (k6 summary inside it).

openai/gpt-6-luna (copilot, effort high) - run 1 - 9 confirmed / 12 reported

Window 20:09:12Z-20:11:15Z (k6's own 122.0 s, summary mtime 20:11:15). 6 anomalies, 6 gaps: 12 items.

• F1 did not hold up: nine POST /ask arrivals and agent_questions_total delta 9 are right, but the ninth is the constant-arrival-rate stage's arrival at t0+120 s - the load generator's boundary, which lands on some runs and not others, not a defect of the services.
• F2 held up: search_products counter 160 against 472 invocations, 312 cache-hit log lines make up the difference; the cache-hit branch returns before _tool_calls.add. Telemetry.
• F3 held up: 22 out-of-stock WARN lines answered 200, catalog_orders_created_total 911 of 933. Behavior.
• F4 held up: api profile query 77.52 % self of 63.16 s; stats_scan 312.6 ms on trace a0e25cd3…. Perf.
• F5 held up: agent spans carry gen_ai.input.messages and the other content keys (names only in the report). Telemetry.
• F6 held up: mcp profile create_default_context 55.83 % self (10.25 s of 18.36 s); catalog.py opens an httpx.Client per call - labelled suspected, numbers exact. Perf.
• G1 held up: no llmbench-mcp span carries the run's User-Agent. Telemetry.
• G2 held up: no service_instance_id among the profile labels. Telemetry.
• G3 held up: memory:alloc_space empty for the three services. Telemetry.
• G4 did not hold up: a caveat on the run's own method (services not restarted, cumulative deltas per existing instance), not a gap in what the services export - the metrics do carry service_instance_id.
• G5 held up: 5 of 944 mcp log lines, all Created new transport with session ID, carry no trace id. Telemetry.
• G6 did not hold up: checks_total{condition="nonzero"} is the passed-check count, so the check outcome is exported.

By kind, confirmed: Telemetry 6 (F2, F5, G1, G2, G3, G5) / Perf 2 (F4, F6) / Behavior 1 (F3).
Figures: launch 20:07:29Z, end 20:41:13Z - preflight 1m43s, drive 2m03s, observation 29m58s, total 33m44s; 63 turns (root + one observe-run subagent), median model-call latency 11.9 s, max 132 s; Input 8,022,813 (uncached 189, cache read 7,464,392, cache write 558,232) / Output 205,995 (reasoning 163,540) / Cache 8,022,624; cost at list (189+558,232)x0.10/M + 7,464,392x0.01/M + 205,995x0.50/M = 0.233484 USD; 1 premium request, 30.31 AIU; signals 4/4 (28 metrics, 22 traces, 10 logs, 14 profiles); 0 source files read before the drive (first at 20:21:49); no traffic of its own; replayable protocol: yes. Copilot CLI 1.0.88, --effort high, COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200.

Stopped second runs

• openai/gpt-6-sol medium, run 2: run 2 stopped at 21:31:51Z, launched 21:20:58Z (10m53s), phase: report skeleton written, 8 sections still ; spend to that point: 0.885264 USD at list, 35 turns, Input 2,502,839 / Output 11,376
• openai/gpt-6-sol high, run 2: run 2 stopped at 21:58:13Z, launched 2026-09-24T21:45:53Z, phase: report skeleton, 8 sections still ; cost at list: (111+162,973)x2.0/M + 2,657,186x0.2/M + 32,137x10.0/M = 1.178975 USD

Review

No sub-agent review on this branch, at the maintainer's request for this session.

🤖 Generated with Claude Code

[using-system]

Rulings - second phase

Opus 5.5 on claude and the four opencode models, both runs per model and effort, the row's run named first.

anthropic/claude-opus-5.5 (claude, effort medium) - run 1 - 17 confirmed / 20 reported - THE ROW (more confirmed than run 2's 13)

Window 22:57:20Z-22:59:21Z (k6's own 120.7 s, summary mtime 22:59:21). 13 anomalies and 8 gaps, of which the third restates F7: 20 items.

• F1 held up: 845 order created lines carry 688 distinct references, 115 of them shared by two to four orders; create_order derives the reference from SELECT COUNT(*) before its insert, with no unique constraint on order_ref. Behavior.
• F2 held up: GET /stats p50 315 / p95 481 ms; stats_scan 247.6 ms of 255.3 ms returning 5,000 rows; api query self dominant. Perf.
• F3 held up: trace e36790f6… carries one listing and 25 sequential detail GETs (247.6 ms); SEARCH_DETAIL_FANOUT defaults to 25. Perf.
• F4 held up: mcp profile create_default_context 60.0 % self; catalog.py opens an httpx.Client per call. Perf.
• F5 held up: search_products counter 161 against 438 spans; the cache-hit branch returns before the increment. Telemetry.
• F6 held up: catalog_orders_created_total has one series, catalog_category="unknown"; the literal is in main.py. Telemetry.
• F7 held up: 22 out-of-stock rejections answered 200 with no error status on any span, the MCP tool logging them at INFO. Behavior.
• F8 held up: _SEARCH_CACHE is a module-level dict with no bound and no invalidation, and its records carry stock from the detail call, while 845 orders decremented stock and 277 searches were served from it - labelled suspected, the mechanism is exact. Behavior.
• F9 held up: trace f26d6303… - category="Lighting" returned results: [], a second model turn retried; list_products filters category = ? against lowercase categories. Behavior.
• F10 held up: every hit is returned, 25 hydrated with descriptions; the final chat call of f26d6303… carried 13,723 input tokens against 481 for an order question; token p95 9,011 against p50 728. Perf.
• F11 held up: _highlight rewrites record["name"] and the rewritten record is what the cache stores (quiet [light]ing lantern in the trace). Behavior.
• F12 held up: 18 tools/list traces at the agent's protocol version, 8 rooted at POST /ask and 10 rooted at llmbench-mcp tools/list itself. Telemetry.
• F13 did not hold up: gen_ai.provider.name=openai names the OpenAI-compatible client the SDK speaks through, with server.address=openrouter.ai beside it; nothing is mislabelled.
• G1 held up: llmbench-mcp exports no server-side duration (http_client_duration_*, mcp_tool_calls_total, target_info only). Telemetry.
• G2 held up: no database child under POST /orders or GET /products/{sku}, unlike get_order. Telemetry.
• G4 held up: no gen_ai_client_operation_duration among the agent's metrics. Telemetry.
• G5 held up: gen_ai.tool.call.result and gen_ai.input.messages stop at exactly 2,048 characters on f26d6303…. Telemetry.
• G6 held up: no instance label on the profiles. Telemetry.
• G7 did not hold up: the older HTTP semantic-convention names are emitted consistently - a naming to query by, not a gap.
• G8 did not hold up: a random service.instance.id per process is the SDK's specified behavior and the metrics carry it; a run identity by name is not something the services owe.
• Not counted: G3 (restates F7).

By kind, confirmed: Telemetry 8 (F5, F6, F12, G1, G2, G4, G5, G6) / Perf 4 (F2, F3, F4, F10) / Behavior 5 (F1, F7, F8, F9, F11).
Figures: launch 22:56:08Z, end 23:08:36Z - preflight 1m12s, drive 2m01s, observation 9m15s, total 12m28s; 49 requests (root + one observe-run subagent, dispatched in the foreground with model: opus, which resolved to claude-opus-5-5 - modelUsage carries that key only, no claude-haiku-4-5 background call on this CLI version), median first-block latency 2.7 s; Input 5,459,257 (uncached 98, cache read 5,201,707, cache write 257,452) / Output 55,968 / Cache 5,459,159; total_cost_usd 3.611513, reconstructed from the transcripts at list (4.00 / 20.00 / 0.20 / 5.00 5m / 8.00 1h USD per million) to the same figure; signals 4/4 (through three helper scripts: 15 metrics, 17 traces, 13 logs, 3 profiles invocations); 0 source files read before the drive (a directory listing at 22:56:53, file reads from 23:03:27); no traffic of its own; replayable protocol: yes. Claude Code 2.1.282, --effort medium. Scratch under /tmp/odd-observe-scratch/ (new location, added to the teardown).

anthropic/claude-opus-5.5 (claude, effort medium) - run 2 - 13 confirmed / 17 reported

Window 23:12:25Z-23:14:28Z (k6's own 122.2 s, summary mtime 23:14:28). 11 anomalies (F3 bundles two defects with two fixes; F11 is F2's consequence by its own stated cause and is not counted) and 7 gaps, of which the third restates F7: 17 items.

• F1 held up: 843 order created lines carry 673 distinct references, 130 of them issued more than once; the reference comes from COUNT(*) before the insert, no unique constraint. Behavior.
• F2 held up: stats_scan 243 of 253 ms; api profile stats 61.76 s of 83.03 s. Perf.
• F3a held up: 106-span trace f857e4f6… - one listing and 25 detail GETs per miss. Perf.
• F3b held up: mcp profile create_default_context 57.2 % self; catalog.py opens an httpx.Client per call. Perf.
• F4 held up: 278 of 438 searches served from _SEARCH_CACHE, which holds stock and prices and is never invalidated. Behavior.
• F5 held up: search_products counter 160 against 438 spans; the increment sits after the cache-hit return. Telemetry.
• F6 held up: the search-bearing conversation 7dbd443f… goes from 269 to 12,438 input tokens on its second model call; token p95 11,780 against p50 780. Perf.
• F7 held up: 22 rejections answered 200, spans UNSET, the MCP tool logging the error body at INFO. Behavior.
• F8 held up: catalog_orders_created_total has one series, catalog_category="unknown". Telemetry.
• F9 did not hold up: a public-looking peer address on host-driven requests is how the machine's container network presents host traffic, not something the services record wrongly - the report itself finds no code that sets it.
• F10 did not hold up: the ninth /ask is the constant-arrival-rate stage's arrival at t0+120 s, the load generator's boundary.
• G1 held up: no database child under GET /products/{sku} or POST /orders. Telemetry.
• G2 held up: llmbench-mcp exports no server-side duration. Telemetry.
• G4 held up: no gen_ai_client_operation_duration among the agent's metrics. Telemetry.
• G5 held up: no instance label on the profiles. Telemetry.
• G6 did not hold up: a random service.instance.id per process is the SDK's specified behavior and the metrics carry it.
• G7 did not hold up: the older HTTP semantic-convention names are emitted consistently - a naming, not a gap.
• Not counted: F11 (the catalog reads slowing in the second half, attributed by the report to F2's CPU-bound scans), G3 (restates F7).

By kind, confirmed: Telemetry 6 (F5, F8, G1, G2, G4, G5) / Perf 4 (F2, F3a, F3b, F6) / Behavior 3 (F1, F4, F7).
Figures: launch 23:11:01Z, end 23:23:44Z - preflight 1m24s, drive 2m03s, observation 9m16s, total 12m43s; 48 requests (observe-run in the foreground with model: opus = claude-opus-5-5, the only modelUsage key), median first-block latency 2.8 s; Input 5,178,264 (uncached 96, cache read 4,952,032, cache write 226,136) / Output 50,498 / Cache 5,178,168; total_cost_usd 3.290502, reconstructed at list to the same figure; signals 4/4; 0 source files read before the drive (first at 23:18:41); no traffic of its own; replayable protocol: yes. Claude Code 2.1.282, --effort medium.

anthropic/claude-opus-5.5 (claude, effort high) - run 1 - 19 confirmed / 21 reported - THE ROW (run 2 stopped by the rule)

Window 23:27:29Z-23:29:48Z (k6's own 139.2 s, summary mtime 23:29:48). 14 anomalies (F4 and F8 each bundle two defects with two fixes) and 7 gaps, of which G4 restates G3 and G7 restates F6: 21 items.

• F1 held up: 189 of 850 catalog get_order spans return more than one row; the order reference comes from COUNT(*) before the insert, no unique constraint. Behavior.
• F2 held up: GET /stats p50 309 / p99 627 ms; api profile stats dominant (the step-up's mechanism is labelled suspected, the cost is exact). Perf.
• F3 held up: 275 of 437 searches served from a never-invalidated, unbounded cache holding stock. Behavior.
• F4a held up: 25 sequential detail GETs per cache miss. Perf.
• F4b held up: mcp profile create_ssl_context path 46.6 %; catalog.py builds an httpx.Client per call. Perf.
• F5 held up: search_products counter 162 against 437 spans and 275 cache-hit log lines. Telemetry.
• F6 held up: 22 out-of-stock rejections answered 200, UNSET spans, INFO on the MCP side. Behavior.
• F7 held up: catalog_orders_created_total has one series, catalog_category="unknown". Telemetry.
• F8a held up: 11 tools/list roots at the agent's protocol version arrive at the MCP detached from their /ask. Telemetry.
• F8b held up: 20 tools/list calls for 9 questions - the agent re-lists the tools on every model request. Perf.
• F9 held up: searching questions carry 12.7k-23k input tokens against 0.6k-1.1k without. Perf.
• F10 held up: trace 6a3b3953… - a chat span of 18,311 ms whose only child is a 10,107 ms provider POST: 8.2 s of it covered by no span (labelled suspected, the figures are exact). Telemetry.
• F11 held up: the agent's http_server_duration buckets stop at 10,000 ms, so /ask p95 and p99 read 1e+04 while that trace lasted 19,198 ms. Telemetry.
• F12 did not hold up: gen_ai.provider.name=openai names the OpenAI-compatible client, with server.address=openrouter.ai beside it.
• F13 did not hold up: the ninth /ask is the constant-arrival-rate stage's arrival at the stage boundary, the load generator's, not a defect of the services.
• F14 held up: the chat, tool and agent spans carry the content-bearing GenAI keys. Telemetry.
• G1 held up: no database child under GET /products/{sku}. Telemetry.
• G2 held up: no database child under POST /orders. Telemetry.
• G3 held up: the MCP's HTTP transport leaves no span and no http_server_* metric. Telemetry.
• G5 held up: a --trace-id profile query for a 445 ms CPU-bound /stats trace answers 0 - profiles carry no span link. Telemetry.
• G6 held up: no instance label on the profiles. Telemetry.
• Not counted: G4 (restates G3's missing server-side metric), G7 (restates F6).

By kind, confirmed: Telemetry 11 (F5, F7, F8a, F10, F11, F14, G1, G2, G3, G5, G6) / Perf 5 (F2, F4a, F4b, F8b, F9) / Behavior 3 (F1, F3, F6).
Figures: launch 23:25:12Z, end 23:42:41Z - preflight 2m17s, drive 2m19s, observation 12m53s, total 17m29s; 63 requests (observe-run in the foreground with model: opus = claude-opus-5-5, the only modelUsage key), median first-block latency 2.5 s; Input 7,527,315 (uncached 126, cache read 7,260,480, cache write 266,709) / Output 82,215 / Cache 7,527,189; total_cost_usd 4.593990, reconstructed at list to the same figure; signals 4/4; 0 source files read before the drive (first at 23:34:35); no traffic of its own; replayable protocol: yes. Claude Code 2.1.282, --effort high.

openai/gpt-6-luna-pro (opencode, effort medium) - run 1 - 11 confirmed / 13 reported - THE ROW (run 2 stopped by the rule)

Window 07:05:41Z-07:07:43Z (k6's own 122.5 s, summary mtime 07:07:43). 8 anomalies and 5 gaps: 13 items.

• 1 held up: trace 795eba0b… - GET /orders/{order_ref} reads two rows for one reference and returns the first; references come from COUNT(*) before the insert. Behavior.
• 2 held up: GET /stats p95 476 ms, stats_scan p95 275 ms, api query self dominant. Perf.
• 3 held up: search_products counter 161 against 443 tool traces. Telemetry.
• 4 held up: the api's /stats histogram p95 476 / p99 495 ms against a trace p95 285 ms and max 304 ms - the p95 falls in the 250-500 ms bucket and the interpolation lands near its top. Telemetry.
• 5 held up: mcp profile create_default_context 58.89 % self; catalog.py builds an httpx.Client per call. Perf.
• 6 did not hold up: the ninth /ask is the constant-arrival-rate stage's arrival at t0+120 s, the load generator's boundary.
• 7 held up: one finish_reason=error model span, its WARN, a retry inside a root answered 200. Behavior.
• 8 held up: 22 out-of-stock rejections answered 200; 855 accepted of 877 attempts. Behavior.
• G1 held up: llmbench-mcp exports no server-side duration and its tool roots carry no HTTP status. Telemetry.
• G2 held up: no database child under POST /orders. Telemetry.
• G3 held up: no llmbench-mcp span carries the run's User-Agent. Telemetry.
• G4 held up: no service_instance_id label on the profiles. Telemetry.
• G5 did not hold up: checks_total{condition="nonzero"} is k6's passed-check count, so the check outcome is exported.

By kind, confirmed: Telemetry 6 (3, 4, G1, G2, G3, G4) / Perf 2 (2, 5) / Behavior 3 (1, 7, 8).
Figures: launch 07:03:30Z, end 07:29:16Z - preflight 2m11s, drive 2m02s, observation 21m33s, total 25m46s; 43 turns, one session (no subagent dispatched), median turn latency 24.1 s, max 153 s; every assistant message carries variant: medium (the model's medium variant sends reasoning.effort: medium); Input 15,908,550 (uncached 861,406, cache read 14,904,871, cache write 142,273) / Output 167,224 (reasoning 128,337) / Cache 15,047,144; cost 0.586663 USD (opencode's recorded figure, reconciled exactly at OpenRouter's prices with the doubled tier applied above 200,000 prompt tokens, 0.10 / 0.01 / 0.125 / 0.50 base); signals 4/4 (24 metrics, 16 traces, 13 logs, 6 profiles); 0 source files read before the drive (first at 07:12:20); no traffic of its own; replayable protocol: yes. opencode 1.18.31 (~/.opencode/bin), --variant medium.

openai/gpt-6-luna-pro (opencode, effort high) - run 1 - 10 confirmed / 11 reported - THE ROW (tie with run 2 on confirmed, cheaper)

Window 08:06:26Z-08:08:28Z (k6's own 122.0 s, summary mtime 08:08:28). 6 anomalies (F4 bundles two defects with two fixes) and 10 gap bullets, of which four are filled (pre-drive emptiness) and the three allocation-profile bullets are one gap: 11 items.

• F1 held up: in trace 6c60197d… get_order returns total_cents: 215608 and the answer reads 215608€ - a 100x price error, one answer of nine (the others converted); the agent's instruction says the tools speak cents. The report masked its trace id; located by searching the run's answers. Behavior.
• F2 did not hold up: the ninth /ask is the constant-arrival-rate stage's arrival at the stage boundary, the load generator's.
• F3 held up: GET /stats p99 495.5 ms, 5,000-row scan and serialisation, api query self dominant. Perf.
• F4a held up: up to 25 detail GETs per cache miss (SEARCH_DETAIL_FANOUT). Perf.
• F4b held up: repeated httpx.Client and SSL-context creation per call, mcp profile. Perf.
• F5 held up: search_products counter misses the cache hits. Telemetry.
• F6 held up: the content-bearing GenAI keys are on the agent's spans (labelled suspected pending a policy; the keys are exact). Telemetry.
• G5 held up: llmbench-mcp has no server-side duration or HTTP status. Telemetry.
• G6 held up: no llmbench-mcp span carries the run's User-Agent. Telemetry.
• G7 held up: 5 of 853 mcp log lines, the transport-creation lines, carry no trace id. Telemetry.
• G8 held up: memory:alloc_space empty for the three services while the store holds allocation data from other emitters. Telemetry.
• Not counted: the four filled bullets; the second and third allocation bullets (same gap, other services).

By kind, confirmed: Telemetry 6 (F5, F6, G5, G6, G7, G8) / Perf 3 (F3, F4a, F4b) / Behavior 1 (F1).
Figures: launch 07:57:59Z, end 09:08:26Z - preflight 8m27s, drive 2m02s, observation 59m58s, total 70m27s; 70 turns (root + one observe-run subagent), median turn latency 40.9 s; every assistant message carries variant: high; Input 27,953,874 (uncached 2,359,179, cache read 25,366,276, cache write 228,419) / Output 504,520 (reasoning 444,547) / Cache 25,594,695; cost 1.319009 USD, reconciled exactly with the doubled tier above 200,000 prompt tokens; signals 4/4 (41 metrics, 18 traces, 12 logs, 15 profiles); 0 source files read before the drive; no traffic of its own; replayable protocol: yes. opencode 1.18.31, --variant high.

openai/gpt-6-luna-pro (opencode, effort high) - run 2 - 10 confirmed / 10 reported

Window 09:16:12Z-09:18:13Z (k6's own 121.3 s, summary mtime 09:18:13). 7 anomaly rows (5 and 7 each bundle two gaps with two fixes) and 4 gap bullets, of which three restate 5a, 7a and 7b: 10 items.

• 1 held up: 420 run-attributed /stats roots, stats_scan p95 273 ms returning 5,000 rows. Perf.
• 2 held up: search_products counter 160 against 425 spans, 265 cache-hit lines (160 + 265 = 425). Telemetry.
• 3 held up: trace 20fcf130… carries 25 detail GETs and a listing under one search; SEARCH_DETAIL_FANOUT defaults to 25 (labelled suspected on attribution; the figures are exact). Perf.
• 4 held up: the /stats histogram p95/p99 480/496 ms against 420 roots whose max is 305 ms - bucket interpolation, the same boundary coarseness on the MCP and /ask series. Telemetry.
• 5a held up: no llmbench-mcp span carries the run's User-Agent. Telemetry.
• 5b held up: the MCP's server spans carry no incoming HTTP status. Telemetry.
• 6 held up: the content-bearing GenAI keys are on the agent's spans (labelled suspected on policy; the keys are exact). Telemetry.
• 7a held up: a service_instance_id selector empties each service's profile. Telemetry.
• 7b held up: memory:alloc_space empty for the three services. Telemetry.
• G4 held up: --trace-id profile queries for three slow exemplars answer 0 frames - no span-profile link. Telemetry.
• Not counted: G1 (restates 5a/5b), G2 (restates 7a), G3 (restates 7b).

By kind, confirmed: Telemetry 8 (2, 4, 5a, 5b, 6, 7a, 7b, G4) / Perf 2 (1, 3) / Behavior 0.
Figures: launch 09:10:16Z, end 10:06:35Z - preflight 5m56s, drive 2m01s, observation 48m22s, total 56m19s; 91 turns (root + one observe-run subagent), median turn latency 26.9 s; every assistant message carries variant: high; Input 40,160,390 (uncached 2,871,721, cache read 37,069,617, cache write 219,052) / Output 447,620 (reasoning 345,410) / Cache 37,288,669; cost 1.577061 USD, reconciled exactly with the doubled tier above 200,000; signals 4/4; 0 source files read before the drive; no traffic of its own; replayable protocol: yes. opencode 1.18.31, --variant high.

openai/gpt-6-sol-pro (opencode, effort medium) - run 1 - 11 confirmed / 11 reported - THE ROW (run 2 stopped by the rule)

Window 10:10:15Z-10:12:16Z (k6's own 121.1 s, summary mtime 10:12:16). 6 anomalies (F2 bundles two defects with two fixes) and 5 gap bullets, the last of which states a non-gap: 11 items.

• F1 held up: GET /stats p50 195 / p99 504 ms, 5,000-row exemplar, api query 82.27 % self. Perf.
• F2a held up: 25 detail GETs on a 239 ms miss against no call on a 1.1 ms hit (SEARCH_DETAIL_FANOUT 25). Perf.
• F2b held up: mcp create_default_context 57.01 % self; catalog.py builds an httpx.Client per call. Perf.
• F3 held up: search_products counter 160 against 443 spans. Telemetry.
• F4 held up: 22 out-of-stock rejections answered 200; 855 created of 877 attempts. Behavior.
• F5 held up: one finish_reason validation WARN, an error model span and a retry inside a root answered 200 at 4,494 ms. Behavior.
• F6 held up: trace 641450f3… - asked for the most expensive lighting products, the answer lists 2338.64 before 2338.78 (labelled suspected; the order is plain in the answer). Behavior.
• G1 held up: no service_instance_id label on the profiles. Telemetry.
• G2 held up: allocation profiles empty for the services while other emitters carry them. Telemetry.
• G3 held up: the errored model call carries no usage - 18 token samples for 19 chat spans. Telemetry.
• G4 held up: no gen_ai_client_operation_duration among the agent's metrics. Telemetry.
• Not counted: the last bullet (the report rules the pre-drive absence a non-gap itself).

By kind, confirmed: Telemetry 5 (F3, G1, G2, G3, G4) / Perf 3 (F1, F2a, F2b) / Behavior 3 (F4, F5, F6).
Figures: launch 10:07:57Z, end 10:18:50Z - preflight 2m18s, drive 2m01s, observation 6m34s, total 10m53s; 34 turns (root + one observe-run subagent), median turn latency 11.5 s; every assistant message carries variant: medium; Input 5,019,602 (uncached 884,722, cache read 4,070,013, cache write 64,867) / Output 38,238 (reasoning 8,398) / Cache 4,134,880; cost 4.253714 USD, reconciled exactly at OpenRouter's prices (2.00 / 0.20 / 2.50 / 10.00 base) with the doubled tier above 200,000 prompt tokens; signals 4/4 (10 metrics, 10 traces, 5 logs, 4 profiles); 0 source files read before the drive; no traffic of its own; replayable protocol: yes. opencode 1.18.31, --variant medium.

openai/gpt-6-sol-pro (opencode, effort high) - run 1 - 14 confirmed / 15 reported - THE ROW (tie with run 2 on confirmed, cheaper)

Window 10:36:02Z-10:38:04Z (k6's own 122.0 s, summary mtime 10:38:04). 8 anomalies (F5 bundles two defects with two fixes) and 8 gap bullets (the third bundles two gaps; two restate F4 and F2; the last is filled): 15 items.

• F1 held up: five orders logged under ORD-000001, 245 get_order reads returning two rows or more; the reference comes from COUNT(*) before the insert. Behavior.
• F2 held up: 22 out-of-stock rejections answered 200; 851 created of 873. Behavior.
• F3 held up: GET /stats p50 290 / p95 479 ms, stats_scan 300 ms over 5,000 rows. Perf.
• F4 held up: search_products counter 162 against 443 calls, 281 cache-hit lines. Telemetry.
• F5a held up: 25 detail reads per cold search. Perf.
• F5b held up: mcp create_default_context 58.14 % self; an httpx.Client per GET. Perf.
• F6 held up: 9 of 9 /ask traces carry gen_ai.input.messages and the tool-argument and result keys. Telemetry.
• F7 did not hold up: the ninth /ask is the constant-arrival-rate stage's arrival at the stage edge, the load generator's.
• F8 held up: a category filter the catalog stores differently returned 0 results and the agent's retry without it pulled 713 hits and 25 detail GETs; list_products matches category = ? exactly. Behavior.
• G1 held up: llmbench-mcp exports no server-side duration. Telemetry.
• G2 held up: no database child under POST /orders. Telemetry.
• G3a held up: no service_instance_id label on the profiles. Telemetry.
• G3b held up: --trace-id profile queries for an api and an mcp exemplar answer 0 frames. Telemetry.
• G4 held up: allocation profiles empty for the three services. Telemetry.
• G7 held up: no gen_ai_client_operation_duration among the agent's metrics. Telemetry.
• Not counted: G5 (restates F4), G6 (restates F2), G8 (filled).

By kind, confirmed: Telemetry 8 (F4, F6, G1, G2, G3a, G3b, G4, G7) / Perf 3 (F3, F5a, F5b) / Behavior 3 (F1, F2, F8).
Figures: launch 10:32:17Z, end 10:51:20Z - preflight 3m45s, drive 2m02s, observation 13m16s, total 19m03s; 46 turns (root + one observe-run subagent), median turn latency 17.6 s; every assistant message carries variant: high; Input 8,162,678 (uncached 1,169,292, cache read 6,848,264, cache write 145,122) / Output 82,202 (reasoning 27,268) / Cache 6,993,386; cost 7.343131 USD, reconciled exactly with the doubled tier above 200,000 prompt tokens; signals 4/4 (28 metrics, 14 traces, 11 logs, 10 profiles); 0 source files read before the drive; no traffic of its own; replayable protocol: yes. opencode 1.18.31, --variant high.

openai/gpt-6-sol-pro (opencode, effort high) - run 2 - 14 confirmed / 15 reported

Window 10:56:11Z-10:58:14Z (k6's own 122.2 s, summary mtime 10:58:14). 9 anomalies (F2, F4 and F8 each bundle two defects with two fixes) and 5 gap bullets (the third restates F8b, the last is a stated non-gap): 15 items.

• F1 held up: 822 order created lines carry 677 distinct references, 113 repeated across 258 orders. Behavior.
• F2a held up: the search cache is never invalidated while 822 orders decrement stock (labelled suspected on outcome; the mechanism is exact). Behavior.
• F2b held up: search_products counter 160 against 427 spans, 267 cache-hit lines. Telemetry.
• F3 held up: GET /stats p50 260 / p95 476 ms, stats_scan 307 ms over 5,000 rows. Perf.
• F4a held up: 25 detail GETs under one search. Perf.
• F4b held up: mcp create_default_context 57.76 % self; an httpx.Client per call. Perf.
• F5 held up: trace 20d79c5c… answers "SKU-00709 is priced at 208100 euros" where the tool returned 208100 cents. Behavior.
• F6 did not hold up: the ninth /ask is the constant-arrival-rate stage's arrival at the stage edge, and what k6's checks assert is the benchmark's, not the services'.
• F7 held up: 21 out-of-stock rejections answered 200. Behavior.
• F8a held up: catalog_orders_created_total has one series, catalog_category="unknown". Telemetry.
• F8b held up: no database child under POST /orders. Telemetry.
• F9 held up: the content-bearing GenAI keys are on the agent's spans (labelled suspected on policy; the keys are exact). Telemetry.
• G1 held up: a service_instance_id selector empties each service's profile. Telemetry.
• G2 held up: allocation profiles empty for the three services. Telemetry.
• G4 held up: no gen_ai_client_operation_duration among the agent's metrics. Telemetry.
• Not counted: G3 (restates F8b), G5 (a stated non-gap).

By kind, confirmed: Telemetry 7 (F2b, F8a, F8b, F9, G1, G2, G4) / Perf 3 (F3, F4a, F4b) / Behavior 4 (F1, F2a, F5, F7).
Figures: launch 10:52:34Z, end 11:12:09Z - preflight 3m37s, drive 2m03s, observation 13m55s, total 19m35s; 41 turns, median turn latency 19.9 s; every assistant message carries variant: high; Input 8,460,853 / Output 82,906 (reasoning 30,414) / Cache 7,319,182; cost 7.530158 USD, reconciled exactly; signals 4/4; 0 source files read before the drive; no traffic of its own; replayable protocol: yes. opencode 1.18.31, --variant high.

x-ai/grok-4.7 (opencode, effort medium) - run 2 - 10 confirmed / 14 reported - THE ROW (more confirmed than run 1's 8)

Window 11:35:19Z-11:37:21Z (k6's own 122.0 s, summary mtime 11:37:21). 7 anomalies (F2 bundles two defects with two fixes; F7 is attributed by the report to F1's contention and is not counted) and 8 gaps, of which the third restates F4: 14 items.

• F1 held up: GET /stats p50 266 / p99 495 ms, stats_scan 351 ms over 5,000 rows, api query 82 % self. Perf.
• F2a held up: 25 detail GETs per cache miss, 4,868 SKU calls against about 868 direct. Perf.
• F2b held up: every hit returned, one search carrying 22,930 input tokens into the next model call. Perf.
• F3 held up: mcp create_default_context 56.46 % self; an httpx.Client per call. Perf.
• F4 held up: search_products counter 160 against 439 spans, 279 cache-hit lines. Telemetry.
• F5 held up: 22 out-of-stock rejections answered 200, no error span. Behavior.
• F6 held up: catalog_orders_created_total carries only catalog_category="unknown". Telemetry.
• G1 held up: no database child under POST /orders or GET /products/{sku}. Telemetry.
• G2 held up: no gen_ai_client_operation_duration among the agent's metrics. Telemetry.
• G4 did not hold up: the older HTTP semantic-convention names are emitted consistently - a naming, not a gap.
• G5 did not hold up: gen_ai.system beside gen_ai.provider.name is a deprecated alias still set, not missing or wrong telemetry.
• G6 did not hold up: gen_ai.provider.name=openai names the OpenAI-compatible client, with server.address=openrouter.ai beside it.
• G7 held up: allocation profiles empty for the three services. Telemetry.
• G8 did not hold up: an HTTP client duration histogram carries no URL label by design (cardinality); per-route client time is on the spans.
• Not counted: F7 (the report ties the POST /orders tail to F1), G3 (restates F4).

By kind, confirmed: Telemetry 5 (F4, F6, G1, G2, G7) / Perf 4 (F1, F2a, F2b, F3) / Behavior 1 (F5).
Figures: launch 11:31:24Z, end 11:47:56Z - preflight 3m55s, drive 2m02s, observation 10m35s, total 16m32s; 43 turns (root + one observe-run subagent), median turn latency 7.6 s; every assistant message carries variant: medium; Input 3,271,994 (uncached 339,258, cache read 2,932,736) / Output 56,790 (reasoning 36,669) / Cache 2,932,736; cost 1.988499 USD, reconciled exactly; signals 4/4; 0 source files read before the drive; no traffic of its own; replayable protocol: yes. opencode 1.18.31, --variant medium.

x-ai/grok-4.7 (opencode, effort medium) - run 1 - 8 confirmed / 9 reported

Window 11:17:57Z-11:19:58Z by k6 (121.1 s, summary mtime 11:19:58; the report's frontmatter ends at 11:20:05, seven seconds past the driver's exit). 3 anomalies (2 bundles two defects with two fixes) and 5 gaps: 9 items.

• 1 held up: GET /stats p50 195 ms against 2.6 ms for a SKU read; api profile stats 73 %. Perf.
• 2a held up: 25 detail GETs per cold search (SEARCH_DETAIL_FANOUT), traces 6b53620d…, 9be384b3…. Perf.
• 2b held up: mcp profile Client.__init__ 48 %; an httpx.Client per GET. Perf.
• 3 held up: trace 043e663e… answers "SKU-00709, priced at 208100 euros" where the tool returned 208100 cents, and offers a truncated SKU-0733. Behavior.
• G1 held up: llmbench-mcp exports no server-side duration. Telemetry.
• G2 held up: no gen_ai_client_operation_duration among the agent's metrics. Telemetry.
• G3 did not hold up: the services emit the older HTTP semantic-convention names consistently and the run is selectable on span.http.user_agent - a naming, not a gap.
• G4 held up: no service_instance_id label on the profiles. Telemetry.
• G5 held up: search_products counter 162 against 453 span calls; the increment sits on the miss path only. Telemetry.

By kind, confirmed: Telemetry 4 (G1, G2, G4, G5) / Perf 3 (1, 2a, 2b) / Behavior 1 (3).
Figures: launch 11:13:26Z, end 11:29:53Z - preflight 4m31s, drive 2m01s, observation 9m55s, total 16m27s; 50 turns (root + one observe-run subagent), median turn latency 12.8 s; every assistant message carries variant: medium (the model's medium variant sends reasoning.effort: medium); Input 4,390,153 (uncached 468,873, cache read 3,921,280) / Output 58,185 (reasoning 38,103) / Cache 3,921,280; cost 3.027826 USD, reconciled exactly at OpenRouter's prices (1.60 / 0.40 / 4.80, doubled above 200,000 prompt tokens); signals 4/4 (13 metrics, 14 traces, 2 logs, 1 profiles); 0 source files read before the drive; no traffic of its own; replayable protocol: yes. opencode 1.18.31, --variant medium.

x-ai/grok-4.7 (opencode, effort high) - run 1 - 10 confirmed / 14 reported - THE ROW (tie with run 2 on confirmed, cheaper)

Window 11:53:56Z-11:55:58Z (k6's own 121.8 s, summary mtime 11:55:58). 7 anomalies (F3 bundles two defects with two fixes) and 6 gaps: 14 items.

• F1 held up: 133 references shared, 181 extra rows; get_order returns one of the colliding rows; the reference comes from COUNT(*) before the insert. Behavior.
• F2 held up: GET /stats p50 216 ms, whole-catalog scan and serialisation, api stats 73 %. Perf.
• F3a held up: 25 detail GETs per cold search, 4,878 SKU calls. Perf.
• F3b held up: an httpx.Client per detail GET, mcp profile. Perf.
• F4 held up: out-of-stock rejections answered 200 with UNSET spans, the WARN naming a reference that is never stored. Behavior.
• F5 held up: search_products counter 160 against 444 spans. Telemetry.
• F6 held up: catalog_orders_created_total carries only catalog_category="unknown". Telemetry.
• F7 did not hold up: the access log's public-looking peer address on host-driven requests is how the machine's container network presents host traffic, not something the services record wrongly.
• G1 did not hold up: the older HTTP semantic-convention names are emitted consistently - a naming, not a gap.
• G2 held up: no gen_ai_client_operation_duration among the agent's metrics. Telemetry.
• G3 did not hold up: gen_ai.system beside gen_ai.provider.name is a deprecated alias still set.
• G4 did not hold up: an HTTP client duration histogram carries no URL label by design; per-path time is on the spans.
• G5 held up: trace d3ff614a… - a cache-hit search_products span carries no attribute saying so; only the MCP log line does. Telemetry.
• G6 held up: no service_instance_id label on the profiles. Telemetry.

By kind, confirmed: Telemetry 5 (F5, F6, G2, G5, G6) / Perf 3 (F2, F3a, F3b) / Behavior 2 (F1, F4).
Figures: launch 11:49:24Z, end 12:13:18Z - preflight 4m32s, drive 2m02s, observation 17m20s, total 23m54s; 41 turns (root + one observe-run subagent), median turn latency 14.5 s; every assistant message carries variant: high; Input 3,663,763 (uncached 526,099, cache read 3,137,664) / Output 85,947 (reasoning 60,854) / Cache 3,137,664; cost 2.702610 USD, reconciled exactly; signals 4/4; 0 source files read before the drive; no traffic of its own; replayable protocol: yes. opencode 1.18.31, --variant high.

x-ai/grok-4.7 (opencode, effort high) - run 2 - 10 confirmed / 13 reported

Window 12:19:18Z-12:21:18Z (k6's own 121.1 s, summary mtime 12:21:19). 6 anomalies (F2 and F6 each bundle two defects with two fixes) and 7 gaps, of which the first restates F6a and the last states a non-gap: 13 items.

• F1 held up: GET /stats mean 242 ms, whole catalog loaded and encoded, api query 86 % self. Perf.
• F2a held up: trace 07200c13… carries 25 SKU children under one search. Perf.
• F2b held up: mcp create_default_context 57 % self; a client per catalog call. Perf.
• F3 held up: search_products counter 161 against 420 traces. Telemetry.
• F4 did not hold up: gen_ai.provider.name=openai names the OpenAI-compatible client and gen_ai.system is its deprecated alias - neither is mislabelled.
• F5 held up: the agent's spans carry the answer text and the tool-call arguments. Telemetry.
• F6a held up: no llmbench-mcp span carries the run's User-Agent. Telemetry.
• F6b held up: 10 of 18 agent tools/list calls arrive at the MCP as unparented roots. Telemetry.
• G2 did not hold up: the older HTTP semantic-convention names are emitted consistently - a naming, not a gap.
• G3 held up: no gen_ai_client_operation_duration among the agent's metrics. Telemetry.
• G4 held up: llmbench-mcp exports no server-side duration. Telemetry.
• G5 did not hold up: an HTTP client duration histogram carries no target label by design.
• G6 held up: no service_instance_id label on the profiles. Telemetry.
• Not counted: G1 (restates F6a), G7 (the report states it is not a gap).

By kind, confirmed: Telemetry 7 (F3, F5, F6a, F6b, G3, G4, G6) / Perf 3 (F1, F2a, F2b) / Behavior 0.
Figures: launch 12:14:30Z, end 12:38:16Z - preflight 4m48s, drive 2m01s, observation 16m57s, total 23m46s; 52 turns, median turn latency 14.5 s; every assistant message carries variant: high; Input 5,186,170 / Output 89,374 (reasoning 64,159) / Cache 4,286,720; cost 3.921398 USD, reconciled exactly; signals 4/4 (profiles through its helper scripts); 0 source files read before the drive; no traffic of its own; replayable protocol: yes. opencode 1.18.31, --variant high.

z-ai/glm-5.3-prime (opencode, effort default - the model has no medium variant) - run 1 - 7 confirmed / 8 reported - THE ROW (run 2 stopped by the rule)

Window 12:44:15Z-12:46:16Z (k6's own 121.1 s, summary mtime 12:46:16). 6 anomalies (the sixth, a /stats p50 drift, is tied by the report to the first and is not counted) and 4 gap bullets, the last a statement of what is not a gap: 8 items.

• 1 held up: GET /stats p50 247 / p95 474 ms over 420 calls, whole catalog read and encoded to count bytes. Perf.
• 2 held up: mcp create_default_context 61.8 % self; an httpx.Client per downstream call. Perf.
• 3 held up: up to 25 detail GETs per uncached search (traces c5b06c7b…, e0389884…). Perf.
• 4 held up: search_products counter 162 against 426 span calls; 162 misses + 264 cache hits in the MCP log. Telemetry.
• 5 held up: catalog_orders_created_total carries only catalog_category="unknown", the literal in main.py. Telemetry.
• G1 held up: no database child under GET /products/{sku} or POST /orders. Telemetry.
• G2 did not hold up: gen_ai.system beside gen_ai.provider.name is a deprecated alias still set.
• G3 held up: no instance label on the profiles. Telemetry.
• Not counted: 6 (the drift of finding 1), the last gap bullet (stated as not a gap).

By kind, confirmed: Telemetry 4 (4, 5, G1, G3) / Perf 3 (1, 2, 3) / Behavior 0.
Figures: launch 12:39:34Z, end 13:02:43Z - preflight 4m41s, drive 2m01s, observation 16m27s, total 23m09s; 49 turns (root + one observe-run subagent), median turn latency 17.7 s; launched without --variant (the model offers low, high and max only) and no assistant message carries a variant - the provider's default effort; Input 5,317,865 (uncached 290,793, cache read 5,027,072) / Output 124,823 (reasoning 100,697) / Cache 5,027,072; cost 4.727823 USD, reconciled exactly at OpenRouter's prices (2.80 / 0.56 / 8.80); signals 4/4; 0 source files read before the drive; no traffic of its own; replayable protocol: yes. opencode 1.18.31.

z-ai/glm-5.3-prime (opencode, effort high) - run 1 - 12 confirmed / 13 reported - THE ROW (more confirmed than run 2's 11)

Window 13:30:04Z-13:32:34Z (k6's own 150.0 s, summary mtime 13:32:34 - the driver waited out a hung /ask). 8 anomalies (F7 bundles two defects with two fixes) and 5 gap bullets, the last stating nothing is unserved: 13 items.

• F1 held up: GET /stats p50 281 ms, 67 % of the api's server time, profile 80 % in query. Perf.
• F2 held up: 4,912 SKU GETs against about 920 direct; trace 01f39e19…; SEARCH_DETAIL_FANOUT 25. Perf.
• F3 held up: mcp create_default_context 58.45 % self; an httpx.Client per call. Perf.
• F4 held up: trace 5aa62117… - POST /ask 120.6 s, a 117.2 s chat span in error with its finish_reason validation WARN, then a retry that re-sends the whole prompt; the call has no bound on its duration. Behavior.
• F5 held up: search_products counter 161 against 462 spans, 301 cache hits. Telemetry.
• F6 held up: _SEARCH_CACHE has no invalidation on place_order and no bound (labelled latent in the run; the code is exact). Behavior.
• F7a held up: 22 out-of-stock rejections of 911 answered 200. Behavior.
• F7b held up: catalog_orders_created_total carries only catalog_category="unknown". Telemetry.
• F8 did not hold up: gen_ai.provider.name=openai names the OpenAI-compatible client, with the router's address beside it.
• G1 held up: llmbench-mcp exports no server-side HTTP metrics. Telemetry.
• G2 held up: no gen_ai_client_operation_duration among the agent's metrics. Telemetry.
• G3 held up: 19 token-usage samples for 22 chat spans - the errored turns carry no usage. Telemetry.
• G4 held up: no service_instance_id label on the profiles. Telemetry.
• Not counted: the last bullet (stated as nothing missing).

By kind, confirmed: Telemetry 6 (F5, F7b, G1, G2, G3, G4) / Perf 3 (F1, F2, F3) / Behavior 3 (F4, F6, F7a).
Figures: launch 13:28:04Z, end 13:45:56Z - preflight 2m00s, drive 2m30s (the scenario's two minutes plus k6 waiting out the hung /ask), observation 13m22s, total 17m52s; 48 turns (root + one observe-run subagent), median turn latency 11.3 s; every assistant message carries variant: high (the model's high variant sends reasoning.effort: high); Input 3,904,309 (uncached 227,893, cache read 3,676,416) / Output 82,501 (reasoning 58,828) / Cache 3,676,416; cost 3.422902 USD, reconciled exactly; signals 4/4; 0 source files read before the drive; no traffic of its own; replayable protocol: yes. opencode 1.18.31, --variant high.

z-ai/glm-5.3-prime (opencode, effort high) - run 2 - 11 confirmed / 14 reported

Window 13:49:17Z-13:51:20Z (k6's own 122.3 s, summary mtime 13:51:20). 9 anomalies and 7 gap bullets (the second restates F4, the last records a deliberate exclusion): 14 items.

• F1 held up: GET /stats p50 288 / p95 479 ms, stats_scan 244-322 ms over 5,000 rows. Perf.
• F2 held up: trace dc3a86bd… - one listing and 25 detail GETs under one search. Perf.
• F3 held up: mcp create_default_context 56.44 % self; an httpx.Client per call. Perf.
• F4 held up: search_products counter 160 against 421 spans, 261 cache hits. Telemetry.
• F5 held up: ORD-000026 issued to two orders for two SKUs; the reference comes from COUNT(*) before the insert. Behavior.
• F6 held up: _SEARCH_CACHE never invalidated by place_order, unbounded. Behavior.
• F7 held up: one finish_reason validation failure retried into the slowest request. Behavior.
• F8 held up: input tokens p95 12.7k for searching questions against p50 768. Perf.
• F9 did not hold up: what the k6 checks assert is the benchmark's, not the services'.
• G1 held up: the MCP's inbound HTTP has no server span and no http_server_* metric. Telemetry.
• G3 held up: no gen_ai_client_operation_duration among the agent's metrics. Telemetry.
• G4 held up: 20 usage samples for 21 chat spans - the errored call carries none. Telemetry.
• G5 did not hold up: the chat spans do carry gen_ai.response.model beside gen_ai.request.model.
• G6 did not hold up: the older HTTP semantic-convention label (http_target) is emitted consistently - a naming, not a gap.
• Not counted: G2 (restates F4), G7 (the /health exclusion is deliberate, as the report says).

By kind, confirmed: Telemetry 4 (F4, G1, G3, G4) / Perf 4 (F1, F2, F3, F8) / Behavior 3 (F5, F6, F7).
Figures: launch 13:47:14Z, end 14:04:44Z - preflight 2m03s, drive 2m03s, observation 13m24s, total 17m30s; 45 turns, median turn latency 7.2 s; every assistant message carries variant: high; Input 4,017,336 / Output 84,906 (reasoning 60,078) / Cache 3,809,792; cost 3.461780 USD, reconciled exactly; signals 4/4; 0 source files read before the drive; no traffic of its own; replayable protocol: yes. opencode 1.18.31, --variant high.