From 8aeb74f0df92b3c23b0bb327acecb0766e81ed40 Mon Sep 17 00:00:00 2001 From: using-system Date: Fri, 25 Sep 2026 00:17:50 +0200 Subject: [PATCH 1/3] feat(bench): an effort column and argument, gpt-6 luna and sol at medium and high The results tables gain an Effort column after Model (every earlier row was measured at medium), and /launch-llms-benchmark takes the effort as an optional third argument, medium by default, passed to each CLI's effort flag; model, effort and CLI identify a row. The copilot launch line lifts prompt mode's 600 s wait on background tasks, which cancelled the first gpt-6-luna observation mid-run. Four rows added: openai/gpt-6-luna and openai/gpt-6-sol on copilot, at medium and high. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/commands/launch-llms-benchmark.md | 83 ++++++++++++-------- .llms-benchmark/README.md | 95 +++++++++++++---------- 2 files changed, 104 insertions(+), 74 deletions(-) diff --git a/.claude/commands/launch-llms-benchmark.md b/.claude/commands/launch-llms-benchmark.md index 3e49899..198ed31 100644 --- a/.claude/commands/launch-llms-benchmark.md +++ b/.claude/commands/launch-llms-benchmark.md @@ -1,6 +1,6 @@ --- description: Benchmark one LLM on the llms-benchmark demo stack - drive it through a coding-agent CLI (opencode, claude or copilot) on the stored scenario, grade the observation report it produced, and propose its row of the results table -argument-hint: " " +argument-hint: " [effort, default medium]" --- Run the whole llms-benchmark protocol for one model on one CLI, end to @@ -20,9 +20,14 @@ the same way you would grade a colleague's incident report. `opencode`, `claude` or `copilot`; the **model** to benchmark, as its canonical `vendor/name` id, the OpenRouter form (`anthropic/claude-sonnet-5`, `openai/gpt-5-mini`, - `google/gemini-3.5-flash-lite`, ...). Those are the only two inputs. - Ask for whichever is missing and stop until you have both. Model and - CLI identify the row: the same model on two CLIs is two rows. + `google/gemini-3.5-flash-lite`, ...); optionally, the **effort** the + CLI runs the model at — `low`, `medium`, `high`, or any other level + the CLI's effort flag accepts — **`medium` when omitted**, the level + every row was measured at before this argument existed. Those are the + only three inputs. Ask for the CLI or the model when missing and stop + until you have both; never ask for the effort. Model, effort and CLI + identify the row: the same model on two CLIs, or at two efforts, is + two rows. - The model id is written the same way whatever the CLI, so the two rows of one model line up. Each CLI is handed its own form of it: `opencode` takes it as `openrouter/`; `claude` takes Anthropic @@ -32,8 +37,9 @@ the same way you would grade a colleague's incident report. dashes; `claude --help` on `--model` names the accepted forms); `copilot` takes the bare name its model picker lists — `openai/gpt-5.6-luna` is `gpt-5.6-luna` (the vendor prefix dropped, - nothing else changed). A model the CLI cannot run is a preflight - failure, not a row. + nothing else changed). A model the CLI cannot run, or cannot run at + the requested effort, is a preflight failure, not a row. Below, + `` is that argument, passed verbatim to the CLI's flag. **Never ask for an API key, and never handle one.** Every credential this protocol needs — the OpenRouter provider in opencode, the Claude Code @@ -67,7 +73,7 @@ Steps: were given - or, when the listing lags OpenRouter's catalog (it did not carry `z-ai/glm-5.3-flashx` on 2026-09-19 while the model ran), a smoke run answers with a `text` event: - `opencode run --model openrouter/ --format json "reply with the single word ok" < /dev/null` + `opencode run --model openrouter/ --variant --format json "reply with the single word ok" < /dev/null` (the package is installed in step 3); - `claude`: `claude --version` answers; the package is installed at **user scope** for Claude Code — `~/.claude/commands/odd-observe.md`, @@ -81,7 +87,7 @@ Steps: configuration (`~/.claude.json`, `mcpServers` carries `oddyssey` — the name only, never its contents); and a smoke run answers with a result naming the model: - `claude -p "Reply with the single word ok" --model --output-format json < /dev/null` + `claude -p "Reply with the single word ok" --model --effort --output-format json < /dev/null` must print a `type: result` JSON whose `modelUsage` carries the model's canonical id. Since Claude Code 2.1.270 a second key, `claude-haiku-4-5`, sits beside it on every run — a background @@ -95,7 +101,7 @@ Steps: (`~/.copilot/config.json` carries a non-empty `loggedInUsers` — the host and login, nothing else lives there); and a smoke run answers with a usage file naming the model: - `copilot -p "Reply with the single word ok" --model --allow-all-tools --usage-output-file /usage.json < /dev/null` + `copilot -p "Reply with the single word ok" --model --effort --allow-all-tools --usage-output-file /usage.json < /dev/null` must leave a `usage.json` whose `modelMetrics` has one key, the model's name. Run it from a scratch directory too — it leaves a session under `~/.copilot/session-state/`. Nothing is installed @@ -107,7 +113,7 @@ Steps: presence, never its value, and never print it. The file is gitignored; `.env.example` next to it says what goes in. -2. **Create the work branch**: `bench/--`, +2. **Create the work branch**: `bench/---`, where `` is the model id with `/` and `.` replaced by `-`. Everything the run installs, configures, and produces happens on this branch, and none of it is what ships. @@ -182,10 +188,11 @@ Steps: 4. **Select the model.** Nothing to configure: the provider is already set up (preflight), and the model and effort are passed on the command line in step 6, never persisted into a config file — `opencode`: - `--model openrouter/ --variant medium`; `claude`: - `--model --effort medium`; `copilot`: - `--model --effort medium`. The three flags name the same - effort level; that is what makes two rows of one model comparable. + `--model openrouter/ --variant `; `claude`: + `--model --effort `; `copilot`: + `--model --effort `. The three flags name the same + effort level; that is what makes two rows of one model at one effort + comparable across CLIs. 5. **Clean what the next run must not read — then recreate the demo stack, never reuse a running one.** Before every run, whatever the @@ -276,7 +283,7 @@ Steps: `opencode`: ``` - caffeinate -i opencode run --model openrouter/ --variant medium \ + caffeinate -i opencode run --model openrouter/ --variant \ --format json --auto --title "llms-benchmark " \ "" < /dev/null ``` @@ -289,7 +296,7 @@ Steps: caffeinate -i env -u CLAUDECODE -u CLAUDE_CODE_CHILD_SESSION -u CLAUDE_CODE_SESSION_ID \ -u CLAUDE_CODE_MESSAGING_SOCKET -u CLAUDE_CODE_MESSAGING_TOKEN -u CLAUDE_PID \ CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS=0 \ - claude -p "" --model --effort medium \ + claude -p "" --model --effort \ --permission-mode bypassPermissions --output-format json \ --session-id "$SID" < /dev/null > /run.json 2> /run.err ``` @@ -298,12 +305,21 @@ Steps: ``` SID=$(uuidgen | tr 'A-Z' 'a-z') - caffeinate -i copilot -p "" --model --effort medium \ + caffeinate -i env COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200 \ + copilot -p "" --model --effort \ --allow-all --no-ask-user --additional-mcp-config @.github/mcp.json \ --session-id "$SID" --output-format json --usage-output-file /usage.json \ < /dev/null > /run.jsonl 2> /run.err ``` + `COPILOT_TASK_WAIT_TIMEOUT_SECONDS=7200` lifts prompt mode's 600 s + wait on background tasks: a root that dispatches `observe-run` with + `mode: background` and ends its turn has its subagent cancelled + 600 s later, report unwritten (`session.warning` + `background_task_wait_timeout`, `subagent.completed` with + `cancelled: true`) - the first `openai/gpt-6-luna` run of + 2026-09-24 lost its whole observation to it, ten minutes in. That + run is void, not a row. `--allow-all` is this CLI's headless auto mode (tools, paths and URLs); `--no-ask-user` removes the tool a run would otherwise use to ask a question nobody answers; `--additional-mcp-config @.github/mcp.json` @@ -985,34 +1001,37 @@ Steps: create one naming the model and the protocol revision, then the PR that closes it. This is a step, not a fallback. - From `main`, freshly pulled, create - `docs/llms-benchmark--` and make **one** change: + `docs/llms-benchmark---` and make **one** change: the row in the results tables of `.llms-benchmark/README.md`. `## Results` holds the two tables below. **A row is identified by - model and CLI together.** The pair is not in the table yet → - append the row; already there → replace that row in place. The - same model driven through two CLIs is two rows - (`google/gemini-3.7-flash` under `opencode` and under another CLI - both appear); the oddyssey version is not part of the key — a new - run of the same model and CLI overwrites the row, whatever version - the old one carried. The table carries no history: one row per - model and CLI, always the latest run. - - **Two tables, not one.** Seventeen columns scroll the model name off + model, effort and CLI together.** The triple is not in the table + yet → append the row; already there → replace that row in place. + The same model driven through two CLIs, or at two efforts, is two + rows (`google/gemini-3.7-flash` under `opencode` and under another + CLI both appear); the oddyssey version is not part of the key — a + new run of the same model, effort and CLI overwrites the row, + whatever version the old one carried. The table carries no + history: one row per model, effort and CLI, always the latest + run. + + **Two tables, not one.** Eighteen columns scroll the model name off the screen and the rows stop being readable, and GitHub keeps no CSS to pin a column. So: - - a **headline table** of nine columns — rank, model, CLI, oddyssey + - a **headline table** of ten columns — rank, model, effort, CLI, oddyssey version, `confirmed / reported`, the findings by kind under a single `Telemetry / Perf / Behavior` header written `X / X / X`, total duration, cost, and cost per confirmed finding. It fits - without scrolling and answers the question on its own. The CLI + without scrolling and answers the question on its own. The effort + column is the `` argument as passed to the CLI's flag + (`medium` by default), right after the model it qualifies. The CLI column names the coding-agent CLI the mission ran in — the `` argument, `opencode`, `claude` or `copilot`, with no version: the version belongs in the pull request, where the row's exact figures already live. The oddyssey version sits right after it because it says which protocol a row was taken under, which a reader needs before any number to its right means anything; - - a **detail table** inside a `
` block — model, CLI, + - a **detail table** inside a `
` block — model, effort, CLI, oddyssey version, the three phase durations, turns, median turn latency, input / output / cache tokens, and signals. Round the token counts (`30.0M`, `79k`): the @@ -1038,7 +1057,7 @@ Steps: The PR body carries the per-finding rulings from step 8 for both runs, so the ratio is auditable and the choice between the two is too, and it names the CLI, its version and the effort flag - used. It also notes three things the table has no column for: how + used, with the effort level. It also notes three things the table has no column for: how many source files the run read **before** the drive, whether it drove any traffic of its own outside the stored scenario, and whether its report carries a replayable verification protocol. diff --git a/.llms-benchmark/README.md b/.llms-benchmark/README.md index b5a0b2f..d155d4d 100644 --- a/.llms-benchmark/README.md +++ b/.llms-benchmark/README.md @@ -4,51 +4,60 @@ A ranking, to pick the model you run the loop with and the coding-agent CLI you drive it through. Each model observes the same running stack through the same replayed -traffic, and its report is graded on evidence. One model on one CLI, one -run, one row: the same model under two CLIs is two rows, ranked against -each other like any other pair. The protocol is fixed and the only -variables are the model and the CLI. +traffic, and its report is graded on evidence. One model at one effort +on one CLI, one run, one row: the same model under two CLIs, or at two +efforts, is two rows, ranked against each other like any other pair. The +protocol is fixed and the only variables are the model, its effort and +the CLI. ## Results -One row per model and CLI, always its latest run. - -| Rank | Model | CLI | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | $/confirmed | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | -| **#1** | `z-ai/glm-5.3-flashx` | opencode | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | $0.030 | -| **#2** | `openai/gpt-5.6-luna` | copilot | 1.13.0 | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | $0.016 | -| **#3** | `openai/gpt-5.6-terra` | copilot | 1.13.0 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | $0.126 | -| **#4** | `openai/gpt-5.6-sol` | copilot | 1.13.0 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | $0.117 | -| **#5** | `google/gemini-3.7-flash` | opencode | 1.13.0 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | $0.135 | -| **#6** | `z-ai/glm-5.3` | opencode | 1.13.0 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | $0.082 | -| **#7** | `deepseek/deepseek-v4.1-flash` | opencode | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **$0.010** | -| **#8** | `google/gemini-3.8-flash` | opencode | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | $0.191 | -| **#9** | `qwen/qwen3.8-max-0902` | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | -| **#10** | `anthropic/claude-opus-5` | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | -| **#11** | `anthropic/claude-fable-5.1` | claude | 1.12.0 ⚠︎ | **17 / 17** | 10 / 5 / 2 | 17m08s | $7.55 | $0.444 | -| **#12** | `z-ai/glm-5.3-flash` | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | -| **#13** | `anthropic/claude-sonnet-5` | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | -| **#14** | `qwen/qwen3.8-27b` | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 | +One row per model, effort and CLI, always its latest run. + +| Rank | Model | Effort | CLI | oddyssey | Confirmed / reported | Telemetry / Perf / Behavior | Total | Cost | $/confirmed | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| **#1** | `z-ai/glm-5.3-flashx` | medium | opencode | 1.13.0 | 11 / 13 | 4 / 4 / 3 | 17m09s | $0.33 | $0.030 | +| **#2** | `openai/gpt-5.6-luna` | medium | copilot | 1.13.0 | 7 / 8 | 3 / 2 / 2 | 6m29s | $0.11 | $0.016 | +| **#3** | `openai/gpt-5.6-terra` | medium | copilot | 1.13.0 | 7 / 8 | 2 / 3 / 2 | **5m58s** | $0.88 | $0.126 | +| **#4** | `openai/gpt-5.6-sol` | medium | copilot | 1.13.0 | 12 / 13 | 8 / 4 / 0 | 9m32s | $1.41 | $0.117 | +| **#5** | `openai/gpt-6-sol` | high | copilot | 1.13.0 | 11 / 13 | 6 / 3 / 2 | 11m49s | $1.46 | $0.133 | +| **#6** | `google/gemini-3.7-flash` | medium | opencode | 1.13.0 | 8 / 9 | 4 / 3 / 1 | 10m17s | $1.08 | $0.135 | +| **#7** | `openai/gpt-6-sol` | medium | copilot | 1.13.0 | 8 / 10 | 5 / 2 / 1 | 10m23s | $1.29 | $0.161 | +| **#8** | `z-ai/glm-5.3` | medium | opencode | 1.13.0 | 17 / 19 | 9 / 4 / 4 | 19m44s | $1.39 | $0.082 | +| **#9** | `openai/gpt-6-luna` | medium | copilot | 1.13.0 | **11 / 11** | 6 / 3 / 2 | 25m34s | $0.18 | $0.016 | +| **#10** | `deepseek/deepseek-v4.1-flash` | medium | opencode | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **$0.010** | +| **#11** | `openai/gpt-6-luna` | high | copilot | 1.13.0 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | $0.018 | +| **#12** | `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | $0.191 | +| **#13** | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | +| **#14** | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | +| **#15** | `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | **17 / 17** | 10 / 5 / 2 | 17m08s | $7.55 | $0.444 | +| **#16** | `z-ai/glm-5.3-flash` | medium | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | +| **#17** | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | +| **#18** | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 |
Run detail — phases, turns, tokens -| Model | CLI | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| `z-ai/glm-5.3-flashx` | opencode | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | -| `openai/gpt-5.6-luna` | copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | -| `openai/gpt-5.6-terra` | copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | -| `openai/gpt-5.6-sol` | copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | -| `google/gemini-3.7-flash` | opencode | 1.13.0 | 2m24s | 2m02s | 5m51s | 90 | 4.3s | 6.5M | 32k | 5.8M | 4/4 | -| `z-ai/glm-5.3` | opencode | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | -| `deepseek/deepseek-v4.1-flash` | opencode | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | -| `google/gemini-3.8-flash` | opencode | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | -| `qwen/qwen3.8-max-0902` | opencode | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | -| `anthropic/claude-opus-5` | claude | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | -| `anthropic/claude-fable-5.1` | claude | 1.12.0 ⚠︎ | 2m52s | 2m02s | 12m14s | 36 | 3.6s | 3.6M | 60k | 3.6M | 4/4 | -| `z-ai/glm-5.3-flash` | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | -| `anthropic/claude-sonnet-5` | claude | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | -| `qwen/qwen3.8-27b` | opencode | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | +| Model | Effort | CLI | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| `z-ai/glm-5.3-flashx` | medium | opencode | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | +| `openai/gpt-5.6-luna` | medium | copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | +| `openai/gpt-5.6-terra` | medium | copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | +| `openai/gpt-5.6-sol` | medium | copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | +| `openai/gpt-6-sol` | high | copilot | 1.13.0 | 2m05s | 2m03s | 7m41s | 51 | 5.1s | 3.9M | 29k | 3.9M | 4/4 | +| `google/gemini-3.7-flash` | medium | opencode | 1.13.0 | 2m24s | 2m02s | 5m51s | 90 | 4.3s | 6.5M | 32k | 5.8M | 4/4 | +| `openai/gpt-6-sol` | medium | copilot | 1.13.0 | 1m38s | 2m03s | 6m42s | 44 | 5.9s | 3.3M | 24k | 3.3M | 4/4 | +| `z-ai/glm-5.3` | medium | opencode | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | +| `openai/gpt-6-luna` | medium | copilot | 1.13.0 | 1m03s | 2m01s | 22m30s | 64 | 7.6s | 7.7M | 138k | 7.7M | 4/4 | +| `deepseek/deepseek-v4.1-flash` | medium | opencode | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | +| `openai/gpt-6-luna` | high | copilot | 1.13.0 | 2m04s | 2m01s | 21m01s | 57 | 10.9s | 6.6M | 165k | 6.6M | 4/4 | +| `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | +| `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | +| `anthropic/claude-opus-5` | medium | claude | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | +| `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | 2m52s | 2m02s | 12m14s | 36 | 3.6s | 3.6M | 60k | 3.6M | 4/4 | +| `z-ai/glm-5.3-flash` | medium | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | +| `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | +| `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | Token counts are rounded; the exact figures are in each run's pull request. Input includes the cached share, so Input and Cache overlap by @@ -62,12 +71,13 @@ design. - **Confirmed / reported** is the grade: how many of the findings the model reported held up when checked against the telemetry it cited and the code it accused. 3 / 3 beats 4 / 12. Anomalies and telemetry gaps both count; a restatement counts once; a row bundling several defects counts once per defect. - **Telemetry / Perf / Behavior** splits the confirmed findings by kind. - **$/confirmed** is what one trustworthy finding costs. -- **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. Model and CLI identify a row; the oddyssey version does not, a new run replaces the row. +- **Effort** is the reasoning effort the CLI ran the model at (`low`, `medium`, `high`, ...), `medium` unless the run asked for another. +- **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. Model, effort and CLI identify a row; the oddyssey version does not, a new run replaces the row. - **Signals**: how many of metrics, traces, logs and profiles the run queried. Not part of the grade, the context to read it in. - **Preflight / Drive / Observation**: the drive is the scenario's fixed two minutes; a long preflight is a model that is lost, a long observation a model that is thorough. **Turns** and **median turn** separate groping (many short turns) from slow answering (few long ones). - **Input / Output / Cache / Cost** come from the CLI's own session store after the run, whole session tree included. Input is the whole prompt processed, cached share included (cache is that share); output includes reasoning; cost is the provider's billed figure, cross-checked against its list prices. -A row measured under an earlier revision of the protocol is marked ⚠︎ and provisional until re-run. The table keeps no history: one row per model and CLI, its latest run. +A row measured under an earlier revision of the protocol is marked ⚠︎ and provisional until re-run. The table keeps no history: one row per model, effort and CLI, its latest run. ## How a row is produced @@ -75,11 +85,12 @@ A row measured under an earlier revision of the protocol is marked ⚠︎ and pr /launch-llms-benchmark opencode anthropic/claude-sonnet-5 /launch-llms-benchmark claude anthropic/claude-haiku-4.5 /launch-llms-benchmark copilot openai/gpt-5.6-luna +/launch-llms-benchmark copilot openai/gpt-5.6-sol high ``` -The CLI and the model id, in `vendor/name` form, are the only arguments. Prerequisites, set up once: an OpenRouter provider in opencode, a Claude Code login with the package installed at user scope, or a Copilot CLI login; and `OPENAI_API_KEY` in `docker-compose/llms-benchmark/.env` for the demo agent's own model calls (`.env.example` next to it). +The CLI and the model id, in `vendor/name` form, are required; an optional third argument sets the effort (`medium` by default). Prerequisites, set up once: an OpenRouter provider in opencode, a Claude Code login with the package installed at user scope, or a Copilot CLI login; and `OPENAI_API_KEY` in `docker-compose/llms-benchmark/.env` for the demo agent's own model calls (`.env.example` next to it). -The command cleans everything a run must not read (stored reports of the three services, leftovers, the local stack's data), recreates the demo stack, drives the model headless at medium effort through one `/odd-observe` mission naming the three services, the stored scenario and the local stack, grades the report finding by finding on evidence, and opens the pull request carrying the row and the rulings. +The command cleans everything a run must not read (stored reports of the three services, leftovers, the local stack's data), recreates the demo stack, drives the model headless at the requested effort through one `/odd-observe` mission naming the three services, the stored scenario and the local stack, grades the report finding by finding on evidence, and opens the pull request carrying the row and the rulings. ## The stack under observation From 8e547a66a1d87341655360e16a8fd2850c0d59d6 Mon Sep 17 00:00:00 2001 From: using-system Date: Fri, 25 Sep 2026 16:42:28 +0200 Subject: [PATCH 2/3] feat(bench): opus 5.5 on claude, four opencode models at medium and high, effective effort per row Adds anthropic/claude-opus-5.5 (claude) and openai/gpt-6-luna-pro, openai/gpt-6-sol-pro, x-ai/grok-4.7 and z-ai/glm-5.3-prime (opencode) at medium and high, and re-sorts the table. opencode accepts any --variant name and sends no effort for an unknown one: the z-ai/glm-5.3* models and deepseek/deepseek-v4.1-flash offer no medium variant, so their rows read `default`, and the preflight now checks the model's variants. The claude teardown also clears /tmp/odd-observe-scratch/. Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/commands/launch-llms-benchmark.md | 17 +++++-- .llms-benchmark/README.md | 54 ++++++++++++++++------- 2 files changed, 51 insertions(+), 20 deletions(-) diff --git a/.claude/commands/launch-llms-benchmark.md b/.claude/commands/launch-llms-benchmark.md index 198ed31..a40b4df 100644 --- a/.claude/commands/launch-llms-benchmark.md +++ b/.claude/commands/launch-llms-benchmark.md @@ -74,7 +74,17 @@ Steps: did not carry `z-ai/glm-5.3-flashx` on 2026-09-19 while the model ran), a smoke run answers with a `text` event: `opencode run --model openrouter/ --variant --format json "reply with the single word ok" < /dev/null` - (the package is installed in step 3); + (the package is installed in step 3) - **and the model has a + variant at the requested effort**: `opencode run` accepts any + `--variant` name, an unknown one included, records it on every + message and sends no effort at all (verified on 2026-09-25 with + `--variant bogus`). Read the model's `variants` from + `opencode models openrouter --verbose`; when `` is not + among them, launch without `--variant` and write `default` in the + Effort column - the provider's default effort is what the model + ran at (the `z-ai/glm-5.3*` models and + `deepseek/deepseek-v4.1-flash` offer `low`, `high` and `max` only, + so every row of theirs measured at "medium" ran at `default`); - `claude`: `claude --version` answers; the package is installed at **user scope** for Claude Code — `~/.claude/commands/odd-observe.md`, `~/.claude/agents/observe-run.md`, `~/.claude/skills/ Run detail — phases, turns, tokens | Model | Effort | CLI | oddyssey | Preflight | Drive | Observation | Turns | Median turn | Input | Output | Cache | Signals | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| `z-ai/glm-5.3-flashx` | medium | opencode | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | +| `z-ai/glm-5.3-flashx` | default | opencode | 1.13.0 | 3m10s | 2m01s | 11m58s | 32 | 13.3s | 2.3M | 68k | 2.1M | 4/4 | | `openai/gpt-5.6-luna` | medium | copilot | 1.13.0 | 1m02s | 2m01s | 3m26s | 38 | 2.8s | 2.9M | 17k | 2.9M | 4/4 | | `openai/gpt-5.6-terra` | medium | copilot | 1.13.0 | 0m36s | 2m01s | 3m21s | 26 | 3.2s | 2.4M | 13k | 2.4M | 4/4 | | `openai/gpt-5.6-sol` | medium | copilot | 1.13.0 | 1m21s | 2m00s | 6m11s | 48 | 4.2s | 3.7M | 28k | 3.5M | 4/4 | | `openai/gpt-6-sol` | high | copilot | 1.13.0 | 2m05s | 2m03s | 7m41s | 51 | 5.1s | 3.9M | 29k | 3.9M | 4/4 | | `google/gemini-3.7-flash` | medium | opencode | 1.13.0 | 2m24s | 2m02s | 5m51s | 90 | 4.3s | 6.5M | 32k | 5.8M | 4/4 | | `openai/gpt-6-sol` | medium | copilot | 1.13.0 | 1m38s | 2m03s | 6m42s | 44 | 5.9s | 3.3M | 24k | 3.3M | 4/4 | -| `z-ai/glm-5.3` | medium | opencode | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | +| `z-ai/glm-5.3` | default | opencode | 1.13.0 | 3m17s | 2m02s | 14m25s | 44 | 9.4s | 4.4M | 121k | 4.0M | 4/4 | +| `anthropic/claude-opus-5.5` | medium | claude | 1.13.0 | 1m12s | 2m01s | 9m15s | 49 | 2.7s | 5.5M | 56k | 5.5M | 4/4 | +| `anthropic/claude-opus-5.5` | high | claude | 1.13.0 | 2m17s | 2m19s | 12m53s | 63 | 2.5s | 7.5M | 82k | 7.5M | 4/4 | | `openai/gpt-6-luna` | medium | copilot | 1.13.0 | 1m03s | 2m01s | 22m30s | 64 | 7.6s | 7.7M | 138k | 7.7M | 4/4 | -| `deepseek/deepseek-v4.1-flash` | medium | opencode | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | +| `openai/gpt-6-luna-pro` | medium | opencode | 1.13.0 | 2m11s | 2m02s | 21m33s | 43 | 24.1s | 15.9M | 167k | 15.0M | 4/4 | +| `deepseek/deepseek-v4.1-flash` | default | opencode | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | +| `x-ai/grok-4.7` | medium | opencode | 1.13.0 | 3m55s | 2m02s | 10m35s | 43 | 7.6s | 3.3M | 57k | 2.9M | 4/4 | | `openai/gpt-6-luna` | high | copilot | 1.13.0 | 2m04s | 2m01s | 21m01s | 57 | 10.9s | 6.6M | 165k | 6.6M | 4/4 | | `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | 9m58s | 2m03s | 8m38s | 148 | 4.4s | 12.8M | 59k | 11.1M | 4/4 | +| `z-ai/glm-5.3-prime` | high | opencode | 1.13.0 | 2m00s | 2m30s | 13m22s | 48 | 11.3s | 3.9M | 83k | 3.7M | 4/4 | +| `x-ai/grok-4.7` | high | opencode | 1.13.0 | 4m32s | 2m02s | 17m20s | 41 | 14.5s | 3.7M | 86k | 3.1M | 4/4 | +| `openai/gpt-6-sol-pro` | medium | opencode | 1.13.0 | 2m18s | 2m01s | 6m34s | 34 | 11.5s | 5.0M | 38k | 4.1M | 4/4 | | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | +| `openai/gpt-6-sol-pro` | high | opencode | 1.13.0 | 3m45s | 2m02s | 13m16s | 46 | 17.6s | 8.2M | 82k | 7.0M | 4/4 | | `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | 2m52s | 2m02s | 12m14s | 36 | 3.6s | 3.6M | 60k | 3.6M | 4/4 | -| `z-ai/glm-5.3-flash` | medium | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | +| `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | +| `z-ai/glm-5.3-prime` | default | opencode | 1.13.0 | 4m41s | 2m01s | 16m27s | 49 | 17.7s | 5.3M | 125k | 5.0M | 4/4 | | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | +| `openai/gpt-6-luna-pro` | high | opencode | 1.13.0 | 8m27s | 2m02s | 59m58s | 70 | 40.9s | 28.0M | 505k | 25.6M | 4/4 | Token counts are rounded; the exact figures are in each run's pull request. Input includes the cached share, so Input and Cache overlap by @@ -71,7 +91,7 @@ design. - **Confirmed / reported** is the grade: how many of the findings the model reported held up when checked against the telemetry it cited and the code it accused. 3 / 3 beats 4 / 12. Anomalies and telemetry gaps both count; a restatement counts once; a row bundling several defects counts once per defect. - **Telemetry / Perf / Behavior** splits the confirmed findings by kind. - **$/confirmed** is what one trustworthy finding costs. -- **Effort** is the reasoning effort the CLI ran the model at (`low`, `medium`, `high`, ...), `medium` unless the run asked for another. +- **Effort** is the reasoning effort the CLI ran the model at (`low`, `medium`, `high`, ...), `medium` unless the run asked for another. `default` means the CLI offers the model no variant at the requested level, so the model ran at its provider's default effort. - **CLI** is the coding-agent CLI the mission ran in; its version is in the row's pull request. Model, effort and CLI identify a row; the oddyssey version does not, a new run replaces the row. - **Signals**: how many of metrics, traces, logs and profiles the run queried. Not part of the grade, the context to read it in. - **Preflight / Drive / Observation**: the drive is the scenario's fixed two minutes; a long preflight is a model that is lost, a long observation a model that is thorough. **Turns** and **median turn** separate groping (many short turns) from slow answering (few long ones). From d8ec1e06778f66a0d0aacef1eef8e17d2cc0fc54 Mon Sep 17 00:00:00 2001 From: using-system Date: Fri, 25 Sep 2026 16:51:24 +0200 Subject: [PATCH 3/3] docs(bench): drop the gpt-6 pro rows and glm-5.3-prime at default effort Removes openai/gpt-6-luna-pro (medium and high), openai/gpt-6-sol-pro (high) and z-ai/glm-5.3-prime (default) from both results tables on the maintainer's decision, and renumbers the ranking (24 rows). Refs #650 Co-Authored-By: Claude Opus 5.5 (1M context) --- .llms-benchmark/README.md | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-) diff --git a/.llms-benchmark/README.md b/.llms-benchmark/README.md index 983f982..f1951db 100644 --- a/.llms-benchmark/README.md +++ b/.llms-benchmark/README.md @@ -27,23 +27,19 @@ One row per model, effort and CLI, always its latest run. | **#9** | `anthropic/claude-opus-5.5` | medium | claude | 1.13.0 | 17 / 20 | 8 / 4 / 5 | 12m28s | $3.61 | $0.212 | | **#10** | `anthropic/claude-opus-5.5` | high | claude | 1.13.0 | 19 / 21 | 11 / 5 / 3 | 17m29s | $4.59 | $0.242 | | **#11** | `openai/gpt-6-luna` | medium | copilot | 1.13.0 | **11 / 11** | 6 / 3 / 2 | 25m34s | $0.18 | $0.016 | -| **#12** | `openai/gpt-6-luna-pro` | medium | opencode | 1.13.0 | 11 / 13 | 6 / 2 / 3 | 25m46s | $0.59 | $0.053 | -| **#13** | `deepseek/deepseek-v4.1-flash` | default | opencode | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **$0.010** | -| **#14** | `x-ai/grok-4.7` | medium | opencode | 1.13.0 | 10 / 14 | 5 / 4 / 1 | 16m32s | $1.99 | $0.199 | -| **#15** | `openai/gpt-6-luna` | high | copilot | 1.13.0 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | $0.018 | -| **#16** | `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | $0.191 | -| **#17** | `z-ai/glm-5.3-prime` | high | opencode | 1.13.0 | 12 / 13 | 6 / 3 / 3 | 17m52s | $3.42 | $0.285 | -| **#18** | `x-ai/grok-4.7` | high | opencode | 1.13.0 | 10 / 14 | 5 / 3 / 2 | 23m54s | $2.70 | $0.270 | -| **#19** | `openai/gpt-6-sol-pro` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 10m53s | $4.25 | $0.387 | -| **#20** | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | -| **#21** | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | -| **#22** | `openai/gpt-6-sol-pro` | high | opencode | 1.13.0 | 14 / 15 | 8 / 3 / 3 | 19m03s | $7.34 | $0.525 | -| **#23** | `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | **17 / 17** | 10 / 5 / 2 | 17m08s | $7.55 | $0.444 | -| **#24** | `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | -| **#25** | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | -| **#26** | `z-ai/glm-5.3-prime` | default | opencode | 1.13.0 | 7 / 8 | 4 / 3 / 0 | 23m09s | $4.73 | $0.675 | -| **#27** | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 | -| **#28** | `openai/gpt-6-luna-pro` | high | opencode | 1.13.0 | 10 / 11 | 6 / 3 / 1 | 70m27s | $1.32 | $0.132 | +| **#12** | `deepseek/deepseek-v4.1-flash` | default | opencode | 1.13.0 | **16 / 16** | 10 / 3 / 3 | 29m05s | $0.16 | **$0.010** | +| **#13** | `x-ai/grok-4.7` | medium | opencode | 1.13.0 | 10 / 14 | 5 / 4 / 1 | 16m32s | $1.99 | $0.199 | +| **#14** | `openai/gpt-6-luna` | high | copilot | 1.13.0 | 10 / 15 | 7 / 3 / 0 | 25m07s | $0.18 | $0.018 | +| **#15** | `google/gemini-3.8-flash` | medium | opencode | 1.13.0 | **12 / 12** | 6 / 4 / 2 | 20m39s | $2.29 | $0.191 | +| **#16** | `z-ai/glm-5.3-prime` | high | opencode | 1.13.0 | 12 / 13 | 6 / 3 / 3 | 17m52s | $3.42 | $0.285 | +| **#17** | `x-ai/grok-4.7` | high | opencode | 1.13.0 | 10 / 14 | 5 / 3 / 2 | 23m54s | $2.70 | $0.270 | +| **#18** | `openai/gpt-6-sol-pro` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 10m53s | $4.25 | $0.387 | +| **#19** | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 15 / 16 | 8 / 4 / 3 | 30m25s | $1.50 | $0.100 | +| **#20** | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | **17 / 17** | 8 / 6 / 3 | 19m31s | $6.15 | $0.362 | +| **#21** | `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | **17 / 17** | 10 / 5 / 2 | 17m08s | $7.55 | $0.444 | +| **#22** | `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 7 / 8 | 3 / 4 / 0 | 32m15s | **$0.10** | $0.014 | +| **#23** | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | **6 / 6** | 2 / 3 / 1 | 13m55s | $3.37 | $0.561 | +| **#24** | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | **11 / 11** | 5 / 3 / 3 | 40m25s | $1.62 | $0.147 |
Run detail — phases, turns, tokens @@ -61,7 +57,6 @@ One row per model, effort and CLI, always its latest run. | `anthropic/claude-opus-5.5` | medium | claude | 1.13.0 | 1m12s | 2m01s | 9m15s | 49 | 2.7s | 5.5M | 56k | 5.5M | 4/4 | | `anthropic/claude-opus-5.5` | high | claude | 1.13.0 | 2m17s | 2m19s | 12m53s | 63 | 2.5s | 7.5M | 82k | 7.5M | 4/4 | | `openai/gpt-6-luna` | medium | copilot | 1.13.0 | 1m03s | 2m01s | 22m30s | 64 | 7.6s | 7.7M | 138k | 7.7M | 4/4 | -| `openai/gpt-6-luna-pro` | medium | opencode | 1.13.0 | 2m11s | 2m02s | 21m33s | 43 | 24.1s | 15.9M | 167k | 15.0M | 4/4 | | `deepseek/deepseek-v4.1-flash` | default | opencode | 1.13.0 | 5m59s | 2m02s | 21m04s | 73 | 11.9s | 8.0M | 94k | 7.4M | 4/4 | | `x-ai/grok-4.7` | medium | opencode | 1.13.0 | 3m55s | 2m02s | 10m35s | 43 | 7.6s | 3.3M | 57k | 2.9M | 4/4 | | `openai/gpt-6-luna` | high | copilot | 1.13.0 | 2m04s | 2m01s | 21m01s | 57 | 10.9s | 6.6M | 165k | 6.6M | 4/4 | @@ -71,13 +66,10 @@ One row per model, effort and CLI, always its latest run. | `openai/gpt-6-sol-pro` | medium | opencode | 1.13.0 | 2m18s | 2m01s | 6m34s | 34 | 11.5s | 5.0M | 38k | 4.1M | 4/4 | | `qwen/qwen3.8-max-0902` | medium | opencode | 1.13.0 | 4m04s | 2m01s | 24m20s | 34 | 22.5s | 2.8M | 67k | 2.5M | 4/4 | | `anthropic/claude-opus-5` | medium | claude | 1.13.0 | 2m47s | 2m02s | 14m42s | 53 | 6.6s | 5.8M | 62k | 5.8M | 4/4 | -| `openai/gpt-6-sol-pro` | high | opencode | 1.13.0 | 3m45s | 2m02s | 13m16s | 46 | 17.6s | 8.2M | 82k | 7.0M | 4/4 | | `anthropic/claude-fable-5.1` | medium | claude | 1.12.0 ⚠︎ | 2m52s | 2m02s | 12m14s | 36 | 3.6s | 3.6M | 60k | 3.6M | 4/4 | | `z-ai/glm-5.3-flash` | default | opencode | 1.13.0 | 6m42s | 2m03s | 23m30s | 34 | 22.3s | 2.3M | 74k | 1.8M | 4/4 | | `anthropic/claude-sonnet-5` | medium | claude | 1.13.0 | 2m36s | 2m02s | 9m17s | 84 | 2.0s | 10.6M | 52k | 10.6M | 4/4 | -| `z-ai/glm-5.3-prime` | default | opencode | 1.13.0 | 4m41s | 2m01s | 16m27s | 49 | 17.7s | 5.3M | 125k | 5.0M | 4/4 | | `qwen/qwen3.8-27b` | medium | opencode | 1.13.0 | 3m49s | 2m00s | 34m36s | 48 | 16.6s | 5.9M | 133k | 3.7M | 4/4 | -| `openai/gpt-6-luna-pro` | high | opencode | 1.13.0 | 8m27s | 2m02s | 59m58s | 70 | 40.9s | 28.0M | 505k | 25.6M | 4/4 | Token counts are rounded; the exact figures are in each run's pull request. Input includes the cached share, so Input and Cache overlap by