From d80ace22369b080579a5900404a50545a628827d Mon Sep 17 00:00:00 2001 From: Jean Mertz Date: Fri, 18 Sep 2026 22:52:56 +0200 Subject: [PATCH] chore(tickets): File harness-design findings `docs/ticket/` gains a record of what the coding-harness study at arXiv:2609.20804 establishes, how each of its three ablated components (planning, action space, context management) maps onto JP's turn loop, and a SWOT drawn from the comparison. Six proposal tickets follow from it and are linked from its closing section. The comparison's sharpest finding is that JP runs the `jp query` path with no context management at all: `jp_llm::window::truncate_to_fit` has two production call sites, `jp_llm::title` and `jp_cli`'s inquiry backend, and neither is the query path. `git log -S` confirms it arrived with the inquiry work and was never wired to `query`, so the absence is original rather than a regression. That gap is filed separately from the compaction proposal it surfaced, because it holds whether or not RFD D50 ever lands. The remaining proposals cover bounded tool output (promote RFD D24), staged elision before summarization in RFD D50, per-response token usage as a conversation event, a turn-scoped cycle cap with identical-call streak detection, and an open question about whether a plan belongs in conversation history. The usage ticket is ranked ahead of the tuning work it enables: JP records no token counts today, so every other proposal here is currently unfalsifiable. Signed-off-by: Jean Mertz --- ...gs-from-arxiv-2609-20804-measured-again.md | 201 ++++++++++++++++++ ...-bounded-tool-output-for-implementation.md | 46 ++++ ...fits-a-conversation-to-the-model-s-cont.md | 60 ++++++ ...nse-token-usage-as-a-conversation-event.md | 63 ++++++ ...summarization-in-d50-s-automatic-compac.md | 74 +++++++ ...all-cycles-and-notice-repeated-identica.md | 64 ++++++ ...-a-plan-belongs-in-conversation-history.md | 57 +++++ 7 files changed, 565 insertions(+) create mode 100644 docs/ticket/0n0nwz3-harness-design-findings-from-arxiv-2609-20804-measured-again.md create mode 100644 docs/ticket/0n0pcr0-promote-d24-and-rank-bounded-tool-output-for-implementation.md create mode 100644 docs/ticket/0n0pex2-the-query-path-never-fits-a-conversation-to-the-model-s-cont.md create mode 100644 docs/ticket/0n0pgw6-record-per-response-token-usage-as-a-conversation-event.md create mode 100644 docs/ticket/0n0pjkk-stage-elision-before-summarization-in-d50-s-automatic-compac.md create mode 100644 docs/ticket/0n0pmyh-bound-a-turn-s-tool-call-cycles-and-notice-repeated-identica.md create mode 100644 docs/ticket/0n0ppbn-decide-whether-a-plan-belongs-in-conversation-history.md diff --git a/docs/ticket/0n0nwz3-harness-design-findings-from-arxiv-2609-20804-measured-again.md b/docs/ticket/0n0nwz3-harness-design-findings-from-arxiv-2609-20804-measured-again.md new file mode 100644 index 000000000..d4615dbe1 --- /dev/null +++ b/docs/ticket/0n0nwz3-harness-design-findings-from-arxiv-2609-20804-measured-again.md @@ -0,0 +1,201 @@ +# Harness-design findings from arXiv 2609.20804, measured against JP + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=llm +- **Label**: domain=tooling +- **Label**: type=task + +"An Empirical Study of Harness Design for Coding Agents" +(, Fan et al., September 2026) ablates three +coding-harness components while holding the execution loop fixed: planning, +action space, and context management. 176 matched settings, four models +(Nemotron-3 30B/120B/550B, Mistral-Medium-3.5-128B), two benchmarks (SWE-Bench +Verified, Terminal-Bench 2.1), four context-window budgets from 32k to 128k. + +This ticket records what the paper establishes, how JP's harness compares, and +where the comparison says JP should spend effort. +The proposals it produced are filed separately and listed at the bottom. + +## What the paper establishes + +1. **Context management's value scales inversely with the window budget, and + almost all of it comes from preventing overflow termination.** Every managed + tier overflowed on zero tasks at every budget. + The unmanaged tier overflowed on 78.7% of SWE-Bench tasks at 32k and 8.7% at + 128k. + Accuracy differences between managed tiers are small; the difference between + managed and unmanaged is not. +2. **Staging cheap elision before expensive summarization wins on cost at equal + accuracy.** Their T4 (elide at a soft threshold, summarize at a hard one) had + the lowest mean cost in seven of eight model/benchmark panels and the lowest + peak-context ratio at all four budgets. +3. **Model-facing recall is dead machinery.** 56% of the settings exposing a + `recall_event` tool never called it; the mean falls to 0.007 calls per task + at 128k; all 16 ablation settings recorded zero. + No accuracy gain over elision alone. +4. **Planning and the action space are model-conditional.** Planning is an + accuracy scaffold for weak models (+11.6 points for the 30B) and a cost saver + for strong ones (roughly 30% cheaper, about 2 points less accurate). + Predefined tools scaffold bash-weak models; bash-only is cheaper and more + accurate for bash-capable ones. + +### What it does not establish + +- The action-space intervention is bundled: tool availability, interface + prompts, file-state tracking, and post-edit diagnostics all vary together. + The paper says so in its own limitations. + It is not evidence that read-before-write or post-edit diagnostics help. +- One run per setting, and Terminal-Bench has 89 tasks, so most Terminal-Bench + contrasts are not significant. + Direction is usable; magnitude is not. +- Every crossover point sits at the weak end of its capability axis. + JP's users run frontier models, where the paper's own data shows planning and + predefined tools helping least. + +## How JP compares + +### Context management + +| Paper | JP | State | +| -------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------- | +| T0 (no management, overflow terminates) | the `jp query` path | shipped behavior | +| M1 elision (stub bulky stale tool observations) | `ToolCallPolicy::Strip { request, response }` with a `PolicySpec` `over` bound | exists, manual only | +| M2 recall (`recall_event` tool) | absent; `--reset` plus always-preserved raw events | correct by design | +| M3 summarization (running summary) | `SummaryPolicy` | exists, manual only | +| Soft / hard thresholds (0.6 / 0.85), verbatim recent window (0.3, floor 2) | none | RFD D50 proposes a single `trigger_ratio` | +| Per-result cap (24k chars) | none | RFD D24 | +| Truncation fallback | `window::truncate_to_fit` | exists; two call sites, neither on the query path | + +JP has every mechanism the paper tested. +It has no trigger. + +### Action space and safety + +| Paper | JP | Verdict | +| ---------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ----------- | +| Compile-time choice: predefined set or bash-only | every tool is a TOML declaration under `conversation.tools`; one built-in (`describe_tools`) | JP stronger | +| Full description in the tool schema, always | `summary` in the schema, full description and examples via `describe_tools` | JP stronger | +| Permission: allow / ask / deny | `RunMode::{Ask, Unattended, Edit, Skip}` plus access grants (RFD 076) | JP stronger | +| Workspace guard, pre- and post-symlink | `jp_tool::AccessPolicy`, pre- and post-canonical | comparable | +| Read-before-write with a content hash | none | absent | +| Post-edit diagnostics | none | absent | +| Stuck detection (5 identical warn, 8 identical fail abort) | none | absent | +| Step budget (300 per task) | none; `TurnState::request_count` is incremented and never read | absent | +| Parallel read-only tools, capped at 8 | all tools, in parallel, uncapped | different | +| Cost and token accounting | provider usage parsed into wire types and discarded | absent | +| Trajectories reconstructed from logs by an LLM judge | durable event stream, stable event IDs (RFD 097), turn markers | JP stronger | + +## SWOT + +### Strengths + +- **The action space is a config axis, not a code axis.** The paper's + action-space finding is that the right tool set depends on the model, and + their harness bakes the choice in at compile time. + JP expresses either condition as config, per conversation and per persona. + What the paper reports as a finding, JP already treats as a knob. +- **Recall is a human operation, not a tool.** RFD 064 made raw events + recoverable by the user and never by the model. + The paper's clearest negative result is that the model-facing version goes + unused. + JP put the boundary in the right place. +- **Truncation is prompt-cache-aware.** `truncate_to_fit` rounds its drop to 10% + of target to keep the prefix stable across calls. + The paper does not model caching at all, which makes its cost figures + optimistic in shape for any harness that thrashes the cache. +- **The trajectory is a first-class durable artifact.** The paper spent an + appendix and an LLM judge reconstructing what JP records natively. +- **Human-in-the-loop is a designed axis.** Inquiries (RFD 005, RFD 028), the + interrupt ladder (RFD 045, RFD 092), the permission model. + The paper's harness has one escape hatch: kill the run. + +### Weaknesses + +- **No context management on the query path.** The failure the paper measures + most directly, and T-0de0hry is in-repo evidence it already bites. +- **Unbounded tool output.** RFD D24 records a 1,293,623-token request against a + 1,000,000 limit, durably persisted. +- **No loop bound and no repetition detection.** The turn loop cycles streaming + to executing with no cap. + The guards are a per-response byte ceiling, an idle timeout, and Ctrl-C. +- **No usage accounting.** Every proposal below is unfalsifiable without it. + The paper's contribution is that it measured; JP cannot. +- **The harness knows nothing about what its tools do.** No concept of "a file + was edited", so no read-before-write, no post-edit diagnostics, no per-turn + touched-file set. + This is the price of the declarative action space, and it is a real price. + +### Opportunities + +- Adopt the paper's tuned constants instead of deriving them: 0.6 and 0.85 + thresholds, a 0.3 recent-window budget floored at two turns, 5 and 8 streak + thresholds. +- The recall result retires a question and saves the work. +- Per-model action-space presets are config in JP and impossible in the paper's + harness. + +### Threats + +- **Model mismatch.** Every crossover point in the paper sits at the weak end of + its capability axis. + In its own strong-model cells, planning reduces accuracy slightly and + predefined tools add cost. +- **Autonomy mismatch.** The paper optimizes for finishing without a human. + Importing its shapes wholesale imports that assumption. + Stuck detection that kills a run is wrong for JP; stuck detection that asks + the user is right. + +## What not to build + +- **A model-facing recall tool.** The evidence against it is strong and JP's + boundary is better. + Worth checking RFD D39 is not recreating it under another name. +- **A benchmark suite.** RFD D65 is a test harness, which is a different thing + and worth having. + A SWE-Bench-style evaluation is not: effects are conditional on model + capability, and JP's users change models faster than a suite stays calibrated. + Instrument real sessions instead. +- **A hard-coded file tool set** to obtain read-before-write and post-edit + diagnostics. + The paper does not establish that those help, and hard-coding them collapses + JP's best structural property. + If the gap ever bites, the orthogonal move is a declarative effect annotation + on tool configs, not a built-in tool set. + +## Proposals + +Ordered by leverage per unit of cost. +The first two attack the failure the paper measures most directly; the third is +what makes any of them checkable. + +1. **T-0n0pcr0**: promote D24 and rank bounded tool output. + The cheapest change, and it sits upstream of the rest: a single oversized + response poisons a conversation permanently, and no compaction trigger + rescues one that has already happened. +2. **T-0n0pjkk**: stage elision before summarization in D50's automatic + compaction. + JP has both policies already; this is about which fires when, and about + borrowing the paper's thresholds rather than guessing new ones. +3. **T-0n0pgw6**: record per-response token usage as a conversation event. + Build alongside 2, not after. + Without it, tuning a threshold is guesswork and no claim about the harness + can be checked. +4. **T-0n0pmyh**: bound a turn's tool-call cycles and notice repeated identical + calls. + Small, and the vestigial `TurnState::request_count` names the missing check. + JP should route a streak to the user rather than nag the model. +5. **T-0n0ppbn**: decide whether a plan belongs in conversation history. + Weakest-supported of the set, and explicitly gated on 3. + Filed to keep the question findable. + +One gap fell out of 2 and is filed on its own, because it holds independently of +whether automatic compaction ever lands: + +- **T-0n0pex2**: the query path never fits a conversation to the model's context + window. + `truncate_to_fit` has two call sites and neither is `query`. + D50 assumes this backstop exists. diff --git a/docs/ticket/0n0pcr0-promote-d24-and-rank-bounded-tool-output-for-implementation.md b/docs/ticket/0n0pcr0-promote-d24-and-rank-bounded-tool-output-for-implementation.md new file mode 100644 index 000000000..0d6a24a4c --- /dev/null +++ b/docs/ticket/0n0pcr0-promote-d24-and-rank-bounded-tool-output-for-implementation.md @@ -0,0 +1,46 @@ +# Promote D24 and rank bounded tool output for implementation + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=tooling +- **Label**: package=jp_cli +- **Label**: type=task + +JP places no ceiling on a tool call response. +`commit_tool_responses` writes whatever the tool produced into the stream and +flushes it, so an oversized response is durable: every later turn re-sends it, +the provider rejects the request, and the conversation needs hand-editing to +recover. +RFD D24 records the case that prompted it, a 1,293,623-token request against a +1,000,000-token limit. + +D24 has been a Draft since 2026-07-27 and sits unranked in the backlog. + +## Why now + +The harness study in T-0n0nwz3 measures what context overflow costs an agent: +with no context management, 78.7% of SWE-Bench Verified tasks terminated on +window overflow at a 32k budget, and 8.7% still did at 128k. +Every managed tier overflowed on zero tasks at every budget. + +Their harness truncates each tool result at 24k characters as part of the fixed +substrate, below the tiers they varied. +It is not one of the interventions; it is the floor the interventions stand on. +JP has no such floor. + +Of the changes that reduce context pressure, this is the cheapest: one config +key resolved through the existing per-tool and `'*'` chain, no new mechanism, no +new axis. + +## What to do + +1. Promote D24 to Discussion. +2. Rank it. + It gates the value of everything else done about context pressure: an + automatic compaction trigger that fires on a conversation already poisoned by + a single 1.2M-token response has nothing useful to do. + +Findings and the rest of the proposals: T-0n0nwz3. diff --git a/docs/ticket/0n0pex2-the-query-path-never-fits-a-conversation-to-the-model-s-cont.md b/docs/ticket/0n0pex2-the-query-path-never-fits-a-conversation-to-the-model-s-cont.md new file mode 100644 index 000000000..281a352ce --- /dev/null +++ b/docs/ticket/0n0pex2-the-query-path-never-fits-a-conversation-to-the-model-s-cont.md @@ -0,0 +1,60 @@ +# The query path never fits a conversation to the model's context window + +- **Status**: Todo +- **Kind**: Feature +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=llm +- **Label**: package=jp_cli +- **Label**: package=jp_llm +- **Label**: type=enhancement + +`jp query` builds a `Thread` from the full conversation stream and hands it to +the provider. +Nothing between the stream and the wire checks it against +`ModelDetails::context_window`. + +`jp_llm::window::truncate_to_fit` exists and documents itself as the entry point +for "fitting a conversation into a model's context window". +It has two production call sites: `jp_llm::title` for title generation and +`jp_cli::cmd::query::tool::inquiry` for inquiry sub-requests. +Neither is the query path. +It was introduced for inquiries in #441 and reused for titles in #895; nothing +removed it from `query`, it was never there. + +## What it costs + +The common case is recoverable and clearly signalled: the provider returns +`ContextWindowExceeded` and the user runs `jp conversation compact`. +That is why this is a gap rather than a defect. + +Two cases are worse: + +- A provider that clamps instead of rejecting produces no signal at all. + See T-0de0hry: Cerebras shortens completions as the window fills and the user + is told nothing. +- RFD D50 (automatic compaction) delegates the single-turn overflow case back to + this path: "A single turn that overflows on its own remains the domain of + hard-fail and truncation." + That backstop does not exist, so D50's own reasoning has a hole in it. + +## Scope worth settling + +Whether dropping the oldest events is the right shape here. +`truncate_to_fit` drops from the front, which on a long session removes the +original request and the early exploration while keeping the most recent tool +output. +The harness study in T-0n0nwz3 keeps the preamble and a verbatim recent window +and compacts only the middle, preferring to stub bulky tool observations before +dropping anything. +JP's compaction policies (RFD 064) already express that shape, so a query-path +backstop could reuse them rather than the blunt drop. + +The related decision is ordering. +With an automatic compaction trigger in place this is a true backstop that +rarely runs; without one it is the only mechanism. +Landing them in either order works. +Landing neither leaves the provider error as the only guard. + +Findings and the rest of the proposals: T-0n0nwz3. diff --git a/docs/ticket/0n0pgw6-record-per-response-token-usage-as-a-conversation-event.md b/docs/ticket/0n0pgw6-record-per-response-token-usage-as-a-conversation-event.md new file mode 100644 index 000000000..b480c3189 --- /dev/null +++ b/docs/ticket/0n0pgw6-record-per-response-token-usage-as-a-conversation-event.md @@ -0,0 +1,63 @@ +# Record per-response token usage as a conversation event + +- **Status**: Todo +- **Kind**: Feature +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=llm +- **Label**: package=jp_conversation +- **Label**: package=jp_llm +- **Label**: type=feature + +Providers report token counts on every response and some report cost. +JP carries none of it into anything durable. +`jp_openrouter::responses::Usage` and `jp_openrouter::types::response::Usage` +exist as wire types with `input_tokens`, `output_tokens` and `cost`; nothing +lifts them into `jp_llm::Event` or the conversation stream. +No other provider's usage is read at all. + +## What it costs + +JP cannot answer "did that change help?" for any change to the harness. +Every other proposal in T-0n0nwz3 is unfalsifiable without this: there is no way +to tune a compaction threshold, judge whether a turn budget fires too early, or +tell whether moving a plan out of history saved anything. + +The harness study's whole contribution is that it measured. +It reports cost per task, peak context as a fraction of the window, turns, and +tool calls per task, and every one of its findings is a comparison between those +numbers. +JP has the richer substrate, a durable event stream with stable event IDs (RFD +097), and none of the measurements. + +Day to day, a user also has no way to see what a conversation has cost. + +## Shape + +A `Usage` event kind on the conversation stream, appended per provider response. +As an event it inherits durability, provider-invisibility under projection, and +the whole `jp conversation` read surface without new machinery. + +Fields worth carrying: input, output, cache-read and cache-write token counts, +the resolved model ID, and cost where the provider reports it. + +Where the pieces belong: + +- Normalization in `jp_llm::Event`, because providers disagree on shape and the + disagreement should not leak past the provider boundary. +- The event type in `jp_conversation::event`. +- Recording in the turn loop, alongside the existing mid-turn flush. + +T-0fedmkq needs the same plumbing from the other end: it wants the served +service tier recorded alongside the turn, and notes that the OpenRouter provider +never sets the request's `usage` flag, so `Usage.cost` arrives unpopulated. +Worth doing together. + +## Caveat + +This is a signal, not a target. +Goodhart applies the moment a number like "tokens per turn" becomes something to +optimize directly. + +Findings and the rest of the proposals: T-0n0nwz3. diff --git a/docs/ticket/0n0pjkk-stage-elision-before-summarization-in-d50-s-automatic-compac.md b/docs/ticket/0n0pjkk-stage-elision-before-summarization-in-d50-s-automatic-compac.md new file mode 100644 index 000000000..53e7630ed --- /dev/null +++ b/docs/ticket/0n0pjkk-stage-elision-before-summarization-in-d50-s-automatic-compac.md @@ -0,0 +1,74 @@ +# Stage elision before summarization in D50's automatic compaction + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=llm +- **Label**: package=jp_config +- **Label**: package=jp_conversation +- **Label**: type=task + +RFD D50 adds automatic compaction with a single `trigger_ratio` and a single +standing rule. +The harness study in T-0n0nwz3 measured that shape against a staged one, and the +staged one won. + +Their T4 elides at a soft threshold and summarizes only at a hard one. +It had the lowest mean cost in seven of eight model/benchmark panels and the +lowest peak-context ratio at all four window budgets, at success rates +comparable to every other managed tier. +The mechanism is that cheap rule-based elision handles most of the pressure +before an LLM summarization call is needed at all. +Summarization alone (their T3) and elision alone (T1, T2) both cost more. + +## What changes in D50 + +JP already has both policies. +This is about which fires when. + +D50's config today: + +``` +trigger_ratio = 0.75 + +[conversation.compaction.auto.rule] +keep_first = 1 +keep_last = 3 +reasoning = "strip" +tool_calls = "strip" +``` + +The staged form needs two thresholds and two rules: an elision rule (`tool_calls += "strip"` with an `over` bound, which is exactly their M1) at the soft +threshold, and a summary rule (their M3) at the hard one. +Their values are 0.6 and 0.85 of the usable window, with the verbatim recent +window budgeted at 0.3 and floored at two turns. + +Adopting their constants is worth more than deriving new ones. +D50 currently notes that 0.75 "is a starting guess". + +## Two other changes worth making + +- **Budget the recent window by size, not only by turn count.** D50's `keep_last + = 3` protects three turns whatever they weigh. + The paper budgets the verbatim window by tokens and floors it at two turns, + which covers the case where the protected tail alone exceeds the budget. + That case is D50's own open question, "what if the projection is still over + the threshold after compacting". +- **The backstop D50 assumes does not exist.** D50 says a single turn that + overflows on its own "remains the domain of hard-fail and truncation". + The query path has no truncation. + See T-0n0pex2. + +## Ranking + +D50 sits at 24 on the priority board, below the Internal Release v0.1 milestone. +On this evidence it belongs higher. +The paper's strongest result is that the gap between no context management and +any context management is large, while the gaps among managed strategies are +small: 35.7 percentage points of success rate at 32k, still 2.7 at 128k, against +single-digit differences between tiers. + +Findings and the rest of the proposals: T-0n0nwz3. diff --git a/docs/ticket/0n0pmyh-bound-a-turn-s-tool-call-cycles-and-notice-repeated-identica.md b/docs/ticket/0n0pmyh-bound-a-turn-s-tool-call-cycles-and-notice-repeated-identica.md new file mode 100644 index 000000000..38591225c --- /dev/null +++ b/docs/ticket/0n0pmyh-bound-a-turn-s-tool-call-cycles-and-notice-repeated-identica.md @@ -0,0 +1,64 @@ +# Bound a turn's tool-call cycles and notice repeated identical calls + +- **Status**: Todo +- **Kind**: Feature +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=tooling +- **Label**: package=jp_cli +- **Label**: type=feature + +The turn loop cycles streaming to executing with no cap. +`run_turn_loop` breaks only when the execution plan is empty, the user +intervenes, or a stream error is fatal. +Nothing counts the cycles, and nothing notices a repeated identical tool call. + +`TurnState::request_count` is incremented at `turn_loop.rs:373` and read +nowhere. +Its doc comment describes a check that does not exist: "Every retry increments +this counter, until a maximum number of retries is reached, after which the turn +ends in an error." + +The guards today are a per-response byte ceiling +(`assistant.request.max_response_bytes`), a per-stream idle timeout, and Ctrl-C. +All three are per-request. +None bounds a turn. + +## What it costs + +A model that reissues the same failing call grinds until the user stops it, on +the user's own API key. +Unattended runs (`run = "unattended"`, a persona driving a long task) have no +user watching. + +The harness study in T-0n0nwz3 caps each task at 300 steps and adds streak +detection: a reminder once a streak reaches five identical calls, and early +termination at eight consecutive identical failing calls. +A streak is calls sharing a tool name and byte-identical arguments, so a +paginated read at a new offset ends it, and permission denials do not count as +failures. +The machinery is small and the thresholds are theirs to borrow. + +## Shape + +Two counters on `TurnState`, which already exists and already half-holds one: + +- A cycle cap, configurable, `0` meaning unbounded. +- A streak counter over `(tool_name, arguments)` byte-identity, excluding + permission denials. + +## Where JP should diverge + +Do not inject a reminder into the model input. +That is the paper's only option; JP has better ones. +In an attended session a streak should reach the user through the inquiry or +interrupt surface, where they can redirect rather than watch. +In an unattended one it should abort. +RFD 104's promptability signal is how the loop tells those apart. + +Picking either threshold is guesswork until per-turn usage is recorded +(T-0n0pgw6), so the counters are worth landing before the numbers are argued +about. + +Findings and the rest of the proposals: T-0n0nwz3. diff --git a/docs/ticket/0n0ppbn-decide-whether-a-plan-belongs-in-conversation-history.md b/docs/ticket/0n0ppbn-decide-whether-a-plan-belongs-in-conversation-history.md new file mode 100644 index 000000000..2e7d957bf --- /dev/null +++ b/docs/ticket/0n0ppbn-decide-whether-a-plan-belongs-in-conversation-history.md @@ -0,0 +1,57 @@ +# Decide whether a plan belongs in conversation history + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=llm +- **Label**: package=jp_cli +- **Label**: type=question + +This workspace's `plan` tool (`.jp/mcp/tools/plan.toml`) writes its state into +tool call responses, which live in the conversation stream. +Every call leaves a copy, so a long task accumulates stale plans in history, and +the built-in compaction rule (`tool_calls = "strip"`) targets exactly those +events. + +The harness study in T-0n0nwz3 deliberately does neither. +Its planning component holds the plan in external state and re-injects the +current version before each model call, "rather than appending previous copies +to the persistent trajectory". + +## Why it might matter + +Their planning result for the two strongest models is a cost result: roughly 30% +cheaper on SWE-Bench, with success rates 2.0 and 0.4 percentage points lower. +The saving is almost entirely post-edit verification, not localization or +repair. +Those are the cells that correspond to JP's users, who run frontier models. + +Their weak-model result (+11.6 points for Nemotron-3 30B, at higher cost) does +not transfer. + +## Why it might not + +- The evidence is one prompt and one update mechanism, ablated only at their + default T4/128k setting. + The paper says so in its limitations. +- JP is interactive. + The human is often the planner, and the value of a machine-maintained plan is + correspondingly lower. +- Re-injection needs a slot in `Thread` for per-request ephemeral content that + does not exist today. + It would have to sit after the cached prefix so prompt caching survives, which + is a constraint the paper never faced. + +## What to do + +Nothing yet. +Record per-turn usage first (T-0n0pgw6), then look at whether verification churn +and accumulated stale plan copies show up in real sessions. +If they do, the design question is where per-request ephemeral content lives in +`Thread`, which is worth an RFD rather than a patch. + +Filed to keep the question findable, not to schedule it. + +Findings and the rest of the proposals: T-0n0nwz3.