diff --git a/docs/ticket/0n0nwz3-harness-design-findings-from-arxiv-2609-20804-measured-again.md b/docs/ticket/0n0nwz3-harness-design-findings-from-arxiv-2609-20804-measured-again.md new file mode 100644 index 000000000..d4615dbe1 --- /dev/null +++ b/docs/ticket/0n0nwz3-harness-design-findings-from-arxiv-2609-20804-measured-again.md @@ -0,0 +1,201 @@ +# Harness-design findings from arXiv 2609.20804, measured against JP + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=llm +- **Label**: domain=tooling +- **Label**: type=task + +"An Empirical Study of Harness Design for Coding Agents" +(, Fan et al., September 2026) ablates three +coding-harness components while holding the execution loop fixed: planning, +action space, and context management. 176 matched settings, four models +(Nemotron-3 30B/120B/550B, Mistral-Medium-3.5-128B), two benchmarks (SWE-Bench +Verified, Terminal-Bench 2.1), four context-window budgets from 32k to 128k. + +This ticket records what the paper establishes, how JP's harness compares, and +where the comparison says JP should spend effort. +The proposals it produced are filed separately and listed at the bottom. + +## What the paper establishes + +1. **Context management's value scales inversely with the window budget, and + almost all of it comes from preventing overflow termination.** Every managed + tier overflowed on zero tasks at every budget. + The unmanaged tier overflowed on 78.7% of SWE-Bench tasks at 32k and 8.7% at + 128k. + Accuracy differences between managed tiers are small; the difference between + managed and unmanaged is not. +2. **Staging cheap elision before expensive summarization wins on cost at equal + accuracy.** Their T4 (elide at a soft threshold, summarize at a hard one) had + the lowest mean cost in seven of eight model/benchmark panels and the lowest + peak-context ratio at all four budgets. +3. **Model-facing recall is dead machinery.** 56% of the settings exposing a + `recall_event` tool never called it; the mean falls to 0.007 calls per task + at 128k; all 16 ablation settings recorded zero. + No accuracy gain over elision alone. +4. **Planning and the action space are model-conditional.** Planning is an + accuracy scaffold for weak models (+11.6 points for the 30B) and a cost saver + for strong ones (roughly 30% cheaper, about 2 points less accurate). + Predefined tools scaffold bash-weak models; bash-only is cheaper and more + accurate for bash-capable ones. + +### What it does not establish + +- The action-space intervention is bundled: tool availability, interface + prompts, file-state tracking, and post-edit diagnostics all vary together. + The paper says so in its own limitations. + It is not evidence that read-before-write or post-edit diagnostics help. +- One run per setting, and Terminal-Bench has 89 tasks, so most Terminal-Bench + contrasts are not significant. + Direction is usable; magnitude is not. +- Every crossover point sits at the weak end of its capability axis. + JP's users run frontier models, where the paper's own data shows planning and + predefined tools helping least. + +## How JP compares + +### Context management + +| Paper | JP | State | +| -------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------- | +| T0 (no management, overflow terminates) | the `jp query` path | shipped behavior | +| M1 elision (stub bulky stale tool observations) | `ToolCallPolicy::Strip { request, response }` with a `PolicySpec` `over` bound | exists, manual only | +| M2 recall (`recall_event` tool) | absent; `--reset` plus always-preserved raw events | correct by design | +| M3 summarization (running summary) | `SummaryPolicy` | exists, manual only | +| Soft / hard thresholds (0.6 / 0.85), verbatim recent window (0.3, floor 2) | none | RFD D50 proposes a single `trigger_ratio` | +| Per-result cap (24k chars) | none | RFD D24 | +| Truncation fallback | `window::truncate_to_fit` | exists; two call sites, neither on the query path | + +JP has every mechanism the paper tested. +It has no trigger. + +### Action space and safety + +| Paper | JP | Verdict | +| ---------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ----------- | +| Compile-time choice: predefined set or bash-only | every tool is a TOML declaration under `conversation.tools`; one built-in (`describe_tools`) | JP stronger | +| Full description in the tool schema, always | `summary` in the schema, full description and examples via `describe_tools` | JP stronger | +| Permission: allow / ask / deny | `RunMode::{Ask, Unattended, Edit, Skip}` plus access grants (RFD 076) | JP stronger | +| Workspace guard, pre- and post-symlink | `jp_tool::AccessPolicy`, pre- and post-canonical | comparable | +| Read-before-write with a content hash | none | absent | +| Post-edit diagnostics | none | absent | +| Stuck detection (5 identical warn, 8 identical fail abort) | none | absent | +| Step budget (300 per task) | none; `TurnState::request_count` is incremented and never read | absent | +| Parallel read-only tools, capped at 8 | all tools, in parallel, uncapped | different | +| Cost and token accounting | provider usage parsed into wire types and discarded | absent | +| Trajectories reconstructed from logs by an LLM judge | durable event stream, stable event IDs (RFD 097), turn markers | JP stronger | + +## SWOT + +### Strengths + +- **The action space is a config axis, not a code axis.** The paper's + action-space finding is that the right tool set depends on the model, and + their harness bakes the choice in at compile time. + JP expresses either condition as config, per conversation and per persona. + What the paper reports as a finding, JP already treats as a knob. +- **Recall is a human operation, not a tool.** RFD 064 made raw events + recoverable by the user and never by the model. + The paper's clearest negative result is that the model-facing version goes + unused. + JP put the boundary in the right place. +- **Truncation is prompt-cache-aware.** `truncate_to_fit` rounds its drop to 10% + of target to keep the prefix stable across calls. + The paper does not model caching at all, which makes its cost figures + optimistic in shape for any harness that thrashes the cache. +- **The trajectory is a first-class durable artifact.** The paper spent an + appendix and an LLM judge reconstructing what JP records natively. +- **Human-in-the-loop is a designed axis.** Inquiries (RFD 005, RFD 028), the + interrupt ladder (RFD 045, RFD 092), the permission model. + The paper's harness has one escape hatch: kill the run. + +### Weaknesses + +- **No context management on the query path.** The failure the paper measures + most directly, and T-0de0hry is in-repo evidence it already bites. +- **Unbounded tool output.** RFD D24 records a 1,293,623-token request against a + 1,000,000 limit, durably persisted. +- **No loop bound and no repetition detection.** The turn loop cycles streaming + to executing with no cap. + The guards are a per-response byte ceiling, an idle timeout, and Ctrl-C. +- **No usage accounting.** Every proposal below is unfalsifiable without it. + The paper's contribution is that it measured; JP cannot. +- **The harness knows nothing about what its tools do.** No concept of "a file + was edited", so no read-before-write, no post-edit diagnostics, no per-turn + touched-file set. + This is the price of the declarative action space, and it is a real price. + +### Opportunities + +- Adopt the paper's tuned constants instead of deriving them: 0.6 and 0.85 + thresholds, a 0.3 recent-window budget floored at two turns, 5 and 8 streak + thresholds. +- The recall result retires a question and saves the work. +- Per-model action-space presets are config in JP and impossible in the paper's + harness. + +### Threats + +- **Model mismatch.** Every crossover point in the paper sits at the weak end of + its capability axis. + In its own strong-model cells, planning reduces accuracy slightly and + predefined tools add cost. +- **Autonomy mismatch.** The paper optimizes for finishing without a human. + Importing its shapes wholesale imports that assumption. + Stuck detection that kills a run is wrong for JP; stuck detection that asks + the user is right. + +## What not to build + +- **A model-facing recall tool.** The evidence against it is strong and JP's + boundary is better. + Worth checking RFD D39 is not recreating it under another name. +- **A benchmark suite.** RFD D65 is a test harness, which is a different thing + and worth having. + A SWE-Bench-style evaluation is not: effects are conditional on model + capability, and JP's users change models faster than a suite stays calibrated. + Instrument real sessions instead. +- **A hard-coded file tool set** to obtain read-before-write and post-edit + diagnostics. + The paper does not establish that those help, and hard-coding them collapses + JP's best structural property. + If the gap ever bites, the orthogonal move is a declarative effect annotation + on tool configs, not a built-in tool set. + +## Proposals + +Ordered by leverage per unit of cost. +The first two attack the failure the paper measures most directly; the third is +what makes any of them checkable. + +1. **T-0n0pcr0**: promote D24 and rank bounded tool output. + The cheapest change, and it sits upstream of the rest: a single oversized + response poisons a conversation permanently, and no compaction trigger + rescues one that has already happened. +2. **T-0n0pjkk**: stage elision before summarization in D50's automatic + compaction. + JP has both policies already; this is about which fires when, and about + borrowing the paper's thresholds rather than guessing new ones. +3. **T-0n0pgw6**: record per-response token usage as a conversation event. + Build alongside 2, not after. + Without it, tuning a threshold is guesswork and no claim about the harness + can be checked. +4. **T-0n0pmyh**: bound a turn's tool-call cycles and notice repeated identical + calls. + Small, and the vestigial `TurnState::request_count` names the missing check. + JP should route a streak to the user rather than nag the model. +5. **T-0n0ppbn**: decide whether a plan belongs in conversation history. + Weakest-supported of the set, and explicitly gated on 3. + Filed to keep the question findable. + +One gap fell out of 2 and is filed on its own, because it holds independently of +whether automatic compaction ever lands: + +- **T-0n0pex2**: the query path never fits a conversation to the model's context + window. + `truncate_to_fit` has two call sites and neither is `query`. + D50 assumes this backstop exists. diff --git a/docs/ticket/0n0pcr0-promote-d24-and-rank-bounded-tool-output-for-implementation.md b/docs/ticket/0n0pcr0-promote-d24-and-rank-bounded-tool-output-for-implementation.md new file mode 100644 index 000000000..0d6a24a4c --- /dev/null +++ b/docs/ticket/0n0pcr0-promote-d24-and-rank-bounded-tool-output-for-implementation.md @@ -0,0 +1,46 @@ +# Promote D24 and rank bounded tool output for implementation + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=tooling +- **Label**: package=jp_cli +- **Label**: type=task + +JP places no ceiling on a tool call response. +`commit_tool_responses` writes whatever the tool produced into the stream and +flushes it, so an oversized response is durable: every later turn re-sends it, +the provider rejects the request, and the conversation needs hand-editing to +recover. +RFD D24 records the case that prompted it, a 1,293,623-token request against a +1,000,000-token limit. + +D24 has been a Draft since 2026-07-27 and sits unranked in the backlog. + +## Why now + +The harness study in T-0n0nwz3 measures what context overflow costs an agent: +with no context management, 78.7% of SWE-Bench Verified tasks terminated on +window overflow at a 32k budget, and 8.7% still did at 128k. +Every managed tier overflowed on zero tasks at every budget. + +Their harness truncates each tool result at 24k characters as part of the fixed +substrate, below the tiers they varied. +It is not one of the interventions; it is the floor the interventions stand on. +JP has no such floor. + +Of the changes that reduce context pressure, this is the cheapest: one config +key resolved through the existing per-tool and `'*'` chain, no new mechanism, no +new axis. + +## What to do + +1. Promote D24 to Discussion. +2. Rank it. + It gates the value of everything else done about context pressure: an + automatic compaction trigger that fires on a conversation already poisoned by + a single 1.2M-token response has nothing useful to do. + +Findings and the rest of the proposals: T-0n0nwz3. diff --git a/docs/ticket/0n0pex2-the-query-path-never-fits-a-conversation-to-the-model-s-cont.md b/docs/ticket/0n0pex2-the-query-path-never-fits-a-conversation-to-the-model-s-cont.md new file mode 100644 index 000000000..281a352ce --- /dev/null +++ b/docs/ticket/0n0pex2-the-query-path-never-fits-a-conversation-to-the-model-s-cont.md @@ -0,0 +1,60 @@ +# The query path never fits a conversation to the model's context window + +- **Status**: Todo +- **Kind**: Feature +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=llm +- **Label**: package=jp_cli +- **Label**: package=jp_llm +- **Label**: type=enhancement + +`jp query` builds a `Thread` from the full conversation stream and hands it to +the provider. +Nothing between the stream and the wire checks it against +`ModelDetails::context_window`. + +`jp_llm::window::truncate_to_fit` exists and documents itself as the entry point +for "fitting a conversation into a model's context window". +It has two production call sites: `jp_llm::title` for title generation and +`jp_cli::cmd::query::tool::inquiry` for inquiry sub-requests. +Neither is the query path. +It was introduced for inquiries in #441 and reused for titles in #895; nothing +removed it from `query`, it was never there. + +## What it costs + +The common case is recoverable and clearly signalled: the provider returns +`ContextWindowExceeded` and the user runs `jp conversation compact`. +That is why this is a gap rather than a defect. + +Two cases are worse: + +- A provider that clamps instead of rejecting produces no signal at all. + See T-0de0hry: Cerebras shortens completions as the window fills and the user + is told nothing. +- RFD D50 (automatic compaction) delegates the single-turn overflow case back to + this path: "A single turn that overflows on its own remains the domain of + hard-fail and truncation." + That backstop does not exist, so D50's own reasoning has a hole in it. + +## Scope worth settling + +Whether dropping the oldest events is the right shape here. +`truncate_to_fit` drops from the front, which on a long session removes the +original request and the early exploration while keeping the most recent tool +output. +The harness study in T-0n0nwz3 keeps the preamble and a verbatim recent window +and compacts only the middle, preferring to stub bulky tool observations before +dropping anything. +JP's compaction policies (RFD 064) already express that shape, so a query-path +backstop could reuse them rather than the blunt drop. + +The related decision is ordering. +With an automatic compaction trigger in place this is a true backstop that +rarely runs; without one it is the only mechanism. +Landing them in either order works. +Landing neither leaves the provider error as the only guard. + +Findings and the rest of the proposals: T-0n0nwz3. diff --git a/docs/ticket/0n0pgw6-record-per-response-token-usage-as-a-conversation-event.md b/docs/ticket/0n0pgw6-record-per-response-token-usage-as-a-conversation-event.md new file mode 100644 index 000000000..b480c3189 --- /dev/null +++ b/docs/ticket/0n0pgw6-record-per-response-token-usage-as-a-conversation-event.md @@ -0,0 +1,63 @@ +# Record per-response token usage as a conversation event + +- **Status**: Todo +- **Kind**: Feature +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=llm +- **Label**: package=jp_conversation +- **Label**: package=jp_llm +- **Label**: type=feature + +Providers report token counts on every response and some report cost. +JP carries none of it into anything durable. +`jp_openrouter::responses::Usage` and `jp_openrouter::types::response::Usage` +exist as wire types with `input_tokens`, `output_tokens` and `cost`; nothing +lifts them into `jp_llm::Event` or the conversation stream. +No other provider's usage is read at all. + +## What it costs + +JP cannot answer "did that change help?" for any change to the harness. +Every other proposal in T-0n0nwz3 is unfalsifiable without this: there is no way +to tune a compaction threshold, judge whether a turn budget fires too early, or +tell whether moving a plan out of history saved anything. + +The harness study's whole contribution is that it measured. +It reports cost per task, peak context as a fraction of the window, turns, and +tool calls per task, and every one of its findings is a comparison between those +numbers. +JP has the richer substrate, a durable event stream with stable event IDs (RFD +097), and none of the measurements. + +Day to day, a user also has no way to see what a conversation has cost. + +## Shape + +A `Usage` event kind on the conversation stream, appended per provider response. +As an event it inherits durability, provider-invisibility under projection, and +the whole `jp conversation` read surface without new machinery. + +Fields worth carrying: input, output, cache-read and cache-write token counts, +the resolved model ID, and cost where the provider reports it. + +Where the pieces belong: + +- Normalization in `jp_llm::Event`, because providers disagree on shape and the + disagreement should not leak past the provider boundary. +- The event type in `jp_conversation::event`. +- Recording in the turn loop, alongside the existing mid-turn flush. + +T-0fedmkq needs the same plumbing from the other end: it wants the served +service tier recorded alongside the turn, and notes that the OpenRouter provider +never sets the request's `usage` flag, so `Usage.cost` arrives unpopulated. +Worth doing together. + +## Caveat + +This is a signal, not a target. +Goodhart applies the moment a number like "tokens per turn" becomes something to +optimize directly. + +Findings and the rest of the proposals: T-0n0nwz3. diff --git a/docs/ticket/0n0pjkk-stage-elision-before-summarization-in-d50-s-automatic-compac.md b/docs/ticket/0n0pjkk-stage-elision-before-summarization-in-d50-s-automatic-compac.md new file mode 100644 index 000000000..53e7630ed --- /dev/null +++ b/docs/ticket/0n0pjkk-stage-elision-before-summarization-in-d50-s-automatic-compac.md @@ -0,0 +1,74 @@ +# Stage elision before summarization in D50's automatic compaction + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=llm +- **Label**: package=jp_config +- **Label**: package=jp_conversation +- **Label**: type=task + +RFD D50 adds automatic compaction with a single `trigger_ratio` and a single +standing rule. +The harness study in T-0n0nwz3 measured that shape against a staged one, and the +staged one won. + +Their T4 elides at a soft threshold and summarizes only at a hard one. +It had the lowest mean cost in seven of eight model/benchmark panels and the +lowest peak-context ratio at all four window budgets, at success rates +comparable to every other managed tier. +The mechanism is that cheap rule-based elision handles most of the pressure +before an LLM summarization call is needed at all. +Summarization alone (their T3) and elision alone (T1, T2) both cost more. + +## What changes in D50 + +JP already has both policies. +This is about which fires when. + +D50's config today: + +``` +trigger_ratio = 0.75 + +[conversation.compaction.auto.rule] +keep_first = 1 +keep_last = 3 +reasoning = "strip" +tool_calls = "strip" +``` + +The staged form needs two thresholds and two rules: an elision rule (`tool_calls += "strip"` with an `over` bound, which is exactly their M1) at the soft +threshold, and a summary rule (their M3) at the hard one. +Their values are 0.6 and 0.85 of the usable window, with the verbatim recent +window budgeted at 0.3 and floored at two turns. + +Adopting their constants is worth more than deriving new ones. +D50 currently notes that 0.75 "is a starting guess". + +## Two other changes worth making + +- **Budget the recent window by size, not only by turn count.** D50's `keep_last + = 3` protects three turns whatever they weigh. + The paper budgets the verbatim window by tokens and floors it at two turns, + which covers the case where the protected tail alone exceeds the budget. + That case is D50's own open question, "what if the projection is still over + the threshold after compacting". +- **The backstop D50 assumes does not exist.** D50 says a single turn that + overflows on its own "remains the domain of hard-fail and truncation". + The query path has no truncation. + See T-0n0pex2. + +## Ranking + +D50 sits at 24 on the priority board, below the Internal Release v0.1 milestone. +On this evidence it belongs higher. +The paper's strongest result is that the gap between no context management and +any context management is large, while the gaps among managed strategies are +small: 35.7 percentage points of success rate at 32k, still 2.7 at 128k, against +single-digit differences between tiers. + +Findings and the rest of the proposals: T-0n0nwz3. diff --git a/docs/ticket/0n0pmyh-bound-a-turn-s-tool-call-cycles-and-notice-repeated-identica.md b/docs/ticket/0n0pmyh-bound-a-turn-s-tool-call-cycles-and-notice-repeated-identica.md new file mode 100644 index 000000000..38591225c --- /dev/null +++ b/docs/ticket/0n0pmyh-bound-a-turn-s-tool-call-cycles-and-notice-repeated-identica.md @@ -0,0 +1,64 @@ +# Bound a turn's tool-call cycles and notice repeated identical calls + +- **Status**: Todo +- **Kind**: Feature +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=tooling +- **Label**: package=jp_cli +- **Label**: type=feature + +The turn loop cycles streaming to executing with no cap. +`run_turn_loop` breaks only when the execution plan is empty, the user +intervenes, or a stream error is fatal. +Nothing counts the cycles, and nothing notices a repeated identical tool call. + +`TurnState::request_count` is incremented at `turn_loop.rs:373` and read +nowhere. +Its doc comment describes a check that does not exist: "Every retry increments +this counter, until a maximum number of retries is reached, after which the turn +ends in an error." + +The guards today are a per-response byte ceiling +(`assistant.request.max_response_bytes`), a per-stream idle timeout, and Ctrl-C. +All three are per-request. +None bounds a turn. + +## What it costs + +A model that reissues the same failing call grinds until the user stops it, on +the user's own API key. +Unattended runs (`run = "unattended"`, a persona driving a long task) have no +user watching. + +The harness study in T-0n0nwz3 caps each task at 300 steps and adds streak +detection: a reminder once a streak reaches five identical calls, and early +termination at eight consecutive identical failing calls. +A streak is calls sharing a tool name and byte-identical arguments, so a +paginated read at a new offset ends it, and permission denials do not count as +failures. +The machinery is small and the thresholds are theirs to borrow. + +## Shape + +Two counters on `TurnState`, which already exists and already half-holds one: + +- A cycle cap, configurable, `0` meaning unbounded. +- A streak counter over `(tool_name, arguments)` byte-identity, excluding + permission denials. + +## Where JP should diverge + +Do not inject a reminder into the model input. +That is the paper's only option; JP has better ones. +In an attended session a streak should reach the user through the inquiry or +interrupt surface, where they can redirect rather than watch. +In an unattended one it should abort. +RFD 104's promptability signal is how the loop tells those apart. + +Picking either threshold is guesswork until per-turn usage is recorded +(T-0n0pgw6), so the counters are worth landing before the numbers are argued +about. + +Findings and the rest of the proposals: T-0n0nwz3. diff --git a/docs/ticket/0n0ppbn-decide-whether-a-plan-belongs-in-conversation-history.md b/docs/ticket/0n0ppbn-decide-whether-a-plan-belongs-in-conversation-history.md new file mode 100644 index 000000000..2e7d957bf --- /dev/null +++ b/docs/ticket/0n0ppbn-decide-whether-a-plan-belongs-in-conversation-history.md @@ -0,0 +1,57 @@ +# Decide whether a plan belongs in conversation history + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-09-18 +- **Label**: domain=conversation +- **Label**: domain=llm +- **Label**: package=jp_cli +- **Label**: type=question + +This workspace's `plan` tool (`.jp/mcp/tools/plan.toml`) writes its state into +tool call responses, which live in the conversation stream. +Every call leaves a copy, so a long task accumulates stale plans in history, and +the built-in compaction rule (`tool_calls = "strip"`) targets exactly those +events. + +The harness study in T-0n0nwz3 deliberately does neither. +Its planning component holds the plan in external state and re-injects the +current version before each model call, "rather than appending previous copies +to the persistent trajectory". + +## Why it might matter + +Their planning result for the two strongest models is a cost result: roughly 30% +cheaper on SWE-Bench, with success rates 2.0 and 0.4 percentage points lower. +The saving is almost entirely post-edit verification, not localization or +repair. +Those are the cells that correspond to JP's users, who run frontier models. + +Their weak-model result (+11.6 points for Nemotron-3 30B, at higher cost) does +not transfer. + +## Why it might not + +- The evidence is one prompt and one update mechanism, ablated only at their + default T4/128k setting. + The paper says so in its limitations. +- JP is interactive. + The human is often the planner, and the value of a machine-maintained plan is + correspondingly lower. +- Re-injection needs a slot in `Thread` for per-request ephemeral content that + does not exist today. + It would have to sit after the cached prefix so prompt caching survives, which + is a constraint the paper never faced. + +## What to do + +Nothing yet. +Record per-turn usage first (T-0n0pgw6), then look at whether verification churn +and accumulated stale plan copies show up in real sessions. +If they do, the design question is where per-request ephemeral content lives in +`Thread`, which is worth an RFD rather than a patch. + +Filed to keep the question findable, not to schedule it. + +Findings and the rest of the proposals: T-0n0nwz3.