Repository navigation
Conversation
…ncile
## Cache savings — wrong baseline (fixes ~3.5x underestimate)
Old formula in lib/resolve.py:
avg_input_price = cost_val / (input_toks + output_toks)
cache_savings = cache_read_toks * avg_input_price * 0.9
This was wrong on two counts:
1. cost_val is the POST-cache cost (already discounted by caching), so
using it as the per-token-input-rate baseline drags the rate down to
what was paid, not what the no-cache baseline would have cost.
2. Dividing by (input + output) mixes output tokens into the denominator,
diluting the per-token rate further.
For a real Sonnet run (497K cache reads, 50K fresh, 11K output, $0.46
total): old formula reported ~$0.38 saved. True savings vs no-cache
baseline: $1.34. The new formula uses LiteLLM's list pricing
(input_cost_per_token, cache_read_input_token_cost,
cache_creation_input_token_cost) directly:
read_savings = cache_read * (list_input_rate - cache_read_rate)
write_overhead = cache_write * (cache_write_rate - list_input_rate)
net = read_savings - write_overhead
Bug only existed in lib/resolve.py — other loops didn't compute savings.
## Distillation savings — overstated by ~10x for long runs
Old formula:
cost_saved = total_tokens_saved * input_cost_per_token
# where total_tokens_saved = (pre - post) * iterations
This treated every iteration as if those tokens would have cost the
full input rate. But after iteration 1, the un-distilled context would
have been a cache hit at 0.1x. New formula prices iter 1 at full rate
and iters 2..N at the cache-read rate.
## Status-log additions
- lib/reconcile.py now tracks cache_read_tokens / cache_creation_tokens
(was: no cache tracking at all) and includes the cache summary in its
status log comment.
- lib/resolve.py distillation summary is now built at status-log-post
time via the new helper, instead of from the dead-code stub left
behind in an earlier refactor.
## Shared helpers
New in lib/formatting.py:
- _model_rates(model) — pulls list + cache rates from LiteLLM with
sensible fallbacks (0.1x / 1.25x) when the cache rates are missing.
- build_cache_savings_summary(usage_path, model)
- build_distillation_summary(pre, post, iters, model)
Both helpers use plain round() instead of math.ceil() (the old code's
ceil-of-float-product produced $0.25 from a true $0.24 due to fp
imprecision).
## Tests
Adds tests/test_cache_distill_savings.py with 16 test cases covering
zero-cache, reads-only, writes-only, mixed reads/writes, write
overhead swamping reads, single-iter distillation, multi-iter
distillation, missing-model fallback, and a regression check that the
new formula doesn't overstate vs. the old naive one.
Full suite: 647 passed (was 631; +16).
Bridge-analysis at 340K tokens fell just over the 300K DISTILL_STRUCT_EXTRACT_LIMIT and skipped distillation entirely — even though the Tier 2 path (structural extract → identify relevant files → read those in full) would have worked fine for it. The structural extract is ~10-15% of total repo tokens (39K for bridge-analysis), so the LLM call inside Tier 2 sees a small input regardless of how big the underlying repo is. The 300K hard cap wasn't protecting anything meaningful for repos in the 300K-1M range. Removed it. Two tiers now: 1. <= 100K tokens (DISTILL_SMALL_REPO_LIMIT): send full codebase 2. otherwise: structural-extract path For very large codebases (~5-10M tokens, where the structural extract itself would overflow a 1M-token model context window) the outer try/except in maybe_distill catches the LLM error and returns the original un-distilled context as a graceful fallback. Added a test to lock in that behavior. Updated docstrings to describe the new two-tier structure, removed the unused DISTILL_STRUCT_EXTRACT_LIMIT constant, replaced the now- obsolete "skips at Tier 3" test with "uses Tier 2 path on larger repos", and added the explicit overflow-fallback test.
Workshop crashed on every run since commit e4209e7 (formatting.py extract) with: File ".remote-dev-bot/lib/workshop.py", line 31, in <module> from lib.formatting import _fmt_tok, ... ModuleNotFoundError: No module named 'lib' The workflow heredocs invoke library modules with `sys.path.insert(0, '.remote-dev-bot/lib')` — only the lib/ directory on sys.path, not the rdb repo root. Three modules used `from lib.X import ...` which requires the rdb root to be on sys.path so that `lib` resolves as a package by name: - lib/workshop.py (line 31, broken since e4209e7) - lib/cumulative_cost.py (line 19, silently broken — wrapped in try/except so cumulative cost tables silently went missing) - lib/design_loop.py (line 13, broken since lib/tools.py extract but never noticed because every callsite also crashed earlier on the workshop import) Changed all three to sibling-style `from X import ...` to match the heredoc style. Script entry points (resolve.py, reconcile.py, post_fallback_cost.py) are unaffected — they each take care of their own sys.path with a `sys.path.insert(0, os.path.dirname( os.path.dirname(__file__)))` so they work under either layout. Tests passed because pytest runs with `PYTHONPATH=.` which puts both the rdb root AND lib/ on the path, masking the bug. Added tests/test_production_imports.py which spawns a subprocess with only lib/ on sys.path (matching production heredoc), and asserts each library module imports cleanly. Locked in for all seven non-entry- point modules. Verified against the failing run: gnovak/bridge-analysis issue #393, workshop run 26121417319 — traceback line-for-line matches.
The workshop crash fixed in PR #623 was present during the v0.9.0 e2e run, but the workshop e2e assertion was lax: both "Workshop Stage 1/2 comment found" and "no workshop comment found" counted as PASS. The crash produced an "exited unexpectedly" fallback comment (which doesn't match the Stage 1/2 substrings) → 0 matches → PASS by the catch-all. That's the kind of false-pass we need e2e to never produce. Same lax pattern was in seven other places. Tightened all of them: design (line 818-825) PASS no "Design analysis" comment found → FAIL workshop (line 826-841) PASS no Stage 1/2 comment found → FAIL build (line 855-867) PASS PR created, no council review comment → FAIL (Council review IS the value-add of build mode over plain resolve; missing it = the test didn't exercise what build is FOR.) delegate (line 868-911) PASS PR created, pipeline complete but no council → FAIL PASS PR created, no pipeline-complete marker → FAIL (Both the council review AND the pipeline-complete marker are required signals that the full pipeline ran end-to-end.) rf-review (line 1041-1052) PASS no "Code review by" comment found → FAIL timeout (line 1063-1088) PASS run completed, no watchdog comment found → FAIL (The whole point of the timeout test is to verify the watchdog fires and posts a "could not fully resolve" / "exited unexpectedly" comment. No comment = the timeout path wasn't exercised.) wrapup (line 1146-1198) PASS run completed, no clear output found → FAIL (Plus: branch search pattern was `openhands-fix-issue-$N`, stale since the v0.6.0 OpenHands removal. Current branches are `rdb-fix-issue-$N-$alias`, so the "agent finished — PR created" branch never fired and the test had been silently coasting on the "no clear output" catch-all PASS for months.) reconcile (line 977-998) PASS force-pushed but no "Reconcile complete" comment → FAIL (Force-push without the completion comment means the workflow crashed between the push and the comment — not a real success.) All substring patterns verified against the actual strings the code emits (lib/workshop.py, lib/reconcile.py, .github/workflows/ remote-dev-bot.yml). No false-negative risk from grep-pattern drift. Pytest still 655 passing.
The most recent full-suite run failed the (now-tightened) timeout
assertion because the agent scope-reduced the vague refactor task and
finished in 50s instead of being killed by the 1-minute watchdog:
Tool: finish({'success': True, 'explanation': 'Created utils.py with
type hints and docstrings, and test_utils.py with 14 passing
unit tests.'})
The test had been silently coasting on the lax "no comment found =
PASS" assertion. PR #624 tightened it to FAIL, which surfaced this:
the watchdog isn't actually being exercised because the task is too
soft and the timeout too long.
## Fix
Two changes, both needed:
- **Concrete uncompressible task**: replace "refactor every file to
follow best practices..." with "create exactly 50 Python files
named day_01.py..day_50.py with this exact body." The agent can
no longer satisfy the requirement by writing one demonstrative
file; it's all 50 or explicit failure.
- **Shorter watchdog**: cut timeout_minutes from 1 to 0.5. Even
scope-reduced work doesn't fit in 30s. Required adding fractional
support: `timeout_minutes` is now `float` in lib/config.py's
ALLOWED_ARGS, and the bash arithmetic in the workflow's 5
watchdog steps changes from `$((TIMEOUT_MINUTES*60))` (bash
integer math, drops fractional values to 0) to
`awk -v m="$TIMEOUT_MINUTES" 'BEGIN{print int(m*60)}'`
(handles int and float, rounds down to whole seconds).
The float type also leaves the door open for users who want
timeout_minutes=1.5 or similar in production configs.
655 unit tests still pass.
Three independent fixes investigated for bridge-analysis #392, where distillation compressed leaderboard.py down to 972 tokens (entire file → ~snippets) and the status log misleadingly reported "5.6K → 972 tokens, ~$0.05 saved". ## 1. Tighten the distill prompt against over-excerpting When an issue references specific line numbers (e.g., "lines 312-313") the distill LLM was extracting tiny snippets and discarding the surrounding file. Prompt now has a hard rule: files to be modified are included in FULL, regardless of how specific the task is. Also addressed the specific line-number-excerpting failure mode in the prompt text. The user-message instructions and system prompt are now consistent on this (no more "[full content or relevant excerpt]" escape hatch). ## 2. Fix the metric (pre_tokens reflects what distillation saw) The status log read: "Distillation: 5.6K → 972 tokens (4.6K saved/iter × 30 iters)" That 5.6K was the size of EXTRA_FILES (README, AGENTS.md, etc.) — NOT what distillation processed. The codebase distillation actually scanned was 368K tokens. maybe_distill() now returns the codebase total as a 6th tuple element so resolve.py can pass the correct value into build_distillation_summary(). Bridge-analysis #392 same numbers, before/after this fix: OLD: 5.6K → 972 tokens, ~$0.05 saved NEW: 368K → 972 tokens, ~$4.30 saved ## 3. Model-dependent Tier-1-vs-Tier-2 cutoff DISTILL_SMALL_REPO_LIMIT was a hardcoded 100K, set when 200K-window models were typical. Modern models we configure (claude-sonnet-4-6, claude-opus-4-7, gpt-5.5, gemini-2.5-flash) all have 1M-token input windows, so we can comfortably send much larger codebases to a single Tier-1 LLM call. Added _small_repo_limit(model) which scales the cutoff to 25% of the model's input window, clamped to [50K, 500K]. Old 100K constant is the fallback for models LiteLLM doesn't know. Effective Tier-1 ceiling per model: - claude-sonnet-4-6, opus-4-7: 250K (1M × 25%) - gpt-5.5: 262K (1.05M × 25%) - gemini-2.5-flash: 262K - claude-sonnet-4-5 (200K win): 50K (floored) - unknown / not in LiteLLM: 100K (legacy default) 662 tests pass (was 655; +7 for _small_repo_limit + prompt content).
…e aggregate Per the design discussion: delegate should look like calling the individual steps in sequence. Per-step cost on Stage 1, 3, 3a, 3c already worked. What was wrong was the trailing "Delegate Stages 1-3c Aggregate Cost" block, which: 1. Used a bespoke label format different from individual /agent-* invocations. 2. Contained a per-step-format cost table (with bold **$X.XX**) that the cumulative-cost scanner would have picked up as a prior step cost — guaranteeing quadratic double-counting on any later /agent-* on the same issue. Replaced with: each per-step cost block in delegate now carries a canonical cumulative table appended (via compute_cumulative_table from lib/cumulative_cost.py — the same code individual invocations use). Cumulative is empty on Stage 1 (no prior cost markers); from Stage 3 onward it includes all prior delegate stages. The cumulative on the last reported stage IS the running total, no aggregate block needed. Council reviewer comments (Stage 2, 3b) intentionally remain cost-table-less — those costs are small and the user has decided they would just be noise. Their costs won't appear in the cumulative, which matches the "what shows in the cumulative is exactly what was reported per-step" rule. No new cost-reporting code; this just routes delegate through the existing per-step + cumulative path that resolve/design/review already use. The `_build_stage_cost_block` helper added in workshop.py is a thin wrapper that calls _build_cost_table + compute_cumulative_table. run_delegate gains two parameters (github_repo, issue_number) so the cumulative scanner can fetch prior comments. The workflow passes them in from the same env vars it already uses elsewhere. 662 tests pass.
Bridge-analysis #436 surfaced the failure mode: vague issue body ("we are getting to the point that I would like to launch this as a web app...") + detailed spec in /agent-delegate comments → agent scope-reduces to a tractable subset (schema + CLI), opens PR #437 (3 files, 40 LOC), declares victory. 85% of the spec dropped on the floor. The fix is NOT delegate-specific. Manual /agent-design → /agent-resolve would hit the same shortcut: the resolve system prompt biases toward small fixes ("a simple fix should take 5-10 iterations; a complex multi-file change rarely needs more than 20-30", "modify existing files rather than creating new ones", "call finish() as soon as the task is done"). For a spec-driven implementation those are actively wrong. ## What changes 1. New SCOPE section in the prompt (wired into build_system_prompt between AGENT_ROLE and the budget paragraph). Says: "If the comments contain a substantial design analysis, implementation spec, or output from a prior /agent-design, /agent-workshop, or /agent-delegate run, treat the most recent revised version as the binding contract for your work — NOT the issue body alone." Plus an explicit anti-shortcut nudge: "Scope-reducing to 'the foundational piece' and shipping a tiny PR is the wrong outcome — it leaves the issue effectively unresolved..." 2. Iteration-budget paragraph acknowledges spec-driven scope: "Implementing a detailed multi-component spec (see ## Scope above): 50-100+ iterations — every file, route, table, and test the spec lists has to be created. Stopping at 20-30 to ship a 'foundational' subset is the wrong call when a spec is the contract." 3. WORKFLOW step 3 ("Implement") softens "modify existing files rather than creating new ones" to apply only to bug fixes — for spec-driven work, the spec defines the file set. ## Why one prompt, not a delegate-specific one Delegate is a macro: it expands into the same agentic calls a user could invoke manually. The user's stated preference: delegate should look like calling the steps in sequence. If we put the scope rule in a delegate-only Stage 4 prompt, the two paths drift and the manual flow keeps shortcutting. Putting it in resolve's general prompt means it triggers for both, no separate prompt to keep in sync. 8 new tests pin the SCOPE wording + budget-paragraph update + the fact that SCOPE is actually wired into build_system_prompt's output. 670 unit tests total pass (was 662).
The parser already enabled council mode for /agent-review (lib/config.py:593 treats council=true as setting council_models output for review mode), but the `review:` workflow job had no council branch — it unconditionally ran the single-reviewer agentic loop. /agent-review with council=true parsed fine but produced exactly one review. Found on bridge-analysis PR #438 (issue #436): user invoked /agent-review council = true Got one review from claude-small only. ## Wiring - New "Run council code review" step in the review job, guarded `if: council_models != '[]' && council_models != ''`. Calls run_build_council (already used by build Stage 2 and delegate Stage 5) with /agent-review-flavored labels. - Existing "Run review loop" and "Post review comment" steps get the opposite guard so they only run in single-reviewer mode. - "Post cost comment (fallback)" stays unguarded; the new council step writes /tmp/llm_usage.json the same way single-reviewer does, so the fallback picks up cost in both modes. ## Label parameterization on run_build_council The function previously hard-coded "Build Stage 2 — Council Code Review" and "/agent-build Stage 2" in its banner / per-reviewer attribution / completion message. Build Stage 2 keeps the defaults. Delegate Stage 5 now passes explicit "Delegate Stage 5/6" labels (was already shipping duplicated banners — the workflow's pre-banner + the function's banner; this commit makes them deliberate). The new review-council path uses plain "Council Code Review" / "/agent-review council=true" labels so users see the command they actually invoked. 664 unit tests pass (was 662; +2 for label parameterization).
…tests Bridge-analysis PR #438 surfaced four reinforcing failure modes that were not addressed by the earlier SCOPE prompt (PR #628): 1. Spec said "BT+EB ranking", agent shipped EB-shrunk averages with no Bradley-Terry, hid the deviation in a docstring. Used 65/150 iterations — budget wasn't the constraint, behavior was. 2. Agent never ran the FastAPI app. Multiple bugs that uvicorn + curl would have surfaced in 30 seconds shipped anyway: 17 wrong-signature TemplateResponse calls, missing row_factory, the precompute function never wired into the pipeline. 3. PR body said "All 315 existing tests pass" — true but useless. Zero tests for the new code. No assertion that BT+EB was actually applied. 4. The old WORKFLOW language — "read only what you need", "complex multi-file change rarely needs more than 20-30 iterations" — was added when we were dropping tool calls due to cost. With caching at ~70-80% hit rate and distillation working, that pressure is gone, and the language now actively discourages reading the 1500-line reference notebook that needs porting. ## Five new sections / changes in build_system_prompt - **METHODOLOGY_FAITHFULNESS** (new): when the spec names an algorithm or references existing code as authoritative, the implementation must reproduce it to within rounding error. If you can't, call finish(success=False) — never ship a stand-in with a TODO. Explicitly addresses the spec-internal-inconsistency case (spec says functions are in module X, they're actually in notebooks/Y) — grep for the real location, don't write a stub. - **DEVIATION_REPORTING** (new): if you simplified, stubbed, or skipped anything, the PR body's FIRST paragraph must say so, with a specific format. Buried docstring comments don't count. - **TESTS** (new): write a test for every new function/class/module. Existing tests passing only shows you didn't break what was there. Load-bearing methodology claims need named tests (e.g., test_leaderboard_matches_bt_eb_reference_within_tolerance). - **END_TO_END** (new): run the thing before finish(success=True). Server tasks: uvicorn + curl, expect 200. CLI: invoke on realistic input. Library: smoke test the new function. This catches the bug class unit tests miss — wrong-signature calls, missing config wiring, never-invoked functions. - **WORKFLOW + _budget_paragraph + EFFICIENCY** (softened): drop the small-fix bias ("rarely needs more than 20-30"), drop "read only what you're about to change", explicitly contemplate algorithm- extraction tasks where reading the full reference is the work. 29 tests pin the new sections + assert the old anti-exploration language is gone. 693 unit tests total pass. Synthesized with the bridge-analysis agent's independent postmortem of PR #438. The bridge agent also flagged upstream causes (spec hallucinated file locations; spec used `...` code placeholders) — those will be addressed in companion PRs to design_loop and workshop.py spec generation, since they're not resolve-prompt issues.
…thodology tests Bridge-analysis PR #438 postmortem identified two failure modes that originate in the implementation-spec generation prompt, not the resolve prompt: 1. The spec said the BT functions were "already present in bridge_analysis" — they were in notebooks/leaderboard.py. The implementer agent tried the obvious import, hit ImportError, and wrote a stub. Spec hallucinating the file location is what set the stub-shipping in motion. 2. The spec used a `def _run_bt_eb_for_cohort(...): ...` template with `...` as the body. Templates with placeholder bodies read as "supply something reasonable here" — exactly how the simplified EB-shrunk-average stand-in got shipped. The fix in PR #631 catches this kind of thing downstream in the resolve prompt (methodology faithfulness, end-to-end verification). This PR fixes the upstream cause: the spec doesn't generate those landmines in the first place. ## SPEC_DESIGN_SYSTEM_PROMPT additions - **Verify every file/function reference before citing it.** Use grep / read_file. If the design says "in module X" but it's actually "in notebooks/Y", write the spec with the correct path. - **Do NOT use `...` placeholders in code templates.** Either spell out the body sufficiently for mechanical fill-in, or write a structured semantic specification that names the canonical reference (path/file.py:line) plus precondition / postcondition / tolerance. - **Tests for load-bearing methodology claims.** Any "uses X algorithm" claim must come with a named acceptance test in the test strategy (e.g., test_leaderboard_matches_bt_eb_reference_within_tolerance). Without it, the methodology is a label, not a contract. ## SPEC_REVISION_SYSTEM_PROMPT additions The revised spec MUST preserve these three properties even if council feedback would weaken them. A reviewer saying "just write 'use the existing impl'" is wrong — preserve the explicit citation and the named test. ## Tests 10 new tests in tests/test_spec_generation_prompts.py pin the prompt content. 682 unit tests total pass. Synthesized with the bridge-analysis agent's postmortem. Companion PR #631 hardens resolve. Design loop prompt change still to come.
…hodology claims Third PR in the trio addressing bridge-analysis PR #438's failure modes. PR #631 hardened resolve. PR #632 hardened spec generation. This PR hardens the design loop — the earliest point in the pipeline where methodology claims and file references first get written down. If the design hallucinates "the existing logic lives in module X" when it's actually in notebooks/Y, that error propagates: into the spec, into the implementation, eventually into a shipped stub. Catching it at the design stage is cheaper than catching it in resolve. ## DEFAULT_SYSTEM_PROMPT additions - **Verify every file/function reference before citing it.** Use grep or read_file. If the canonical impl lives in a notebook, say so explicitly — don't pretend it's in a clean module. - **Methodology claims need named acceptance tests.** Any "uses BT+EB" / "applies BH-FDR" claim in the design must come with a proposed acceptance test in the analysis (e.g., test_leaderboard_matches_bt_eb_reference_within_tolerance with a tolerance value). Without it, the methodology is a label not a contract, and the implementer agent can ship a stand-in that the existing test suite won't catch. ## Tests 4 new tests in TestDesignPromptFidelityRules. 676 total pass. Same diagnostic synthesis as PR #631 and #632 — derived from the bridge-analysis agent's postmortem + my analysis. The three together address the failure mode at every layer it could be caught: design generation (this PR), spec generation (#632), implementation (#631).
Sequential-subagent review of rdb done 2026-06-11 against dev @ de72856 (immediately after PRs #631-633 merged). Archived here so it's accessible from anywhere, not just one workstation, and so future reviews have a template. Layout: comprehensive-review/ README.md — index + navigator 2026-06-11/ SUMMARY.md — the synthesis / entry point findings-*.md — per-phase incremental findings (5 files) PROCESS.md — how the review was conducted AGENT-PROMPTS.md — subagent prompts used per phase Findings headline (full detail in SUMMARY.md): Tier 1 (ship-stoppers): pr_title shell injection (resolve.py:1104); extra_instructions interpolated into Python heredocs in 5 council/loop steps; install.md curls a deleted yaml template; test.yml never runs on PRs to dev. Tier 2 (real degradations): ContextWindowExceededError recovery is dead code; distillation silently broken in /agent-design (heredoc unpacks 5 values from a 6-tuple); design_max_iterations / review_max_iterations declared in config but missing from parse outputs; reconcile job missing context-config env plumbing. Highest-leverage next feature: close the council loop. PR #438 showed all 3 council reviewers caught the methodology shortcut while the pipeline self-reported ✅ complete — the council is the working defense against premature victory and it's currently advisory only. No code changes in this PR; just the review record.
Check in the 2026-06-11 comprehensive review
Four fixes in lib/resolve.py, all from the 2026-06-11 comprehensive review (see comprehensive-review/2026-06-11/). 1. create_pr now builds the gh command as an argv list (no shell) instead of interpolating the PR title into a shell=True string with only double-quote escaping. The title is LLM-authored from issue text, so shell metacharacters could reach the runner. argv form removes the shell entirely. 2. Reordered the completion() exception handlers so ContextWindowExceededError is caught before BadRequestError. CWE subclasses BadRequestError, so the old order made the context-overflow recovery (emergency trim + graceful wrap-up) unreachable dead code; overflows died with a generic "Bad request". 3. Cache-write token tracking read getattr(prompt_details, "cache_creation_input_tokens"), but litellm's PromptTokensDetailsWrapper field is cache_creation_tokens. Verified against the installed litellm (model_fields has cache_creation_tokens, not the _input_ variant). The old name always read 0, so cache-write tokens never appeared in status logs and cache-savings figures were overstated. 4. The two "no tool calls for 3 consecutive iterations" break paths now write_status() before breaking. The post-loop code only writes a status when the loop ends naturally, so these breaks previously ended the run with no failure explanation at all. 707 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Harden create_pr against title injection + three resolve.py loop bugs
Four fixes in .github/workflows/remote-dev-bot.yml from the 2026-06-11 comprehensive review (T1.2, T2.2, T2.3, T2.4): 1. Route extra_instructions / model_extra_instructions / model / alias through step `env:` blocks instead of interpolating them as Python string literals inside heredocs (8 sites: build council, review loop, review council, design loop, workshop, delegate stages 1-3, delegate stage-5 council). The resolve steps already used this pattern; the heredoc interpolation crashed on multi-line block-scalar values (the documented config format) and let target-repo config inject Python. 2. Fix the design-job maybe_distill() unpack: the heredoc unpacked 5 values but lib/distill.py returns a 6-tuple, so the swallowed ValueError silently disabled distillation for every /agent-design run. Now matches lib/design_loop.py's 6-value unpack. 3. Declare design_max_iterations and review_max_iterations as parse-job outputs. config.py emits both to GITHUB_OUTPUT but the outputs block never re-exported them, so design/review loops always ran the hardcoded default 10 and per-invocation max_iterations overrides were silently ignored (design's configured budget is 15, and its wrapup fires at 12 — unreachable in a 10-iteration loop). 4. Plumb CONTEXT_KEEP_TOOL_RESULTS / BASH_OUTPUT_LIMIT / MAX_CONTEXT_TOKENS / COMPACTION_COVERAGE / COMPACTION_FACTOR into the reconcile step env. lib/reconcile.py consumes all five but the job never set them, so reconcile ran with truncation=8000 and cache-defeating keep_n=10 against the config-mandated 0s. Behavior changes: design runs get distillation and their configured iteration budget back; reconcile gets its configured context settings; multi-line extra_instructions no longer crashes council/review/design/ workshop/delegate steps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Route config strings through workflow step env + fix parse-output/env plumbing
resolve.py and reconcile.py place a moving-tail cache_control marker on the last message before each API call (restored in #621 after the #606 regression); design_loop never got the fix, so standalone /agent-design and workshop/delegate design stages (Stage 1, Stage 3a) run at 0% cache hit on Anthropic models and pay full input price every iteration. Port the same strip-and-place logic, provider-gated with an explicit TTL for Gemini, matching the sibling loops verbatim. Closes the last Tier-3 sibling-drift item from the 2026-06-11 comprehensive review. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolve: use shared read/write-aware trim_tool_results
Delegate: fix Stage 3 blocking, design-stage wrapup, distillation cost totals
Design loop: add moving-tail prompt-cache markers
…oops The three loops had drifted into three implementations of the same nudge: resolve re-injected every iteration past the threshold, reconcile injected exactly once (==), design_loop injected once via flag (>=). The 2026-06-11 comprehensive review flagged the divergence (SUMMARY.md Tier 3); resolve's repeated re-injection was confirmed as deliberate escalation — a single nudge gets buried under subsequent tool results. Unify all three on resolve's semantics: inject every iteration once iteration + 1 >= threshold. reconcile switches == to >=; design_loop drops its injected-once flag. Message content stays per-loop (each already recomputes the remaining-iteration count per injection). The deliberate-escalation rationale is now documented at all three sites. Behavior change: reconcile and design_loop agents past the threshold now receive the wrap-up nudge every iteration instead of once. With the moving-tail cache markers now in all three loops, the appended nudges are cheap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Unify wrapup injection on every-iteration escalation across the three loops
Stage 6 has been an agentic code revision since #521 (April 2026): the workflow re-invokes resolve.py on the PR branch with the Stage 5 council reviews and design/spec artifacts injected via EXTRA_FILES. Three stale references still described the pre-#521 one-shot "revision plan": run_delegate's docstring, a nearby comment, and the delegate result comment's stage checklist. The stale docstring also misled the 2026-06-11 comprehensive review into calling the council loop open and ranking "close the council loop" as the top feature — add CORRECTIONS.md to the review directory recording the error and what survives it, with a pointer at the top of SUMMARY.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion Correct stale Stage 6 docs; record review correction
The reconcile e2e test's pass/fail was verified thoroughly (force-push SHA change + "Reconcile complete" comment) and printed in the results table, but RECONCILE_FAIL was never added to total_fail and had no summary line — a reconcile failure exited 0 and the suite read as green. From the 2026-06-11 comprehensive review (findings-workflows top finding 7, first item). The other two accounting items in that finding (rf-feedback sub-outcome not counted; duplicated wrapup run) are left as-is for now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…odels Pre-1.0 breaking change to the alias convention, made now while there are no external users to break. Aliases become three capability tiers per provider — small (value coding workhorse), medium (provider flagship), large (frontier above the flagship, where one exists) — with the reasoning documented in the config: - Tiers anchor on capability at agentic coding, not price; the ladder is monotone (up a tier is never weaker, never cheaper). - Every alias always resolves to a runnable model: providers lacking a distinct model for a tier co-point at the tier below (visible ladder compression, not a hole, and no surprise-pricing or can't-code traps). - The alias set is append-only post-1.0; repointing is expected and gets a changelog entry. Users who would draw the lines differently define their own aliases in their repo config. New mappings: claude-small=sonnet-5, claude-medium=opus-4-8 (was claude-large), claude-large=fable-5 (new frontier tier), gpt-small=gpt-5.3-codex, gpt-medium=gpt-5.6-sol (gpt-large co-points — no distinct OpenAI frontier tier suits agentic coding today), gemini-small/medium/large=gemini-3.5-flash (co-pointed; medium repoints to Gemini 3.5 Pro at GA). Docs updated to the tier language (README tier table + rationale, how-it-works, onboarding, install, demo); default escalation advice now points at *-medium. Everything load-bearing (auto default_model, councils, e2e smoke tests) references *-small, which did not move. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two weeks passed between cutting this PR and merging it, and the market moved — which is the tier system's first live exercise, absorbed with zero alias-name changes: - claude-medium: opus-4-8 -> opus-5 (released Jul 24; same $5/$25 band, agentic-coding focused — a textbook flagship repoint) - gemini-small/medium/large: 3.5-flash -> 3.6-flash (stable, better at code/agentic AND cheaper — dominates, so the co-pointed trio moves together; Gemini 3.5 Pro slipped again, so medium stays co-pointed) Model IDs verified against the Anthropic models overview and the Gemini API docs. OpenAI lineup unchanged since the 5.6 family. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Model aliases: three capability tiers, current models across providers
The 2026-08-05 full-test-suite run failed before any LLM call because RDB_TESTER_PAT_TOKEN (set Feb 25) had expired: every gh call returned HTTP 401, the quota check spewed a bash syntax error trying to compare a JSON error body against an integer, and the failure surfaced as three confusing per-test errors instead of one diagnosis. Both e2e scripts now verify the token up front (gh api user) and exit with rotation instructions when it's invalid, and the shared check_graphql_quota copies guard against non-numeric parses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The account table said the token belongs to the remote-dev-bot account, but that account was flagged by GitHub as a bot and authorized-trigger duty moved to gnovak — every e2e test issue on rdb-test is authored by gnovak. Documented the actual state, recommended a fine-grained PAT scoped to the test repo with the permissions the suite really needs, and noted the expiry failure mode (the 2026-08-05 suite failure). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oth tokens The previous commit documented the gnovak-PAT workaround as the standing setup. The bot-account arrangement is the intended design — the account is just flagged by GitHub right now (its profile 404s on the public API) pending a support conversation. Restore the intended setup as primary in CONTRIBUTING with a clearly-marked temporary-state note carrying the stopgap guidance and the restore path. Pre-flight hardening while here: print the token's expiration date (GitHub-Authentication-Token-Expiration header) alongside the login so future runs self-document how long the credential has left, and validate UNAUTHORIZED_PAT up front in e2e-security.sh — its failures otherwise surface only mid-gating-test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
e2e: fail fast on dead GH_TOKEN; guard quota parse
…ol loop)
The 2026-08-08 full-suite run showed openai/gpt-5.6-sol rejecting
function tools on chat completions ("use /v1/responses or set
reasoning_effort to 'none'"), failing both aliases at iteration 1.
Per the alias rules — capability at agentic coding is the admission
test and every alias must resolve to a runnable model — the OpenAI
ladder compresses fully onto the Codex line until a Responses API
path exists (tracked in issue 653).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-point gpt-medium/gpt-large at gpt-5.3-codex until a Responses API path exists
GitHub Actions has no `workflows` permissions scope; validation began
hard-rejecting it around 2026-05-01 ("Invalid workflow file ... Line 26:
Unexpected value 'workflows'"). Since then every push to any branch has
spawned a phantom failed dogfood run (100+ red Xs since May), and real
/dogfood comment triggers fail the same validation. Workflow-file write
access comes from the GitHub App's Workflows permission, not a job
permissions key, so nothing is lost by removing it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_dogfood_has_workflows_write_permission (from issue #463) asserted the presence of the exact key that now invalidates the workflow file. The key never did anything — GITHUB_TOKEN cannot modify workflow files regardless of job permissions; that capability comes from the GitHub App's Workflows permission. Replaced with the inverse guard across all workflow files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… models Two fixes from the 2026-08-08 full-suite runs: 1. gemini-3.6-flash fails rdb's loop 4-of-6 times today on the litellm thought-signature converter bug (issue 654; upstream litellm 16893 / 25322 family — no fixed release as of 1.95.0/1.96.0rc1). Per the alias admission test, the gemini trio repoints to gemini-2.5-flash — the best Gemini that actually runs the loop (it predates thought signatures entirely and passed every May e2e window). Restore 3.6-flash when litellm's handling stabilizes. 2. The timeout test was outrun a second time: a model scripted the 50-file task in one bash call inside the 30s watchdog window. Third hardening makes wall-clock the constraint instead of reasoning speed: 200 files in 200 individual git commits (~40-60s of git execution alone), which no model can compress below the watchdog regardless of how fast it thinks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix dogfood.yml: remove invalid permissions key breaking every push since May
Repoint gemini to 2.5-flash (litellm bug); harden timeout test
e2e: count reconcile failures in the suite exit code
Comprehensive AI draft covering v0.9.0..dev (31 merges): the alias tier system, zero-config install, the review-driven security and reliability campaign, cost/caching correctness, delegate fixes, test infrastructure, and known issues. Awaiting the mandatory human pare-down before tagging. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Draft v1.0.0 changelog (awaiting pare-down)
These were pointer files whose body just said "read AGENTS.md". That indirection has a failure mode: an agent can stop at the pointer and miss edits to AGENTS.md. A real symlink makes CLAUDE.md / GEMINI.md *be* AGENTS.md at the filesystem level, so the canonical content is always what's read. Git stores symlinks natively (mode 120000), so this travels correctly on clone. CLAUDE.md -> AGENTS.md; .gemini/GEMINI.md -> ../AGENTS.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backport #634 (agent-pointer symlinks) from main to dev
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The v1.0.0 release: merges dev → main, moving main forward from v0.9.0 (May 2026) across the entire stabilization cycle — the 2026-06-11 comprehensive review and its fix campaign, the model-alias tier system, zero-config install, and the test-infrastructure work.
Release gate (all green, 2026-08-08): 725 unit tests; e2e across all nine model aliases on three providers plus review-feedback, timeout, and wrapup phases; full security suite. Run: https://github.com/gnovak/remote-dev-bot/actions/runs/31263517654
See CHANGELOG.md (included in this merge) for the user-facing summary. Known issues at release: #653 (gpt tiers on codex pending a Responses API path), #654 (gemini tiers on 2.5-flash pending litellm's thought-signature fix).
After merge: tag v1.0.0 on the merge commit.
🤖 Generated with Claude Code