Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
0dceacc
Fix cache savings formula; add cache + distillation summaries to reco…
gnovak May 15, 2026
5845926
Remove upper cap on distillation; let Tier 2 handle any repo size
gnovak May 15, 2026
30cb81b
Merge pull request #621 from gnovak/fix-cache-and-distill-savings
gnovak May 16, 2026
c1f4695
Merge pull request #622 from gnovak/remove-distill-upper-cap
gnovak May 16, 2026
8e40e55
Fix module imports that break under production sys.path layout
gnovak May 20, 2026
42cc054
Merge pull request #623 from gnovak/fix-workshop-import
gnovak May 20, 2026
b02ee84
Tighten e2e assertions so missing-output cases fail instead of pass
gnovak May 20, 2026
7a74349
Merge pull request #624 from gnovak/tighten-e2e-assertions
gnovak May 20, 2026
93bc290
Make timeout e2e test reliably exercise the watchdog
gnovak May 23, 2026
43aae99
Distillation: tighten prompt, fix metric, model-dependent Tier-1 limit
gnovak May 23, 2026
a12967e
Merge pull request #625 from gnovak/fix-timeout-e2e-test
gnovak May 23, 2026
a2374ba
Merge pull request #626 from gnovak/fix-distillation-prompt-and-metrics
gnovak May 23, 2026
1ccd5b3
Delegate cost reporting: use canonical cumulative format, drop bespok…
gnovak May 27, 2026
c01ca37
Resolve: teach the prompt to treat design/spec comments as binding scope
gnovak May 27, 2026
eefa7c7
Merge pull request #627 from gnovak/delegate-cost-reporting-cumulative
gnovak May 27, 2026
dcf6c63
Merge pull request #628 from gnovak/resolve-scope-prompt
gnovak May 27, 2026
cc723de
Wire /agent-review council=true into the review job
gnovak May 27, 2026
dd97c0b
Merge pull request #629 from gnovak/review-council-wiring
gnovak May 27, 2026
3bf1d24
Resolve prompt: methodology fidelity, end-to-end verification, write …
gnovak May 31, 2026
4e0e450
Spec generation: verify file refs, ban `...` placeholders, require me…
gnovak May 31, 2026
b4e870e
Design loop: verify file references, require acceptance tests for met…
gnovak May 31, 2026
b0d688b
Merge pull request #631 from gnovak/resolve-methodology-fidelity
gnovak Jun 3, 2026
0c08c59
Merge pull request #632 from gnovak/spec-generation-prompt-fidelity
gnovak Jun 3, 2026
de72856
Merge pull request #633 from gnovak/design-loop-verify-references
gnovak Jun 3, 2026
80860d3
Check in the 2026-06-11 comprehensive review
gnovak Jun 15, 2026
91a1620
Merge pull request #635 from gnovak/add-comprehensive-review-2026-06-11
gnovak Jun 15, 2026
3d1600c
resolve.py: harden create_pr against title injection + three loop bugs
gnovak Jul 5, 2026
03f7eda
Merge pull request #636 from gnovak/fix-resolve-security-and-bugs
gnovak Jul 5, 2026
9ae1bec
workflow: route config strings through step env + fix plumbing gaps
gnovak Jul 5, 2026
5dc5ebc
Merge pull request #637 from gnovak/fix-workflow-env-and-parse-outputs
gnovak Jul 5, 2026
e9be96f
test.yml: run CI on PRs targeting dev, not just main
gnovak Jul 5, 2026
99d62a0
Merge pull request #638 from gnovak/fix-test-ci-dev-trigger
gnovak Jul 5, 2026
bf80f3e
context.py: keep tool-call pairs intact across the compaction boundary
gnovak Jul 5, 2026
6799d2b
distill.py: fix signature extraction dropping everything but arg names
gnovak Jul 5, 2026
dd51d36
reconcile.py: four loop fixes from the comprehensive review
gnovak Jul 5, 2026
33eeb81
design_loop.py: read cache-write tokens from litellm's actual field name
gnovak Jul 5, 2026
fd2965d
Merge pull request #639 from gnovak/fix-context-compaction-pairing
gnovak Jul 5, 2026
6dc264b
Merge pull request #640 from gnovak/fix-distill-signature-extraction
gnovak Jul 5, 2026
10194be
Merge pull request #641 from gnovak/fix-reconcile-loop-bugs
gnovak Jul 5, 2026
ca87e43
Merge pull request #642 from gnovak/fix-design-loop-cache-field
gnovak Jul 5, 2026
227c135
Auto-detect default_model from configured API keys; fix install docs
gnovak Jul 5, 2026
962f80e
Merge pull request #643 from gnovak/install-default-model-autodetect
gnovak Jul 5, 2026
93c1936
resolve.py: use lib/context.py's trim_tool_results, drop stale local …
gnovak Jul 5, 2026
39b1108
workshop.py: three delegate fixes from the comprehensive review
gnovak Jul 5, 2026
df3fb9a
design_loop.py: add moving-tail prompt-cache markers
gnovak Jul 5, 2026
d2e4802
Merge pull request #644 from gnovak/resolve-use-shared-trim
gnovak Jul 5, 2026
e2e710e
Merge pull request #645 from gnovak/workshop-delegate-tier4-fixes
gnovak Jul 5, 2026
aaf5abc
Merge pull request #646 from gnovak/design-loop-cache-markers
gnovak Jul 5, 2026
b9ca05c
Unify wrapup-injection mechanics: every-iteration escalation in all l…
gnovak Jul 5, 2026
4dd2684
Merge pull request #647 from gnovak/unify-wrapup-injection
gnovak Jul 6, 2026
f0f55cf
Correct stale "revision plan" descriptions of delegate Stage 6
gnovak Jul 6, 2026
e723195
Merge pull request #648 from gnovak/fix-stage6-docs-and-review-correc…
gnovak Jul 6, 2026
05dd821
e2e.sh: count reconcile failures in the exit code and summary
gnovak Jul 6, 2026
e744adc
Model aliases: three capability tiers (small/medium/large), current m…
gnovak Jul 20, 2026
f0a11b1
Refresh alias targets for early-August model releases
gnovak Aug 5, 2026
7fe5141
Merge pull request #650 from gnovak/update-model-aliases-1.0
gnovak Aug 5, 2026
1059cc9
e2e: fail fast with a clear message when GH_TOKEN is dead
gnovak Aug 5, 2026
f22ccdb
CONTRIBUTING: document that RDB_TESTER_PAT_TOKEN is a gnovak PAT
gnovak Aug 8, 2026
a311854
Rework token docs to intent-first; pre-flight prints expiry, checks b…
gnovak Aug 8, 2026
65c5ca6
Merge pull request #652 from gnovak/e2e-token-preflight
gnovak Aug 8, 2026
fe5b877
gpt-medium/gpt-large: co-point at gpt-5.3-codex (Sol can't run the to…
gnovak Aug 8, 2026
9f3c018
Merge pull request #655 from gnovak/gpt-copoint-codex
gnovak Aug 8, 2026
16e1d54
dogfood.yml: remove invalid `workflows: write` permissions key
gnovak Aug 8, 2026
65fabf8
test_yaml: invert the workflows-permission regression test
gnovak Aug 8, 2026
8533e72
Repoint gemini aliases to 2.5-flash; harden timeout test against fast…
gnovak Aug 8, 2026
ee0de1c
Merge pull request #656 from gnovak/fix-dogfood-permissions
gnovak Aug 8, 2026
4eaec05
Merge pull request #657 from gnovak/gemini-25-flash-and-timeout-fix
gnovak Aug 8, 2026
0ad1795
Merge pull request #649 from gnovak/e2e-count-reconcile-failures
gnovak Aug 19, 2026
4ae130d
Draft v1.0.0 changelog
gnovak Aug 10, 2026
78ce625
Pare down v1.0.0 changelog per review
gnovak Aug 19, 2026
cf6a0dc
Merge pull request #658 from gnovak/changelog-v1.0.0
gnovak Aug 19, 2026
2652253
agents: make CLAUDE.md / GEMINI.md real symlinks to AGENTS.md
gnovak Jun 15, 2026
2fec6b2
Merge pull request #659 from gnovak/backport-agent-symlinks
gnovak Sep 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .gemini/GEMINI.md

This file was deleted.

1 change: 1 addition & 0 deletions .gemini/GEMINI.md
6 changes: 5 additions & 1 deletion .github/workflows/dogfood.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,15 @@ on:
pull_request_review_comment:
types: [created]

# NOTE: `workflows: write` is not a valid permissions scope — GitHub began
# hard-rejecting it ~2026-05-01 ("Unexpected value 'workflows'"), which made
# every push spawn a phantom failed dogfood run and broke /dogfood comments.
# Workflow-file write access is granted by the App's Workflows permission
# (see CONTRIBUTING.md), not by a job permissions key.
permissions:
contents: write
issues: write
pull-requests: write
workflows: write

jobs:
resolve:
Expand Down
191 changes: 167 additions & 24 deletions .github/workflows/remote-dev-bot.yml

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Tests

on:
pull_request:
branches: [main]
branches: [main, dev]

jobs:
test:
Expand Down
35 changes: 35 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,40 @@
# Changelog

## v1.0.0 (Aug 2026)

- **Model aliases: three capability tiers.** Aliases are now
`{provider}-{small,medium,large}`: small = best value-for-money
agentic-coding workhorse, medium = the provider's flagship, large =
frontier-above-flagship where one exists. Tiers are anchored on
capability at agentic coding (price is an expectation, not the
admission test) and the ladder is monotone — going up a tier never
gets a weaker model. The model landscape doesn't always map cleanly
onto three tiers, so aliases sometimes point at the same model rather
than leaving a tier undefined or pointing it at something that can't
do the job. Repointing rules and reasoning are documented in
`remote-dev-bot.yaml` and the README.
- **Zero-config default model** (`default_model: auto`): fresh installs
run without configuring a model at all — the default resolves from
whichever provider API key secret is present. Explicit configuration
always wins.
- **Lots of small fixes and cleanup in preparation for 1.0**: security
hardening, reliability and prompt-cache correctness, cost-accounting
accuracy, doc repair, and test-infrastructure work, driven by a
comprehensive pre-1.0 review (`comprehensive-review/2026-06-11/`).

### Known issues

- The gemini tiers point at `gemini-2.5-flash` rather than the newer
3.x models: litellm's Gemini 3.x thought-signature handling
intermittently breaks multi-turn tool calling (#654).
- The gpt tiers point at `gpt-5.3-codex` rather than GPT-5.6 Sol: the
5.6 frontier family requires OpenAI's Responses API for tool use,
which rdb's litellm path doesn't use yet (#653).

**Breaking changes:** model aliases renamed as part of the tier system
(pre-1.0, no compatibility guarantee was in effect). Post-1.0, alias
names are stable and the alias set is append-only.

## v0.9.0 — Delegate mode, reconcile mode, council reviews (May 2026)

- **Delegate mode** (`/agent-delegate`): Full design-to-implementation pipeline
Expand Down
6 changes: 0 additions & 6 deletions CLAUDE.md

This file was deleted.

1 change: 1 addition & 0 deletions CLAUDE.md
17 changes: 15 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,13 @@ Three separate GitHub identities are used so each role is cleanly separated:
| Account | Purpose | Credentials stored as |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `gnovak` | Repo owner. Used for normal development and testing. | `RDB_PAT_TOKEN` (on both repos), or GitHub App (`RDB_APP_ID` variable + `RDB_APP_PRIVATE_KEY` secret) |
| `remote-dev-bot` | Dedicated bot account. Collaborator on `remote-dev-bot-test`. Posts authorized test comments that trigger agent runs without attributing activity to `gnovak`. | `RDB_TESTER_PAT_TOKEN` (on remote-dev-bot) |
| `remote-dev-bot` | Dedicated bot account. Collaborator on `remote-dev-bot-test`. Posts authorized test comments that trigger agent runs without attributing activity to `gnovak`. | `RDB_TESTER_PAT_TOKEN` (on remote-dev-bot) — but see the temporary-state note below |
| `remote-dev-bot-tester` | Simulates an unauthorized external user. NOT a collaborator on any repo. | `RDB_TESTER_UNAUTHORIZED_PAT_TOKEN` (on remote-dev-bot) |

### remote-dev-bot (bot account) details

The intended setup:

- Classic PAT with `public_repo` + `workflow` scopes, no expiration
- **`public_repo`** — create issues, post comments, open PRs in public repos
(standard e2e test flow)
Expand All @@ -40,6 +42,17 @@ Three separate GitHub identities are used so each role is cleanly separated:
its trigger comments
- Keeps test activity out of `gnovak`'s GitHub contribution stats

> **Temporary state (as of 2026-08):** the `remote-dev-bot` account is
> flagged by GitHub (its profile 404s on the public API), so tokens cannot
> be minted on it until that's resolved with GitHub support. Until then,
> `RDB_TESTER_PAT_TOKEN` holds a PAT minted on `gnovak` instead (verifiable:
> e2e test issues on rdb-test are authored by `gnovak`). If minting the
> stopgap on `gnovak`, prefer a fine-grained PAT scoped to
> `gnovak/remote-dev-bot-test` only (Contents/Issues/Pull requests
> read-write + Actions read-write) with a long expiry — the 2026-08-05
> full-suite failure was a 90-day stopgap token quietly expiring. Once the
> bot account is restored, move the token back per the intended setup above.

### remote-dev-bot-tester details

- Classic PAT with `public_repo` scope, no expiration
Expand Down Expand Up @@ -81,7 +94,7 @@ Secrets stored on `gnovak/remote-dev-bot`:
| `OPENAI_API_KEY` | OpenAI API key (for GPT models) |
| `GEMINI_API_KEY` | Google AI API key (for Gemini models) |
| `RDB_APP_PRIVATE_KEY` | GitHub App private key, for bot identity on comments/PRs. |
| `RDB_TESTER_PAT_TOKEN` | PAT for `remote-dev-bot` account (collaborator on rdb-test). Used by e2e tests to post authorized trigger comments. |
| `RDB_TESTER_PAT_TOKEN` | PAT for `remote-dev-bot` account (collaborator on rdb-test; temporarily a `gnovak` PAT — see account details). Used by e2e tests to post authorized trigger comments. |
| `RDB_TESTER_UNAUTHORIZED_PAT_TOKEN` | PAT for `remote-dev-bot-tester` (not a collaborator). Used by security e2e tests to verify unauthorized users are blocked. |

Variables stored on `gnovak/remote-dev-bot`:
Expand Down
48 changes: 34 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ to design agents that can autonomously handle real development tasks.
## How It Works

1. Create a GitHub issue describing a feature or bug
2. Comment `/agent-resolve` (or `/agent-resolve-claude-large`, etc.) to trigger
2. Comment `/agent-resolve` (or `/agent-resolve-claude-medium`, etc.) to trigger
implementation
3. A GitHub Action spins up an AI agent that runs a custom LiteLLM agent loop
that:
Expand Down Expand Up @@ -65,11 +65,11 @@ comment (no code changes).
| Command | What it does |
| ------------------------------- | ---------------------------------------------------- |
| `/agent-resolve` | Resolve the issue and open a PR (default model) |
| `/agent-resolve-claude-large` | Resolve with a specific model |
| `/agent-resolve-claude-medium` | Resolve with a specific model |
| `/agent-design` | Explore codebase and post a design analysis comment |
| `/agent-design-claude-large` | Design analysis with a specific model |
| `/agent-design-claude-medium` | Design analysis with a specific model |
| `/agent-review` | Post a code review comment on a PR (no code changes) |
| `/agent-review-claude-large` | Code review with a specific model |
| `/agent-review-claude-medium` | Code review with a specific model |
| `/agent-workshop[-<model>]` | Design analysis + multi-model council critique |
| `/agent-build[-<model>]` | Implement issue + multi-model council code review |
| `/agent-delegate[-<model>]` | Full design-to-implementation cycle with no human pauses |
Expand All @@ -78,8 +78,8 @@ comment (no code changes).
Modes and model aliases are configured in `remote-dev-bot.yaml`.

**Mobile-friendly syntax:** Commands are case-insensitive and you can use spaces
instead of dashes: `/agent resolve claude large` works the same as
`/agent-resolve-claude-large`.
instead of dashes: `/agent resolve claude medium` works the same as
`/agent-resolve-claude-medium`.

### Per-Invocation Arguments

Expand Down Expand Up @@ -113,13 +113,30 @@ Model aliases (like `claude-small`) map to **LiteLLM model identifiers** in
`remote-dev-bot.yaml`. Remote Dev Bot uses LiteLLM to talk to different LLM
providers through a unified interface.

Aliases come in three capability tiers per provider, so you never have to
track model releases:

| Tier | Meaning | Typical use |
| ---------- | ---------------------------------------------------- | ----------- |
| `*-small` | Best value-for-money coding workhorse | The default; most tasks |
| `*-medium` | The provider's flagship | Complex multi-file features |
| `*-large` | Frontier model above the flagship, where one exists | Hardest long-horizon work |

Tiers are anchored on capability at agentic coding, not price, and the ladder
is monotone — going up a tier never gets you a weaker model. Where a provider
has no distinct model for a tier, the alias points at the tier below it, so
every alias always runs. The model landscape doesn't always divide cleanly
into three tiers; these mappings are judgment calls, and you can always
define your own aliases (see "Adding Models" below) if you'd draw the lines
differently. An abstraction doesn't have to be perfect to be useful.

**Model ID format:** `provider/model-name`

| Provider | Prefix | Example |
| ------------------ | ------------ | ----------------------------- |
| Anthropic (Claude) | `anthropic/` | `anthropic/claude-sonnet-4-5` |
| OpenAI (GPT) | `openai/` | `openai/gpt-5.1-codex-mini` |
| Google (Gemini) | `gemini/` | `gemini/gemini-2.5-flash` |
| Anthropic (Claude) | `anthropic/` | `anthropic/claude-sonnet-5` |
| OpenAI (GPT) | `openai/` | `openai/gpt-5.3-codex` |
| Google (Gemini) | `gemini/` | `gemini/gemini-3.6-flash` |

### Supported Providers

Expand Down Expand Up @@ -161,12 +178,15 @@ Prefix the model string with the provider name in `remote-dev-bot.yaml` (e.g.,
**For most tasks:** Use the default (`/agent-resolve`). Claude Sonnet
(`claude-small`) offers a good balance of capability and cost.

**For complex multi-file features:** Use `/agent-resolve-claude-large` (Opus) or
`/agent-resolve-gpt-large` (GPT Codex). These models handle larger contexts and
more intricate reasoning.
**For complex multi-file features:** Use `/agent-resolve-claude-medium` (Opus)
or `/agent-resolve-gpt-medium`. Flagship models handle larger contexts and more
intricate reasoning.

**For the hardest long-horizon work:** `/agent-resolve-claude-large` (Fable)
is the frontier tier — expect roughly 2-4x the cost of a medium run.

**For coding-heavy tasks:** Models with "codex" in the name (e.g.,
`openai/gpt-5.1-codex-mini`) are specifically tuned for code generation and may
`openai/gpt-5.3-codex`) are specifically tuned for code generation and may
perform better on implementation tasks.

### Commit Trailers
Expand Down Expand Up @@ -435,7 +455,7 @@ The agent ran, posted a comment with its evaluation, but didn't open a PR. This
means the agent judged that it couldn't fully resolve the issue — it hit the
iteration limit, got confused, or determined its changes were incomplete.

Try a more capable model (`/agent-resolve-claude-large`) or add more detail to
Try a more capable model (`/agent-resolve-claude-medium`) or add more detail to
the issue description. The agent's evaluation comment will say what it attempted
and why it stopped.

Expand Down
Loading