From 8533e725d9286b8155b443dc478f8f227f702846 Mon Sep 17 00:00:00 2001 From: Greg Novak Date: Sat, 8 Aug 2026 06:39:57 +0000 Subject: [PATCH] Repoint gemini aliases to 2.5-flash; harden timeout test against fast models MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two fixes from the 2026-08-08 full-suite runs: 1. gemini-3.6-flash fails rdb's loop 4-of-6 times today on the litellm thought-signature converter bug (issue 654; upstream litellm 16893 / 25322 family — no fixed release as of 1.95.0/1.96.0rc1). Per the alias admission test, the gemini trio repoints to gemini-2.5-flash — the best Gemini that actually runs the loop (it predates thought signatures entirely and passed every May e2e window). Restore 3.6-flash when litellm's handling stabilizes. 2. The timeout test was outrun a second time: a model scripted the 50-file task in one bash call inside the 30s watchdog window. Third hardening makes wall-clock the constraint instead of reasoning speed: 200 files in 200 individual git commits (~40-60s of git execution alone), which no model can compress below the watchdog regardless of how fast it thinks. Co-Authored-By: Claude Fable 5 --- remote-dev-bot.yaml | 10 +++++----- tests/e2e.sh | 16 ++++++++++------ 2 files changed, 15 insertions(+), 11 deletions(-) diff --git a/remote-dev-bot.yaml b/remote-dev-bot.yaml index 318bb52..3edb00a 100644 --- a/remote-dev-bot.yaml +++ b/remote-dev-bot.yaml @@ -179,15 +179,15 @@ models: description: "Co-pointed at gpt-small — no OpenAI model above the Codex line runs rdb's tool loop today" gemini-small: - id: gemini/gemini-3.6-flash - description: "Google workhorse — strongest Gemini for agentic coding today" + id: gemini/gemini-2.5-flash + description: "Google workhorse — best Gemini that runs rdb's loop today (3.6-flash blocked on litellm thought-signature bug, issue #654)" gemini-medium: - id: gemini/gemini-3.6-flash - description: "Co-pointed at gemini-small — repoint to Gemini 3.5 Pro if/when it reaches GA" + id: gemini/gemini-2.5-flash + description: "Co-pointed at gemini-small — repoint when a stronger Gemini clears the tool-loop admission test" gemini-large: - id: gemini/gemini-3.6-flash + id: gemini/gemini-2.5-flash description: "Co-pointed — no distinct Google frontier tier today" # Agent settings — apply to all modes unless overridden per-mode above. diff --git a/tests/e2e.sh b/tests/e2e.sh index 2c2b255..f911b35 100755 --- a/tests/e2e.sh +++ b/tests/e2e.sh @@ -393,23 +393,27 @@ post_issue_comment "$RF_ISSUE_NUM" "$TEST_REPO" "/agent-resolve" # --- Trigger timeout test --- # Purpose: verify the watchdog kills the agent process when it exceeds # timeout_minutes. The task must be concrete enough that the agent cannot -# scope-reduce its way to an early finish() — and the timeout short enough -# that even fast scope-reduced work doesn't fit. (The previous vague -# "refactor everything" prompt with timeout_minutes=1 let the agent finish -# in 50s by creating one file with type hints; watchdog never fired.) +# scope-reduce its way to an early finish() — and slow enough in WALL-CLOCK +# terms that no model can outrun the watchdog. Two prior hardenings: +# (1) the vague "refactor everything" prompt with timeout_minutes=1 let the +# agent finish in 50s by creating one file; (2) the 50-files-one-PR version +# was outrun on 2026-08-08 by a model that scripted all 50 files in a +# single bash call inside the 30s window. The current design demands 200 +# individual git commits — tool-execution wall time (~40-60s of git alone) +# exceeds the watchdog no matter how fast the model reasons. log "Creating timeout test issue..." timeout_ts=$(date +%s) timeout_title="Test: timeout enforcement (e2e-timeout-$timeout_ts)" timeout_issue_url=$(gh issue create --repo "$TEST_REPO" \ --title "$timeout_title" \ - --body "Create exactly 50 Python files named day_01.py through day_50.py in a new \`days/\` directory at the repo root. Each file must contain exactly one function with this exact signature and body (substitute NN with the day number, NOT zero-padded in the body): + --body "Create exactly 200 Python files named day_001.py through day_200.py in a new \`days/\` directory at the repo root. Each file must contain exactly one function with this exact signature and body (substitute NN with the day number, NOT zero-padded in the body): \`\`\`python def visits_on_day_NN() -> int: return NN \`\`\` -All 50 files must be created and committed in a single PR. This is a concrete, fully-specified task that cannot be scope-reduced — produce all 50 files or report failure.") +IMPORTANT: each file must be added in its own individual git commit — 200 separate commits, one file per commit, with the commit message 'Add day NNN'. Do not batch files into shared commits. This is a concrete, fully-specified task that cannot be scope-reduced — produce all 200 files in 200 commits or report failure.") timeout_issue_num="${timeout_issue_url##*/}" cleanup_issues+=("$timeout_issue_num")