Skip to content

Repoint gemini to 2.5-flash (litellm bug); harden timeout test #657

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gnovak merged 1 commit into from
Aug 8, 2026
Merged

Repoint gemini to 2.5-flash (litellm bug); harden timeout test #657

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions remote-dev-bot.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,15 +179,15 @@ models:
description: "Co-pointed at gpt-small — no OpenAI model above the Codex line runs rdb's tool loop today"

gemini-small:
id: gemini/gemini-3.6-flash
description: "Google workhorse — strongest Gemini for agentic coding today"
id: gemini/gemini-2.5-flash
description: "Google workhorse — best Gemini that runs rdb's loop today (3.6-flash blocked on litellm thought-signature bug, issue #654)"

gemini-medium:
id: gemini/gemini-3.6-flash
description: "Co-pointed at gemini-small — repoint to Gemini 3.5 Pro if/when it reaches GA"
id: gemini/gemini-2.5-flash
description: "Co-pointed at gemini-small — repoint when a stronger Gemini clears the tool-loop admission test"

gemini-large:
id: gemini/gemini-3.6-flash
id: gemini/gemini-2.5-flash
description: "Co-pointed — no distinct Google frontier tier today"

# Agent settings — apply to all modes unless overridden per-mode above.
Expand Down
16 changes: 10 additions & 6 deletions tests/e2e.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -393,23 +393,27 @@ post_issue_comment "$RF_ISSUE_NUM" "$TEST_REPO" "/agent-resolve"
# --- Trigger timeout test ---
# Purpose: verify the watchdog kills the agent process when it exceeds
# timeout_minutes. The task must be concrete enough that the agent cannot
# scope-reduce its way to an early finish() — and the timeout short enough
# that even fast scope-reduced work doesn't fit. (The previous vague
# "refactor everything" prompt with timeout_minutes=1 let the agent finish
# in 50s by creating one file with type hints; watchdog never fired.)
# scope-reduce its way to an early finish() — and slow enough in WALL-CLOCK
# terms that no model can outrun the watchdog. Two prior hardenings:
# (1) the vague "refactor everything" prompt with timeout_minutes=1 let the
# agent finish in 50s by creating one file; (2) the 50-files-one-PR version
# was outrun on 2026-08-08 by a model that scripted all 50 files in a
# single bash call inside the 30s window. The current design demands 200
# individual git commits — tool-execution wall time (~40-60s of git alone)
# exceeds the watchdog no matter how fast the model reasons.
log "Creating timeout test issue..."
timeout_ts=$(date +%s)
timeout_title="Test: timeout enforcement (e2e-timeout-$timeout_ts)"
timeout_issue_url=$(gh issue create --repo "$TEST_REPO" \
--title "$timeout_title" \
--body "Create exactly 50 Python files named day_01.py through day_50.py in a new \`days/\` directory at the repo root. Each file must contain exactly one function with this exact signature and body (substitute NN with the day number, NOT zero-padded in the body):
--body "Create exactly 200 Python files named day_001.py through day_200.py in a new \`days/\` directory at the repo root. Each file must contain exactly one function with this exact signature and body (substitute NN with the day number, NOT zero-padded in the body):

\`\`\`python
def visits_on_day_NN() -> int:
return NN
\`\`\`

All 50 files must be created and committed in a single PR. This is a concrete, fully-specified task that cannot be scope-reduced — produce all 50 files or report failure.")
IMPORTANT: each file must be added in its own individual git commit — 200 separate commits, one file per commit, with the commit message 'Add day NNN'. Do not batch files into shared commits. This is a concrete, fully-specified task that cannot be scope-reduced — produce all 200 files in 200 commits or report failure.")
timeout_issue_num="${timeout_issue_url##*/}"
cleanup_issues+=("$timeout_issue_num")

Expand Down