Skip to content

fix(devin): resolve models through catalog family metadata (carry #6091) - #6172

Merged
lidge-jun merged 5 commits into
devfrom
codex/rt5-devin-models
Sep 28, 2026
Merged

lidge-jun merged 5 commits into
devfrom
codex/rt5-devin-models

Conversation

@lidge-jun

@lidge-jun lidge-jun commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

Carries #6091 by @wtfsayo onto current dev, with two review fixes on top. Devin model resolution guessed tiers from uid suffixes, which sent the wrong model on live accounts: swe-1-7 at medium effort went out as the Max row, a missed uid could fall into a different family (claude-opus-5 → claude-opus-5-5-*), family defaults were ignored (SWE-2 defaults to High, GLM-5.2 to its bare row), uncapped turns stopped at 8192 output tokens, and swe-1-6 silently dropped images.

After this PR the adapter resolves through the catalog's family metadata (ClientModelConfig #23/#30/#31):

  • A family id with no effort selects the family's catalog default member; an effort moves only the effort axis, to the lowest rung at or above it (else the highest below). Resolution never leaves the family.
  • Fast Mode and 1M Context stay at the named row's values unless the caller asks for fast or a 1m value.
  • The picker shows one row per family with the real ladder and the family default as defaultReasoningEffort; swe-1-6 is text-only so images fall back to a vision model.
  • CompletionConfiguration [codex] Refresh Codex cache after provider changes #3 is max_newlines and is sent at a fixed 128000; it was being fed the context window.

Lane fixes on top of the contributor commits (8ad177fbd5):

  • A disabled named row is kept when the request still lands on it. Before, swe-2-max asked for at max with that row disabled on the account quietly became swe-2-high. Now it is sent as named, so the chat preflight reports that row's tier refusal. Asking for a different variant, or naming the family, still passes over disabled rows.
  • A requested effort is never lowered to keep a toggle. In a family whose Fast rows stop at Medium, a Fast row asked for at high used to drop to Medium Fast. It now falls back to the regular High row. A Fast row at or above the request still wins, and a member with no effort rung counts as lower. none is exempt, so Thinking-off still wins for it.

Release note: Devin turns with no output cap from the caller or config now use the model's catalog output limit (usually 128000; 384000 for DeepSeek V4 Pro) instead of 8192. A caller or configured cap still wins. A saved bare swe-1-7 sent without an effort now resolves to the family default (swe-1-7-medium); Codex sends an effort on every turn, and high/xhigh/max still reach the Max row.

Carries #6091. Contributor commits are cherry-picked with their authorship; the lane commit carries the trailer.

This is the first PR of a short chain: the Devin wire fixes (#6092 carry) and reasoning signatures (#6116 carry) are stacked on this branch because all three edit src/adapters/devin.ts.

Co-authored-by: Sayo hi@sayo.wtf

  • Codex review follow-ups: ee1fd1c839 advertises the minimal rung, so Gemini Flash shows its [minimal, medium] ladder; 0ca17f3e77 publishes the effective family default when the marked default row is disabled, so the picker and request resolution agree.

Verification

  • Contributor live evidence (from fix(devin): resolve models through catalog family metadata #6091, real Devin account, 267 catalog rows): swe-1-7@high → swe-1-7 (Max), swe-1-7@low → swe-1-7-medium, swe-2@xhigh → swe-2-max, kimi-k3@medium → kimi-k3-high; [codex] Refresh Codex cache after provider changes #3 = 128000 accepted 135,583 prompt tokens on swe-1-6 and 238,954 on a 1M-window Gemini row. Not re-run live in this lane.
  • New cases in tests/providers/devin-family-resolution.test.ts: "a disabled named row keeps its uid when the request lands on it", "a requested effort is not lowered to keep a toggle" (both fail without 8ad177fbd5), and "none keeps Thinking off when the Thinking-off row has no rung" (guard).
  • Local tests were skipped by maintainer instruction; the PR CI on this head is the test evidence. Before that instruction arrived, the lane ran bun run typecheck (pass), all 15 tests/providers/devin-*.test.ts files (all pass, 0 fail), bun run structure:check and bun run privacy:scan (pass). bun run test:changed was not completed locally.
  • Independent reviews: a read-only reviewer found the two defects fixed above; the fix design was re-audited to NEAR-PASS and its remaining point folded.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

wtfsayo and others added 3 commits September 28, 2026 16:40
Parse ClientModelConfig #23 ModelInfo, #30 ModelFamilyMetadata and #31
default flag. A family id with no effort now selects the family's default
member (swe-1-7 -> swe-1-7-medium, not the Max row), an effort moves only
the effort axis to the nearest rung, Fast Mode and 1M Context follow only
explicit requests, and resolution never leaves the family. The suffix
fallback's variant scan matches the collapsed base instead of a string
prefix, so claude-opus-5 no longer lands on claude-opus-5-5-*.

The picker collapses by family, reads ladders from the effort axis, and
advertises the family default as defaultReasoningEffort; swe-1-6-fast
stays its own row. swe-1-6 is marked text-only (live: it drops images
silently) so the vision fallback applies.

Uncapped turns send the selected row's catalog maxOutputTokens instead of
8192. CompletionConfiguration #3 is max_newlines, now a fixed value; the
input-ceiling plumbing that fed it a context window is removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 09b9e6d)
An effort a family lacks now selects the lowest rung at or above it, then
the highest below. Nearest-rung sent `high` on SWE-1.7 (Medium, Max) to
Medium; it now selects the Max row. A disabled default rounds up the same
way and falls back down only when nothing above it is enabled.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit ed6449e)
…ggle

A disabled exact uid asked for with an effort that still selects it (swe-2-max
at max) is now sent as named, so the preflight reports that row's refusal
instead of quietly serving swe-2-high. Family selection now ranks "effort not
lowered" ahead of the non-effort axes: a Fast anchor asked for at a rung the
Fast rows lack falls back to the regular row at or above that rung, and an
unranked member counts as lower. none is exempt, so Thinking-off still wins.

Co-authored-by: Sayo <hi@sayo.wtf>
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner September 28, 2026 08:21
@coderabbitai

coderabbitai Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Next included review available in 3 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: lidge-jun/opencodex/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: c5a67cce-d742-484f-bd2f-dc0fb48fefdf

📥 Commits

Reviewing files that changed from the base of the PR and between cbe0d40 and 0ca17f3.

📒 Files selected for processing (16)
  • docs-site/src/content/docs/reference/adapters.md
  • scripts/test-layout/layout.json
  • src/adapters/devin.ts
  • src/adapters/devin/cloud-direct/catalog.ts
  • src/adapters/devin/cloud-direct/chat.ts
  • src/adapters/devin/live-models.ts
  • src/codex/catalog/provider-models.ts
  • structure/catalog.md
  • structure/providers-and-adapters.md
  • tests/fixtures/test-layout-expected.json
  • tests/providers/devin-adapter.test.ts
  • tests/providers/devin-family-resolution.test.ts
  • tests/providers/devin-hardening.test.ts
  • tests/providers/devin-live-models.test.ts
  • tests/providers/devin-output-budget.test.ts
  • tests/providers/devin-prompt-cache.test.ts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-28T08:25:42.965055Z 8ad177f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8ad177fbd5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/adapters/devin/live-models.ts
Comment thread src/adapters/devin/live-models.ts Outdated
@github-actions

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the bug Something isn't working label Sep 28, 2026
@lidge-jun

Copy link
Copy Markdown
Owner Author

리뷰 · 우선순위 62 / 80

이 PR은 Devin이 모델을 고르는 방식을 고쳐요. 바탕은 dev예요. #6091을 지금 dev 위로 옮기고, 그 위에 리뷰에서 나온 두 가지를 더 고쳤어요.

예전에는 이름 끝글자를 보고 단계를 짐작했어요. swe-1-7에 medium을 적어도 Max 줄이 나갔고, 없는 이름은 claude-opus-5가 claude-opus-5-5로 넘어가기도 했어요. 가족 기본값도 무시됐고, 출력 한도를 안 적으면 8192에서 멈췄어요. swe-1-6은 그림을 조용히 버렸어요.

이제는 카탈로그의 가족 정보를 읽어요. 노력을 안 적으면 그 가족의 기본 줄을 골라요. swe-1-7은 swe-1-7-medium, swe-2는 swe-2-high예요. 노력을 적으면 그 가족 안에서만, 그 값 이상인 가장 낮은 줄로 올려요. 위에 없으면 아래의 가장 높은 줄이에요. swe-1-7에 high를 주면 Max 줄 swe-1-7이 나가요. Fast와 1M은 달라고 할 때만 바뀌고, 그 토글을 지키려고 노력을 내리지는 않아요. 꺼진 줄을 이름까지 찍어서 그 단계가 그대로면, 다른 줄로 갈아타지 않고 그 줄의 거절이 나와요. 출력 한도를 안 적으면 그 줄의 카탈로그 한도를 보내요. 보통 128000이고, DeepSeek V4 Pro는 384000이에요. 요청 3번 칸은 줄 수 한도라서 128000으로 고정해요. 본문은 그 값으로 135,583토큰과 238,954토큰 입력이 통과했다고 적어요.

라인 - src/adapters/devin/live-models.ts 344행. 가족 축의 Minimal을 피커 사다리에서 빼요. devinFamilyEffortOf는 minimal을 알아보는데, 344행 필터는 106행 REASONING_RUNG_TOKENS에 minimal이 없어서 버려요. 테스트에 넣은 Gemini Flash는 Minimal과 Medium뿐이에요. Medium만 남으면 줄이 하나라서 사다리 자체를 안 보여 줘요. Minimal 줄을 고를 칸이 사라져요. 106행에 넣어도 128행 RUNG_ORDER에 없으면 정렬이 맨 앞으로 밀려요.

라인 - src/adapters/devin/live-models.ts 335행. 꺼진 줄은 기본값을 적기 전에 건너뛰어요. 기본 줄이 꺼지고 형제만 켜져 있으면 피커의 defaultEfforts가 비어요. 요청을 처리할 때는 꺼진 기본 줄의 노력을 기준으로 가까운 켜진 줄을 골라요. 테스트도 그래요. SWE-2 기본 High가 꺼지면 노력을 안 적은 swe-2는 swe-2-max가 돼요. 피커는 그 기본을 안 알려 주니, 화면은 목록 첫 칸인 medium을 보내고 실제 생략 요청은 Max로 갈 수 있어요.

메인테이너의 판단이 필요한 지점

한도를 안 적은 턴은 8192가 아니라 카탈로그 한도까지 답을 만들 수 있어요. 요금과 대기 시간이 커져요. 호출자가 적은 한도는 그대로 이겨요.

저장해 둔 맨 이름 swe-1-7은 Max 행 이름이지만, 이제는 가족 기본인 medium으로 풀려요. Codex는 매 턴 노력을 보내서 high 이상은 Max로 가요. 노력을 안 붙인 저장 값만 바뀌어요.

swe-1-6을 그림 불가로 박아 둔 것은 2026-09-27 실측이에요. 카탈로그 칸은 그림을 받는다고 적어요.

CI의 test 2/4는 종료 코드 124예요. 파일을 하나씩 돌리면 통과했고, 이 PR 테스트가 깨진 로그는 아니에요. 다른 테스트 샤드는 이 글을 쓸 때 아직 돌아가고 있었어요.

#6091은 아직 열려 있어요. 이 PR이 그 내용을 dev 기준으로 대신하고, 없는 단계를 위로 올리는 고침과 꺼진 줄 처리를 더 넣었어요. #6178과 #6116이 이 브랜치 위에 쌓여 있어요.

너의 추천

바탕 dev를 유지하세요. Minimal을 106행 집합과 128행 순서에 넣고, Gemini Flash 피커가 minimal과 medium을 보여 주는지 테스트를 추가하세요. 꺼진 기본 줄의 피커 기본값은, 요청을 풀 때와 같은 규칙으로 켜진 줄에서 계산하세요. 화면이 medium을 가리키고 전송은 Max로 가는 일이 없어야 해요. 그 둘을 고친 뒤 머지하세요. #6091은 중복으로 닫으세요. 위에 쌓인 #6178과 #6116은 이 커밋이 dev에 들어간 다음에 다시 맞추세요.

이 댓글은 grok-bot이 작성했습니다

lidge-jun and others added 2 commits September 28, 2026 17:44
Co-authored-by: Sayo <hi@sayo.wtf>
Co-authored-by: Sayo <hi@sayo.wtf>
@lidge-jun

Copy link
Copy Markdown
Owner Author

Maintainer integration into dev under the MAINTAINERS.md dev exception (lidge-jun, admin).

Exact head 0ca17f3e7774d493abf0cb2454481a5b879d2d41: the aggregate ci job passed on this head. enforce-target runs were cancelled or queued behind the runner backlog, not failed on the merits. Codex review findings on this PR are fixed on this head. No maintainer change requests are outstanding. Carries #6091 by @wtfsayo with a Co-authored-by trailer.

@lidge-jun
lidge-jun merged commit dce4469 into dev Sep 28, 2026
32 of 34 checks passed
@lidge-jun
lidge-jun deleted the codex/rt5-devin-models branch September 28, 2026 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants