Skip to content

Honor Codex proxy-owned model selection - #1076

Merged
kywch merged 5 commits into
mainfrom
fix/codex-proxy-model-selection
Sep 2, 2026
Merged

Honor Codex proxy-owned model selection#1076
kywch merged 5 commits into
mainfrom
fix/codex-proxy-model-selection

Conversation

@kywch

@kywch kywch commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • treat exact CODEX_CONFIG.model ownership as authoritative
  • skip ACP set_model only when proxy config proves it owns requested model
  • fail closed to normal set_model behavior for missing, empty, malformed, non-object, missing-model, or mismatched config

Validation

  • 109 focused tests passed
  • Ruff, format, and git diff --check passed
  • exact-head arXiv integration condition codex-gpt54 completed at reward 1 with no provider, sandbox, or trial errors
  • full 10-condition matrix completed; sole zero reward was unrelated Gemini provider 429

Review

Maintainer or other-person review required. Author will not self-merge.


Devin Review

@kywch
kywch deployed to pypi-internal-preview August 31, 2026 01:45 — with GitHub Actions Active

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment thread src/benchflow/acp/runtime.py Outdated
@kywch
kywch deployed to pypi-internal-preview August 31, 2026 02:17 — with GitHub Actions Active
@kywch

kywch commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the proxy-owned Codex effort finding. Requested model_reasoning_effort now enters launch config without unsupported ACP model/effort calls. Regression covers no-effort and high; all CI checks pass.

@bingran-you

Copy link
Copy Markdown
Collaborator

Thermo-nuclear review at exact head 5476cff22e9692c00d727de37d2ee4d8fe9b26fc: the behavior is verified, but I am requesting a structural change before merge.

I ran a real Docker BenchFlow trial through the live Azure route with codex-acp, gpt-5.5, and xhigh. It completed with reward 1, the proxy selected gpt-5.5, both provider responses recorded reasoning.effort=xhigh, and usage was captured. The full suite passes (5871 passed, 58 skipped, 7 deselected).

The remaining concern is ownership. This PR adds a second CODEX_CONFIG parser in ACP runtime, then mutates model_reasoning_effort inside connect_acp(). Parsing and applying Codex launch configuration already has a canonical home in agents/codex_config.py. Moving part of that policy into transport orchestration creates two definitions of malformed-config behavior and makes a general ACP path understand Codex-specific JSON internals.

I think the code-judo move is to extend the canonical Codex configuration helper so it applies provider selection and requested reasoning effort atomically, then have runtime make one direct helper call. That should delete the local json dependency, _codex_config(), and the Codex-specific mutation branch from connect_acp(). The strict malformed-config contract can then be tested once at the canonical boundary, with only a thin integration assertion in runtime.

This is not cosmetic: merging the current #1076 and #1080 heads takes src/benchflow/acp/runtime.py from 911 to 1,001 lines. The feature works, but the queued implementation crosses the explicit 1k decomposition threshold and places policy in the wrong layer. Please keep the verified behavior and restructure the ownership before merge.

@kywch

kywch commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

@bingran-you Addressed requested ownership change in f5f7ce8d.

Codex launch configuration now has one canonical owner in codex_config.py: it parses/validates CODEX_CONFIG, applies reasoning effort before launch, and reports model ownership. Runtime makes one helper call and consumes that result; local JSON parsing and Codex-specific mutation are removed.

Tests now cover malformed config at canonical boundary, with thin runtime ownership/fallback checks. runtime.py: 946 → 929 LOC.

Validation: affected tests 138 passed; Ruff/type/diff checks pass; live ClawsBench Codex run succeeds. Full suite: 5872 passed, with 2 known baseline credential failures.

@kywch
kywch deployed to pypi-internal-preview September 2, 2026 04:54 — with GitHub Actions Active
@bingran-you
bingran-you deployed to pypi-internal-preview September 2, 2026 08:19 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator

Thermo-nuclear exact-head follow-up at c50e4c12648f7370ae9c9fc2eb846104bd54b9d6: the code and E2E behavior are ready; independent human review is the remaining merge gate.

The ownership restructuring is correct: agents/codex_config.py is now the single owner of Codex launch configuration, and the final repair binds proxy ownership to the requested model alias so stale proxy config cannot suppress ACP model selection for a different model.

The exact-head Docker run through the real Azure route completed end to end with reward 1 and one tool call. Both provider responses used gpt-5.5, both recorded reasoning.effort=xhigh, usage came from provider responses (14,760 input, 216 output, 7,168 cache-read tokens), and verifier/error fields were clean. Validation also passed: full suite 5912 passed, 86 skipped, 7 deselected, Ruff, ty, diff checks, and all current CI checks.

For completeness, forcing --sandbox-user root causes the verifier service to disappear after an ACP run, but the same failure reproduces on untouched main; the supported default sandbox-user run above is fully green, so that is not introduced by this PR.

I authored the final alias-binding repair during this review, so an independent maintainer must review this exact SHA before squash merge under the repository policy.

@bingran-you
bingran-you requested a review from xdotli September 2, 2026 09:35
@kywch
kywch merged commit 5d73839 into main Sep 2, 2026
11 checks passed
@bingran-you
bingran-you deleted the fix/codex-proxy-model-selection branch September 2, 2026 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants