Skip to content

Retry with max_completion_tokens when the endpoint rejects max_tokens - #1232

Open
Schro-code wants to merge 2 commits into
deeplethe:devfrom
Schro-code:fix/compat-retries-unsupported-max-tokens
Open

Schro-code wants to merge 2 commits into
deeplethe:devfrom
Schro-code:fix/compat-retries-unsupported-max-tokens

Conversation

@Schro-code

@Schro-code Schro-code commented Oct 14, 2026 •

Copy link
Copy Markdown

Why

Newer OpenAI reasoning models (o1, o3, GPT-5 family) renamed max_tokens to max_completion_tokens. Sending the old name returns a 400 with error.code: "unsupported_parameter", and every extraction chunk fails. The same models also reject temperature values other than their default (1) with error.code: "unsupported_value".

What changes

A send_with_compat_retries method inspects error.param and error.code in a 400 response and retries once with the corrected parameter: max_tokens is renamed to max_completion_tokens, and a rejected temperature is dropped. chat_at (non-streaming) routes through it. The same detection is woven into chat_at_streaming's existing retry loop so the streaming path also handles the transition.

Each adjustment applies at most once per request — the already-applied key is removed from the body before retrying, so the loop terminates even if the endpoint keeps rejecting.

How it was checked

Tested against an Azure OpenAI endpoint running gpt-5.6-sol, which rejects max_tokens. Before the fix: 70/70 extraction chunks fail with 400. After: all pass on the first retry.

Before review

  • Every commit is signed off (git commit -s)
  • cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace pass
  • For changes under web/: pnpm build and pnpm test pass — N/A, no web changes
  • SQL under crates/utopia-store/ was tested with UTOPIA_DATABASE_URL set — N/A, no SQL changes
  • A new migration takes the next free number on dev — N/A, no migrations
  • UI strings are in both web/src/i18n/en.ts and zh.ts — N/A, no UI strings
  • A change to the data model, the ontology contract or a public API has its ADR in docs/decisions/ — N/A

Zhoulu added 2 commits October 14, 2026 18:08
Signed-off-by: Zhoulu <ldazzh@lineadirecta.es>
Signed-off-by: Zhoulu <ldazzh@lineadirecta.es>

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, the problem is real and reading error.param / error.code is the right way to detect it.

Two things to change:

  • In chat_at_streaming each attempt rewrites body["max_tokens"], so after the rename the next request carries it again and the retry can loop without end.
  • Every call pays the 400 again (once per chunk). Learn the parameter name and a rejected temperature once on the client, as completion_ceiling already does.

Please add a wiremock test like the existing "max_tokens is too large" one, and run cargo fmt (the lint failure). Thanks again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants