Skip to content

Add Z.ai Coding Plan routing - #1074

Merged
kywch merged 7 commits into
mainfrom
fix/zai-coding-routing
Sep 2, 2026
Merged

Add Z.ai Coding Plan routing#1074
kywch merged 7 commits into
mainfrom
fix/zai-coding-routing

Conversation

@kywch

@kywch kywch commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add explicit Z.ai Coding Plan provider, key, and endpoint routing
  • reject registry-derived base URLs that do not match proxy protocol
  • forward only validated generation parameters, using correct token field for Responses versus chat and native routes
  • support current catalog entries and unlisted future GLM 5.x model IDs

Validation

  • 296 focused tests passed
  • Ruff, format, and git diff --check passed
  • exact-head integration matrix completed all 10 selected arXiv conditions; OpenClaw and Claude ACP Z.ai conditions both earned reward 1
  • full matrix had zero sandbox or trial errors; sole zero reward was unrelated Gemini provider 429

Review

This PR is intentionally stacked on #1030. After #1030 merges, retarget to main and verify final diff.

Maintainer or other-person review required. Author will not self-merge.

@kywch
kywch deployed to pypi-internal-preview August 31, 2026 01:45 — with GitHub Actions Active

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

Comment thread src/benchflow/providers/litellm_config.py Outdated
@kywch
kywch deployed to pypi-internal-preview August 31, 2026 02:19 — with GitHub Actions Active
@kywch

kywch commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed endpoint-provenance finding: registry-generated provider env now carries explicit provenance, while user-supplied endpoints remain authoritative. Added direct, resolved-explicit, and registry-generated regressions. Refreshed onto #1030 head 5fbf930; all applicable CI checks pass.

@bingran-you

Copy link
Copy Markdown
Collaborator

Thermo-nuclear review at exact head 6409916c5836105cb7a775e1653603f817e88d3f: requesting changes. The full local suite passes (5900 passed, 58 skipped, 7 deselected), but direct contract probes found three blockers that the current tests miss.

  1. Endpoint provenance is tracked for the key/base pair instead of per field. In resolve_provider_env(), any explicit generic key suppresses the registry marker even when the base URL was filled by the registry. With a native Z.AI key plus an explicit BENCHFLOW_PROVIDER_API_KEY, no explicit base URL, claude-agent-acp, and a zai-coding/* model, the exact head resolved the registry Anthropic endpoint but left provenance unset. LiteLLM then produced an openai/glm-* route while retaining the Anthropic endpoint. That is a protocol mismatch. Please track base-URL and key provenance independently, ideally from the existing explicit-agent-env-key metadata, rather than inferring ownership from not any(BASE_URL, API_KEY).

  2. The advertised model catalog is not the current Coding Plan contract. The registry includes glm-5.2, which is absent from the current official Coding Plan list, while omitting GLM-4.7 and GLM-4.5-Air. Current source: https://docs.z.ai/devpack/overview. Please catalog only currently supported IDs; explicit future IDs can still use the provider-prefixed routing path without being advertised as known-good.

  3. The generic generation-parameter validator accepts values Z.AI rejects. A direct probe through _apply_model_generation_params() forwarded temperature=1.1, top_p=0, and max_tokens=200000. Z.AI documents temperature <= 1, top_p >= 0.01, and max_tokens <= 131072: https://docs.z.ai/api-reference/llm/chat-completion.

The cleaner shape is to keep this PR about Z.AI routing, represent endpoint/key origin explicitly per field, and split the unrelated generic generation-override mechanism unless it is backed by provider/model capability metadata. That removes magic inference instead of adding more special cases.

The live Z.AI credential reached the Coding endpoint but returned quota-exhausted HTTP 429, so authentication/endpoint acceptance is established but a successful end-to-end completion is not. Also, this PR still targets the open fix/gemini-acp-model-id/#1030 branch rather than main; after the blockers and base dependency are resolved, please retarget/refresh and rerun the end-to-end check.

@kywch
kywch deployed to pypi-internal-preview September 2, 2026 03:34 — with GitHub Actions Active
@kywch

kywch commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

@bingran-you Addressed review findings at d4708464: fixed mixed endpoint/key provenance, limited advertised Coding Plan models to GLM-5.3/GLM-5.3-Flash, removed generic generation overrides, and added reconnect coverage. ClawsBench tested with Claude ACP and OpenClaw; focused suites, Ruff, type checks, and diff checks pass.

@bingran-you
bingran-you deployed to pypi-internal-preview September 2, 2026 08:39 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator

Thermo-nuclear exact-head follow-up at 4ee7f78d8d9619abb0dbf91dfa542baf89d838cb: the earlier code findings are resolved, but this is not merge-ready yet.

The current implementation now has explicit per-field endpoint/key provenance, advertises only the current GLM-5.3 Coding Plan models, keeps the LiteLLM route on the Coding endpoint across agent protocols and reconnects, and removes the unrelated generic generation overrides. Validation passed locally: 419 focused tests, full suite 5932 passed, 86 skipped, 7 deselected, Ruff, ty, and diff checks.

Two merge gates remain:

  1. The only supplied Z.AI credential authenticates at the Coding Plan endpoint but returns HTTP 429 Weekly/Monthly Limit Exhausted with a stated reset of 2026-09-04 22:19:23. This proves endpoint/key acceptance, not a successful model completion.
  2. This PR is still stacked on fix/gemini-acp-model-id / Fix Gemini ACP model IDs with Google provider prefixes #1030. Because the base is not main, the current check set omits the full test and parity jobs. Fix Gemini ACP model IDs with Google provider prefixes #1030 is now green, but still awaits independent review and merge.

After #1030 lands, retarget/refresh this branch onto main, rerun the complete CI set, and run one successful real Coding Plan completion after quota resets. An independent exact-head human review is also required before squash merge.

@kywch
kywch changed the base branch from fix/gemini-acp-model-id to main September 2, 2026 19:39
@kywch
kywch deployed to pypi-internal-preview September 2, 2026 19:39 — with GitHub Actions Active
@kywch

kywch commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

@bingran-you Refreshed onto main at e88df27. Z.AI/ClawsBench validation completed; all applicable CI checks pass. Exact-head review found no blockers.

@kywch
kywch merged commit 1efdcfa into main Sep 2, 2026
11 checks passed
@bingran-you
bingran-you deleted the fix/zai-coding-routing branch September 2, 2026 19:47
@bingran-you

Copy link
Copy Markdown
Collaborator

Post-merge thermo-nuclear verification for exact PR head e88df27aafdc8d233536e8e15c02cfe7932fd512 and resulting main commit 1efdcfaf5cfbd7a4b547f346f54db93163d0fa53.

The head change from the previously reviewed 4ee7f78d is a merge of the newly landed #1030, #1069, #1076, and #1080 changes; the Z.AI-specific implementation diff is unchanged. The complete main-targeted CI set is green. I also validated the composed main tree locally: 5961 passed, 86 skipped, 7 deselected, with Ruff and ty check src/ clean.

Cross-feature real-backend smokes on the merged tree passed:

  • Gemini CLI + google/gemma-4-31b-it: reward 1, one tool, provider-response usage captured;
  • Codex ACP + Azure gpt-5.5 at xhigh: reward 1, three tools, provider-response usage captured;
  • native Claude OAuth timeout: honest 60-second timeout, pending tool retained, native ACP usage retained, terminal_trajectory_complete=false, verifier completed without error.

The successful Z.AI Coding Plan completion requested in the prior review is still not established: the only supplied Z.AI key remains quota-exhausted until its stated reset. This PR was merged before that real-provider gate cleared, so I will retry the Z.AI route against merged main after reset rather than retroactively describing it as pre-merge validated. The remote fix/zai-coding-routing branch has been deleted.

@bingran-you

Copy link
Copy Markdown
Collaborator

Post-reset provider verification on 2026-09-05 at 05:26 UTC: one minimal request using the supplied credential was sent directly to https://api.z.ai/api/coding/paas/v4/chat/completions with the merged registry model glm-5.3-flash. The endpoint again returned HTTP 429, now with provider code 1302 and message Rate limit reached for requests; no completion or usage object was produced. Because the direct provider gate failed, I did not spend a second call on the Docker E2E. This is renewed external provider/quota state rather than evidence of a merged routing defect, so the successful Coding Plan E2E evidence gap remains open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants