Skip to content

Add a Claude Code compatibility matrix and release gate #78

Closed

Description

@NOirBRight

Part of #73.

Blocked by: #75, #76, #77 (native dependencies). This is the release gate: it proves the combination, it does not repair it.

Objective

Prove the combined Claude Code client integration with a versioned compatibility matrix, extend the client E2E matrix with Anthropic Messages cases, and gate the "stable" claim on recorded evidence.

Authority

ADR-0014 "Evidence and release gates": every upstream path must show real-CLI multi-turn text, incremental streaming, tool call/result round trips, cancellation, and error handling; the native Anthropic path additionally proves semantic-field and SSE preservation. Publish verified CLI versions and limitations in English and Chinese, and do not claim Anthropic endorses non-Claude model compatibility.

Scope

1. Real-client matrix

  • Extend scripts/e2e_gateway_client_matrix.py with Anthropic Messages request, JSON response, and streaming SSE cases.
  • Run against all three upstream protocols: native Anthropic Messages, OpenAI Responses, and Chat Completions — using provider identities that are explicitly approved for live evidence.
  • Cover main-session text, multi-turn history, tool call + result, parallel tools, images, compaction/resume, subagents, thinking/effort, prompt caching, cancellation, and error recovery to the extent Spike Claude Code Messages compatibility through CodexHub #74 classified them; unsupported cases are recorded as unsupported, not skipped silently.
  • Record the minimum supported and the latest verified Claude Code versions, with the exact executable and platform for every claim.

2. Credential contract — explicit prerequisite

docs/agents/real-client-e2e.md requires a new Issue and a dedicated credential contract before a client is added to the live matrix; the same rule that governs xAI Grok applies here. Before any live Claude Code row is added, record on this issue:

  • the credential contract shape (input file, schema id, provider identity) and the Issue that owns it;
  • which provider identities are approved for live evidence and who approved them;
  • that no credential material enters this issue, fixtures, logs, or commits.

If approval is not granted, leave this issue blocked and state the concrete blocker. Do not infer consent from a configured provider, and do not substitute mock-only runs for live acceptance.

3. Support table

  • Produce a support table distinguishing preserved, adapted (named policy), unsupported, unknown, and provider-specific behavior, traceable to Spike Claude Code Messages compatibility through CodexHub #74's classification and Add inbound Anthropic Messages API for Claude Code clients #75's implementation.
  • Unsupported features are visible in the matrix and in the client UI (card compatibility state / request diagnostics). No feature is silently claimed.
  • Version drift produces an actionable report with sanitized fixtures; scheduled/report-only upstream compatibility checks become blocking only after stability criteria are approved.

4. Release gate

  • Release notes and documentation state the verified Claude Code version range, the experimental scope, and the known limitations in English and Chinese.
  • A maintainer records the criteria for moving from experimental to stable, and the change that would satisfy them.
  • Relevant Python, Rust, and frontend suites pass at the candidate commit; the quality report runs once and remains non-blocking.

Out of scope

Protocol implementation (#75), client configuration and UI (#76), model contract (#77), and any ACP/AgentProvider work (#85).

Acceptance criteria

  • Automated E2E covers the supported text/tool/SSE contract against every approved upstream protocol.
  • A real pinned Claude Code CLI passes multi-turn text, incremental streaming, tool call/result round trips, cancellation, and error handling on each approved upstream path.
  • The native Anthropic path separately proves semantic-field and SSE preservation.
  • The support table is complete, classified, and traceable to evidence for every capability in scope.
  • Unsupported features appear in the matrix and the client UI; no silent claim remains.
  • Minimum and latest verified Claude Code versions are recorded with exact executables.
  • Version drift generates an actionable report with sanitized fixtures.
  • The credential contract and its owning Issue are recorded before the first live Claude Code matrix row; no credentials are committed or pasted into issues.
  • English and Chinese release documentation state the version range, experimental scope, and limitations.
  • A maintainer records the experimental → stable criteria.
  • Relevant Python, Rust, and frontend suites pass on the reviewed SHA; report-only quality findings are recorded without becoming a release gate.

Required verification

Risk class: strict — release gate, live runtime evidence, security boundary. Use the repository launcher; never a bare python/pytest.

.\scripts\codexhub-python.cmd -m pytest -q --ignore=tests/test_real_client_e2e.py
.\scripts\codexhub-python.cmd -m pytest -q tests/test_real_client_e2e.py
.\scripts\codexhub-python.cmd scripts/report_quality_gates.py
Push-Location src-tauri
cargo test --locked
cargo clippy --locked --all-targets -- -D warnings
Pop-Location
Push-Location frontend
npm run build
npm run test:ui-contract
Pop-Location

Linux: ./scripts/verify-linux.sh. Real-client acceptance is a local release-operator procedure per docs/agents/real-client-e2e.md; a pass on one operating system never replaces the other.

References

  • scripts/e2e_gateway_client_matrix.py, tests/test_e2e_gateway_client_matrix.py
  • docs/agents/real-client-e2e.md, docs/agents/verification-policy.md, docs/agents/release.md
  • docs/reviews/v0.3-agent-provider-roadmap.md
  • E2E third-party: official DeepSeek Flash #497 (OpenCode Go credential-contract precedent)

Document status

ADR-0014 is committed in draft PR #556 at ed76f0a6e22d3188879504638b2943f5bb855d41, not yet merged to main. The pinned link is readable now; the document is accepted direction, not live protocol evidence.

Activity

  1. NOirBRight commented on Sep 21, 2026

    @NOirBRight
    OwnerAuthor

    PARTIAL: harness/matrix now emits claude-code anthropic-messages rows and the experimental support table is in docs/evidence/claude-code-support-table.md. Live Claude Code rows remain blocked: no dedicated credential-contract Issue has been recorded, so this ticket cannot add live matrix evidence. Pinned observed CLI is 2.1.278 (loopback only).

  2. NOirBRight commented on Sep 21, 2026

    @NOirBRight
    OwnerAuthor

    Credential contract recorded as #558. Approved live identities: Official Codex gpt-5.6-luna (Responses conversion) and CommandCode deepseek/deepseek-v4.1-flash (Chat conversion). Native Anthropic upstream is not approved. No credentials in this comment.

  3. NOirBRight commented on Sep 21, 2026

    @NOirBRight
    OwnerAuthor

    Linux checkpoint at 011a312/c468a3b lineage, current campaign HEAD after test-fix merge: core pytest 3143 passed, clippy -D warnings, UI contract 7 passed. Live Linux CLI/text/tool/cancel recorded on #558. Windows/release-operator evidence is still required; Linux does not replace it. Draft PR 556 remains experimental, not ready.

  4. NOirBRight commented on Sep 21, 2026

    @NOirBRight
    OwnerAuthor

    Windows yoga observation: Claude Code 2.1.232 at C:\Users\noirb.local\bin\claude.exe. Pin is 2.1.278. Version-gate would exit 2 (drift). No Windows live matrix was run; Linux 2.1.278 evidence does not cover yoga. Isolated apply/readback matrix still passes on Linux including claude.

  5. NOirBRight commented on Sep 21, 2026

    @NOirBRight
    OwnerAuthor

    yoga Claude Code upgraded 2.1.232 → 2.1.278 via claude update. Version-gate matches pin. Windows live matrix not started; operator will log in.

  6. NOirBRight commented on Sep 21, 2026

    @NOirBRight
    OwnerAuthor

    Windows live on yoga: isolated checkout D:\Workstation\CodexHub-claude-live at 9eb0634. Claude Code 2.1.278 print-run through isolated Gateway Official gpt-5.6-luna returned is_error=false, result=ok. Operator worktree D:\Workstation\CodexHub left untouched. CommandCode/DeepSeek are not configured on yoga so those Windows rows were not run. No credentials in this comment.

  7. NOirBRight commented on Sep 21, 2026

    @NOirBRight
    OwnerAuthor

    Windows yoga isolated Gateway now has the approved CommandCode and DeepSeek Anthropic providers (copied into the isolated CODEX_HOME only). Claude Code 2.1.278 print-runs: CommandCode result=ok; DeepSeek Anthropic is_error=false result=好的. Luna already passed. Operator D:\Workstation\CodexHub untouched. No credentials in this comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestneeds-triageMaintainer needs to evaluate this issuewayfinder:taskWayfinder implementation task

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions