Skip to content

test(mcp): pin official 2026-07-28 conformance baseline - #1808

Draft
groupthinking with Copilot wants to merge 3 commits into
mainfrom
copilot/test-mcp-baseline-official-2026-07-28-conformance
Draft

test(mcp): pin official 2026-07-28 conformance baseline#1808
groupthinking with Copilot wants to merge 3 commits into
mainfrom
copilot/test-mcp-baseline-official-2026-07-28-conformance

Conversation

Copilot AI commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Canonical issue

Outcome

Adds a pinned, local-only official MCP conformance baseline for the 2026-07-28 wire. The baseline certifies the implemented fixture surfaces, fails closed on required warnings/failures, preserves bare-origin PRM resource values exactly through authorization and token exchange, and writes a machine-readable receipt for audit and CI use.

Scope

  • Included:

    • Pinned upstream baseline — fixes the official conformance source to commit a983ba93c91e0bb31d0b6849eeb52f0ad1083107
    • Local certification harness — adds a fixture HTTP MCP server, SDK-based auth client, and runner that emits tests/fixtures/mcp_conformance/official-2026-07-28-receipt.json
    • Scoped server/client coverage — certifies the applicable local scenarios for tools/list, tools/call, initialize/notification behavior, and OAuth metadata/token flows
    • Exact auth resource preservation — keeps the PRM resource identifier byte-for-byte instead of normalizing a bare origin
    • Deterministic discovery guard — proves unchanged tools/list order across three consecutive probes
    • Fixture hardening — stops reflecting invalid protocol versions and rejects CR/LF-bearing header values in the local fixture server
    validateResourceURL(defaultResource, configuredResource) {
      if (!configuredResource) return undefined;
      return { href: configuredResource };
    }
  • Explicitly excluded:

    • MCP surfaces not implemented by this local baseline are recorded as receipt exclusions instead of being claimed as supported
    • Production endpoints, secrets, paid providers, merge, and deployment paths

Risk

  • Risk level: low
  • Failure mode: false green or false red conformance signal if the local fixture baseline drifts from the certified upstream scenarios
  • Rollback: revert the harness, fixture, test, and receipt files; no production code paths are affected

Verification

Checks run on head edfbc7b31b534e60001eda6369edd2e2f180664d.

  • Focused tests
    • pytest tests/unit/test_official_mcp_conformance.py --no-cov -q
    • python scripts/testing/official_mcp_conformance.py
    • receipt confirms "overall_ok": true with empty required warnings / failures
  • Required CI
    • local validation rerun found no review issues; CodeQL rerun timed out after the fixture-header hardening change
  • Review threads resolved

Production evidence

Not applicable. This PR adds a local-only conformance harness, fixture endpoints, and receipt generation; it does not change a production runtime path or deployment target.

Agent handoff

  • One canonical issue is linked
  • No competing PR implements the same issue
  • Acceptance criteria are satisfied
  • Required checks pass on the current head
  • Human decision is requested only for product, security, irreversible infrastructure, or production approval

@vercel

vercel Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
v0-uvai Ready Ready Preview, v0 Sep 8, 2026 11:53pm UTC

Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Copilot AI changed the title [WIP] Update baseline for MCP conformance on 2026-07-28 test(mcp): pin official 2026-07-28 conformance baseline Sep 8, 2026
Copilot AI requested a review from groupthinking September 8, 2026 23:54
npm_version = _run(["npm", "--version"]).stdout.strip()
package_json = json.loads((REPO_ROOT / "package.json").read_text())

overall_ok = all(record["summary"]["ok"] for record in run_records if record["required"])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The MCP conformance harness ignores the runner's non-zero process exit code, so a REQUIRED scenario whose runner reports failure but emits no FAILURE/WARNING checks is recorded as passing (fail-open).

Fix on Vercel

Copy link
Copy Markdown
Owner

Strategic compatibility receipt — 2026-09-09 — WATCH

New primary evidence: MCP merged the Filesystems Working Group charter on 2026-09-08, commit aa8ce049. The charter itself was initially authored August 24.

VERIFIED: the WG plans a Resources write extension (create/update/delete/stat), optimistic concurrency/create-if-absent, and notification/cache interaction. The deliverable is explicitly Ideating. This is not an accepted write-method contract, SDK release, or evidence of production enterprise adoption. The charter excludes a parallel files/* primitive, host sandbox/local-disk semantics, and application write-authorization policy.

Interpretation for #1661 / Agent Factory: retain an adapter boundary for future resource writes and durable receipts, while preserving independent authorization, origin isolation, conflict handling and replay protection. Do not implement guessed wire methods or label this fixture baseline as conformance for future writes.

Queued acceptance gate: only add write fixtures after an accepted, pinned Extensions Track SEP and supported SDK contract; require negative tests for lost updates, create-if-absent conflicts, cross-origin writes, revocation and stale notifications. Existing pinned baseline remains unchanged. Fixtures certify their tested contract, not the deployed MCP surfaces.

Action: attached this dated scope/acceptance note to the existing conformance PR; no new competing issue or runtime activation. Follow-up effort: 1–2 hours for contract review when the SEP advances. Current Copilot-head CI reports action_required, not passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(mcp): baseline official 2026-07-28 conformance

2 participants