Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .claude/skills/mcp-sdk-tier-audit/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,8 @@ npm run --silent tier-check -- \
--output json
```

Omit `--requirements` only if the user did not name a revision. When it is set the
Omit `--requirements` only if the user did not name a revision (the CLI then
scores against the latest shipped revision's requirement set alone). When it is set the
scorecard reports `requirements_revision`, both pass rates count exactly the
scenarios that revision requires, and anything run but not scored carries a
`notScoredReason` of `extension` or `added-after-release`. `requirements_revisions`
Expand All @@ -93,7 +94,7 @@ If no client-cmd was detected, omit the `--client-cmd` flag (client conformance

The CLI output includes server conformance pass rate, client conformance pass rate (with per-spec-version breakdown), issue triage compliance, P0 resolution times, label taxonomy, stable release status, policy signal files, and spec tracking gap. Parse the JSON output to feed into Step 4.

The conformance results now include a `specVersions` field on each detail entry, enabling per-version pass rate analysis. The `list` command also shows spec version tags: `node dist/index.js list` shows `[2025-06-18]`, `[2025-11-25]`, `[draft]`, or `[extension]` next to each scenario.
The conformance results now include a `specVersions` field on each detail entry, enabling per-version pass rate analysis. The `list` command also shows spec version tags: `node dist/index.js list` shows the dated revisions a scenario applies to (`[2025-06-18,2025-11-25,2026-07-28]`), `[draft]` for a scenario targeting the unreleased revision after the latest, or `[extension]`.

### Conformance Baseline Check

Expand Down Expand Up @@ -188,7 +189,7 @@ Informational (not scored for tier):
| ------------ | ----- | --------- |
| Client: Auth | 0/1 | 0/2 |

The tier-scoring table only includes date-versioned scenarios. `draft` and `extension` scenarios are shown separately as informational — they do not affect tier advancement.
The tier-scoring table only includes date-versioned scenarios (`2026-07-28` is a dated release and scores). `draft` (requirements newer than the latest release) and `extension` scenarios are shown separately as informational — they do not affect tier advancement.

This immediately shows where failures concentrate. Failures clustered in Client: Auth / `2025-11-25` means "new auth features not yet implemented" — a scope gap, not a quality problem. Failures in Server or Client: Core are more concerning.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,9 @@ Source: `modelcontextprotocol/docs/community/sdk-tiers.mdx` in the spec reposito

Every scenario in the conformance suite has a `specVersions` field indicating which spec version it targets. The valid values are defined as the `SpecVersion` type (as a list) in `src/types.ts` — run `node dist/index.js list` to see the current mapping of scenarios to spec versions.

Date-versioned scenarios (e.g. `2025-06-18`, `2025-11-25`) count toward tier scoring. `draft` and `extension` scenarios are listed separately as informational.
Date-versioned scenarios (e.g. `2025-06-18`, `2025-11-25`, `2026-07-28`) count toward tier scoring. `draft` (requirements newer than the latest dated release) and `extension` scenarios are listed separately as informational.

The `--spec-version` CLI flag filters scenarios cumulatively for date versions (e.g. `--spec-version 2025-06-18` includes `2025-03-26` + `2025-06-18`). For `draft`/`extension`, it returns exact matches only.
The `--spec-version` CLI flag selects the scenarios applicable at a revision: cumulative for date versions (e.g. `--spec-version 2025-06-18` includes `2025-03-26` + `2025-06-18`) minus anything that revision removed (`2026-07-28` drops the stateful-lifecycle scenarios); `draft` adds any draft-only scenarios on top of the latest release. Extension scenarios are never selected by `--spec-version` (use `--suite extensions`).

The tier-check output includes a per-version pass rate breakdown alongside the aggregate.

Expand Down
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ Keep scenarios separate when they're genuinely independent features or when they
- Start with **one end-to-end scenario** covering the happy path with many checks along the way.
- Don't add "step 1 only" and "step 1+2" as separate scenarios — the second subsumes the first.
- Register the scenario in the appropriate suite list in `src/scenarios/index.ts` (`core`, `extensions`, `backcompat`, etc.).
- Set `source.introducedIn` to the dated revision the requirement shipped in (e.g. `'2026-07-28'`). Use `DRAFT_SPEC_VERSION` only for a requirement that is in `docs/specification/draft/` but not yet in a dated release; such scenarios run under `--suite draft` / `--spec-version draft` and stay out of the default suite and tier scoring until the draft ships and they are retagged to its date. `DRAFT_PROTOCOL_VERSION` is the draft's wire string, not a tag.

## Check conventions

Expand Down
18 changes: 11 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,15 +63,15 @@ npx @modelcontextprotocol/conformance client --command "<client-command>" --scen

- `--command` - The command to run your MCP client (can include flags)
- `--scenario` - The test scenario to run (e.g., "initialize")
- `--suite` - Run a suite of tests in parallel: `all`, `core`, `extensions`, `backcompat`, `auth`, `metadata`, `draft` (scenarios targeting the in-progress draft spec), or `sep-835`
- `--spec-version <version>` - Filter scenarios by spec version (e.g., `2025-11-25`, `2026-07-28`; `draft` is accepted as an alias for the current draft identifier). The draft version selects the latest dated release plus any draft-only scenarios. When omitted, the version is inferred from the scenario's spec applicability (draft-only scenarios run at the draft version, everything else at the latest dated release); an explicitly requested version outside a scenario's applicability window skips the scenario (exit 0) unless `--force` is passed
- `--suite` - Run a suite of tests in parallel: `all`, `core`, `extensions`, `backcompat`, `auth`, `metadata`, `draft` (scenarios for requirements newer than the latest dated release — currently empty, since everything that was in it shipped in `2026-07-28`), or `sep-835`
- `--spec-version <version>` - Target a spec revision: a dated release (`2025-03-26`, `2025-06-18`, `2025-11-25`, `2026-07-28`) or `draft` for the unreleased revision after the latest. A dated version selects every scenario applicable at that revision (cumulative, minus anything the revision removed); `draft` selects the latest release's set plus any draft-only scenarios and runs them at the draft's wire `protocolVersion` (currently also `2026-07-28`, until the spec repo gives the next draft its own). When omitted, each scenario runs at the latest dated release if it applies there, otherwise at the newest revision it does apply to (draft-only scenarios at the draft, scenarios removed in `2026-07-28` at `2025-11-25`); an explicitly requested version outside a scenario's applicability window skips the scenario (exit 0) unless `--force` is passed
- `--force` - Run a scenario even if it is not applicable at the requested `--spec-version`
- `--requirements <revision>` - Run exactly what a spec revision requires, frozen at its release (see [Conformance Requirements](#conformance-requirements))
- `--expected-failures <path>` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures))
- `--timeout` - Timeout in milliseconds (default: 30000)
- `--verbose` - Show verbose output

The framework appends `<server-url>` as an argument to your command and sets the `MCP_CONFORMANCE_SCENARIO` environment variable to the scenario name. For scenarios that require additional context (e.g., client credentials), the `MCP_CONFORMANCE_CONTEXT` environment variable contains a JSON object with scenario-specific data. When `--spec-version` is passed, its resolved value is forwarded to the client process as `MCP_CONFORMANCE_PROTOCOL_VERSION`; example clients can use this value directly as their `protocolVersion`. SDKs that hard-code their protocol version can ignore it. Clients under test must derive the lifecycle from the protocol version they are asked to run: dated versions through `2025-11-25` use the stateful lifecycle (initialize handshake), while the 2026 draft (`2026-07-28`) uses the stateless lifecycle (per-request `_meta`).
The framework appends `<server-url>` as an argument to your command and sets the `MCP_CONFORMANCE_SCENARIO` environment variable to the scenario name. For scenarios that require additional context (e.g., client credentials), the `MCP_CONFORMANCE_CONTEXT` environment variable contains a JSON object with scenario-specific data. The wire `protocolVersion` of the revision a scenario runs at is forwarded to the client process as `MCP_CONFORMANCE_PROTOCOL_VERSION`; example clients can use this value directly as their `protocolVersion`. SDKs that hard-code their protocol version can ignore it. Clients under test must derive the lifecycle from the protocol version they are asked to run: dated versions through `2025-11-25` use the stateful lifecycle (initialize handshake), while `2026-07-28` and later use the stateless lifecycle (per-request `_meta`).

### Server Testing

Expand All @@ -83,7 +83,8 @@ npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]

- `--url` - URL of the server to test
- `--scenario <scenario>` - Test scenario to run (e.g., "server-initialize"). Runs all available scenarios by default
- `--suite <suite>` - Suite to run: "active" (default; excludes pending and draft-spec scenarios), "all", "draft" (scenarios targeting the in-progress draft spec), or "pending"
- `--suite <suite>` - Suite to run: "active" (default; excludes pending and draft-only scenarios), "all", "draft" (scenarios for requirements newer than the latest dated release — currently empty), or "pending"
- `--spec-version <version>` - Target a spec revision (see Client Testing above). When omitted, scenarios that apply to `2026-07-28` run at that (stateless) wire and scenarios it removed run at `2025-11-25`; pass `--spec-version 2025-11-25` to test a server that only implements the stateful lifecycle
- `--requirements <revision>` - Run exactly what a spec revision requires, frozen at its release (see [Conformance Requirements](#conformance-requirements))
- `--expected-failures <path>` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures))
- `--verbose` - Show verbose output
Expand Down Expand Up @@ -364,7 +365,7 @@ npm start -- sdk typescript-sdk --mode client --suite auth
# Target a specific spec version (passed through to the underlying run).
# When omitted, the SDK's `specVersion` from KNOWN_SDKS is used, if set —
# e.g. typescript-sdk-v1 defaults to 2025-11-25.
npm start -- sdk typescript-sdk --mode client --spec-version draft
npm start -- sdk typescript-sdk --mode client --spec-version 2026-07-28
```

Build/run commands for each official SDK are looked up by name from [`src/sdk-runner/known-sdks.ts`](src/sdk-runner/known-sdks.ts) — no config file is required in the SDK repo. Resolution order is **CLI flag > built-in entry**, so any field can be overridden on the command line for refs that diverge from the built-in.
Expand Down Expand Up @@ -434,8 +435,11 @@ npx @modelcontextprotocol/conformance tier-check \
```

Omit `--client-cmd` and the client leg is skipped and reported as a gap. Omit
`--requirements` and scoring falls back to the suite as it stands today, which is
not what you want for a tier claim; see [Conformance Requirements](#conformance-requirements).
`--requirements` (and `--spec-version`) and scoring uses the latest shipped
revision's requirement set alone (`2026-07-28`), which claims one revision, not
every one; pass `--requirements` explicitly for a tier claim. `--spec-version`
without `--requirements` scores the suite as it stands today at that revision,
which is not frozen; see [Conformance Requirements](#conformance-requirements).

The exit code is a CI verdict on the machine-checkable half: nonzero when any
scored conformance scenario fails or a leg could not be measured. Governance
Expand Down
22 changes: 13 additions & 9 deletions examples/clients/typescript/everything-client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ import type {
import { JWT_BEARER_GRANT_TYPE } from '../../../src/scenarios/client/auth/helpers/createWorkloadJwt.js';
import { ElicitRequestSchema } from '@modelcontextprotocol/sdk/types.js';
import { ClientConformanceContextSchema } from '../../../src/schemas/context.js';
import { DRAFT_PROTOCOL_VERSION } from '../../../src/types.js';
import { STATELESS_SPEC_VERSIONS } from '../../../src/connection/select.js';
import { LATEST_SPEC_VERSION } from '../../../src/types.js';
import { STATELESS_PROTOCOL_VERSIONS } from '../../../src/connection/select.js';
import {
auth,
extractWWWAuthenticateParams
Expand Down Expand Up @@ -94,13 +94,14 @@ const PROTOCOL_VERSION = process.env.MCP_CONFORMANCE_PROTOCOL_VERSION;
// in-repo client imports the stateless version set from src/ so it cannot
// drift from the runner's mapping.
const USE_STATELESS_LIFECYCLE = PROTOCOL_VERSION
? (STATELESS_SPEC_VERSIONS as readonly string[]).includes(PROTOCOL_VERSION)
? STATELESS_PROTOCOL_VERSIONS.includes(PROTOCOL_VERSION)
: false;

// Wire protocolVersion for stateless requests: the runner-resolved version
// when available (so a dated stateless release is exercised under its own
// identifier), the current draft otherwise.
const STATELESS_PROTOCOL_VERSION = PROTOCOL_VERSION ?? DRAFT_PROTOCOL_VERSION;
// when available (so each stateless revision is exercised under its own
// identifier), the latest release otherwise.
const STATELESS_PROTOCOL_VERSION: string =
PROTOCOL_VERSION ?? LATEST_SPEC_VERSION;

const STATELESS_META_BASE = {
'io.modelcontextprotocol/clientInfo': {
Expand Down Expand Up @@ -336,15 +337,18 @@ async function runRequestMetadataClient(serverUrl: string): Promise<void> {
const clone = response.clone();
try {
const errorResult = await clone.json();
// UnsupportedProtocolVersionError is -32022 in the draft schema.
// UnsupportedProtocolVersionError is -32022 (2026-07-28 onward).
if (errorResult.error?.code === -32022) {
logger.debug(
'Received UnsupportedProtocolVersionError, starting negotiation...'
);
const serverSupported: string[] =
errorResult.error.data?.supported || [];
const clientSupported = [
...new Set([STATELESS_PROTOCOL_VERSION, DRAFT_PROTOCOL_VERSION])
...new Set([
STATELESS_PROTOCOL_VERSION,
...STATELESS_PROTOCOL_VERSIONS
])
];
const mutuallySupported = clientSupported.filter((v) =>
serverSupported.includes(v)
Expand Down Expand Up @@ -459,7 +463,7 @@ registerScenarios(
'auth/token-endpoint-auth-none',
// Resource mismatch (client should error when PRM resource doesn't match)
'auth/resource-mismatch',
// SEP-2207: Offline access / refresh token guidance (draft)
// SEP-2207: Offline access / refresh token guidance (2026-07-28)
'auth/offline-access-scope',
'auth/offline-access-not-supported',
// SEP-2468: ISS parameter - positive scenarios (standard client is fine)
Expand Down
11 changes: 6 additions & 5 deletions examples/servers/typescript/everything-server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1235,7 +1235,7 @@ const LEGACY_SESSION_PROTOCOL_VERSIONS = [
'2025-11-25'
];

// Stateless (draft) operations whose results MUST carry the SEP-2549 caching
// Stateless (2026-07-28+) operations whose results MUST carry the SEP-2549 caching
// hints (`ttlMs`, `cacheScope`).
const STATELESS_CACHEABLE_METHODS: ReadonlySet<string> = new Set([
'server/discover',
Expand All @@ -1246,9 +1246,9 @@ const STATELESS_CACHEABLE_METHODS: ReadonlySet<string> = new Set([
'resources/read'
]);

/** Normalize a stateless (draft) JSON-RPC response. Draft results MUST carry `resultType`
/** Normalize a stateless (2026-07-28+) JSON-RPC response. These results MUST carry `resultType`
* and cacheable operations the SEP-2549 caching hints; stamp any the dispatch site did
* not set so every stateless result is draft-schema-valid. Errors pass through untouched. */
* not set so every stateless result is schema-valid. Errors pass through untouched. */
function normalizeStatelessResponse(
method: string,
payload: { result?: Record<string, unknown>; [key: string]: unknown }
Expand Down Expand Up @@ -1620,7 +1620,8 @@ app.post('/mcp', async (req, res) => {
{
uri: 'test://stateless-static-text',
name: 'Stateless Static Text',
description: 'A static text resource served on the draft path',
description:
'A static text resource served on the stateless path',
mimeType: 'text/plain'
}
],
Expand Down Expand Up @@ -1677,7 +1678,7 @@ app.post('/mcp', async (req, res) => {
{
uri,
mimeType: 'text/plain',
text: 'Static text content from the stateless draft path.'
text: 'Static text content from the stateless path.'
}
],
ttlMs: 300000,
Expand Down
34 changes: 32 additions & 2 deletions scripts/sync-schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,24 @@
/**
* Vendor schema/{version}/schema.ts and schema/{version}/schema.json from the
* modelcontextprotocol spec repo into src/spec-types/{version}.ts and
* src/spec-types/{version}.schema.json at a pinned SHA.
* src/spec-types/{version}.schema.json at a pinned SHA, for every entry of
* DATED_SPEC_VERSIONS plus `draft`, and regenerate src/spec-types/schemas.ts
* (the version -> JSON Schema map the wire validator compiles).
*
* Publishing a new spec revision: append its date to DATED_SPEC_VERSIONS in
* src/types.ts, then run this at the release tag. Nothing else in the schema
* plumbing needs editing.
*
* Usage: npm run sync-schema -- <sha-or-ref>
*/
import { execFileSync } from 'node:child_process';
import { mkdirSync, writeFileSync, rmSync, copyFileSync } from 'node:fs';
import { join } from 'node:path';
import { DATED_SPEC_VERSIONS, DRAFT_SPEC_VERSION } from '../src/types';

const VERSIONS = ['2025-03-26', '2025-06-18', '2025-11-25', 'draft'] as const;
// DRAFT_SPEC_VERSION is the literal 'draft', which is also the spec repo's
// directory name for the in-progress revision.
const VERSIONS = [...DATED_SPEC_VERSIONS, DRAFT_SPEC_VERSION] as const;
const SPEC_REPO =
'https://github.com/modelcontextprotocol/modelcontextprotocol.git';
const OUT_DIR = join(process.cwd(), 'src', 'spec-types');
Expand All @@ -29,6 +38,13 @@ mkdirSync(OUT_DIR, { recursive: true });
const git = (args: string[]) =>
execFileSync('git', args, { cwd: tmp, encoding: 'utf8' });

/** `2025-03-26` -> `schema2025_03_26`, `draft` -> `schemaDraft`. */
function importName(v: string): string {
return v === DRAFT_SPEC_VERSION
? 'schemaDraft'
: `schema${v.replace(/-/g, '_')}`;
}

try {
console.log(`Fetching ${SPEC_REPO} @ ${ref} ...`);
git(['init', '-q']);
Expand All @@ -47,6 +63,20 @@ try {
console.log(` ${v} -> src/spec-types/${v}.schema.json`);
}

const schemasTs =
`// Generated by scripts/sync-schema.ts from DATED_SPEC_VERSIONS + draft.\n` +
`// Do not edit: add the version to src/types.ts and re-run the script.\n` +
`import type { SpecVersion } from '../types';\n` +
VERSIONS.map(
(v) => `import ${importName(v)} from './${v}.schema.json';\n`
).join('') +
`\n/** Vendored JSON Schema for every spec version the suite can target. */\n` +
`export const SPEC_SCHEMAS: Record<SpecVersion, Record<string, unknown>> = {\n` +
VERSIONS.map((v) => ` '${v}': ${importName(v)},\n`).join('') +
`};\n`;
writeFileSync(join(OUT_DIR, 'schemas.ts'), schemasTs, 'utf8');
console.log(' version map -> src/spec-types/schemas.ts');

writeFileSync(
join(OUT_DIR, 'SOURCE'),
`modelcontextprotocol@${sha}\n`,
Expand Down
Loading
Loading