feat(client): load scenarios from outside the repository

[claude]

Requested by Felix Weinberger · Slack thread

Adds --scenario-file <path> to client and list, so a team can run its own client scenarios through the runner without changing this repository.

Why

Client scenarios can only be added by changing this repository. That is right for conformance, but it leaves no place for two things people want to do with the same runner:

• Checks that belong to one product, not to the spec. For example "our client tags every tool call with a trace id". These will never be conformance requirements, and today the only way to run them is to fork.
• Trying a scenario out before proposing it. Client coverage outside auth is thin. Scenarios for pagination, tool results, resources or prompts are easier to propose once they have run against real clients for a while.

registerScenario() already exists in src/scenarios/index.ts and has no caller; this gives it one.

Try it

After npm install && npm run build:

node dist/index.js client --scenario-file examples/scenarios/trace-id.mjs --scenario example/trace-id --command "node examples/scenarios/trace-id-client.mjs"

A scenario file is a JavaScript module whose default export is one scenario (name, description, source, start(ctx), stop(), getChecks()) or an array of them. It imports nothing from the package, and the same file serves every spec version the scenario applies to.

What is in it

• src/scenarios/custom.ts: loads the modules, validates names and shapes, registers through the existing registerScenario(), and applies the selection rules.
• src/index.ts: the option on client and list, the custom suite, the (custom) label.
• src/scenarios/index.ts: listScenariosForSpec() reads the registry, so loaded scenarios follow --spec-version.
• examples/scenarios/: trace-id.mjs, a complete commented scenario for a product-specific check, and a client that passes it.
• README.md: "Writing Your Own Scenarios".
• src/scenarios/custom.test.ts: 51 tests, including the example end to end through runConformanceTest.

How it stays apart from conformance

• A run with --scenario-file never yields official results: it cannot be combined with --requirements, a built-in suite or a built-in scenario, and that is checked before any file is imported.
• Names must be path-safe and must not match any built-in scenario, of any command, in any case.
• Every run prints which scenarios were loaded and that they are not part of MCP conformance; list and the suite summary mark them (custom).
• The checks a loaded scenario returns are validated, because it is untyped: status: 'PASS' would otherwise print "Passed: 0/0 … OVERALL: PASSED".
• The interface is documented as experimental.

Behaviour without the option

Unchanged. list output and the membership and order of every suite at every spec version are byte-identical to main.

Notes for review

• The example adds a small client file where AGENTS.md prefers extending everything-client.ts. A scenario that lives outside the suite needs a client that lives outside it too; the everything-client dispatches on built-in scenario names.
• Loading a file runs its code (import()), the same trust model as --command. The README says so.
• Suite mode is used whenever no --scenario is given, also for a file with one scenario, so that exit codes do not depend on how many scenarios a file defines.
• Not covered here: a module or a start() that never settles ends the process with exit 0. The same is true of built-in scenarios, so the fix belongs in the runner and can be sent separately.
• Out of scope: the server command and the hosted deployment, which keeps running exactly the scenarios in this repository.
• May need a rebase if fix: bound client suite concurrency #509, Treat 2026-07-28 as a released spec revision; keep draft for what comes after it #495 or Standardize scenario setup/execution failure reporting (#248) #327 land first.

Open question

traceability --results <dir> reads every checks.json under a directory and counts check ids of the form sep-<number>-…. A custom run saved into the same directory would be counted. This PR only documents "use your own id prefix". Should loaded scenarios be refused such ids, or should checks.json carry a marker?

How it was tested

• npm run check and npm test: clean, 48 files, 673 tests.
• The built CLI driven end to end: passing client, client that does nothing, untagged call, name clashes, hostile names, missing file, syntax error, every conflicting option combination, -o, --expected-failures, --spec-version, interactive mode, from another folder.
• A first-time user, given only the README and the example, wrote a three-page pagination scenario and a tool-error scenario with passing and failing clients on the first attempt, and ran the same file at 2025-11-25 and 2026-07-28.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AvvJSiaDgQMo5FftSjEohh

---

Generated by Claude Code

[pkg-pr-new]

Open in StackBlitz

npx https://pkg.pr.new/@modelcontextprotocol/conformance@520

commit: 6647735