Skip to content

feat(sep-xxxx): Conformance suite for triggers and events - #504

Closed
panyam wants to merge 27 commits into
modelcontextprotocol:mainfrom
panyam:feat/events-conformance-suite
Closed

panyam wants to merge 27 commits into
modelcontextprotocol:mainfrom
panyam:feat/events-conformance-suite

Conversation

@panyam

@panyam panyam commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Draft, and it must not merge as it stands. MCP Events has no SEP number, so every check id here carries a placeholder sep-9999- prefix. Renaming it is a merge blocker rather than a follow-up: plan.modelcontextprotocol.io builds from main, so a draft is invisible to it, but a merge would publish SEP 9999 as though it were real. See The number question below, which is still the one decision I would most like input on.

What changes

Adds a requirement-traceability yaml for MCP Events, plus the five server scenarios that score an implementation against it. Everything here is extracted from the design sketch that merged on main of modelcontextprotocol/experimental-ext-triggers-events on 2026-09-08.

Scenario What it grades Rows
events-discovery the capability declaration, events/list, the descriptor fields, the error-code range 12
events-poll poll delivery, the EventOccurrence shape, the cursor lifecycle 33
events-push a long-lived events/stream: confirmation, heartbeats, event frames, cancellation 17
events-webhook events/subscribe, the subscription key, TTL negotiation, events/unsubscribe 27
events-webhook-delivery what the server POSTs: verification, Standard Webhooks signing, retries, control envelopes, SSRF 28

Together they emit 117 of the 135 declared rows. The other 18 need a prerequisite no implementation exposes over the wire yet — a catalog that can change mid-run, two principals, or the four capability rows the sketch has only just gained — and are listed under Out of scope with what each needs. Five is the whole plan, so what is left is rows inside these files rather than more files.

Three scenarios (events-poll, events-webhook, events-webhook-delivery) landed in the two earlier commits on this branch; the first revision of this description covered only discovery.ts and poll.ts, and this rewrite catches it up to the diff.

Provenance

panyam offered an Events conformance suite in #triggers-events-wg on 2026-08-28 and Peter Alexander accepted on 2026-08-31 ("No objections, that would be great!"). It follows the SEP-2640 suite (#330) in shape: extract the requirements into a traceability yaml first, then write scenarios that emit one check per row.

Per AGENTS.md this would normally start as an issue. The WG conversation is that discussion, and the extraction is easier to argue about as a diff than as prose, so it is here as a draft instead. Happy to split it into an issue if you would rather.

Prerequisite knowledge

  • Events design sketch — the document every row is extracted from. Cursor Lifecycle carries most of events-poll; Subscribing and Event Delivery is the densest normative section and is what the three later scenarios cover.
  • Standard Webhooks — the signing scheme the sketch adopts. sep-9999-delivery-signature-* grades the v1,<base64> signature over {id}.{timestamp}.{body}, keyed on the base64-decoded bytes after the whsec_ prefix.
  • AGENTS.md § Scenario design and § Check conventions — why this is five scenarios with 117 checks rather than 117 scenarios, and why a missing prerequisite fails rather than skips (Standardize how scenarios report setup/execution failures #248).
  • src/traceability/ — how a declared row becomes tested or untested. This is what makes declaring all 131 rows up front the right move rather than an overreach.
  • src/seps/sep-2640.yaml — the closest precedent for the yaml's shape and for how much provenance belongs in its header.

The number question

This is the part I would most like a decision on, because it is the only thing blocking merge.

SEP numbers are PR numbers in modelcontextprotocol/modelcontextprotocol, and Events has no PR there — the work lives in its own repository. Both traceability gates are numeric: the directory filter at src/traceability/index.ts:174 (/^sep-\d+\.yaml$/) and the check-id filter at line 41 (/^sep-\d+-/). A file named events.yaml is silently dropped from the manifest rather than rejected, and src/new-sep/index.ts:163 refuses any argument that is not a positive integer. So there is no honest name that works today.

Three ways out, and I have no strong preference between the first two:

  1. Reserve a number. Cleanest, and it makes this mergeable as-is after a mechanical rename.
  2. Let the traceability layer carry pre-SEP extensions. Relax both regexes to accept a non-numeric key, so work like this can be tracked before it has a number. Wider change, but Events will not be the last extension in this position.
  3. Keep it unnumbered and drop the yaml. Loses the traceability artifact, which is most of the value.

I used 9999 to unblock the extraction, deliberately far from the live range. The rename touches the filename, the sep: field, every sep-9999- id in the yaml, and the same ids under src/scenarios/server/events/ — mechanical, but wide enough that it is worth doing once, in the right direction.

Reviewer's guide

A restaurant can cook a dish that is not on the menu. The kitchen has the ingredients and the line knows the recipe, so if you somehow know to ask you get it — but nobody reading the menu ever orders it, so from the outside it may as well not exist. That is the state this suite found its first implementation in: events/list answers, all three delivery modes work, and nothing appears in capabilities.extensions, which is the menu a spec-following client reads before deciding what to ask for.

That shaped the whole suite, because the obvious design misses it. A harness normally treats an undeclared optional capability as not-applicable and skips, which is right for a restaurant that genuinely does not serve the dish and wrong for one that serves it off-menu: the skip reports a clean run over a surface no client can reach. So the capability gate asks before it skips — it calls events/list, and only a server that both declares nothing and implements nothing gets the skip. The same instinct runs through the rest of the diff: wherever a check cannot be exercised, the scenario names the missing prerequisite and fails, because the one thing worse than a red conformance run is a green one that tested nothing.

Read in this order. The first four are the whole argument; the rest is mechanism.

  1. src/seps/sep-9999.yaml — start here, header first. It covers the placeholder, how 144 keyword occurrences became 135 declared rows and 30 excluded ones, which rows each scenario emits, and what the 18 unemitted rows are waiting for. The rows themselves are skimmable once the header lands.
  2. src/scenarios/server/events/discovery.ts — the off-menu gate is at the top of checks(). Everything after it grades whatever the server actually serves, field by field, reporting the first offender per field so ids stay stable regardless of catalog size.
  3. src/scenarios/server/events/webhook-delivery.ts — the densest file, and the one whose shape is least obvious. Note the first branch of deliveryChecks: with no EVENTS_WEBHOOK_CALLBACK_BASE the callback is a loopback URL, and that is the SSRF probe rather than a degraded run. A server that refuses it passes the SSRF rows and reports the delivery rows untestable; one that POSTs to 127.0.0.1 fails the SSRF rows and hands over real deliveries to grade everything else against. Both outcomes say something, and neither is a false green.
  4. src/scenarios/server/events/webhook.ts — subscription management only; anything the server then POSTs belongs to the file above. identityChecks is the interesting part: the key is (principal, delivery.url, name, arguments), a run holds exactly one principal, so the three components the harness can vary are graded and the principal component is reported untestable by name rather than passing a cross-tenant row nothing exercised. Every subscription it creates is released before it returns — against a public server a leak is state held for a callback nobody reads.
  5. src/scenarios/server/events/push.ts — one 35s observation window serves every timing row. The window has to outlast the document's 30s heartbeat SHOULD, which is why this scenario needs --timeout 60000: a shorter one cannot tell a silent server from a slow one, and reporting a compliant slow server as broken is worse than taking the time.
  6. src/scenarios/server/events/stream.ts — why events-push cannot use conn.request(). It resolves on the response for its id, and that is exactly what a push stream withholds until the subscription ends, so awaiting it would block for the life of the subscription and see none of the notifications the scenario grades. This opens the POST itself and hands back a session while the stream is still live. Aborting it is also the Streamable HTTP cancel, so the cancellation rows come out of the same mechanism rather than a second code path.
  7. src/scenarios/server/events/receiver.ts — the harness has to be the callback endpoint, because the signature, the headers, the verification challenge and the retry cadence are only observable from there. It records raw body bytes, since the signature covers those and re-serializing the JSON would change them. Per-path behaviour (accept, redirect, refuse, fail-then-accept, wrong-challenge) lets one receiver serve every probe.
  8. src/scenarios/server/events/negative-fixture.ts and the four negative*.test.ts files — the negative controls, 100 cases over one fixture. The fixture is a stateless events server whose every option breaks exactly one rule, which is what makes a control a control: with two things wrong at once, a flipped check does not say which one it caught. It also does the six things no client can ask a real server for, so the rows that report untestable against both implementations are graded here at least once.
  9. poll.ts, helpers.ts, index.ts, types.ts — in poll.ts, quietAdvanceChecks is what makes cursor advancement gradeable against a server where nothing is happening, which is the state a conformance fixture is usually in. In helpers.ts, EVENTS_EXTENSION_ID does double duty: ScenarioSource carries it as { extensionId }, which keeps these scenarios off the --spec-version timeline, and it is the key the capability itself lives under in capabilities.extensions. Those were two different things until 2026-09-22, and the story of how they became one is under Before/after. The rest is registration.

How it works

discovery:
  declared, value <- capabilities.extensions["io.modelcontextprotocol/events"]
  if not declared:
      probe <- events/list
      if probe is -32601:  skip everything          # genuinely does not do events
      else:                fail the declaration check, keep grading
  grade events/list, then each descriptor field across all descriptors
  probe one unknown event name -> grades NotFound and the server-range rule

poll:
  pick the first descriptor advertising "poll"; none -> untestable, not skip
  p1 <- poll(cursor=null)        # bootstrap: must replay nothing
  p2 <- poll(cursor=p1.cursor)   # quiet period: a cursor must still come back
  p3 <- poll(no cursor field)    # absent must be accepted as null
  p4 <- poll(maxEvents=1)        # batch must be capped
  p5/p6 <- maxAgeMs with and without a cursor
  poll forward from p1.cursor, waiting nextPollMs, to catch a real occurrence;
  grade the EventOccurrence rows only if one arrived

push:
  pick the first descriptor advertising "push"
  open events/stream and keep the POST open
  expect notifications/events/active within 3s -> cursor, truncated, _meta
  settle 35s                     # outlasts the 30s heartbeat SHOULD
  grade heartbeat cadence, event frames, foreign frames, SSE comments
  abort the request              # the Streamable HTTP cancel
  assert nothing more arrives, then grade the final result and the
  concurrency rules with a second and third stream

webhook:
  pick the first descriptor advertising "webhook"
  subscribe(https callback, fresh secret, ttlMs) -> id, refreshBefore, status
  same key again                 # must be an idempotent upsert, same id
  vary url / name / arguments    # must each produce a different id
  ttlMs omitted / null / absurd  # default, no-expiry, capped-not-refused
  reject: http:// url, a missing secret, one too short to be a secret,
    and a type whose delivery does not list webhook
  unsubscribe the key, then unsubscribe a key never held -> NotFound
  release every subscription created, including ones the server should
  have refused and did not

webhook-delivery:
  start an HTTP receiver; callback is EVENTS_WEBHOOK_CALLBACK_BASE or loopback
  subscribe with a fresh secret
  refused + loopback  -> the SSRF rows pass, delivery rows untestable
  refused + routable  -> nothing can be delivered, everything untestable
  accepted + loopback -> the SSRF rows fail, and grade the rest anyway
  wait up to 20s for the first POST, then 3s for more
  grade the verification handshake, transport and headers, the signature over
  the raw bytes, the event and control envelopes, redirect refusal, and the
  retry cadence on a path that fails before it accepts

Cursors are opaque by definition, so the harness cannot mint one old enough to fall outside a retention window; the truncated rows that need a stale cursor report untestable rather than skip. The same is true of DNS rebinding: sep-9999-ssrf-validate-at-delivery-time needs a hostname that resolves differently between subscribe and delivery, so it names that and fails rather than passing on the strength of the subscribe-time check.

Decision log

Carried forward from the first revision:

  • All 135 rows are declared now, though only 117 are emitted. An undeclared requirement is invisible; an undeclared-and-unemitted one is indistinguishable from a requirement nobody noticed. Declaring the full set makes the outstanding rows surface as untested, which is the manifest working rather than a gap in this file.
  • Pure MAY and OPTIONAL sentences get no row. Five initially got one and were demoted to excluded: rather than deleted, so the keyword sweep stays auditable against the source.
  • Client, host, receiver and SDK-guidance obligations are excluded rather than declared. Roughly a fifth of the normative sentences address the client or the callback endpoint. Declaring them would inflate the denominator with rows no server-side scenario could ever emit.
  • 88 of the 135 rows quote no RFC 2119 keyword. They come from the EventOccurrence field table, the error-code table and the descriptor fields, which the sketch states declaratively. Since "severity follows the keyword" is undefined for them, the yaml header states the rule used: a field the document marks required, or a wire fact a client cannot work without, is FAILURE; anything else drawn from an example payload is WARNING.
  • The sketch is not a spec diff, so new-sep's PR lookup does not apply and specPathToUrl (src/new-sep/index.ts:19) hard-requires a docs/specification/draft/ prefix that does not exist here. Scaffolded with --spec-url. There are no section anchors to cite, so rows carry no per-row url.

Decided since:

  • Five scenarios rather than one. A server may implement any non-empty subset of the three delivery modes, so poll-only and push-only servers are both conformant and grading them together would misreport each. Webhook splits again at the point where the server stops answering requests and starts making them: subscription management works against any reachable server, while delivery needs a callback URL the server under test can reach over https, which is the one prerequisite a local run cannot satisfy — the SSRF rules this suite checks require a conformant server to refuse localhost.
  • The delivery scenario runs a real HTTP server rather than mocking the callback. Signing, headers, the challenge handshake and retry timing exist only at the endpoint. A mock would grade what the harness believes the server sent.
  • The signature is verified over the raw bytes, per the document's own instruction to receivers, so the suite cannot accidentally pass a server that signs a re-serialization of its own JSON. This caught a real formula bug in the first implementation that its own client and receiver both agreed with.
  • events-push sets its own clock. 35s of watching is most of the wall time in this suite. The alternative is a window shorter than the cadence the document permits, which grades the harness's patience rather than the server.
  • Writing the controls found two bugs in the scenarios, which is the best argument for having them. webhook.ts listed 26 ids in the set it reports untestable when it bails, where its gradeable path emits 27: sep-9999-error-unsupported simply vanished from every run against a server with no webhook-capable type, instead of naming the prerequisite it was missing. And webhook-delivery.ts reported the gap and terminated envelope rows untestable unconditionally, even when the server had just POSTed one — the envelope was sitting in what the receiver recorded and nothing looked at it. Both are fixed here, and both are the class of defect a green run cannot show you.
  • EVENTS_DELIVERY_SETTLE_MS is new, and it is how long the delivery scenario lets retries and the two non-retryable probes play out. It was three hard-coded 5s sleeps, which is most of that scenario's wall time and pure waste against a fixture that answers in microseconds. The default does not change.
  • Rows the principal owns report untestable rather than passing. Cross-tenant isolation, per-principal verification caching and subscribe-time authorization all need a second credential. The runner cannot send even the first one today, which is the next thing I would like to add (see Out of scope).

Risk / blast radius

  • Affects: additive. Five new scenarios, one new yaml, one new entry in EXTENSION_IDS. No existing scenario, suite list or check id changes.
  • Not selected by default. source is { extensionId }, so matchesSpecVersion returns false for every --spec-version and these never join a dated run. They are also in the pending list. Reaching them takes --scenario events-* or --suite all.
  • Tests: npm run build, npm run lint and npm test pass — 725 tests across 51 files. The controls take the suite from 163s to 254s, most of it the delivery ones waiting out retry windows. The two timing-heavy files import the scenario once and run their cases concurrently for that reason; sequentially they were 425s, which is a lot to pay on every pre-push.
  • Negative controls cover all five scenarios — 100 cases across negative.test.ts, negative-push.test.ts, negative-webhook.test.ts and negative-delivery.test.ts, over one shared fixture in negative-fixture.ts. Each pairs a conformant in-process server with one broken in exactly one way, and every divergence in the table below has a case, so a red run against a real server can be trusted rather than re-derived. The fixture also does the six things no client can ask a real server for — fail upstream, lose its replay window, terminate a subscription, close a stream itself, and send the gap and terminated control envelopes — which are the only evidence those checks work, since they report untestable against both implementations.
  • Be paranoid about: the capability gate. It is the one place this suite deliberately departs from how every other extension scenario here behaves, and it is load-bearing for the headline finding. If you think an undeclared-but-serving server should skip, say so — that choice is the difference between a green run and a red one. There is an inline comment on it.
  • src/seps/traceability.json will drift once these ids are emitted in a real run. Per AGENTS.md the traceability workflow refreshes it by PR and it is not a gate, so this leaves it alone.

Before / after

There was no Events suite before, so what matters is what an implementation looks like when scored. Two were, both re-measured 2026-09-21: mcpkit's examples/events/kitchen-sink, and metronome-mcp.fly.dev, Peter's own server, announced 2026-09-10 and written by the author of the sketch.

Scenario kitchen-sink metronome
events-discovery 8/11 10/11
events-poll 26/29 27/29
events-push 11/15 13/15
events-webhook 17/23 20/23
events-webhook-delivery 12/21 3/27

metronome's delivery number is not a result about metronome: it refuses the loopback callback, correctly, so 24 rows report untestable for want of a routable one. Point EVENTS_WEBHOOK_CALLBACK_BASE at a tunnel and that run becomes meaningful. Its other four scenarios fail nothing real except the capability placement row below.

Fourteen rows fail in kitchen-sink for a real reason rather than a missing prerequisite, each observed against the running server rather than read out of its source. Nine are new here, from the three later scenarios:

Check What the server does
sep-9999-stream-subscription-id-meta puts the stream's correlation id in params.requestId, not params._meta["io.modelcontextprotocol/subscriptionId"], so a client holding two streams cannot route by what the document tells it to read
sep-9999-stream-exempt-from-concurrency-cap counts events/stream opens against the per-principal subscription cap, which the document exempts them from
sep-9999-subscribe-url-https-required accepts an http:// callback and subscribes to it
sep-9999-unsubscribe-unknown-not-found answers success for a subscription key it never held, so a client cannot tell teardown from a typo
sep-9999-delivery-signature-formula keys the HMAC on the literal whsec_… string instead of the base64-decoded bytes after the prefix, so no Standard Webhooks receiver verifies its signatures
sep-9999-verification-required-before-delivery delivers with no challenge handshake — the handshake does not exist yet
sep-9999-ssrf-validate-callback-url delivers to http://127.0.0.1

The signature and verification rows are the same shape of problem and worth reading together: nothing proves the callback endpoint wanted the deliveries, and what arrives there cannot be authenticated. metronome fails neither. The SSRF row is fixture configuration rather than a library hole — that fixture sets an allow-private-networks flag so its demo can reach a local receiver — but the row fails and should, because the suite grades the server it was pointed at.

Both implementations failed sep-9999-capability-events-object, for different reasons, and that row is where this got interesting. mcpkit declared nothing at all. metronome declared events under capabilities.extensions["io.modelcontextprotocol/events"], the SEP-2133 extensions map, where the document said top-level capabilities.events — so the suite was scoring the sketch author's own server as non-conformant against his own document.

Rather than pick a side I asked, on 2026-09-22. The document was the thing that was wrong. Extension negotiation is where a capability belongs, metronome had been right all along, and PR 7 moves the sketch there. This suite now reads the extensions map, and mcpkit's original pass-then-fail on that row turns out to be an artifact of following a document that did not yet say what its author meant.

Two smaller things came out of the same conversation. metronome's -32603-instead-of--32602 on arguments that violate inputSchema was a framework default surfacing, and was fixed the same day. And the reason events only appear on the 2026-07-28 wire is that extension negotiation did not exist before it, so a 2025-11-25 run reporting the extension as absent is correct rather than a gap.

That is the case for the suite better than any pass count: scoring two implementations found a defect in the specification. The tables above are dated 2026-09-21 and predate all three fixes, so they need re-measuring before anyone quotes them.

Running a suite against an implementation nobody wrote it for is also what finds the suite's own bugs. Two are fixed in this branch and both were invisible against a single target: minimalArguments gave up on any inputSchema with a required property, which scored metronome's poll at 1/34, and the occurrence rows were graded off a cursor: null poll that a conformant server answers empty. Same pattern as SEP-2640.

flowchart TD
    A[server under test] --> B{extension declared?}
    B -->|yes| G[grade everything]
    B -->|no| C{events/list answers?}
    C -->|"-32601"| D[SKIP all: does not do events]
    C -->|returns a catalog| E[FAIL the declaration check]
    E --> G
    G --> H{check exercisable?}
    H -->|yes| I[SUCCESS or FAILURE]
    H -->|no| J[untestable: FAIL and name the prerequisite]
Loading

Running them

node dist/index.js server --url <url> --scenario events-discovery
node dist/index.js server --url <url> --scenario events-poll
node dist/index.js server --url <url> --scenario events-webhook
node dist/index.js server --url <url> --scenario events-push --timeout 60000
node dist/index.js server --url <url> --scenario events-webhook-delivery --timeout 180000

Use a spec version of 2026-07-28 or later. Extension negotiation did not exist before it, so nothing earlier can declare Events, and an older run reports the extension as absent by design. The runner's default already lands there.

events-push needs the longer timeout for the reason above; EVENTS_PUSH_WATCH_MS overrides the window. EVENTS_WEBHOOK_CALLBACK_BASE makes the delivery rows gradeable against a server that correctly refuses loopback. One caution from running all five in sequence against one fixture process: subscriptions the earlier scenarios hold can exhaust a per-principal cap and starve the last one, which then reports almost everything untestable. Each scenario releases what it creates, so this is a property of the fixture's cap rather than of the suite, but it is worth knowing before reading a 1/29.

Out of scope

  • The 18 unemitted rows. Seven (schema-evolution-additive, breaking-change-new-name, list-changed-notification, four removal-*) need a server whose catalog can be made to change mid-run. Four are error-table rows already provoked by other checks and simply not claimed in their own right yet; two of those look cheap. Two need a second principal. Four are the capability rows the sketch gained on 2026-09-23, of which fallback-method-not-found is the one already within reach. payload-minimality is a SHOULD about payload content, and delivery-body-size probably measures as much of it as a harness can see. Each is declared and reports untested, which is the traceability layer doing its job.
  • An auth option on the server runner — a bearer token or a repeatable --header. Scoring metronome at all needs a local proxy in front of it to inject the header, and the two authz rows need two credentials. Its own PR, and it would retire the proxy.
  • The sep-9999-* rename, once the number question above is settled. Merge blocker.

… phase 1

Adds the requirement-traceability yaml for MCP Events plus the first two of
five server scenarios, scoring against the design sketch that merged on main
of modelcontextprotocol/experimental-ext-triggers-events on 2026-09-08.

The SEP number is a placeholder. Events has no PR in
modelcontextprotocol/modelcontextprotocol, and both traceability gates are
numeric (the filename regex at src/traceability/index.ts:174, the check-id
regex at line 41), so an unnumbered file is dropped from the manifest without
an error. 9999 is far from the live range; the rename is mechanical and the
yaml header names the rename as a merge blocker rather than a follow-up: a
merge under 9999 would publish it as a real SEP. The reservation question is
open with the WG.

src/seps/sep-9999.yaml declares 131 checks and 30 excluded rows against 144
RFC 2119 keyword occurrences. Pure MAY and OPTIONAL sentences get no check.
Client, host, receiver and SDK-guidance obligations are excluded rather than
declared, so the denominator holds only what a server-side run can observe.

events-discovery covers the capability, events/list, the descriptor fields and
the error-code contract. events-poll covers poll delivery, the EventOccurrence
shape and cursor lifecycle, driving a real two-poll quiet-period loop so
cursor advancement is gradeable against a server with no traffic. Together
they emit 45 rows; the other 86 report untested until the push and webhook
scenarios land.

The capability gate asks before it skips. An optional capability a server
never declared is not a defect, but a server that answers events/list while
declaring nothing has a surface no spec-following client would reach, and a
SKIP would report that as a clean run. mcpkit is in exactly that state.

Against examples/events/kitchen-sink at mcpkit main: discovery 8/11, poll
18/28. Six divergences, five of them server-side defects not previously
tracked, one confirming the known nextPollSeconds drift. The yaml header lists
each.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017kyEZdgh1zjdhURgdcZ4nL
@pkg-pr-new

pkg-pr-new Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npx https://pkg.pr.new/@modelcontextprotocol/conformance@504

commit: 2323122

'Server does not declare the `events` capability and does not implement `events/list`; the extension is optional.'
);
}
checks.push(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flagging this deliberately, because it is the one place this suite departs from how every other extension scenario in the repo behaves, and the departure is what turns the run against mcpkit red rather than green.

Everywhere else here, an optional capability the server never declared is a clean SKIP. That is correct when the server genuinely does not implement the feature. It is wrong when the server implements it and never says so: mcpkit answers events/list, serves all three delivery modes, and declares nothing under capabilities.events, so a client that reads capabilities to decide what to call never reaches any of it. A SKIP reports that as a clean run over a surface no client can use.

So the gate asks before it skips. One extra events/list separates "does not do events" (-32601, skip everything) from "does events off-menu" (returns a catalog, fail the declaration check and keep grading the rest).

The precedent is declaredSkillsCapability in src/scenarios/server/skills/helpers.ts, split out from skillsCapability for the same reason: a malformed declaration was being folded into "not declared" and skipping the whole suite, which "reads as a clean run against a server that is plainly wrong". This applies that argument to an absent declaration rather than a malformed one.

The cost is one extra request against servers that do not implement events at all. If you would rather pay nothing there and accept the false SKIP, say so, but the finding seems worth the request.

@panyam panyam changed the title feat(events): server conformance for the merged Events design sketch, phase 1 feat(sep-xxxx): Conformance suite for triggers and events Sep 15, 2026
panyam added a commit to panyam/mcpkit that referenced this pull request Sep 15, 2026
Adds the testconf-events target, its runner-agnostic script, and the
local-suites manifest entry, driving examples/events/kitchen-sink against the
two phase-1 scenarios (events-discovery, events-poll). Those are proposed
upstream as a draft in modelcontextprotocol/conformance#504 and tracked on the
panyam/mcpconformance fork branch until it lands.

INFO rather than a gate. The suite scores against the Events design sketch
that merged upstream on 2026-09-08, which has no SEP number yet, so its check
IDs still carry a placeholder sep-9999- prefix. And the run is red on purpose:
the failures are our divergences from that document, which is what the suite
was built to surface.

27 pass / 13 fail / 5 warn against kitchen-sink at c9160cf. Five divergences,
all tracked in #1380: events/list answers while no capabilities.events is
declared, so the whole surface is unreachable for a client that reads
capabilities first; events/poll omits the events key instead of returning an
empty array; and the events.topology meta-source does not keep the descriptor
contract, sending delivery null, carrying no inputSchema, and answering
events/poll despite advertising no poll delivery.

#1379 closed the nextPollSeconds rename and #1381 added list_changed and
termination plus inputSchema on the three real sources, so those no longer
show up here. Flip this to a gate once #1380 closes and the spec text
stabilises.

path-defaults.{mk,sh,just} are regenerated via gen_conf_paths.py rather than
hand-edited. check_local_suites.py reports no drift across 9 suites.

Refs #1374

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017kyEZdgh1zjdhURgdcZ4nL
panyam added a commit to panyam/mcpkit that referenced this pull request Sep 15, 2026
Adds the testconf-events target, its runner-agnostic script, and the
local-suites manifest entry, driving examples/events/kitchen-sink against the
two phase-1 scenarios (events-discovery, events-poll). Those are proposed
upstream as a draft in modelcontextprotocol/conformance#504 and tracked on the
panyam/mcpconformance fork branch until it lands.

INFO rather than a gate. The suite scores against the Events design sketch
that merged upstream on 2026-09-08, which has no SEP number yet, so its check
IDs still carry a placeholder sep-9999- prefix. And the run is red on purpose:
the failures are our divergences from that document, which is what the suite
was built to surface.

27 pass / 13 fail / 5 warn against kitchen-sink at c9160cf. Five divergences,
all tracked in #1380: events/list answers while no capabilities.events is
declared, so the whole surface is unreachable for a client that reads
capabilities first; events/poll omits the events key instead of returning an
empty array; and the events.topology meta-source does not keep the descriptor
contract, sending delivery null, carrying no inputSchema, and answering
events/poll despite advertising no poll delivery.

#1379 closed the nextPollSeconds rename and #1381 added list_changed and
termination plus inputSchema on the three real sources, so those no longer
show up here. Flip this to a gate once #1380 closes and the spec text
stabilises.

path-defaults.{mk,sh,just} are regenerated via gen_conf_paths.py rather than
hand-edited. check_local_suites.py reports no drift across 9 suites.

Refs #1374

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017kyEZdgh1zjdhURgdcZ4nL
panyam and others added 10 commits September 17, 2026 21:52
…e occurrences off a poll that can carry them

Running the suite against a second implementation — Peter Alexander's
metronome-mcp.fly.dev, written by the author of the design sketch — found two
bugs that kitchen-sink alone could not surface.

minimalArguments gave up on any inputSchema with a required property. The
document's own examples are filters and transforms with nothing required, and
the helper took that for a rule. metronome.tick requires periodSeconds and
label, so every poll row reported untestable and the scenario scored 1/34. It
now derives a value per required property from the schema itself — default,
const, first enum or examples entry, then minimum for numbers or a fixed
string — and still declines to guess when a required property offers nothing.

The sep-9999-occurrence-* rows were graded off the bootstrap poll, which passes
cursor: null. Null means start from now, so a conformant server returns no
events there: the suite could only score those seven rows against a server that
violated sep-9999-cursor-null-starts-from-now. They now poll forward from the
returned cursor, waiting nextPollMs between attempts (clamped to 250ms-2s) up
to a 6s ceiling, and still report untestable against a quiet server.

Scores with the fixes. kitchen-sink at d2950655: discovery 8/11, poll 26/29,
up from 18/28 with no change to mcpkit. metronome: discovery 10/11, poll 27/29.

Two metronome divergences, in the yaml header. It answers -32603 rather than
-32602 for arguments that violate inputSchema. And it declares events under
capabilities.extensions["io.modelcontextprotocol/events"] where this document
puts it top-level, which is a question for the WG before the row moves either
way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the third of five events scenarios, covering the 17 sep-9999-stream-* rows.

The Connection abstraction cannot express events/stream: conn.request resolves
on the response for its id, and a push stream withholds that until the
subscription ends, so a scenario awaiting it would block for the life of the
stream and see none of the notifications it grades. stream.ts opens the POST
itself and hands back a session while the stream is still open. Aborting that
request is also the document's client-side cancel on Streamable HTTP, so the
cancel row falls out of the same mechanism.

The watch window is 35s by default, over the runner's 30s timeout, so the
scenario needs --timeout 60000. That is deliberate: the heartbeat is a MUST
with a 30s SHOULD on cadence, so a shorter window cannot tell a silent server
from a slow one. Under 30s the heartbeat rows report untestable rather than
failing a server that may be conformant. The interval check carries 2s of
tolerance, because kitchen-sink times its heartbeat at exactly 30s and measured
30005ms once scheduling had its say.

Rows needing something the client cannot ask for - an upstream failure, a
retention gap, a termination, a server-initiated close - report untestable with
the prerequisite named, per src/scenarios/untestable.ts.

kitchen-sink at d2950655: 11/15. metronome: 13/15, where both failures are
untestable rows rather than divergences.

One new divergence, not previously tracked: kitchen-sink puts the correlation
id in params.requestId, where the document requires
params._meta["io.modelcontextprotocol/subscriptionId"]. A client holding two
streams cannot route by what the document tells it to read.
…agement

The fourth of five scenarios: the 26 subscribe, TTL and unsubscribe rows.
Delivery itself - signatures, the verification handshake, control envelopes -
waits for events-webhook-delivery and a reachable callback.

Two things shape it. A run holds one principal, so the key components the
harness can vary (url, name, arguments) are graded and the principal half is
reported untestable rather than passing a cross-tenant row that was never
exercised across two tenants. And a server may cap concurrent subscriptions per
principal: kitchen-sink allows two per event type, so every throwaway
subscription is released as soon as its check is graded, including a probe the
server was supposed to reject but accepted. Without that the scenario grades
the cap error instead of the rule it was probing, which it did three times
while being written.

Every subscription is unsubscribed before the scenario returns, which matters
against a public server.

kitchen-sink at d2950655: 17/23, three divergences. It accepts an http://
callback URL where the document requires https and -32602 (two rows, one
probe), and unsubscribing a key it never held succeeds instead of answering
-32011 NotFound, so a client cannot tell teardown from a typo.

metronome: 20/23, no divergences. The three failures are untestable rows: the
unsupported-mode path (its only event type offers every mode), the auth
requirement, and cross-tenant isolation.
…nario

Grades what the server POSTs: the verification handshake, Standard Webhooks
signing, retries, control envelopes and the SSRF rules. receiver.ts runs a real
HTTP endpoint and keeps the raw body bytes, because the signature covers those
and re-serializing the JSON would change them.

The scenario needs a callback the server can reach, and
EVENTS_WEBHOOK_CALLBACK_BASE supplies one. Without it the receiver binds to
loopback, which is not a degraded mode but a different question: the document
requires a server to refuse a non-routable callback, so a loopback URL is the
SSRF probe. A server that refuses it passes the SSRF rows and reports delivery
untestable; one that POSTs to 127.0.0.1 fails them and hands over real
deliveries to grade everything else against.

kitchen-sink at d2950655: 12/21, four divergences, two of them security-shaped.

  It accepts and delivers to a loopback callback, so a caller can aim
  deliveries at anything the server can reach and the caller cannot.

  It signs with the literal whsec_ string as the HMAC key, where the document
  says the key is the base64-decoded bytes after that prefix. No Standard
  Webhooks receiver verifies these signatures; confirmed by hand against four
  candidate formulas before filing it here.

  It delivers without the verification handshake, so an unverified third-party
  URL receives POSTs.

metronome refuses the loopback callback outright, which passes the SSRF rows and
leaves the delivery rows untestable. Grading it needs a public callback.

One harness bug, found while reading the results: the retry row folded signature
validity into its freshness check, so a server with the wrong formula was
reported as reusing timestamps it had not reused. Freshness is now graded on
distinct timestamps and signatures alone.

Coverage is now 117 of the 131 declared rows across the five scenarios. The
remaining 14 need a server that changes at runtime (schema evolution,
list_changed, event-type removal) or error paths this run did not reach.
The SSRF row is fixture configuration, not a library defect. kitchen-sink
passes WithWebhookAllowPrivateNetworks(true) so the demos can reach a local
receiver, and that one flag disables both the subscribe-time and dial-time
guards; mcpkit's default path blocks loopback and the other non-routable
ranges. The row still fails, because the suite grades the server it was pointed
at, but it means the target runs with protection off rather than that mcpkit
lacks it.

Redirects are not followed either: webhook.go sets CheckRedirect to
ErrUseLastResponse, which is what the suite already scored.

Adds the root cause behind each of the other five, read out of mcpkit at
5cfb7c51. The signature bug is the widest: the whsec_ format arrived two days
after signing landed and key derivation was never revisited, so both verifiers,
the Go and Python clients, the whole-enchilada receiver and the telegram tests
all share it. Server and clients agree with each other and none agrees with the
document, which is the failure a conformance suite exists to catch.
webhook.ts listed 26 ids in ALL_IDS, the set it reports untestable when it
cannot get far enough to grade, where its gradeable path emits 27.
sep-9999-error-unsupported was the missing one, so against a server with no
webhook-capable event type the row did not report a missing prerequisite — it
vanished from the run. A suite that emits a different row set depending on the
server is a suite whose pass count cannot be compared across SDKs.

webhook-delivery.ts reported sep-9999-envelope-gap and -terminated untestable
unconditionally, on the reasoning that no client can ask a server for a
retention gap or a revocation. True, and beside the point: if the server sends
one anyway the envelope is sitting in what the receiver recorded, and nothing
looked. Both are now graded when they arrive and untestable when they do not,
which is how push.ts has always handled its gap row.

Also makes the settle window a knob. Three hard-coded 5s sleeps let retries and
the two non-retryable probes play out, which is most of that scenario's wall
time and pure waste against a fixture that answers in microseconds.
EVENTS_DELIVERY_SETTLE_MS overrides it; the default does not change.
A green run against kitchen-sink proves a check does not false-positive. It
proves nothing about whether the check catches anything, and only discovery and
poll had controls that did. These add 71 cases over the other three scenarios,
100 across all five, each pairing a conformant in-process server with one broken
in exactly one way.

One fixture serves all five. negative-fixture.ts holds the stateless events
server that negative.test.ts had inline, extracted unchanged, plus three things
the later scenarios need: events/stream as a real SSE response written on timers,
because the heartbeat cadence and the cancellation rows are about timing; a
subscription store keyed the way the document keys it, minus the principal a
single run cannot vary; and a delivery engine that POSTs signed Standard
Webhooks payloads to the callback and retries them, because the signature and
the retry cadence are only observable from the endpoint.

Every divergence the suite found in a real implementation has a case here, so a
red run can be read rather than re-derived: the correlation id in
params.requestId, streams counted against the subscription cap, an http://
callback accepted, unsubscribe answering success for a key it never held, the
HMAC keyed on the literal whsec_ string, and delivery with no verification
handshake.

The fixture also does the six things no client can ask a real server for — fail
upstream, lose its replay window, terminate a subscription, close a stream
itself, and send the gap and terminated control envelopes. Those rows report
untestable against both implementations, so these are the only evidence those
checks work at all.

Two notes for whoever writes the next one. vi.resetModules() is needed for the
scenarios that read their timing knobs at module load, and a reset registry hands
back a second copy of the connection module, so the run context has to be built
from the same fresh import or instanceof JsonRpcError is false across the two
graphs. And the delivery fixture spaces an out-of-order event 60ms ahead of the
verification POST, because the check compares millisecond arrival times and two
loopback POSTs land inside the same millisecond often enough to read as
in-order.

npm test: 725 passing across 51 files.
The two files that grade timing spent almost all of their wall time asleep: the
delivery controls waited out five settle windows per case and took 180s, the push
controls 80s. Both together took the suite from 163s to 425s, which every
contributor pays on pre-push.

The cases are independent — each builds its own fixture on its own port and
touches no shared state — so the only thing keeping them sequential was
`vi.resetModules()` per case, needed because both scenarios read their timing
knobs when the module is first evaluated. Importing once per file in `beforeAll`
removes that, and the cases then run concurrently: delivery 45s, push 43s, suite
254s.

One graph means one set of timings, and one push case has to outlast the 30s
heartbeat cadence the document permits, because under 30s a silent server and a
slow one are indistinguishable. It takes its own graph in a sequential block at
the foot of the file, after the concurrent cases are done with theirs; resetting
the registry while they were still running would pull it out from under them.

725 passing, unchanged.
The design sketch put it top-level under `capabilities.events`. metronome, the
sketch author's own server, used `capabilities.extensions`. Raised with him
2026-09-22 rather than resolved by picking a side, and the document turned out
to be the thing that was wrong: upstream PR 7 moves the sketch into Extension
Negotiation, keyed by `io.modelcontextprotocol/events`.

`declaredEventsCapability` and the four scenarios that probe the declaration
now read the extensions map. `EVENTS_CAPABILITY` is gone; `EVENTS_EXTENSION_ID`
absorbed its job, having previously carried a comment saying in as many words
not to read the capability at that key.

Two rows are restated from PR 7 and four are new: the empty settings object,
the optional client declaration, `listChanged` gating whether the server may
send `notifications/events/list_changed`, and the `-32601` fallback for a
server that does not offer the extension. The four are declared and not yet
emitted, so they report untested.

The header's record of both metronome divergences becomes a record of how they
resolved. He fixed the `-32603`-instead-of-`-32602` one the same day.

These rows track an unmerged PR, which the head of this file now says.

Refs modelcontextprotocol#504
@panyam

panyam commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Updated for the capability-location resolution (d3d26e0).

The rows put the declaration at the top level, under capabilities.events, because that is what the design sketch said. Two implementations read that sentence and landed in different places: this suite's target followed the document, and metronome-mcp.fly.dev used the SEP-2133 extensions map. Rather than pick a side or accept both, I asked @Poita on 2026-09-22.

The document was the thing that was wrong. experimental-ext-triggers-events#7 moves the declaration into Extension Negotiation, keyed by io.modelcontextprotocol/events. These rows now follow that PR.

What changed here

  • declaredEventsCapability and the four scenarios that probe the declaration read capabilities.extensions[...]. EVENTS_CAPABILITY is gone, and EVENTS_EXTENSION_ID absorbed its job — it previously carried a comment saying in as many words not to read the capability at that key.
  • sep-9999-capability-events-object and -capability-list-changed-flag restated from PR 7.
  • Four new rows for what else PR 7 specifies: the empty settings object, the optional client declaration, listChanged gating whether a server may send notifications/events/list_changed, and the -32601 fallback for a server that does not offer the extension. All four are declared and not yet emitted, so they report untested.
  • The negative controls moved with them, which is how the one message I had missed got caught.

These rows track an unmerged PR, which the yaml header now says. Same posture as the SEP-2350 scope-challenge scenarios against #481: a red row means the head moved rather than that an implementation regressed.

One other thing this suite turned up and @Poita has since fixed: arguments violating inputSchema answered -32603 where the document requires -32602.

So across two implementations the suite found a defect in each and one in the document. That last one only surfaced because the rows were not softened to let both sides pass, which felt pedantic at the time and is the reason the sketch is being corrected now.

Six push and webhook rows describe what a server does when something goes
wrong upstream, and a healthy server does none of it during a run. The
scenarios watched a window, saw nothing, and reported untestable, which under
src/scenarios/untestable.ts is red and stays red.

A fixture MAY now expose the conditions as ordinary tools. This calls the two
that are safe to fire mid-stream — a transient upstream failure and a
retention gap — neither of which ends the subscription, so the rows that grade
heartbeats and deliveries still see what they need. Termination is left alone
deliberately: it is terminal for the source and these scenarios share one
fixture process, so firing it here would poison whatever runs next.

The controls are called over their own connection rather than the streaming
one. That is the more faithful simulation, since an upstream failure does not
arrive as a request from the subscriber watching for it, and the signals fan
out to every live subscriber regardless.

Discovery happens on the connection run() already holds, not around the
stream. The first version probed later and cost a connect plus two tools/list
round trips inside the observation window, which was enough to miss a
termination arriving at 250ms and turned negative-push.test.ts red. A server
without the controls now pays nothing for the question and opens no second
connection.

Servers without controls are unaffected and keep reporting untestable, with
the reason naming the specific tool that would make the row gradeable rather
than describing the gap in the abstract.

Refs modelcontextprotocol#504
sep-9999-stream-terminated-ends-subscription and both
sep-9999-stream-final-result-* could not be graded on the main session, and not
for want of a control: the harness cancels that stream to test
stream-cancel-stops-delivery, and on Streamable HTTP a client-side abort is
terminal, so no final frame is ever sent. A server-ended stream is a different
stream.

terminateChecks opens its own, fires the terminate control, and grades all
three from it. It refuses to terminate the event type the rest of the run
depends on, because termination is one-shot for the source and the four events
scenarios share one fixture process; a fixture offering a single push-capable
type gets untestable with that reason rather than a poisoned suite.

The main session keeps grading these opportunistically. A server that closes
the stream itself has already answered them, and dedupe is first-wins, so the
control path never runs for ids the observation window already produced. That
ordering is what the negative fixture exercises, and getting it wrong turned
negative-push.test.ts red twice: once by removing the opportunistic path
entirely, once by flattening three rows to one severity when terminating is a
MUST and the two final-result rows are SHOULDs.

Also fixes a bug in the final-result shape check that mcpkit surfaced. It
counted `resultType` as information the frame carries, but `resultType` is a
base-protocol field on the common Result interface that servers MUST include,
which sep-2640.yaml already records. Every conformant server failed that row.
`_meta` was already excluded; `resultType` now is too.

Against mcpkit: events-push 14/16 to 17/18, zero warnings, the remaining
failure being a real missing subscriptionId.

Refs modelcontextprotocol#504
…ontrols

The rule is that two tenants subscribing to the same `(name, arguments)` with
the same callback get distinct subscriptions, because the principal is part of
the key. A run authenticates as one principal for its lifetime, so the harness
could only ever supply one side of that comparison and the row reported
untestable.

A fixture MAY now register on another principal's behalf and answer whether a
given principal's subscription still exists. With both, the check does the
comparison the rule is about: same event, same callback, principal the only
thing that differs.

Two things are graded, and the second is the one that bites. Distinct ids show
the principal reached the key at all. A surviving subscription after the other
tenant unsubscribes shows the two are genuinely independent, which is what a
tenant would notice if it were not true — one tenant able to cancel another's
subscription.

askControl joins fireControl in helpers for the controls that answer with text
rather than just succeeding.

Servers without the controls are unaffected and keep reporting untestable, now
naming the two tools that would make the row gradeable.

Against mcpkit: events-webhook 17/23 to 18/23.

Refs modelcontextprotocol#504
…iled one

Once a server verifies inside events/subscribe, which is how the document
reads ("returned synchronously from events/subscribe"), three things in the
webhook-delivery scenario stopped measuring what they claim to.

Every probe callback answered every POST with its failure status, the
challenge included, so a verifying server could not subscribe the 410, 413,
retry or redirect probes and all four rows went untestable. The receiver
now answers the challenge on every path before the path's behaviour
applies; wrong-challenge is the one path that still fails it. The 410/413
and retry graders also waited for the first POST on their path, which is
now the challenge, answered 200 and never retried, so they wait for the
first non-verification POST instead. The redirect grader waits for a POST
it actually answered with 302.

verification-failure-error was untestable on the assumption that the
handshake is asynchronous. It now subscribes a wrong-challenge callback:
-32015 with reason challenge_failed passes; any other error, or an accepted
subscription that is then delivered to, fails; an accepted subscription with
no delivery is a WARNING, since the endpoint is safe but the subscriber was
never told.

delivery-retry-regenerates-signature failed on any two retries inside one
second, because webhook-timestamp is whole seconds and a regenerated stamp
is indistinguishable from a reused one there. It flapped against mcpkit,
whose first retry is 500ms out. Only a stamp repeated across attempts a
second or more apart now fails; retries all inside one second are a WARNING.

The fixture gains synchronousVerification (and ignoreFailedEcho for the
negative case) and retryGapMs. Against mcpkit kitchen-sink --serve the
scenario goes from 12 to 18 SUCCESS; with verification disabled in the
library the verification rows go red again.
ttl-long-grant-retained, ttl-no-expiry-persisted and ttl-no-expiry-gc-terminated
were untestable because they need a restart the harness cannot ask for over
the wire. A fixture exposing three controls now makes them gradeable:

- events_conformance_restart rebuilds the server over the same subscription
  store and answers the generation it is moving to
- events_conformance_generation answers the current one
- events_conformance_subscription_state answers active, suspended or absent
  for a derived id, which the exists control cannot, since it hides
  suspended subscriptions

The generation pair is what makes the restart detectable on both kinds of
server: a stateful one ends the old session, so the scenario reconnects; a
stateless one keeps answering on the same connection, first from the old
build and then the new.

The scenario subscribes once with ttlMs:null and once with a one-day grant,
restarts, and reads both back. A no-expiry grant that is not honoured leaves
its two rows untestable at WARNING rather than failing, since granting one is
optional. The GC row watches the no-expiry subscription, whose placeholder
callback fails every delivery, for EVENTS_GC_WAIT_MS: dropped passes, still
there is a WARNING because dropping is a MAY, and already lost across the
restart is untestable. The terminated envelope goes to the placeholder origin
and cannot be observed, which the check says.

The durability checks run last, since a restart ends the scenario's
connection. Wait limits are read at call time so the negative controls can
shorten them. Against mcpkit kitchen-sink --conformance-events the scenario
goes from 22/23 to 25/26; a build whose restart loses the store fails the
two retention rows.
The document gives a server four ways to confirm a callback wants deliveries:
the challenge handshake, a configured allowlist, prior out-of-band verification,
or a receiver-published /.well-known/mcp-webhook-receiver.json. The suite only
knew the first, so it reported a server that read the document and delivered
without challenging as one that "delivered to a callback that was never asked to
prove intent". mcpkit 1449 landed that path today, which would have turned the
row red for a server satisfying it.

The receiver now publishes the document and counts fetches, and consent counts
as established by a challenge or by a fetch that precedes the first delivery.
It is declared over one path prefix and no other, so the main delivery path
exercises the document route while every probe path still takes the handshake:
one run covers both rather than whichever the server prefers.
EVENTS_RECEIVER_WELL_KNOWN=0 withholds it. The document is honoured only on an
https origin, so against a loopback receiver the handshake decides everything,
the same way the SSRF rows only mean something over a tunnel.

Two rows come out of that. sep-9999-verification-no-raw-endpoint-responses was
an unconditional SUCCESS whose details read "no endpoint response body was
echoed back in any server error observed during this run" — nothing had observed
anything, so it could not have caught a leak. It now grades the error from the
wrong-challenge probe, where the receiver echoes a distinctive string, and
reports untestable on a run where no endpoint failed.
sep-9999-error-callback-endpoint-error, declared since the first commit and never
emitted, is claimed off that same error: the -32015 code and whether data.reason
is one of the documented lastError categories.

Why the failure branch stays a failure when nothing is observable: the allowlist
and out-of-band paths cannot apply to these callbacks, because the path is minted
fresh for the run and no operator has ever seen it.

7 new controls, 31 in negative-delivery.test.ts. Against the previous scenario
the well-known case fails with "expected 'FAILURE' to be 'SUCCESS'", which is
the regression this prevents.
Four declared rows emitted nothing while the request that would grade each was
already being sent. Claiming them costs no new round trips.

sep-9999-fallback-method-not-found — what a server that does not offer the
extension answers. discovery.ts already probes events/list on a server that
declares nothing and keys its SKIP off -32601, so the one run that ever meets
such a server now grades the rule instead of folding it into skipAll. A server
answering some other code fails it: a client cannot tell "no such extension"
from a real failure. Everything else still skips, because every other events
rule is genuinely inapplicable there.

sep-9999-capability-empty-settings — `{}` declares support with no list-change
notifications. Graded off the declared value, and inapplicable rather than failed
when the server declared settings of its own, since the rule says what an empty
object means and nothing about a populated one.

sep-9999-error-invalid-params and -resource-exhausted — the error-table rows for
-32602 and -32013. The first rides the poll invalid-arguments probe, the way
sep-9999-subscribe-url-https-required already rides its enforcement probe, with
details.gradedBy naming where the verdict came from. The second grades the three
concurrent streams in push.ts: a refusal for a quota must name the quota in
data.limit, and a server that reaches no limit reports untestable rather than
passing.

117 of 135 rows emitted before this and the commit before it; 122 now. Of the 13
left, seven wait on a control that can change a catalog mid-run, two on a second
credential, and sep-9999-error-forbidden on a principal the harness can be
refused as.

5 new controls. One existing assertion changes: the "skips the suite" case now
asserts the fallback row passes and everything else skips, which adds an
assertion rather than relaxing one. No assertion weakened.
… track

The extensions-map change added four rows and restated two, and the header's
arithmetic did not follow: it still said 131 declared, 117 of 131 emitted, and
"the 14 rows nothing emits yet". It is 135, 122 and 13.

It also said "see the tracking note at the head of this file" and there was no
such note. There is now, as a banner: the six rows that quote text only PR 7
has, why the suite is ahead of the document, and to re-point spec_source at main
when it merges. spec_source and spec_url named `main`, where a reader who fetched
it found text contradicting those rows; both now name PR 7's head.

Seven strings still told implementers the capability lives at
`capabilities.events`, including the description attached to
sep-9999-capability-events-object itself, so one check carried two contradictory
statements and the one a reader of a failing run sees was the wrong one. The
survivors were exactly the strings no control asserts on, which is also where to
add an assertion. The note in helpers.ts keeps the old spelling deliberately,
because it is explaining the history.

declaredEventsCapability now calls extensionsOf rather than inlining the same
lookup, which is what that helper's doc comment promises.
The comment introducing poll's error-table row named the webhook precedent by its
exact slug, which makes `grep sep-9999-subscribe-url-https-required` report two
scenarios emitting it when only one does. It now names the rule in prose. The
header's count for poll.ts followed the same miscount and read 35 where the file
emits 34; the suite total of 122 was right either way.
Measuring against kitchen-sink at mcpkit d7624d68 found two rows going missing
rather than reporting a missing prerequisite, both in paths I had just added to.

sep-9999-fallback-method-not-found was only emitted when the server declared
nothing, so against every real events server it vanished. It is now SKIPPED there
with the reason: a server that declares the extension is not a server that does
not offer it, so the rule cannot apply.

events-webhook-delivery emitted 27 rows when it refused a loopback callback and
29 when it delivered. The refusal answers two of the four SSRF rows by itself;
the other two need a delivery to revalidate and a redirect to refuse, and were
simply absent. Both now report untestable there.

This is the defect the suite's own untestable policy exists to prevent: a row
that appears on one path and not another makes two servers' pass counts
incomparable, which is the whole point of a fixed denominator. Controls for both,
asserting the row is present and inapplicable rather than absent.
… a time

events-webhook-delivery grades 29 rows and reaches 2 of them against a server
with its SSRF guards on. Not a defect on either side: the harness must be the
receiver, because the signature, the headers, the handshake and the retry cadence
are only visible from the endpoint being POSTed to, so it listens on loopback —
and a hardened server refuses a loopback callback, which is what two of those
rows grade. Both cannot hold for one subscription.

The way out is a fixture control, the same shape as the other eight:
events_conformance_allow_callback_origin permits one named origin past the
callback guards for the rest of the process. The order is what keeps the SSRF
verdict honest. Subscribe with the guards on, grade the two SSRF rows against the
server as configured, then lift for this receiver's origin and subscribe again,
so signing, headers, retries, envelopes and verification grade off real
deliveries. Nothing about hardening is concluded after the override.

sep-9999-ssrf-validate-at-delivery-time now reports SKIPPED when the origin was
permitted on purpose, because a delivery to a deliberately-allowed origin is
evidence of nothing; proving revalidation still needs a hostname whose DNS answer
changes between subscribe and delivery.

The control does not exist yet: mcpkit 1457 has the contract, and this side sits
behind hasControl, so a server without it reports untestable exactly as before
and names both ways out, the tunnel and the control. 26 of the 27 rows become
reachable when it lands.

Two controls cover both paths. Writing them turned up the fixture refusing a
permitted origin anyway, because its https check ran first — which also showed
the "harden" flag I had added was redundant, since a fixture without
acceptHttpUrl already refuses the harness's loopback callback. Gone, and the
permitted set now bypasses that check the way the real control will.

npm test: 751 passing.
A missing prerequisite reports as a failure on purpose: SKIPPED is excluded from
counts, exit codes and the expected-failures baseline, so a skipped row is one
nobody can burn down. The cost is that a run reads as broken when it is mostly
unchecked. events-webhook-delivery against a hardened server printed

  Passed: 2/29, 27 failed, 0 warnings

where nothing was violated and 27 rows had no reachable callback to grade
against. A reader cannot tell that from a server with 27 real defects.

The distinction was already in the data. notTestable() writes a stable prefix and
untestableCheck() sets details.untestable, both so consumers can separate
"violated" from "could not be verified" without a new status. The runners just
did not print it:

  Passed: 21/21, 0 failed, 8 warnings (8 of those unverified, not violated)

The clause appears only when there is something to say, so a clean run prints
exactly what it printed before.

The four numbers were computed by three copies of the same four filters in
server.ts, client.ts and authorization-server.ts, so a fifth number had to go in
three places or in none. They now share tallyChecks/formatTally. No verdict
changes: unverified counts rows already inside failed and warnings, and every
exit code still keys off those, so a run that failed before fails now.

Verified against a Go server, per AGENTS.md on shared infrastructure: mcpkit's
kitchen-sink at eec13596 across all five events scenarios, 101 of 122 rows
passing and 0 real failures, plus 10 unit cases covering the subset rule, the
prefix-only path and that SKIPPED counts as neither.
…not refuse

Both SSRF rows were graded from one refusal of the harness's
http://127.0.0.1 receiver. The https rule refuses that URL on its own, so
a server that never checks routability passed reject-non-routable anyway.
mcpkit's conformance build was in exactly that state until
panyam/mcpkit#1462 and passed the row throughout:

  -32602 delivery.url rejected: delivery.url must use https (got http)
  SUCCESS sep-9999-ssrf-reject-non-routable

The row now has its own probe, run before anything can lift a guard: an
https://127.0.0.1 callback aimed at a bare TCP listener that records
whether the server connected. The scheme is valid, so only routability
can refuse it, and the listener separates a refusal from a dial that
failed afterwards. A server that skips the check and then fails its own
verification POST answers -32015, which reads as a refusal from the
subscribe alone.

  refused, never dialled   SUCCESS
  dialled, either way      FAILURE
  accepted, never dialled  WARNING (may be delivery-time validation)

Over a tunnel the server may be on another host, where its dial to
127.0.0.1 never reaches the listener, so there only -32602 counts as a
refusal. The probe also runs in that mode, where the row used to be
untestable.

validate-callback-url keeps grading from the http refusal, since either
rule refusing it is validation.

The negative fixture gains rejectNonRoutableUrl. The existing "refusing
the loopback callback passes the SSRF rows" case used a fixture that
enforced https alone and asserted this row SUCCESS, which certified the
bug; it now uses a fixture that refuses non-routable hosts, and still
asserts SUCCESS. Four new cases cover the table above, all red on the
old scenario.

Refs panyam/mcpkit#1460
error-resource-exhausted was graded only inside the concurrency probe,
which opens three streams on one event type and needs all three to stay
open for stream-exempt-from-concurrency-cap. A server capped on that type
passes the error row and fails the MUST beside it, so on one type the two
cannot both pass, and a server that caps anything else never has the row
provoked at all. mcpkit reports it untestable for exactly that reason
(panyam/mcpkit#1461), though it answers -32013 with data.limit on every
subscribe path.

New read-only control, events_conformance_quota, answering
{"name": "<event type>", "max": <n>}. When present, the scenario opens
streams on that type one at a time, up to max+1, and grades the first
refusal:

  -32013 with data.limit     SUCCESS
  -32013 without it          WARNING
  another code               FAILURE
  max+1 opened, no refusal   untestable, naming the cap that never bit

A refusal before max+1 counts, since other scenarios may hold
subscriptions under the same principal. The probe's row is pushed ahead
of the concurrency probe's, so dedupe keeps it; without the control the
concurrency probe's opportunistic grading stands as before, and its
untestable message now names the control.

The negative fixture gains a per-type quota with the control, and four
cases cover the table and the absent control, all red on the old
scenario. The existing maxConcurrent cases are unchanged.

Refs panyam/mcpkit#1461
…o's own subscription

envelope-gap and envelope-terminated reported untestable against every
server, because nothing in a run produces either. The push scenario solved
the same problem with yield_gap and terminate, but those cannot carry over:
in mcpkit a source's gap and terminal signals reach push streams only, so
no webhook envelope results, and terminate ends a whole event type, which
for a webhook-capable type ends it for every scenario after this one.

Two per-subscription controls instead, taking { id }:
events_conformance_webhook_gap and events_conformance_webhook_terminate.
The scenario fires them after the retry and redirect probes, terminate
last since it ends the main subscription, and waits up to 5s for each
envelope.

  envelope arrived, well formed     SUCCESS
  arrived without cursor / error    WARNING
  acknowledged, nothing arrived     gap WARNING, terminated FAILURE
  control absent or declined        untestable, naming the control

The envelope rows now grade after the probes, from what arrived, so the
signalled envelopes also count toward the discriminator, signing and
webhook-id rows.

The negative fixture gains both controls, reusing its envelope sender with
the event and handshake switched off. Four cases, all red on the old
scenario; the malformed-envelope case asserts on the graded body as well as
the status, since an untestable row is also a WARNING.

Refs panyam/mcpkit#1463
Only the poll scenario graded this row, and only against its own target,
so a server whose replay-capable types it polls reported it SKIPPED
while its push streams broke the rule. mcpkit did exactly that
(panyam/mcpkit#1468): a gap on a cursorless type sent
notifications/events/active {cursor: null, truncated: true}.

The push scenario now finds a type without replay by opening each
push-capable type until one confirms its stream with cursor: null, the
push analogue of poll reading cursor: null off its result, since
events/list declares nothing about replay. It asks for a gap there with
yield_gap and watches 1.5s:

  no truncated:true active     SUCCESS
  truncated:true active        WARNING (SHOULD), with the frame
  every type replays           SKIPPED, as on poll
  no yield_gap control         untestable, naming it

run() fills the row in on every early return, so runs stay comparable.

The negative fixture gains one no-replay type and the yield_gap control,
which sends the conformant fresh active for any other type. Five cases,
all red on the old scenario.

Refs panyam/mcpkit#1468
@panyam

panyam commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

Closing in favour of a fresh PR from the same branch, so the description reads against the finished diff rather than as a sixth revision of one written when two of the five scenarios existed. Same commits, nothing dropped. Link in a moment.

@panyam panyam closed this Sep 24, 2026
@panyam

panyam commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

Replaced by #521, same branch and same commits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant