Conversation
… phase 1 Adds the requirement-traceability yaml for MCP Events plus the first two of five server scenarios, scoring against the design sketch that merged on main of modelcontextprotocol/experimental-ext-triggers-events on 2026-09-08. The SEP number is a placeholder. Events has no PR in modelcontextprotocol/modelcontextprotocol, and both traceability gates are numeric (the filename regex at src/traceability/index.ts:174, the check-id regex at line 41), so an unnumbered file is dropped from the manifest without an error. 9999 is far from the live range; the rename is mechanical and the yaml header names the rename as a merge blocker rather than a follow-up: a merge under 9999 would publish it as a real SEP. The reservation question is open with the WG. src/seps/sep-9999.yaml declares 131 checks and 30 excluded rows against 144 RFC 2119 keyword occurrences. Pure MAY and OPTIONAL sentences get no check. Client, host, receiver and SDK-guidance obligations are excluded rather than declared, so the denominator holds only what a server-side run can observe. events-discovery covers the capability, events/list, the descriptor fields and the error-code contract. events-poll covers poll delivery, the EventOccurrence shape and cursor lifecycle, driving a real two-poll quiet-period loop so cursor advancement is gradeable against a server with no traffic. Together they emit 45 rows; the other 86 report untested until the push and webhook scenarios land. The capability gate asks before it skips. An optional capability a server never declared is not a defect, but a server that answers events/list while declaring nothing has a surface no spec-following client would reach, and a SKIP would report that as a clean run. mcpkit is in exactly that state. Against examples/events/kitchen-sink at mcpkit main: discovery 8/11, poll 18/28. Six divergences, five of them server-side defects not previously tracked, one confirming the known nextPollSeconds drift. The yaml header lists each. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017kyEZdgh1zjdhURgdcZ4nL
…e occurrences off a poll that can carry them Running the suite against a second implementation — Peter Alexander's metronome-mcp.fly.dev, written by the author of the design sketch — found two bugs that kitchen-sink alone could not surface. minimalArguments gave up on any inputSchema with a required property. The document's own examples are filters and transforms with nothing required, and the helper took that for a rule. metronome.tick requires periodSeconds and label, so every poll row reported untestable and the scenario scored 1/34. It now derives a value per required property from the schema itself — default, const, first enum or examples entry, then minimum for numbers or a fixed string — and still declines to guess when a required property offers nothing. The sep-9999-occurrence-* rows were graded off the bootstrap poll, which passes cursor: null. Null means start from now, so a conformant server returns no events there: the suite could only score those seven rows against a server that violated sep-9999-cursor-null-starts-from-now. They now poll forward from the returned cursor, waiting nextPollMs between attempts (clamped to 250ms-2s) up to a 6s ceiling, and still report untestable against a quiet server. Scores with the fixes. kitchen-sink at d2950655: discovery 8/11, poll 26/29, up from 18/28 with no change to mcpkit. metronome: discovery 10/11, poll 27/29. Two metronome divergences, in the yaml header. It answers -32603 rather than -32602 for arguments that violate inputSchema. And it declares events under capabilities.extensions["io.modelcontextprotocol/events"] where this document puts it top-level, which is a question for the WG before the row moves either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the third of five events scenarios, covering the 17 sep-9999-stream-* rows. The Connection abstraction cannot express events/stream: conn.request resolves on the response for its id, and a push stream withholds that until the subscription ends, so a scenario awaiting it would block for the life of the stream and see none of the notifications it grades. stream.ts opens the POST itself and hands back a session while the stream is still open. Aborting that request is also the document's client-side cancel on Streamable HTTP, so the cancel row falls out of the same mechanism. The watch window is 35s by default, over the runner's 30s timeout, so the scenario needs --timeout 60000. That is deliberate: the heartbeat is a MUST with a 30s SHOULD on cadence, so a shorter window cannot tell a silent server from a slow one. Under 30s the heartbeat rows report untestable rather than failing a server that may be conformant. The interval check carries 2s of tolerance, because kitchen-sink times its heartbeat at exactly 30s and measured 30005ms once scheduling had its say. Rows needing something the client cannot ask for - an upstream failure, a retention gap, a termination, a server-initiated close - report untestable with the prerequisite named, per src/scenarios/untestable.ts. kitchen-sink at d2950655: 11/15. metronome: 13/15, where both failures are untestable rows rather than divergences. One new divergence, not previously tracked: kitchen-sink puts the correlation id in params.requestId, where the document requires params._meta["io.modelcontextprotocol/subscriptionId"]. A client holding two streams cannot route by what the document tells it to read.
…agement The fourth of five scenarios: the 26 subscribe, TTL and unsubscribe rows. Delivery itself - signatures, the verification handshake, control envelopes - waits for events-webhook-delivery and a reachable callback. Two things shape it. A run holds one principal, so the key components the harness can vary (url, name, arguments) are graded and the principal half is reported untestable rather than passing a cross-tenant row that was never exercised across two tenants. And a server may cap concurrent subscriptions per principal: kitchen-sink allows two per event type, so every throwaway subscription is released as soon as its check is graded, including a probe the server was supposed to reject but accepted. Without that the scenario grades the cap error instead of the rule it was probing, which it did three times while being written. Every subscription is unsubscribed before the scenario returns, which matters against a public server. kitchen-sink at d2950655: 17/23, three divergences. It accepts an http:// callback URL where the document requires https and -32602 (two rows, one probe), and unsubscribing a key it never held succeeds instead of answering -32011 NotFound, so a client cannot tell teardown from a typo. metronome: 20/23, no divergences. The three failures are untestable rows: the unsupported-mode path (its only event type offers every mode), the auth requirement, and cross-tenant isolation.
…nario Grades what the server POSTs: the verification handshake, Standard Webhooks signing, retries, control envelopes and the SSRF rules. receiver.ts runs a real HTTP endpoint and keeps the raw body bytes, because the signature covers those and re-serializing the JSON would change them. The scenario needs a callback the server can reach, and EVENTS_WEBHOOK_CALLBACK_BASE supplies one. Without it the receiver binds to loopback, which is not a degraded mode but a different question: the document requires a server to refuse a non-routable callback, so a loopback URL is the SSRF probe. A server that refuses it passes the SSRF rows and reports delivery untestable; one that POSTs to 127.0.0.1 fails them and hands over real deliveries to grade everything else against. kitchen-sink at d2950655: 12/21, four divergences, two of them security-shaped. It accepts and delivers to a loopback callback, so a caller can aim deliveries at anything the server can reach and the caller cannot. It signs with the literal whsec_ string as the HMAC key, where the document says the key is the base64-decoded bytes after that prefix. No Standard Webhooks receiver verifies these signatures; confirmed by hand against four candidate formulas before filing it here. It delivers without the verification handshake, so an unverified third-party URL receives POSTs. metronome refuses the loopback callback outright, which passes the SSRF rows and leaves the delivery rows untestable. Grading it needs a public callback. One harness bug, found while reading the results: the retry row folded signature validity into its freshness check, so a server with the wrong formula was reported as reusing timestamps it had not reused. Freshness is now graded on distinct timestamps and signatures alone. Coverage is now 117 of the 131 declared rows across the five scenarios. The remaining 14 need a server that changes at runtime (schema evolution, list_changed, event-type removal) or error paths this run did not reach.
The SSRF row is fixture configuration, not a library defect. kitchen-sink passes WithWebhookAllowPrivateNetworks(true) so the demos can reach a local receiver, and that one flag disables both the subscribe-time and dial-time guards; mcpkit's default path blocks loopback and the other non-routable ranges. The row still fails, because the suite grades the server it was pointed at, but it means the target runs with protection off rather than that mcpkit lacks it. Redirects are not followed either: webhook.go sets CheckRedirect to ErrUseLastResponse, which is what the suite already scored. Adds the root cause behind each of the other five, read out of mcpkit at 5cfb7c51. The signature bug is the widest: the whsec_ format arrived two days after signing landed and key derivation was never revisited, so both verifiers, the Go and Python clients, the whole-enchilada receiver and the telegram tests all share it. Server and clients agree with each other and none agrees with the document, which is the failure a conformance suite exists to catch.
webhook.ts listed 26 ids in ALL_IDS, the set it reports untestable when it cannot get far enough to grade, where its gradeable path emits 27. sep-9999-error-unsupported was the missing one, so against a server with no webhook-capable event type the row did not report a missing prerequisite — it vanished from the run. A suite that emits a different row set depending on the server is a suite whose pass count cannot be compared across SDKs. webhook-delivery.ts reported sep-9999-envelope-gap and -terminated untestable unconditionally, on the reasoning that no client can ask a server for a retention gap or a revocation. True, and beside the point: if the server sends one anyway the envelope is sitting in what the receiver recorded, and nothing looked. Both are now graded when they arrive and untestable when they do not, which is how push.ts has always handled its gap row. Also makes the settle window a knob. Three hard-coded 5s sleeps let retries and the two non-retryable probes play out, which is most of that scenario's wall time and pure waste against a fixture that answers in microseconds. EVENTS_DELIVERY_SETTLE_MS overrides it; the default does not change.
A green run against kitchen-sink proves a check does not false-positive. It proves nothing about whether the check catches anything, and only discovery and poll had controls that did. These add 71 cases over the other three scenarios, 100 across all five, each pairing a conformant in-process server with one broken in exactly one way. One fixture serves all five. negative-fixture.ts holds the stateless events server that negative.test.ts had inline, extracted unchanged, plus three things the later scenarios need: events/stream as a real SSE response written on timers, because the heartbeat cadence and the cancellation rows are about timing; a subscription store keyed the way the document keys it, minus the principal a single run cannot vary; and a delivery engine that POSTs signed Standard Webhooks payloads to the callback and retries them, because the signature and the retry cadence are only observable from the endpoint. Every divergence the suite found in a real implementation has a case here, so a red run can be read rather than re-derived: the correlation id in params.requestId, streams counted against the subscription cap, an http:// callback accepted, unsubscribe answering success for a key it never held, the HMAC keyed on the literal whsec_ string, and delivery with no verification handshake. The fixture also does the six things no client can ask a real server for — fail upstream, lose its replay window, terminate a subscription, close a stream itself, and send the gap and terminated control envelopes. Those rows report untestable against both implementations, so these are the only evidence those checks work at all. Two notes for whoever writes the next one. vi.resetModules() is needed for the scenarios that read their timing knobs at module load, and a reset registry hands back a second copy of the connection module, so the run context has to be built from the same fresh import or instanceof JsonRpcError is false across the two graphs. And the delivery fixture spaces an out-of-order event 60ms ahead of the verification POST, because the check compares millisecond arrival times and two loopback POSTs land inside the same millisecond often enough to read as in-order. npm test: 725 passing across 51 files.
The two files that grade timing spent almost all of their wall time asleep: the delivery controls waited out five settle windows per case and took 180s, the push controls 80s. Both together took the suite from 163s to 425s, which every contributor pays on pre-push. The cases are independent — each builds its own fixture on its own port and touches no shared state — so the only thing keeping them sequential was `vi.resetModules()` per case, needed because both scenarios read their timing knobs when the module is first evaluated. Importing once per file in `beforeAll` removes that, and the cases then run concurrently: delivery 45s, push 43s, suite 254s. One graph means one set of timings, and one push case has to outlast the 30s heartbeat cadence the document permits, because under 30s a silent server and a slow one are indistinguishable. It takes its own graph in a sequential block at the foot of the file, after the concurrent cases are done with theirs; resetting the registry while they were still running would pull it out from under them. 725 passing, unchanged.
The design sketch put it top-level under `capabilities.events`. metronome, the sketch author's own server, used `capabilities.extensions`. Raised with him 2026-09-22 rather than resolved by picking a side, and the document turned out to be the thing that was wrong: upstream PR 7 moves the sketch into Extension Negotiation, keyed by `io.modelcontextprotocol/events`. `declaredEventsCapability` and the four scenarios that probe the declaration now read the extensions map. `EVENTS_CAPABILITY` is gone; `EVENTS_EXTENSION_ID` absorbed its job, having previously carried a comment saying in as many words not to read the capability at that key. Two rows are restated from PR 7 and four are new: the empty settings object, the optional client declaration, `listChanged` gating whether the server may send `notifications/events/list_changed`, and the `-32601` fallback for a server that does not offer the extension. The four are declared and not yet emitted, so they report untested. The header's record of both metronome divergences becomes a record of how they resolved. He fixed the `-32603`-instead-of-`-32602` one the same day. These rows track an unmerged PR, which the head of this file now says. Refs modelcontextprotocol#504
Six push and webhook rows describe what a server does when something goes wrong upstream, and a healthy server does none of it during a run. The scenarios watched a window, saw nothing, and reported untestable, which under src/scenarios/untestable.ts is red and stays red. A fixture MAY now expose the conditions as ordinary tools. This calls the two that are safe to fire mid-stream — a transient upstream failure and a retention gap — neither of which ends the subscription, so the rows that grade heartbeats and deliveries still see what they need. Termination is left alone deliberately: it is terminal for the source and these scenarios share one fixture process, so firing it here would poison whatever runs next. The controls are called over their own connection rather than the streaming one. That is the more faithful simulation, since an upstream failure does not arrive as a request from the subscriber watching for it, and the signals fan out to every live subscriber regardless. Discovery happens on the connection run() already holds, not around the stream. The first version probed later and cost a connect plus two tools/list round trips inside the observation window, which was enough to miss a termination arriving at 250ms and turned negative-push.test.ts red. A server without the controls now pays nothing for the question and opens no second connection. Servers without controls are unaffected and keep reporting untestable, with the reason naming the specific tool that would make the row gradeable rather than describing the gap in the abstract. Refs modelcontextprotocol#504
sep-9999-stream-terminated-ends-subscription and both sep-9999-stream-final-result-* could not be graded on the main session, and not for want of a control: the harness cancels that stream to test stream-cancel-stops-delivery, and on Streamable HTTP a client-side abort is terminal, so no final frame is ever sent. A server-ended stream is a different stream. terminateChecks opens its own, fires the terminate control, and grades all three from it. It refuses to terminate the event type the rest of the run depends on, because termination is one-shot for the source and the four events scenarios share one fixture process; a fixture offering a single push-capable type gets untestable with that reason rather than a poisoned suite. The main session keeps grading these opportunistically. A server that closes the stream itself has already answered them, and dedupe is first-wins, so the control path never runs for ids the observation window already produced. That ordering is what the negative fixture exercises, and getting it wrong turned negative-push.test.ts red twice: once by removing the opportunistic path entirely, once by flattening three rows to one severity when terminating is a MUST and the two final-result rows are SHOULDs. Also fixes a bug in the final-result shape check that mcpkit surfaced. It counted `resultType` as information the frame carries, but `resultType` is a base-protocol field on the common Result interface that servers MUST include, which sep-2640.yaml already records. Every conformant server failed that row. `_meta` was already excluded; `resultType` now is too. Against mcpkit: events-push 14/16 to 17/18, zero warnings, the remaining failure being a real missing subscriptionId. Refs modelcontextprotocol#504
…ontrols The rule is that two tenants subscribing to the same `(name, arguments)` with the same callback get distinct subscriptions, because the principal is part of the key. A run authenticates as one principal for its lifetime, so the harness could only ever supply one side of that comparison and the row reported untestable. A fixture MAY now register on another principal's behalf and answer whether a given principal's subscription still exists. With both, the check does the comparison the rule is about: same event, same callback, principal the only thing that differs. Two things are graded, and the second is the one that bites. Distinct ids show the principal reached the key at all. A surviving subscription after the other tenant unsubscribes shows the two are genuinely independent, which is what a tenant would notice if it were not true — one tenant able to cancel another's subscription. askControl joins fireControl in helpers for the controls that answer with text rather than just succeeding. Servers without the controls are unaffected and keep reporting untestable, now naming the two tools that would make the row gradeable. Against mcpkit: events-webhook 17/23 to 18/23. Refs modelcontextprotocol#504
…iled one
Once a server verifies inside events/subscribe, which is how the document
reads ("returned synchronously from events/subscribe"), three things in the
webhook-delivery scenario stopped measuring what they claim to.
Every probe callback answered every POST with its failure status, the
challenge included, so a verifying server could not subscribe the 410, 413,
retry or redirect probes and all four rows went untestable. The receiver
now answers the challenge on every path before the path's behaviour
applies; wrong-challenge is the one path that still fails it. The 410/413
and retry graders also waited for the first POST on their path, which is
now the challenge, answered 200 and never retried, so they wait for the
first non-verification POST instead. The redirect grader waits for a POST
it actually answered with 302.
verification-failure-error was untestable on the assumption that the
handshake is asynchronous. It now subscribes a wrong-challenge callback:
-32015 with reason challenge_failed passes; any other error, or an accepted
subscription that is then delivered to, fails; an accepted subscription with
no delivery is a WARNING, since the endpoint is safe but the subscriber was
never told.
delivery-retry-regenerates-signature failed on any two retries inside one
second, because webhook-timestamp is whole seconds and a regenerated stamp
is indistinguishable from a reused one there. It flapped against mcpkit,
whose first retry is 500ms out. Only a stamp repeated across attempts a
second or more apart now fails; retries all inside one second are a WARNING.
The fixture gains synchronousVerification (and ignoreFailedEcho for the
negative case) and retryGapMs. Against mcpkit kitchen-sink --serve the
scenario goes from 12 to 18 SUCCESS; with verification disabled in the
library the verification rows go red again.
ttl-long-grant-retained, ttl-no-expiry-persisted and ttl-no-expiry-gc-terminated were untestable because they need a restart the harness cannot ask for over the wire. A fixture exposing three controls now makes them gradeable: - events_conformance_restart rebuilds the server over the same subscription store and answers the generation it is moving to - events_conformance_generation answers the current one - events_conformance_subscription_state answers active, suspended or absent for a derived id, which the exists control cannot, since it hides suspended subscriptions The generation pair is what makes the restart detectable on both kinds of server: a stateful one ends the old session, so the scenario reconnects; a stateless one keeps answering on the same connection, first from the old build and then the new. The scenario subscribes once with ttlMs:null and once with a one-day grant, restarts, and reads both back. A no-expiry grant that is not honoured leaves its two rows untestable at WARNING rather than failing, since granting one is optional. The GC row watches the no-expiry subscription, whose placeholder callback fails every delivery, for EVENTS_GC_WAIT_MS: dropped passes, still there is a WARNING because dropping is a MAY, and already lost across the restart is untestable. The terminated envelope goes to the placeholder origin and cannot be observed, which the check says. The durability checks run last, since a restart ends the scenario's connection. Wait limits are read at call time so the negative controls can shorten them. Against mcpkit kitchen-sink --conformance-events the scenario goes from 22/23 to 25/26; a build whose restart loses the store fails the two retention rows.
The document gives a server four ways to confirm a callback wants deliveries: the challenge handshake, a configured allowlist, prior out-of-band verification, or a receiver-published /.well-known/mcp-webhook-receiver.json. The suite only knew the first, so it reported a server that read the document and delivered without challenging as one that "delivered to a callback that was never asked to prove intent". mcpkit 1449 landed that path today, which would have turned the row red for a server satisfying it. The receiver now publishes the document and counts fetches, and consent counts as established by a challenge or by a fetch that precedes the first delivery. It is declared over one path prefix and no other, so the main delivery path exercises the document route while every probe path still takes the handshake: one run covers both rather than whichever the server prefers. EVENTS_RECEIVER_WELL_KNOWN=0 withholds it. The document is honoured only on an https origin, so against a loopback receiver the handshake decides everything, the same way the SSRF rows only mean something over a tunnel. Two rows come out of that. sep-9999-verification-no-raw-endpoint-responses was an unconditional SUCCESS whose details read "no endpoint response body was echoed back in any server error observed during this run" — nothing had observed anything, so it could not have caught a leak. It now grades the error from the wrong-challenge probe, where the receiver echoes a distinctive string, and reports untestable on a run where no endpoint failed. sep-9999-error-callback-endpoint-error, declared since the first commit and never emitted, is claimed off that same error: the -32015 code and whether data.reason is one of the documented lastError categories. Why the failure branch stays a failure when nothing is observable: the allowlist and out-of-band paths cannot apply to these callbacks, because the path is minted fresh for the run and no operator has ever seen it. 7 new controls, 31 in negative-delivery.test.ts. Against the previous scenario the well-known case fails with "expected 'FAILURE' to be 'SUCCESS'", which is the regression this prevents.
Four declared rows emitted nothing while the request that would grade each was
already being sent. Claiming them costs no new round trips.
sep-9999-fallback-method-not-found — what a server that does not offer the
extension answers. discovery.ts already probes events/list on a server that
declares nothing and keys its SKIP off -32601, so the one run that ever meets
such a server now grades the rule instead of folding it into skipAll. A server
answering some other code fails it: a client cannot tell "no such extension"
from a real failure. Everything else still skips, because every other events
rule is genuinely inapplicable there.
sep-9999-capability-empty-settings — `{}` declares support with no list-change
notifications. Graded off the declared value, and inapplicable rather than failed
when the server declared settings of its own, since the rule says what an empty
object means and nothing about a populated one.
sep-9999-error-invalid-params and -resource-exhausted — the error-table rows for
-32602 and -32013. The first rides the poll invalid-arguments probe, the way
sep-9999-subscribe-url-https-required already rides its enforcement probe, with
details.gradedBy naming where the verdict came from. The second grades the three
concurrent streams in push.ts: a refusal for a quota must name the quota in
data.limit, and a server that reaches no limit reports untestable rather than
passing.
117 of 135 rows emitted before this and the commit before it; 122 now. Of the 13
left, seven wait on a control that can change a catalog mid-run, two on a second
credential, and sep-9999-error-forbidden on a principal the harness can be
refused as.
5 new controls. One existing assertion changes: the "skips the suite" case now
asserts the fallback row passes and everything else skips, which adds an
assertion rather than relaxing one. No assertion weakened.
… track The extensions-map change added four rows and restated two, and the header's arithmetic did not follow: it still said 131 declared, 117 of 131 emitted, and "the 14 rows nothing emits yet". It is 135, 122 and 13. It also said "see the tracking note at the head of this file" and there was no such note. There is now, as a banner: the six rows that quote text only PR 7 has, why the suite is ahead of the document, and to re-point spec_source at main when it merges. spec_source and spec_url named `main`, where a reader who fetched it found text contradicting those rows; both now name PR 7's head. Seven strings still told implementers the capability lives at `capabilities.events`, including the description attached to sep-9999-capability-events-object itself, so one check carried two contradictory statements and the one a reader of a failing run sees was the wrong one. The survivors were exactly the strings no control asserts on, which is also where to add an assertion. The note in helpers.ts keeps the old spelling deliberately, because it is explaining the history. declaredEventsCapability now calls extensionsOf rather than inlining the same lookup, which is what that helper's doc comment promises.
The comment introducing poll's error-table row named the webhook precedent by its exact slug, which makes `grep sep-9999-subscribe-url-https-required` report two scenarios emitting it when only one does. It now names the rule in prose. The header's count for poll.ts followed the same miscount and read 35 where the file emits 34; the suite total of 122 was right either way.
Measuring against kitchen-sink at mcpkit d7624d68 found two rows going missing rather than reporting a missing prerequisite, both in paths I had just added to. sep-9999-fallback-method-not-found was only emitted when the server declared nothing, so against every real events server it vanished. It is now SKIPPED there with the reason: a server that declares the extension is not a server that does not offer it, so the rule cannot apply. events-webhook-delivery emitted 27 rows when it refused a loopback callback and 29 when it delivered. The refusal answers two of the four SSRF rows by itself; the other two need a delivery to revalidate and a redirect to refuse, and were simply absent. Both now report untestable there. This is the defect the suite's own untestable policy exists to prevent: a row that appears on one path and not another makes two servers' pass counts incomparable, which is the whole point of a fixed denominator. Controls for both, asserting the row is present and inapplicable rather than absent.
… a time events-webhook-delivery grades 29 rows and reaches 2 of them against a server with its SSRF guards on. Not a defect on either side: the harness must be the receiver, because the signature, the headers, the handshake and the retry cadence are only visible from the endpoint being POSTed to, so it listens on loopback — and a hardened server refuses a loopback callback, which is what two of those rows grade. Both cannot hold for one subscription. The way out is a fixture control, the same shape as the other eight: events_conformance_allow_callback_origin permits one named origin past the callback guards for the rest of the process. The order is what keeps the SSRF verdict honest. Subscribe with the guards on, grade the two SSRF rows against the server as configured, then lift for this receiver's origin and subscribe again, so signing, headers, retries, envelopes and verification grade off real deliveries. Nothing about hardening is concluded after the override. sep-9999-ssrf-validate-at-delivery-time now reports SKIPPED when the origin was permitted on purpose, because a delivery to a deliberately-allowed origin is evidence of nothing; proving revalidation still needs a hostname whose DNS answer changes between subscribe and delivery. The control does not exist yet: mcpkit 1457 has the contract, and this side sits behind hasControl, so a server without it reports untestable exactly as before and names both ways out, the tunnel and the control. 26 of the 27 rows become reachable when it lands. Two controls cover both paths. Writing them turned up the fixture refusing a permitted origin anyway, because its https check ran first — which also showed the "harden" flag I had added was redundant, since a fixture without acceptHttpUrl already refuses the harness's loopback callback. Gone, and the permitted set now bypasses that check the way the real control will. npm test: 751 passing.
A missing prerequisite reports as a failure on purpose: SKIPPED is excluded from counts, exit codes and the expected-failures baseline, so a skipped row is one nobody can burn down. The cost is that a run reads as broken when it is mostly unchecked. events-webhook-delivery against a hardened server printed Passed: 2/29, 27 failed, 0 warnings where nothing was violated and 27 rows had no reachable callback to grade against. A reader cannot tell that from a server with 27 real defects. The distinction was already in the data. notTestable() writes a stable prefix and untestableCheck() sets details.untestable, both so consumers can separate "violated" from "could not be verified" without a new status. The runners just did not print it: Passed: 21/21, 0 failed, 8 warnings (8 of those unverified, not violated) The clause appears only when there is something to say, so a clean run prints exactly what it printed before. The four numbers were computed by three copies of the same four filters in server.ts, client.ts and authorization-server.ts, so a fifth number had to go in three places or in none. They now share tallyChecks/formatTally. No verdict changes: unverified counts rows already inside failed and warnings, and every exit code still keys off those, so a run that failed before fails now. Verified against a Go server, per AGENTS.md on shared infrastructure: mcpkit's kitchen-sink at eec13596 across all five events scenarios, 101 of 122 rows passing and 0 real failures, plus 10 unit cases covering the subset rule, the prefix-only path and that SKIPPED counts as neither.
…not refuse Both SSRF rows were graded from one refusal of the harness's http://127.0.0.1 receiver. The https rule refuses that URL on its own, so a server that never checks routability passed reject-non-routable anyway. mcpkit's conformance build was in exactly that state until panyam/mcpkit#1462 and passed the row throughout: -32602 delivery.url rejected: delivery.url must use https (got http) SUCCESS sep-9999-ssrf-reject-non-routable The row now has its own probe, run before anything can lift a guard: an https://127.0.0.1 callback aimed at a bare TCP listener that records whether the server connected. The scheme is valid, so only routability can refuse it, and the listener separates a refusal from a dial that failed afterwards. A server that skips the check and then fails its own verification POST answers -32015, which reads as a refusal from the subscribe alone. refused, never dialled SUCCESS dialled, either way FAILURE accepted, never dialled WARNING (may be delivery-time validation) Over a tunnel the server may be on another host, where its dial to 127.0.0.1 never reaches the listener, so there only -32602 counts as a refusal. The probe also runs in that mode, where the row used to be untestable. validate-callback-url keeps grading from the http refusal, since either rule refusing it is validation. The negative fixture gains rejectNonRoutableUrl. The existing "refusing the loopback callback passes the SSRF rows" case used a fixture that enforced https alone and asserted this row SUCCESS, which certified the bug; it now uses a fixture that refuses non-routable hosts, and still asserts SUCCESS. Four new cases cover the table above, all red on the old scenario. Refs panyam/mcpkit#1460
error-resource-exhausted was graded only inside the concurrency probe, which opens three streams on one event type and needs all three to stay open for stream-exempt-from-concurrency-cap. A server capped on that type passes the error row and fails the MUST beside it, so on one type the two cannot both pass, and a server that caps anything else never has the row provoked at all. mcpkit reports it untestable for exactly that reason (panyam/mcpkit#1461), though it answers -32013 with data.limit on every subscribe path. New read-only control, events_conformance_quota, answering {"name": "<event type>", "max": <n>}. When present, the scenario opens streams on that type one at a time, up to max+1, and grades the first refusal: -32013 with data.limit SUCCESS -32013 without it WARNING another code FAILURE max+1 opened, no refusal untestable, naming the cap that never bit A refusal before max+1 counts, since other scenarios may hold subscriptions under the same principal. The probe's row is pushed ahead of the concurrency probe's, so dedupe keeps it; without the control the concurrency probe's opportunistic grading stands as before, and its untestable message now names the control. The negative fixture gains a per-type quota with the control, and four cases cover the table and the absent control, all red on the old scenario. The existing maxConcurrent cases are unchanged. Refs panyam/mcpkit#1461
…o's own subscription
envelope-gap and envelope-terminated reported untestable against every
server, because nothing in a run produces either. The push scenario solved
the same problem with yield_gap and terminate, but those cannot carry over:
in mcpkit a source's gap and terminal signals reach push streams only, so
no webhook envelope results, and terminate ends a whole event type, which
for a webhook-capable type ends it for every scenario after this one.
Two per-subscription controls instead, taking { id }:
events_conformance_webhook_gap and events_conformance_webhook_terminate.
The scenario fires them after the retry and redirect probes, terminate
last since it ends the main subscription, and waits up to 5s for each
envelope.
envelope arrived, well formed SUCCESS
arrived without cursor / error WARNING
acknowledged, nothing arrived gap WARNING, terminated FAILURE
control absent or declined untestable, naming the control
The envelope rows now grade after the probes, from what arrived, so the
signalled envelopes also count toward the discriminator, signing and
webhook-id rows.
The negative fixture gains both controls, reusing its envelope sender with
the event and handshake switched off. Four cases, all red on the old
scenario; the malformed-envelope case asserts on the graded body as well as
the status, since an untestable row is also a WARNING.
Refs panyam/mcpkit#1463
Only the poll scenario graded this row, and only against its own target, so a server whose replay-capable types it polls reported it SKIPPED while its push streams broke the rule. mcpkit did exactly that (panyam/mcpkit#1468): a gap on a cursorless type sent notifications/events/active {cursor: null, truncated: true}. The push scenario now finds a type without replay by opening each push-capable type until one confirms its stream with cursor: null, the push analogue of poll reading cursor: null off its result, since events/list declares nothing about replay. It asks for a gap there with yield_gap and watches 1.5s: no truncated:true active SUCCESS truncated:true active WARNING (SHOULD), with the frame every type replays SKIPPED, as on poll no yield_gap control untestable, naming it run() fills the row in on every early return, so runs stay comparable. The negative fixture gains one no-replay type and the yield_gap control, which sends the conformant fresh active for any other type. Five cases, all red on the old scenario. Refs panyam/mcpkit#1468
commit: |
panyam
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
Adds a requirement-traceability yaml for MCP Events and the five server scenarios that score an implementation against it, plus 143 negative controls and one runner change. Everything here is extracted from the design sketch on
mainofmodelcontextprotocol/experimental-ext-triggers-events, as amended by PR 7.events-discoveryevents/list, the descriptor fields, the error-code range, the-32601fallbackevents-pollEventOccurrenceshape, the cursor lifecycle,maxAgeMs, truncationevents-pushevents/stream: confirmation, heartbeats, event frames, gaps, termination, cancellationevents-webhookevents/subscribe, the subscription key, TTL negotiation and durability,events/unsubscribeevents-webhook-delivery122 of the 135 declared rows are emitted. The other 13 need a prerequisite nothing exposes yet, and each says which under Out of scope. Five scenarios is the whole plan; what remains is rows inside these files rather than more files.
Provenance
panyam offered an Events conformance suite in
#triggers-events-wgon 2026-08-28 and @Poita accepted on 2026-08-31 ("No objections, that would be great!"). It follows the SEP-2640 suite (#330) in shape: extract the requirements into a traceability yaml first, then write scenarios that emit one check per row.Per
AGENTS.mdthis would normally start as an issue. The WG conversation is that discussion, and an extraction is easier to argue about as a diff than as prose, so it is here as a draft. Happy to split it into an issue if you would rather.Prerequisite knowledge
events-poll; Subscribing and Event Delivery is the densest normative section and is what the three later scenarios cover.sep-9999-delivery-signature-formulagradesv1,<base64>over{id}.{timestamp}.{body}, keyed on the base64-decoded bytes after thewhsec_prefix, computed over the raw request bytes.AGENTS.md§ Scenario design and § Check conventions — why this is five scenarios with 122 checks rather than 122 scenarios, and why a missing prerequisite fails rather than skips (Standardize how scenarios report setup/execution failures #248). That rule shapes more of this diff than anything else.src/traceability/— how a declared row becomestestedoruntested, which is what makes declaring all 135 rows up front the right move rather than an overreach.The number question
The part I would most like a decision on, because it is the only thing blocking merge that this branch can act on.
SEP numbers are PR numbers in
modelcontextprotocol/modelcontextprotocol, and Events has no PR there — the work lives in its own repository. Both traceability gates are numeric: the directory filter atsrc/traceability/index.ts:174(/^sep-\d+\.yaml$/) and the check-id filter at line 41 (/^sep-\d+-/). A file namedevents.yamlis silently dropped from the manifest rather than rejected, andsrc/new-sep/index.ts:163refuses any argument that is not a positive integer. So there is no honest name that works today.Three ways out, and I have no strong preference between the first two:
9999unblocked the extraction and sits far from the live range. The rename touches the filename, thesep:field, every id in the yaml and the same ids undersrc/scenarios/server/events/— mechanical, but wide enough to do once, in the right direction.The second blocker is not mine to clear: six capability rows quote PR 7, which moves the declaration into Extension Negotiation and is unmerged.
spec_sourceandspec_urlname that PR's head rather thanmain, and a banner at the head of the yaml says so and says to re-point them on merge.Reviewer's guide
A building inspector who cannot get onto the roof writes "could not access the roof" on the report. What they must not do is leave that line blank, because a blank line and a passed line look identical to whoever reads it next, and the owner then sells the house on a report that never mentioned the roof. Every awkward decision in this diff comes from that: a conformance run is a report somebody will act on, so the suite is built so silence is never mistaken for success. A check that cannot be exercised says which prerequisite was missing and counts as red (
AGENTS.md#248, because SKIPPED is excluded from counts, exit codes and the expected-failures baseline, so a skipped row is one nobody can burn down). The capability gate asks before it skips, since a server that answersevents/listwhile declaring nothing has an events surface no spec-following client can find. And where the inspector genuinely cannot reach — a retention gap, a revoked subscription, a callback the server will not POST to — the fixture grows a ladder, which is what the nineevents_conformance_*controls are.Read in this order. The first four are the argument; the rest is mechanism.
checks(). Everything after it grades what the server actually serves, field by field, reporting the first offender per field so ids stay stable regardless of catalog size.verificationChecksaccepts consent by challenge or by a fetch of the receiver's/.well-known/mcp-webhook-receiver.json, because the document allows four consent paths and grading only the handshake fails a server that took another; anddeliveryChecksgrades the SSRF rows against the server as configured before asking it to permit this one origin, so no verdict about hardening is taken after the override.identityChecksis the interesting part: the key is(principal, delivery.url, name, arguments), a run holds one principal, and the fixture'ssubscribe_ascontrol supplies the second so cross-tenant isolation is graded rather than assumed. Every subscription it creates is released before it returns.conn.request()resolves on the response for its id, and a push stream withholds that until the subscription ends, so stream.ts opens the POST itself and hands back a live session. receiver.ts is the other half: the harness has to be the callback endpoint, because signatures, headers, the challenge and retry cadence are only visible from there, and it keeps raw body bytes because the signature covers those.negative*.test.tsfiles — 143 controls over one fixture whose every option breaks exactly one rule. With two things wrong at once a flipped check does not say which one it caught. It also does the things no client can ask a real server for, so rows that report untestable against real implementations are graded here at least once.quietAdvanceChecksin poll.ts makes cursor advancement gradeable against a server where nothing is happening, which is the state a fixture is usually in. The rest is registration.How it works
Decision log
untested, which is the manifest working rather than a gap in this file.MAYandOPTIONALsentences get no row, and client, host and receiver obligations are excluded rather than declared. Roughly a fifth of the normative sentences address the client or the callback endpoint; declaring them would inflate the denominator with rows no server-side scenario could ever emit. The five that initially got a row were demoted toexcluded:rather than deleted, so the keyword sweep stays auditable.EventOccurrencefield table, the error-code table and the descriptor fields, which the sketch states declaratively. Since "severity follows the keyword" is undefined for them, the yaml header states the rule used: a field the document marks required, or a wire fact a client cannot work without, isFAILURE; anything else drawn from an example payload isWARNING.unverifiedis a subset offailedandwarnings, so no exit code moves and a clean run prints exactly what it printed before.Risk / blast radius
src/runner/summary.ts. Five new scenarios, one new yaml, one new entry inEXTENSION_IDS. No existing scenario, suite list or check id changes.sourceis{ extensionId }, somatchesSpecVersionreturns false for every--spec-versionand these never join a dated run. Reaching them takes--scenario events-*or--suite all.npm run build,npm run lintandnpm testpass — 774 across 52 files, of which 143 are the events negative controls.tallyChecks/formatTally. Verified against a Go server across all five scenarios perAGENTS.mdon shared infrastructure, plus unit cases covering the subset rule, the prefix-only path, and that SKIPPED counts as neither.src/seps/traceability.jsonwill drift once these ids are emitted in a real run. PerAGENTS.mdthe traceability workflow refreshes it by PR and it is not a gate, so this leaves it alone.Before / after
There was no Events suite before, so what matters is what an implementation looks like when scored, and what changed in the one being scored. Measured 2026-09-24 against mcpkit's
examples/events/kitchen-sinkat0b84dbca, with the fixture's conformance controls enabled:events-discoveryevents-pollevents-pushevents-webhookevents-webhook-deliveryZero real failures is the end of a nine-day argument, not the starting point. The first full run, on 2026-09-21, was 8/11, 26/29, 11/15, 17/23 and 12/21, with fourteen divergences that were all real defects. Every one is now closed. They are worth reading as a list, because they are the case for the suite:
capabilitiesdeclared nothing whileevents/listansweredwhsec_…stringhttp://callbacks accepted at subscribeparams.requestIdevents/streamcounted against the subscription capnextPollSecondsafter the rename tonextPollMsmaxAgestill in secondsdelivery: nulland noinputSchemaevents/pollanswering for a type advertising no polleventsomitted rather than[]when nothing happenedAnd one of them was not a defect in the implementation at all.
sep-9999-capability-events-objectfailed against both implementations for different reasons: mcpkit declared nothing, whilemetronome-mcp.fly.dev— written by the sketch's author — declared undercapabilities.extensions, where the document said top-level. Rather than soften the row or pick a side, I asked, on 2026-09-22. The document was the thing that was wrong, and PR 7 now moves it. Scoring two implementations against one unsoftened row found a defect in the specification, which is the strongest argument available for the shape of this suite.Two suite bugs came out of the same exercise and are fixed here:
minimalArgumentsgave up on anyinputSchemawith a required property, which scored metronome's poll at 1/34, and the occurrence rows were graded off acursor: nullpoll that a conformant server answers empty. Neither was visible against a single target.flowchart TD A[check runs] --> B{could it be exercised?} B -->|yes| C{requirement met?} C -->|yes| D[SUCCESS] C -->|no| E[FAILURE / WARNING by keyword] B -->|no, prerequisite missing| F[fails, naming the prerequisite] B -->|no, rule cannot apply here| G[SKIPPED] F --> H[counted as unverified in the summary]Running them
Use a spec version of
2026-07-28or later: extension negotiation did not exist before it, so nothing earlier can declare Events, and an older run reports the extension absent by design. The runner's default already lands there.events-pushneeds the longer timeout for the reason above (EVENTS_PUSH_WATCH_MSoverrides the window).EVENTS_WEBHOOK_CALLBACK_BASEpoints delivery at a public https callback and is the only way to grade those rows on a server that neither exposes the origin control nor accepts loopback.EVENTS_RECEIVER_WELL_KNOWN=0withholds the receiver document to force the handshake path.Out of scope
schema-evolution-additive,breaking-change-new-name,list-changed-notification, fourremoval-*) need a server whose catalog can change mid-run, tracked as a control in the reference implementation. Three (error-forbidden, bothauthz-*) need a runner that can send or drop a credential, which is feat: Allow fixed request headers in server conformance tests #453.capability-list-changed-gatedneeds a server that declareslistChanged: falseand then sends the notification anyway.capability-client-declarationis a MAY about client behaviour that no server scenario can emit, and belongs inexcluded:by this file's own rules.payload-minimalityis a SHOULD about payload content thatdelivery-body-sizemeasures the observable half of.--headeron theserverrunner (feat: Allow fixed request headers in server conformance tests #453). Four of the rows above want it, and scoring a second implementation currently needs a local proxy to inject a bearer token.sep-9999-*once the number question is settled, and re-pointingspec_sourceatmainwhen PR 7 merges. Both are merge blockers.