Repository navigation
feat: check direct-tool responses against forge-generated response schemas - #301
Merged
Merged
Conversation
mattzcarey
force-pushed
the
feat/direct-tool-response-validation
branch
3 times, most recently
from
October 21, 2026 10:33
acb6e56 to
3600fe8
Compare
3 of 5 tasks
mattzcarey
force-pushed
the
feat/direct-tool-response-validation
branch
from
October 21, 2026 11:12
3600fe8 to
609c3a3
Compare
mattzcarey
force-pushed
the
feat/direct-tool-response-validation
branch
2 times, most recently
from
October 21, 2026 17:55
45e7627 to
aab56a4
Compare
…hemas Forge's MCP target now generates each tool's response schemas (forge.json pins cloudflare/forge's forge-mcp branch, which adds them; tools/list never sends them). Both direct-tool surfaces check every JSON response, success or error, against them and record each mismatch as a response_schema_mismatch Analytics Engine event, with schema locations and keywords only, never response values. - The check runs in the forge tools service, where the generated schemas already live: the main Worker trims each array to 25 items and calls its checkResponse RPC in waitUntil, catching everything, so it can't change, fail or slow a tool result, and loads nothing into the main Worker. - generate-tools.ts records the openapi.forge.json release in source.json, so each mismatch names the spec it was checked against. - scripts/schema-mismatch-report.ts groups mismatches by product, operation and violation, naming the generated tool and linking the spec release. - RESPONSE_VALIDATION_SAMPLE_RATE samples checks; adds @cfworker/json-schema. Regenerated from openapi@fdbc7d9, whose account_or_zone carries its accounts/zones enum and description (cloudflare/forge#59), so the forge tools no longer add them.
mattzcarey
force-pushed
the
feat/direct-tool-response-validation
branch
from
October 21, 2026 18:37
aab56a4 to
90ccefa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The direct tools (
?codemode=false, with or withoutexperimentalForge) now check every JSON response, success or error, against the response schemas forge generates, and record each mismatch as aresponse_schema_mismatchAnalytics Engine event.npm run report:schema-mismatcheslists them by product → operation → violation for the owning teams.We want to advertise
outputSchemaon direct tools, but clients that validatestructuredContentwould reject correct responses that don't match their schemas. This soaks the schemas we'd advertise first, so we can fix what breaks. Part of #320.responses(forge.json→cloudflare/forge'sforge-mcpbranch at9561734, kept rebased on its main).tools/listnever sends them.checkResponseover RPC inwaitUntil. Tool results never change or fail, and a check costs at most ~10 ms.*.generate-tools.tsrecords theopenapi.forge.jsonrelease insource.json. Each mismatch names it and the generated tool, and the report links to it.openapi@fdbc7d9, the first release whoseaccount_or_zonehas itsaccounts/zonesenum and description (account_or_zone has no enum or description in openapi.forge.json forge#59). The forge tools stop adding them and only makeaccount_or_zone_idoptional.writeOnlyyet, so required write-only fields (Hyperdrive'sorigin.password, for one) show up as missing-property violations until Read writeOnly so required write-only fields don't break response schemas forge#63 lands.RESPONSE_VALIDATION_SAMPLE_RATEvar (1everywhere) and new dependency@cfworker/json-schema.Tests ran (
npm run check), including worker tests through both surfaces against the real generated schemas.scripts/bench-response-validation.tsmeasures the check.