Skip to content

feat: check direct-tool responses against forge-generated response schemas - #301

Merged
mattzcarey merged 1 commit into
mainfrom
feat/direct-tool-response-validation
Oct 21, 2026
Merged

mattzcarey merged 1 commit into
mainfrom
feat/direct-tool-response-validation

Conversation

@mattzcarey

@mattzcarey mattzcarey commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

The direct tools (?codemode=false, with or without experimentalForge) now check every JSON response, success or error, against the response schemas forge generates, and record each mismatch as a response_schema_mismatch Analytics Engine event. npm run report:schema-mismatches lists them by product → operation → violation for the owning teams.

We want to advertise outputSchema on direct tools, but clients that validate structuredContent would reject correct responses that don't match their schemas. This soaks the schemas we'd advertise first, so we can fix what breaks. Part of #320.

  • Schemas: forge's MCP target now generates each tool's responses (forge.json → cloudflare/forge's forge-mcp branch at 9561734, kept rebased on its main). tools/list never sends them.
  • Where it runs: the check runs in the forge tools service, which already holds the schemas, so nothing is fetched or parsed per isolate and the main Worker never loads them. The main Worker trims each array to 25 items and calls checkResponse over RPC in waitUntil. Tool results never change or fail, and a check costs at most ~10 ms.
  • No customer data: events and logs hold keywords, schema locations and instance paths, with indices and undeclared keys as *.
  • Traceable: generate-tools.ts records the openapi.forge.json release in source.json. Each mismatch names it and the generated tool, and the report links to it.
  • Cost: the forge tools bundle grows from 0.76 MB to 1.2 MB gzipped, and its startup evaluation from ~70 ms to ~135 ms (Node).
  • Spec: regenerated from openapi@fdbc7d9, the first release whose account_or_zone has its accounts/zones enum and description (account_or_zone has no enum or description in openapi.forge.json forge#59). The forge tools stop adding them and only make account_or_zone_id optional.
  • Known false positive: forge doesn't read writeOnly yet, so required write-only fields (Hyperdrive's origin.password, for one) show up as missing-property violations until Read writeOnly so required write-only fields don't break response schemas forge#63 lands.
  • Config: new RESPONSE_VALIDATION_SAMPLE_RATE var (1 everywhere) and new dependency @cfworker/json-schema.
checkResponse({ toolName, method, pathTemplate, status, body, bodyChars, props })

Tests ran (npm run check), including worker tests through both surfaces against the real generated schemas. scripts/bench-response-validation.ts measures the check.

@mattzcarey
mattzcarey force-pushed the feat/direct-tool-response-validation branch 3 times, most recently from acb6e56 to 3600fe8 Compare October 21, 2026 10:33
@mattzcarey
mattzcarey force-pushed the feat/direct-tool-response-validation branch from 3600fe8 to 609c3a3 Compare October 21, 2026 11:12
@mattzcarey mattzcarey changed the title feat: record direct-tool responses that don't match their OpenAPI response schema feat: check direct-tool responses against forge-generated response schemas Oct 21, 2026
@mattzcarey
mattzcarey force-pushed the feat/direct-tool-response-validation branch 2 times, most recently from 45e7627 to aab56a4 Compare October 21, 2026 17:55
…hemas

Forge's MCP target now generates each tool's response schemas (forge.json
pins cloudflare/forge's forge-mcp branch, which adds them; tools/list never
sends them). Both direct-tool surfaces check every JSON response, success or
error, against them and record each mismatch as a response_schema_mismatch
Analytics Engine event, with schema locations and keywords only, never
response values.

- The check runs in the forge tools service, where the generated schemas
  already live: the main Worker trims each array to 25 items and calls its
  checkResponse RPC in waitUntil, catching everything, so it can't change,
  fail or slow a tool result, and loads nothing into the main Worker.
- generate-tools.ts records the openapi.forge.json release in source.json,
  so each mismatch names the spec it was checked against.
- scripts/schema-mismatch-report.ts groups mismatches by product, operation
  and violation, naming the generated tool and linking the spec release.
- RESPONSE_VALIDATION_SAMPLE_RATE samples checks; adds @cfworker/json-schema.

Regenerated from openapi@fdbc7d9, whose account_or_zone carries its
accounts/zones enum and description (cloudflare/forge#59), so the forge
tools no longer add them.
@mattzcarey
mattzcarey force-pushed the feat/direct-tool-response-validation branch from aab56a4 to 90ccefa Compare October 21, 2026 18:37
@mattzcarey
mattzcarey merged commit 617e1da into main Oct 21, 2026
5 checks passed
@mattzcarey
mattzcarey deleted the feat/direct-tool-response-validation branch October 21, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant