Skip to content

Align action API validation across SDKs with shared conformance cases - #1

Open
chinnuteja wants to merge 1 commit into
codag-megalith:mainfrom
chinnuteja:conformance-action-api
Open

chinnuteja wants to merge 1 commit into
codag-megalith:mainfrom
chinnuteja:conformance-action-api

Conversation

@chinnuteja

Copy link
Copy Markdown

Python and TypeScript currently send metrics that omit required action_kind or decision values, while Go rejects them locally. The action endpoints also disagree with the published ActionKind enum and required ToolCall.name. This change aligns those validation paths and adds shared regression cases.

Reproduced inconsistencies

These are client-side results against upstream e24b983. Accept means the client reaches the request path; reject means it fails locally.

Input Published contract Python TypeScript Go
Metric missing action_kind Reject Accept Accept Reject
Metric with decision: "" Reject Accept Accept Reject
Action with kind: "tests" Reject Accept Accept Accept
Action with tool: {} Reject Accept Reject Reject

Missing decision also passes Python and TypeScript validation: their token checks default absent values to an empty string, which matches the token pattern.

Changes

  • Require action_kind and nonempty decision in Python and TypeScript metrics.
  • Validate action and metric kinds against the nine published enum values in all three clients. The TypeScript declaration already exposes this exact union.
  • Require tool.name in Python actions, and report non-string kinds as ValidationError.
  • Add 36 shared accept/reject cases. Nine cases cover dynamic inputs that cannot be represented by Go's typed structs and are skipped there.
  • Assert zero requests for rejected inputs and exactly one request for accepted inputs in all three suites.
  • Extend the existing contract check: every required field of MetricEvent and ActionEnvelope needs a reject-case that differs from an accepted input only by omission of that field. Invalid-value cases and descriptive field labels cannot substitute for omission coverage. Four regression tests exercise this guard.

No new runtime or test dependencies and no Makefile changes.

Verification

  • Python: 85 tests pass.
  • TypeScript/JavaScript: 74 tests pass.
  • Go: go test -count=1 ./... passes.
  • Contract check and test-count floor pass.
  • Windows equivalents of the packaged-artifact checks pass: build/install/import the Python wheel (including py.typed), pack/install/import the npm package, typecheck its declarations with strict NodeNext settings, and build/run a Go downstream consumer. The repository's Bash make test-dist remains for Linux CI.

Server behavior has not been tested. These changes enforce the published request contract locally and tighten previously permissive validation, including rejection of unknown enum values.

All three clients still reject result: "", although the schema permits it. This existing discrepancy is left for a separate decision about whether the schema or clients should change.

Require metric action_kind and decision, validate ActionKind values, and
check Python tool.name. Reject malformed Python kinds with ValidationError.

Add 36 shared cases with request-count assertions. Require each mandatory
field omission to pair with an otherwise identical accepted input, and
exercise the coverage guard with regression tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant