Skip to content

feat(metrics): add JSON schema conformance metric - #370

Open
aayushi-sing wants to merge 1 commit into
OpenAgentHQ:mainfrom
aayushi-sing:feat/schema-conformance
Open

feat(metrics): add JSON schema conformance metric#370
aayushi-sing wants to merge 1 commit into
OpenAgentHQ:mainfrom
aayushi-sing:feat/schema-conformance

Conversation

@aayushi-sing

Copy link
Copy Markdown
Contributor

Summary

Adds a deterministic JSON Schema conformance metric for evaluating structured model outputs.

Changes

  • Add SchemaConformance generation metric.
  • Parse generated answers as JSON and validate them against JSON Schema Draft 2020-12.
  • Return 1.0 for fully conformant outputs and 0.0 for invalid JSON or non-conformant outputs.
  • Add deterministic top-level field-level partial scoring for object schemas.
  • Pass dataset metadata through the evaluation pipeline so metrics can access the expected JSON schema.
  • Register schema_conformance in METRIC_REGISTRY.
  • Add jsonschema dependency and update uv.lock.
  • Add unit tests covering valid, invalid, partial, missing-schema, empty-schema, and extra-property cases.

Testing

  • uv run pytest tests/unit/test_metrics/test_generation.py -q — 41 passed
  • uv run ruff check openagent_eval/metrics/generation/schema_conformance.py tests/unit/test_metrics/test_generation.py — passed
  • uv run ruff format --check openagent_eval/metrics/generation/schema_conformance.py tests/unit/test_metrics/test_generation.py — passed
  • git diff --check — passed

The full test suite was also run locally. It had one unrelated Windows-specific failure in test_oaeval_help_runs_without_pytest caused by a cp1252 Unicode encoding error; no changes were made to that unrelated test.
Closes #347

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a structured-output / JSON-schema conformance metric

1 participant