Skip to content

fix: make EvaluationReport.to_file() reject non-strict JSON - #393

Open
serenearyal wants to merge 1 commit into
strands-agents:mainfrom
serenearyal:fix/384-report-to-file-strict-json
Open

fix: make EvaluationReport.to_file() reject non-strict JSON#393
serenearyal wants to merge 1 commit into
strands-agents:mainfrom
serenearyal:fix/384-report-to-file-strict-json

Conversation

@serenearyal

Copy link
Copy Markdown

Description

EvaluationReport.to_file() had the same three problems that #383 fixed for Experiment.to_file():

  • json.dump ran with defaults, so a NaN or Infinity score was written as the bare literal NaN. Python's lenient json.loads reads it back, but strict parsers (jq, JavaScript JSON.parse, serde) reject the file. overall_score: float and scores: list[float] make NaN reachable from ordinary aggregation.
  • The destination was opened with mode "w" before serialization, so a failure mid-write truncated the file. On main, a valid 224-byte report was cut to 106 bytes of invalid JSON after a save with a non-serializable case value. If to_dict() itself raised, the file was left at 0 bytes.
  • The file was opened without encoding=. This was harmless while ensure_ascii defaulted to True, but it becomes a real bug on non-UTF-8 locales once the writer emits UTF-8 text like fix: make Experiment.to_file() reject non-strict JSON instead of writing invalid files #383 does.

Changes

  • EvaluationReport.to_file() now mirrors Experiment.to_file(): serialize first with ensure_ascii=False, allow_nan=False, encode to UTF-8, and write the bytes only on success. NaN, Infinity, and unpaired surrogates raise a ValueError with the original error as __cause__. Non-serializable objects raise TypeError before the file is touched. The Raises: docstring is extended to match.
  • EvaluationReport.from_file() reads with encoding="utf-8" so the read side matches the write side, the same as Experiment.from_file().
  • Four regression tests in tests/strands_evals/types/test_evaluation_report.py: NaN rejected with the existing file intact, unpaired surrogates rejected without writing, non-serializable value rejected with the existing file intact, and a UTF-8 round trip.

This is a plain mirror of #383 rather than a shared helper. Happy to extract one if you prefer; the issue notes this is the third copy of the pattern.

Related Issues

Fixes #384. Same pattern as #383 (Experiment.to_file), which also covers #382.

Documentation PR

None needed. Behaviour change is limited to raising on data that could not produce valid JSON.

Type of Change

Bug fix

Testing

  • Added four regression tests; all fail on main and pass with this change.

  • hatch test on Python 3.10, 3.11, 3.12, 3.13, 3.14: 2144 passed on each.

  • hatch fmt --formatter, hatch fmt --linter, mypy, and pre-commit run on the changed files: clean.

  • hatch run test-integ not run: no integration test exercises EvaluationReport.to_file(), and the suite needs AWS credentials I do not have locally.

  • Reproduced the truncation and NaN cases with a scratch script on main, and confirmed the fixed writer raises before touching the file.

  • I ran hatch run prepare

Checklist

  • I have read the CONTRIBUTING document
  • I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
  • My change is focused and reasonably small; I have split unrelated work into separate PRs
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@serenearyal
serenearyal requested a review from a team as a code owner September 4, 2026 04:48
@serenearyal
serenearyal requested a review from pgrayy September 4, 2026 04:48
@github-actions github-actions Bot added area-core Core eval framework: Case, Experiment, task handler, evaluation data stores area-devx Developer experience: papercuts, confusing public APIs, error messages, ergonomics, usability bug Something isn't working labels Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-core Core eval framework: Case, Experiment, task handler, evaluation data stores area-devx Developer experience: papercuts, confusing public APIs, error messages, ergonomics, usability bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] EvaluationReport.to_file() writes RFC-8259-invalid JSON (literal NaN), has no explicit encoding, and can leave a truncated file

1 participant