Add a check for stale generated sample fixtures #300

Activity

1. appleweiping commented on Sep 4, 2026

   

   appleweiping

   on Sep 4, 2026

   More actions

   Ran through the acceptance criteria against current main (040a279) to see what is left here, since #302 and #306 already landed a checker.

   Criterion 1 — regenerate and report drift: done. scripts/check_sample_fixtures.py regenerates sample_std.json from each dataset's sample_atif.json and diffs it, with --dataset to narrow and --update to rewrite.

   Criterion 2 — usable in CI: not yet. grep -rn check_sample_fixtures .github/ tests/ pyproject.toml .pre-commit-config.yaml returns nothing, so the checker only runs when someone remembers to invoke it.

   Wiring it in today would fail immediately. Running the script unmodified over all datasets:

   52 ok
   4 drift: agenttuning_alfworld, agenttuning_webshop, CharlieDreemur_OpenManus-RL, enterpriselab
   

   These are semantic, not formatting. In agenttuning_alfworld, record 0 step 1 is source: agent, message: "OK. I'll follow your instructions and try my best to solve the task." in the committed fixture, while the current converter emits the source: user task description there — the committed file still carries an agent acknowledgement step the converter no longer produces. So the four fixtures need a decision (regenerate, or treat the converter change as the regression) before a CI gate can be green.

   Criterion 4 — documenting refresh: not yet. --update exists but is not mentioned in CONTRIBUTING.md, AGENTS.md or README.md, so the recovery path is only discoverable by reading the script.

   Criterion 3 — optional-dependency skips: the only skip path is a missing sample_std.json; a converter that cannot import an optional dependency surfaces as error ... converter failed with the stderr attached. That may already be clear enough, or you may want it classified as a skip — that seemed like your call rather than something to assume.

   One environment note in case it bites someone else: on a non-UTF-8 Windows locale the converters fail with UnicodeEncodeError: 'gbk' codec and the run reports ~30 spurious errors. Setting PYTHONUTF8=1 makes it clean. Pinning the child environment's encoding inside the script would make results machine-independent.

   Happy to open a PR for the CI wiring plus a short CONTRIBUTING section on --update, if you would like it split that way. I have deliberately not touched the four drifted fixtures, since whether the current converter output is the intended result is a judgement about the data rather than the tooling.