Skip to content

Reject unequal parse counts in the CASPER metric helper - #269

Open
sylvesterkaczmarek wants to merge 2 commits into
google-research:masterfrom
sylvesterkaczmarek:fix/casper-metric-input-alignment-20260928
Open

sylvesterkaczmarek wants to merge 2 commits into
google-research:masterfrom
sylvesterkaczmarek:fix/casper-metric-input-alignment-20260928

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Fixes #268.

Correction

Reject unequal target/prediction counts before top_metrics evaluates any pairs. Ordinary zip previously truncated to the shorter list, so a direct caller could receive perfect metrics for only a prefix of its targets. The error identifies both counts and follows the wording already used by the MTOP command.

The CLI already validates input alignment; this change protects direct calls to the metric helper. Equal-length inputs, metric calculations, empty matching lists and both invalid-target policies retain their behavior.

Verification

All 17 new cases and the 37-test CASPER metric/utility/augmentation selection pass locally on macOS/Python 3.11 and in exact-commit hosted validation on Ubuntu/Python 3.10 and 3.11. The jobs test e2f02c1a305d102668c73fb7c08b32904bc3c63b and use TensorFlow 2.15.1 for the existing CLI.

Tests cover shorter/longer/empty prediction lists under both invalid-target policies, exact existing metric values, formatted parses, tuple/list inputs and preserved validation errors. Two subprocess tests exercise the existing CLI with real temporary TSV files, confirming its successful aligned case and existing mismatch rejection.

Each hosted job reproduces eight failures with original production code and nine passing controls, passes all 20 pre-existing selected tests, restores the submitted bytes and passes the new cases again. The complete 37-test selection also passes from a separate exported checkout with module imports verified. New-test lint, source correctness lint, dependency, compilation and patch checks pass. Logs, JUnit reports, dependency versions and coverage are retained as workflow artifacts. Validation configuration remains on a separate fork branch.

Scope

This is a focused helper validation change with a new test module. The full Language repository, model training and external MTOP datasets were not tested. No production dependency, upstream workflow, metric formula or existing test expectation changes are included.

@sylvesterkaczmarek

Copy link
Copy Markdown
Author

Any feedback on the CASPER parse-count validation? The workflow is still waiting for maintainer approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CASPER TOP metric helper silently truncates unequal parse lists

1 participant