Repository navigation
Reject unequal parse counts in the CASPER metric helper - #269
Open
sylvesterkaczmarek wants to merge 2 commits into
Open
sylvesterkaczmarek wants to merge 2 commits into
sylvesterkaczmarek wants to merge 2 commits into
Conversation
Author
|
Any feedback on the CASPER parse-count validation? The workflow is still waiting for maintainer approval. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #268.
Correction
Reject unequal target/prediction counts before
top_metricsevaluates any pairs. Ordinaryzippreviously truncated to the shorter list, so a direct caller could receive perfect metrics for only a prefix of its targets. The error identifies both counts and follows the wording already used by the MTOP command.The CLI already validates input alignment; this change protects direct calls to the metric helper. Equal-length inputs, metric calculations, empty matching lists and both invalid-target policies retain their behavior.
Verification
All 17 new cases and the 37-test CASPER metric/utility/augmentation selection pass locally on macOS/Python 3.11 and in exact-commit hosted validation on Ubuntu/Python 3.10 and 3.11. The jobs test
e2f02c1a305d102668c73fb7c08b32904bc3c63band use TensorFlow 2.15.1 for the existing CLI.Tests cover shorter/longer/empty prediction lists under both invalid-target policies, exact existing metric values, formatted parses, tuple/list inputs and preserved validation errors. Two subprocess tests exercise the existing CLI with real temporary TSV files, confirming its successful aligned case and existing mismatch rejection.
Each hosted job reproduces eight failures with original production code and nine passing controls, passes all 20 pre-existing selected tests, restores the submitted bytes and passes the new cases again. The complete 37-test selection also passes from a separate exported checkout with module imports verified. New-test lint, source correctness lint, dependency, compilation and patch checks pass. Logs, JUnit reports, dependency versions and coverage are retained as workflow artifacts. Validation configuration remains on a separate fork branch.
Scope
This is a focused helper validation change with a new test module. The full Language repository, model training and external MTOP datasets were not tested. No production dependency, upstream workflow, metric formula or existing test expectation changes are included.