Repository navigation
Add isolated benchmark coverage experiments and blind review workflow - #51
Draft
CompleteDotTech wants to merge 2 commits into
Draft
CompleteDotTech wants to merge 2 commits into
CompleteDotTech wants to merge 2 commits into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Abstentions in the Kaggle benchmark made coverage difficult to interpret. This adds isolated diagnostic tools that distinguish wording changes, gate changes, uncertain primary scores and unreadable or missing context, while preserving the existing production behavior.
The tools provide frozen paired experiments, blind two-reviewer packets, canonical holdout deduplication, separate positive/negative cutoff sweeps, targeted second-pass text/image trials, and larger matching controls by comparison mode. Source labels remain explicitly identified as proxies. Human-reviewed metrics and candidate selection remain pending; no heldout predictions or enforcement changes are made.
Runs bind inputs, assets, native runtime, pricing and tooling hashes, persist each paid attempt, refuse uncertain replay, and retain exact-hash provenance with no provider usage. A narrow recovery archives local signal-limit failures and proves they had no external attempts before requeuing them. Completed paid text results are retained.
Validation: 367 application tests passed; 10 offline benchmark tests passed; 5 cost arithmetic tests and 6 label-audit tests passed. CI runs the new offline coverage tests without provider calls or dataset downloads. Live diagnostic runs and spending receipts are stored outside the repository on G:, with independent annotations still pending.
Completed coverage diagnostics: 477 cases / 447 provider calls, estimated $0.078493, with reconciled accounting and no provider errors. In the conservative three-way pipeline, text coverage rose 65% to 73% but proxy false flags rose 1 to 5; image coverage rose 71% to 80%, with unchanged proxy false flags and three additional source-positive negatives. Matching at 0.7 detected 79/90 controlled copies with 0/300 different-source proxy flags. These are diagnostic results, not independent policy accuracy or production threshold approval. All four hosted CI checks passed.