Skip to content

Add isolated benchmark coverage experiments and blind review workflow - #51

Draft
CompleteDotTech wants to merge 2 commits into
mainfrom
experiment/benchmark-gates-v2
Draft

CompleteDotTech wants to merge 2 commits into
mainfrom
experiment/benchmark-gates-v2

Conversation

@CompleteDotTech

@CompleteDotTech CompleteDotTech commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Abstentions in the Kaggle benchmark made coverage difficult to interpret. This adds isolated diagnostic tools that distinguish wording changes, gate changes, uncertain primary scores and unreadable or missing context, while preserving the existing production behavior.

The tools provide frozen paired experiments, blind two-reviewer packets, canonical holdout deduplication, separate positive/negative cutoff sweeps, targeted second-pass text/image trials, and larger matching controls by comparison mode. Source labels remain explicitly identified as proxies. Human-reviewed metrics and candidate selection remain pending; no heldout predictions or enforcement changes are made.

Runs bind inputs, assets, native runtime, pricing and tooling hashes, persist each paid attempt, refuse uncertain replay, and retain exact-hash provenance with no provider usage. A narrow recovery archives local signal-limit failures and proves they had no external attempts before requeuing them. Completed paid text results are retained.

Validation: 367 application tests passed; 10 offline benchmark tests passed; 5 cost arithmetic tests and 6 label-audit tests passed. CI runs the new offline coverage tests without provider calls or dataset downloads. Live diagnostic runs and spending receipts are stored outside the repository on G:, with independent annotations still pending.

Completed coverage diagnostics: 477 cases / 447 provider calls, estimated $0.078493, with reconciled accounting and no provider errors. In the conservative three-way pipeline, text coverage rose 65% to 73% but proxy false flags rose 1 to 5; image coverage rose 71% to 80%, with unchanged proxy false flags and three additional source-positive negatives. Matching at 0.7 detected 79/90 controlled copies with 0/300 different-source proxy flags. These are diagnostic results, not independent policy accuracy or production threshold approval. All four hosted CI checks passed.

@CompleteDotTech CompleteDotTech changed the title Add isolated gate experiments and blind benchmark review Add isolated benchmark coverage experiments and blind review workflow Oct 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant