Repository navigation
Keep empty SQL fields out of numeric benchmark scores - #1102
Merged
Merged
Conversation
Signed-off-by: Virtue039 <virtue039@qq.com>
…ow of their own with a Postgres service Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
WaylandYang
previously approved these changes
Oct 8, 2026
WaylandYang
left a comment
Contributor
There was a problem hiding this comment.
Thank you, this is a real scoring bug and the fix is the right size.
Checked locally at 688a5bf:
lib.test.mjs: 15 of 15 pass with the fix, and 7 of the 15 fail ondev'slib.mjs.lib.pg.test.mjspasses against a real PostgreSQL 16.
One change, pushed to your branch (f70c16b): the regressions now run as a step of the existing web job, and the separate Bench parsers workflow with its Postgres service is gone. The dependency-free tests cover the parser; the real-psql test stays opt-in, as documented in the README.
Merging when CI is green. Thanks again.
WaylandYang
approved these changes
Oct 8, 2026
WaylandYang
left a comment
Contributor
There was a problem hiding this comment.
Approving again on f70c16b: the first approval was dismissed when the push was processed.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The answer benchmark searches every numeric field in the first SQL result row. With the default psql output,
SELECT 7, NULL::numericprints7|.firstRow()converts the blank field withNumber(""), producing[7, 0]and falsely matching a zero-valued gold answer. Empty strings and whitespace-only fields have the same problem.value()also converts a blank first column to zero when another column keeps the row nonempty.What changes
firstRow()excludes blank fields from numeric candidates;value()returns{ empty: true }when the first column is blank.Comparison tolerances and the existing handling of blank rows are unchanged. This does not distinguish SQL NULL from an empty string, since psql's default output represents both as an empty field.
How it was checked
SELECT 0yields zero. Applying the answer benchmark's numeric matching predicate toSELECT 7, NULL::numeric,SELECT 7, ''::text,SELECT 7, ' '::text, andSELECT NULL::numeric, 7changed from true to false.SELECT 7, 0still matches. For the scalar reader,SELECT NULL::numeric, 7changed from{ n: 0 }to{ empty: true }..mjsfiles passed syntax checks.git diff --checkand the workflow YAML check passed.These are parser and scoring-predicate checks against real PostgreSQL, not a complete model-backed benchmark run. The temporary server was stopped after validation.
Before review
git commit -s).cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings, andcargo test --workspace: Rust/cargo are unavailable in the local environment.pnpm build(including type checks), andpnpm testpassed: 229 tests across 26 files, using Node 20.19.5 and the repository-pinned pnpm 10.2.1.