Skip to content

Keep empty SQL fields out of numeric benchmark scores - #1102

Merged
WaylandYang merged 2 commits into
deeplethe:devfrom
virtue039:fix/bench-empty-sql-fields
Oct 8, 2026
Merged

WaylandYang merged 2 commits into
deeplethe:devfrom
virtue039:fix/bench-empty-sql-fields

Conversation

@virtue039

@virtue039 virtue039 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Why

The answer benchmark searches every numeric field in the first SQL result row. With the default psql output, SELECT 7, NULL::numeric prints 7|. firstRow() converts the blank field with Number(""), producing [7, 0] and falsely matching a zero-valued gold answer. Empty strings and whitespace-only fields have the same problem. value() also converts a blank first column to zero when another column keeps the row nonempty.

What changes

  • Both readers use the same blank-field guard. firstRow() excludes blank fields from numeric candidates; value() returns { empty: true } when the first column is blank.
  • Preserve genuine zero, negative and decimal values, numeric candidates from all columns in the first nonblank result row, and the scalar reader's first-column contract.
  • Add dependency-free parser/scoring regressions and an opt-in real-PostgreSQL test, document their commands, and run both in a dedicated Node 20 / PostgreSQL 16 workflow with read-only repository permissions.

Comparison tolerances and the existing handling of blank rows are unchanged. This does not distinguish SQL NULL from an empty string, since psql's default output represents both as an empty field.

How it was checked

  • Before the fix, 7 of the 15 parser regressions failed; after the fix all 15 pass.
  • Node 20.19.5 with a dedicated temporary PostgreSQL 16.14 server and psql 16.15: all 16 tests passed, with no skips.
  • Against that real server, gold SQL SELECT 0 yields zero. Applying the answer benchmark's numeric matching predicate to SELECT 7, NULL::numeric, SELECT 7, ''::text, SELECT 7, ' '::text, and SELECT NULL::numeric, 7 changed from true to false. SELECT 7, 0 still matches. For the scalar reader, SELECT NULL::numeric, 7 changed from { n: 0 } to { empty: true }.
  • All 24 benchmark .mjs files passed syntax checks. git diff --check and the workflow YAML check passed.

These are parser and scoring-predicate checks against real PostgreSQL, not a complete model-backed benchmark run. The temporary server was stopped after validation.

Before review

  • Every commit is signed off (git commit -s).
  • Run cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, and cargo test --workspace: Rust/cargo are unavailable in the local environment.
  • Frontend frozen-lockfile installation, pnpm build (including type checks), and pnpm test passed: 229 tests across 26 files, using Node 20.19.5 and the repository-pinned pnpm 10.2.1.
  • The new benchmark workflow and repository-wide checks pass on GitHub.

virtue039 and others added 2 commits October 8, 2026 16:14
Signed-off-by: Virtue039 <virtue039@qq.com>
…ow of their own with a Postgres service

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
WaylandYang
WaylandYang previously approved these changes Oct 8, 2026

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, this is a real scoring bug and the fix is the right size.

Checked locally at 688a5bf:

  • lib.test.mjs: 15 of 15 pass with the fix, and 7 of the 15 fail on dev's lib.mjs.
  • lib.pg.test.mjs passes against a real PostgreSQL 16.

One change, pushed to your branch (f70c16b): the regressions now run as a step of the existing web job, and the separate Bench parsers workflow with its Postgres service is gone. The dependency-free tests cover the parser; the real-psql test stays opt-in, as documented in the README.

Merging when CI is green. Thanks again.

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving again on f70c16b: the first approval was dismissed when the push was processed.

@WaylandYang
WaylandYang merged commit 4c8836f into deeplethe:dev Oct 8, 2026
6 checks passed
@WaylandYang WaylandYang mentioned this pull request Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants