Skip to content

perf: post codec and JDBC benchmark results on pull requests - #292

Merged
magnusp merged 2 commits into
mainfrom
perf/pr-benchmark-comment
Oct 31, 2026
Merged

magnusp merged 2 commits into
mainfrom
perf/pr-benchmark-comment

Conversation

@magnusp

@magnusp magnusp commented Oct 31, 2026

Copy link
Copy Markdown
Owner

What

Adds libsql-perf, a driver-side benchmark for the JSON codec and the JDBC result-set getters, and .github/workflows/perf.yml, which builds the base and PR merge commits, runs three interleaved rounds, and upserts a sticky comment with the comparison.

This is the harness that produced #291.

Why

#291 found DirectHranaJsonCodec allocating ~55% more than the Jackson baseline per scan and encoding ~19% slower. Those numbers came from a throwaway harness. Landing the harness gives every PR the same metrics, so regressions show up in review instead of in a later investigation.

Details

  • Workloads: codec decode 5k and 20k rows, encode one execute, JDBC scan with typed getters and with getObject, plus Jackson baselines for the codec rows.
  • No server: server and network time dominate on shared runners and would bury codec changes. End-to-end throughput stays with libsql-itests.
  • Base vs head on the same runner, three interleaved rounds, median per metric. Throughput drops and allocation increases over 10% are flagged; the check stays green so reviewers judge the numbers.
  • Sticky comment keyed by <!-- libsql-perf -->, updated on each push. The raw perf-results JSON is uploaded as an artifact.
  • maven.deploy.skip keeps the module out of publishing; it is in the reactor so it compiles and passes Spotless in normal CI.
  • For fork PRs the benchmark still runs; the comment step is skipped because the token is read-only.

First run

The base commit has no libsql-perf module yet, so this PR's own comment reports head-only numbers. Deltas start with the next PR.

Test plan

  • ./mvnw -pl libsql-perf -am -DskipTests package
  • java -cp 'libsql-perf/target/classes:libsql-perf/target/lib/*' io.github.magnusp.libsql.perf.PerfBenchmark --quick
  • ./mvnw spotless:check
  • Renderer exercised with base/head JSON fixtures, including the missing-base path; comment.sh syntax-checked with bash -n.

Refs #291.

Add a driver-side benchmark module and a workflow that posts its result as a
sticky comment on every pull request.

The benchmark measures the direct and Jackson codec decode/encode paths and
the JDBC result-set getters on fixed payloads. It starts no server, so server
and network time cannot bury codec changes in shared-runner noise.

The workflow measures the base commit and the PR merge commit, three
interleaved rounds each, and reports medians. Throughput drops and allocation
increases over 10% are flagged; the check stays green so reviewers judge the
numbers. The base side is skipped until the module exists on the base branch,
so this PR's first comment reports absolute head numbers.

Refs #291.
@magnusp magnusp added the P2 Normal priority: edge case, feature gap, or test/process improvement label Oct 31, 2026
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Performance benchmark

Head 93ea3b6fd6d3: absolute numbers only, because the benchmark is not present in the base commit. Deltas start with the next PR.

Driver-side microbenchmarks only; server and network time are excluded. Shared-runner noise is about ±10%, so only larger regressions are flagged.

JDK 25.0.5, Linux amd64, 4 CPUs.

Throughput

Workload Base Head Δ
Decode 5k rows (direct) — 1.04M rows/s n/a
Decode 5k rows (Jackson baseline) — 935.6k rows/s n/a
Decode 20k rows (direct) — 1.02M rows/s n/a
Encode one execute (direct) — 984.0k req/s n/a
Encode one execute (Jackson baseline) — 930.9k req/s n/a
JDBC scan + getters (20k rows) — 25.16M rows/s n/a
JDBC scan + getObject (20k rows) — 13.01M rows/s n/a

Allocation

Workload Base Head Δ
Decode 5k rows (direct) — 8.18 MB/op n/a
Decode 5k rows (Jackson baseline) — 5.22 MB/op n/a
Decode 20k rows (direct) — 32.66 MB/op n/a
Encode one execute (direct) — 1.38 kB/op n/a
Encode one execute (Jackson baseline) — 1.84 kB/op n/a
JDBC scan + getters (20k rows) — 1.04 kB/op n/a
JDBC scan + getObject (20k rows) — 1.12 MB/op n/a

Workloads and thresholds live in tools/perf/render-comment.py. Run locally with ./mvnw -pl libsql-perf -am -DskipTests package and see libsql-perf/README.md.

@magnusp
magnusp merged commit 2769e69 into main Oct 31, 2026
7 checks passed
@magnusp
magnusp deleted the perf/pr-benchmark-comment branch October 31, 2026 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority: edge case, feature gap, or test/process improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant