Benchmark overhead distorts results in favor of slower databases

[kochelmonster]

The current benchmark includes a non-trivial amount of benchmark-side work in the measured interval. This can distort comparisons in favor of slower databases.

I profiled fillrandbatch with perf and attributed libc/syscall samples back to the first owning frame in the call stack.

Command shape:

perf record -F 997 -g --call-graph dwarf -- ./dbbench/bin/<backend> --benchmarks=fillrandbatch

Only benchmark-interval samples were considered. Startup, database open, and setup work were excluded.

Results

Database	Time in benchmark	Time in DB	Percentage of benchmark code	Measurement error
Leaves	~0.067 s	~0.316 s	17.50%	+21.2%
LMDB	~0.116 s	~0.687 s	14.50%	+16.9%
LevelDB	~0.047 s	~0.695 s	6.37%	+6.8%

Suggested Fix: Add a Null Backend Calibration Run

Add a null database backend that performs all benchmark-side work but does not actually store data. Before running the actual benchmarks, execute a calibration run with the null backend to measure the overhead introduced by the benchmark harness itself.

[hyc]

A calibration run might work, if the amount of overhead was always constant and independent of the backend, but your result shows that it varies with the backend.

A more pertinent question is, how much of the overhead can actually be eliminated? If the answer is "none of it" then the results are still valid because no real-world application will be able to do better than that. Ultimately, a few percent of bias toward slower databases is irrelevant because they're still the slower DBs. And many of these aren't directly comparable anyway because they aren't really implementing equivalent features. E.g., LevelDB has no ACID transactions. All that you can really expect is to get a high level view of relative performance.

[kochelmonster]

I wouldn't put too much stock in the absolute times. perf affects the measurement, and you should only use the relative figures. I would assume that a pure calibration run will yield pretty much the same results.