Skip to content

Add benchmarking suite for parallel.sh performance - #2

Merged
inxilpro merged 1 commit into
mainfrom
claude/readme-build-graph-benchmarks-8u9r2d
Aug 13, 2026
Merged

inxilpro merged 1 commit into
mainfrom
claude/readme-build-graph-benchmarks-8u9r2d

Conversation

@inxilpro

Copy link
Copy Markdown
Contributor

This PR adds a comprehensive benchmarking suite to measure the performance gains of using parallel.sh chains versus sequential script execution.

Summary

A new bench/bench.sh script has been added that benchmarks four realistic build scenarios, comparing execution time when steps run sequentially (traditional set -e script) versus in parallel chains. The benchmarks are integrated into the CI pipeline and documented in the README.

Key Changes

  • New bench/bench.sh — A POSIX shell benchmarking script that:

    • Defines four realistic build scenarios (PHP app with frontend, CI checks, failing lint, dominant step)
    • Runs each scenario twice: once sequentially and once with chains
    • Uses sleep commands to simulate build steps without competing for system resources
    • Supports configurable repetitions (REPS), time scaling (SCALE), and shell selection (SHELL_UNDER_TEST)
    • Automatically detects the best available timing mechanism (GNU date, Perl, Python, or fallback to seconds)
    • Reports results as a table and fails only if chains are not faster than sequential execution
  • Updated Makefile — Added bench target to run benchmarks (not part of default check target since it takes several minutes)

  • Updated .github/workflows/ci.yml — Added bench job that:

    • Runs benchmarks on every push
    • Publishes results to the GitHub Actions run summary
    • Only fails if a scenario was not faster in chains than in order (tolerates noise from hosted runners)
  • Updated README.md — Added documentation including:

    • Gantt diagrams showing parallel vs sequential execution timelines
    • Benchmark results table with speedup metrics
    • Explanation of each scenario and what it demonstrates
    • Instructions for running benchmarks locally with different configurations

Notable Implementation Details

  • The benchmark uses sleep instead of real work to isolate the parallelization benefit from CPU/IO contention
  • Results report the best of multiple runs rather than the mean, since slower runs only reflect machine noise
  • The library overhead is measured separately (eight empty chains) to contextualize results
  • Timing precision adapts to available tools, with millisecond precision preferred over whole seconds
  • Scenarios demonstrate different parallelization benefits: 1.6x–5.8x speedup depending on build shape

https://claude.ai/code/session_01Bn7kapsRo75cCUfU6nfNpS

The README said chains are faster without ever showing by how much. Two
gantt charts now draw the same build in order and in chains, one for a
build that passes and one for a build whose lint fails, and a benchmark
suite puts numbers under them.

bench/bench.sh describes four builds as the chains they are made of, with
every step a sleep as long as that step usually takes, and runs each one
twice: in declaration order, the way a set -e script runs it, and as
chains. Sleeps because the shape of a build is what changes here, not the
work in it, and the report says so. Four scenarios, because the gain
depends entirely on the shape: two independent chains, four independent
checks, a failure found early, and one step long enough that nothing can
be done about it.

The only assertion is that a scenario was faster in chains than in order.
The times are a report, and a hosted runner is far too noisy to hold a
particular number against a build. CI runs it on every push and puts the
table in the run summary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bn7kapsRo75cCUfU6nfNpS
@inxilpro
inxilpro merged commit e589507 into main Aug 13, 2026
22 checks passed
@inxilpro
inxilpro deleted the claude/readme-build-graph-benchmarks-8u9r2d branch August 13, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants