Add benchmarking suite for parallel.sh performance

.github/workflows/ci.yml

@@ -72,6 +72,28 @@ jobs:
       - run: make install-dev
       - run: make test SHELLSPEC_FLAGS="--shell ${{ matrix.shell }}"
 
+  # The benchmarks are a report, and a hosted runner is a noisy place to
+  # take timings on, so the job only fails when a scenario was not faster in
+  # chains than in order at all. That is a claim no amount of a busy runner
+  # should be able to break, and the table lands in the run summary either
+  # way.
+  bench:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v7
+      - name: Benchmark
+        run: make bench 2>&1 | tee bench.txt
+      - name: Publish the table
+        if: always()
+        run: |
+          [ -f bench.txt ] || exit 0
+          {
+            echo '```'
+            cat bench.txt
+            echo '```'
+          } >>"$GITHUB_STEP_SUMMARY"
+
   freebsd:
     runs-on: ubuntu-latest
     timeout-minutes: 30

Makefile

@@ -7,15 +7,16 @@ SHELLSPEC = .tools/bin/shellspec
 # Extra flags for the suite, e.g. `make test SHELLSPEC_FLAGS="--shell dash"`.
 SHELLSPEC_FLAGS =
 
-# The library and the example: POSIX sh, formatted, no bashisms.
-SCRIPTS = parallel.sh examples/build.sh
+# The library, the example and the benchmarks: POSIX sh, formatted, no
+# bashisms.
+SCRIPTS = parallel.sh examples/build.sh bench/bench.sh
 
 # The spec files are POSIX sh as well and get checked, but not formatted:
 # shellspec's Describe, It and End are ordinary commands, so shfmt sees no
 # nesting to keep and flattens the whole file to one column.
 SPECS = spec/parallel_spec.sh spec/spec_helper.sh
 
-.PHONY: check lint test install-dev
+.PHONY: check lint test bench install-dev
 
 check: lint test
 
@@ -28,6 +29,11 @@ test:
 	@[ -x $(SHELLSPEC) ] || { echo "$(SHELLSPEC) is missing: make install-dev" >&2; exit 1; }
 	$(SHELLSPEC) $(SHELLSPEC_FLAGS)
 
+# Not part of `check`: it is minutes of sleeping, and it reports rather than
+# asserts. Takes REPS, SCALE and SHELL_UNDER_TEST from the environment.
+bench:
+	./bench/bench.sh
+
 # Into .tools, which is gitignored.
 install-dev:
 	curl -fsSL https://raw.githubusercontent.com/shellspec/shellspec/$(SHELLSPEC_VERSION)/install.sh \

README.md

@@ -30,6 +30,45 @@ chain "npm ci --audit false" "npm run build"
 run
 ```
 
+Drawn out, with the times off a middling Laravel app. The second one
+finishes when its longest chain does rather than when its last command
+does:
+
+```mermaid
+gantt
+    title The build above, in order and in chains
+    dateFormat HH:mm:ss
+    axisFormat %M:%S
+    todayMarker off
+    section In order
+    composer install :done, 00:00:00, 00:00:40
+    npm ci           :done, 00:00:40, 00:01:30
+    npm run build    :done, 00:01:30, 00:01:50
+    section In chains
+    composer install :active, 00:00:00, 00:00:40
+    npm ci           :active, 00:00:00, 00:00:50
+    npm run build    :active, 00:00:50, 00:01:10
+```
+
+A failure arrives sooner for the same reason. The step that fails is not
+waiting on the steps that would have run before it, so the build stops
+about when the failure happens instead of once everything ahead of it is
+done:
+
+```mermaid
+gantt
+    title A lint that fails, at the end of a script and in a chain
+    dateFormat HH:mm:ss
+    axisFormat %M:%S
+    todayMarker off
+    section In order
+    phpunit            :done, 00:00:00, 00:00:50
+    lint, exits 1      :crit, 00:00:50, 00:01:00
+    section In chains
+    phpunit, cancelled :active, 00:00:00, 00:00:10
+    lint, exits 1      :crit, 00:00:00, 00:00:10
+```
+
 When something fails, the failure is at the bottom and the rest is grouped
 above it:
 
@@ -92,6 +131,58 @@ a different interval and `POLL_WHOLE` for a different fallback:
 POLL=0.5 . ./parallel.sh
 ```
 
+## Benchmarks
+
+`bench/bench.sh` runs four builds twice each, once in declaration order the
+way a `set -e` script runs them and once as chains. Every step in them is a
+`sleep`, at a tenth of the length the step it stands for would take, so the
+suite is minutes rather than hours. On a four-core Linux box, under dash,
+best of three runs:
+
+| Scenario | In order | In chains | Speedup |
+| --- | ---: | ---: | ---: |
+| PHP app with a front end | 11.01s | 7.09s | 1.6x |
+| CI checks, four of them | 12.01s | 5.03s | 2.4x |
+| A failing lint | 6.01s | 1.03s | 5.8x |
+| One dominant step | 10.01s | 6.04s | 1.7x |
+
+- **PHP app with a front end** — `composer install` (4s) in one chain,
+  `npm ci` (5s) then `npm run build` (2s) in the other. The build at the top
+  of this file.
+- **CI checks, four of them** — lint (1s), typecheck (3s), unit tests (5s)
+  and a build (3s), none of them waiting on any other. The shape chains are
+  best at: the job takes as long as its slowest check instead of as long as
+  all of them.
+- **A failing lint** — unit tests (5s) alongside a lint that exits 1 after
+  1s. In order the failure turns up last because that is where the step is,
+  and a lint at the top of the script would be found just as early. That is
+  the point: in chains it does not matter where it is.
+- **One dominant step** — `npm run build` (6s) alongside `composer install`
+  (3s) and `php artisan migrate` (1s). The ceiling on all of this: a build
+  cannot finish before its longest chain does, so the most chains can do is
+  hide the rest behind it.
+
+Eight chains that do nothing at all finish in 0.15s: one `mktemp`, eight
+forks, and a poll or two. That is about what the library costs a build with
+nothing to gain.
+
+The table is a report on shapes, not a measurement of any real build. A
+`sleep` waits without competing for a core, a disk or a link, so these are
+the times a build gets when its steps are mostly waiting on something other
+than each other. Steps that each saturate the machine will not see them.
+
+`make bench` runs it. `REPS` is how many runs each scenario gets, `SCALE`
+multiplies every duration, and `SHELL_UNDER_TEST` picks the shell:
+
+```sh
+REPS=1 SCALE=0.25 SHELL_UNDER_TEST=/bin/bash make bench
+```
+
+CI runs it on every push and puts the table in the run summary. It fails the
+job only when a scenario was not faster in chains than in order at all:
+anything tighter is a number to hold against a hosted runner, and they are
+too noisy to be held to one.
+
 ## Caveats
 
 `kill` stops a cancelled chain's shell, not its grandchildren. A command