Skip to content

ci(benchmark): add syncpack, Bun and Yarn repositories, and interactive pickers - #189

Merged
donfear merged 3 commits into
mainfrom
ci/benchmark-more-tools
Oct 21, 2026
Merged

donfear merged 3 commits into
mainfrom
ci/benchmark-more-tools

Conversation

@donfear

@donfear donfear commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Extends the Benchmark workflow with more tools, two bigger repositories, and timings for the interactive pickers, yarn upgrade-interactive included.

What's new

Tools

  • syncpack (syncpack update --check; 1.5M weekly downloads; a native Rust binary) runs on every repository.
  • bun outdated --recursive runs on the new Bun repository.

Repositories

  • vendurehq/vendure: Bun workspace, 54 package.json files, 323 dependencies.
  • backstage/backstage: Yarn 4 workspace, 269 package.json files, 591 dependencies. Now the largest.

Picker benchmark. Each tool that has an interactive list of updates opens it in a 160×50 terminal (scripts/bench-pty.py), and is timed until every update is loaded:

Picker Runs on
inup every repository
yarn upgrade-interactive Yarn
pnpm update -i --latest pnpm
bun update -i Bun
taze -I every repository
syncpack update -i every repository
npm-check -u single package only
ncu -i single package only (with --workspaces it shows one package.json at a time)

npm has no picker.

"Loaded" means the last thing the picker draws or writes to its cache before 5 s of quiet. That counts rows fetched off screen: yarn upgrade-interactive stops drawing after about 2.5 s on backstage and keeps fetching for several seconds more. Pickers run in rounds, one run per tool per round. A picker that exits, draws nothing, or never goes quiet fails the job.

README and docs. The README section gets "Listing outdated dependencies" and "Opening the picker" tables, each with a first run and a repeat run. The docs page explains both measurements and lists every command.

Left out: deno outdated lists only about 11 of the roughly 600 updates the other tools find on these npm/pnpm repositories, so its numbers wouldn't be comparable.

Fixes

  • Sandbox: it now also redirects npm_config_cache, BUN_INSTALL, BUN_INSTALL_CACHE_DIR and YARN_GLOBAL_FOLDER. A shell that sets BUN_INSTALL made Bun's "first run" read a warm cache: 0.05 s instead of 7.7 s.
  • Picker environment: pickers run without the CI variables, because inup and other tools refuse to open a picker in CI.
  • Clean git status: whatever the install leaves behind is hidden from git status, because inup's picker asks before working on a dirty tree.
  • Package managers: detected from packageManager, else the lockfile. pnpm and Bun run as their native binaries; Yarn runs as the release the repository commits.

Preview from this PR's check (3 runs per tool, GitHub Actions; run)

The published numbers will come from a 10-run dispatch.

Listing, first run express vuejs/core TanStack/query vendure backstage
inup 0.40 s 1.07 s 2.09 s 3.16 s 4.22 s
npm-check-updates 0.67 s 4.31 s 11.1 s 3.28 s 13.7 s
taze 0.49 s 2.63 s 9.05 s 4.79 s 12.5 s
syncpack 0.30 s 1.08 s 2.35 s 3.72 s 3.96 s
npm-check 1.06 s — — — —
npm / pnpm / bun outdated 0.80 s 1.52 s 2.36 s 1.72 s —
Picker, first run express vuejs/core TanStack/query vendure backstage
inup 0.46 s 1.19 s 2.33 s 3.63 s 4.88 s
npm-check-updates 0.68 s — — — —
taze 0.47 s 1.82 s 8.39 s 4.46 s 11.9 s
syncpack 0.29 s 1.04 s 2.40 s 3.37 s 4.08 s
npm-check 1.09 s — — — —
pnpm / bun update -i / yarn upgrade-interactive — 1.51 s 2.35 s 1.66 s 9.53 s

What the preview shows:

  • inup vs yarn upgrade-interactive on backstage: inup's picker wins first runs (4.88 s vs 9.53 s) and repeat runs (1.66 s vs 3.38 s).
  • syncpack: on GitHub's runners it's the fastest on express and backstage, and within a few hundredths of inup on vuejs/core. It wins most repeat runs from its 30-minute cache.
  • Bun: fastest on vendure, both listing and picker.
  • Repeat runs: taze and Bun also win some, from caches that skip the registry. inup revalidates every package on every run.

The docs page says all of this.

- syncpack (`update --check`) runs on every repository.
- vendurehq/vendure (Bun workspace, 54 package.json files) adds
  `bun outdated --recursive`.
- backstage/backstage (Yarn 4 workspace, 269 package.json files) is the
  largest repository. Yarn 2+ lists updates only in its interactive picker,
  so it has no package-manager row here.
- The package manager comes from `packageManager`, else the lockfile. pnpm
  and Bun run as the native binaries their npm packages ship; Yarn runs as
  the release the repository commits.
- The sandbox also redirects npm's, Bun's and Yarn's own cache variables:
  a shell that sets BUN_INSTALL made Bun's "first run" read a warm cache.
- Rows shared by several package managers read "npm / pnpm / bun outdated";
  the chart is more compact to fit five repositories.
Each tool that has an interactive list of updates opens it in a 160x50
terminal (scripts/bench-pty.py) and is timed until every update is loaded:
the last thing it draws or writes to its cache before 5 s of quiet. That
counts the rows a picker fetches off screen, as yarn upgrade-interactive
does for seconds after its screen settles.

Pickers: inup, yarn upgrade-interactive, pnpm update -i --latest,
bun update -i, taze -I, syncpack update -i, npm-check -u and ncu -i. ncu
shows one package.json at a time with --workspaces, so like npm-check it
only runs on the single-package repository; npm has no picker. They run in
rounds, every tool once per round, with first and repeat runs as for
listing, outside CI (where pickers refuse to open) and with what installing
left behind hidden from git status (inup asks before working on a dirty
tree).

The README section and docs page gain picker tables next to the listing
ones.
"pnpm update -i", "bun update -i" and "yarn upgrade-interactive" read as
"pnpm / bun update -i / yarn upgrade-interactive".
@donfear
donfear marked this pull request as ready for review October 21, 2026 14:54
@donfear
donfear merged commit 4103ffa into main Oct 21, 2026
39 checks passed
@donfear
donfear deleted the ci/benchmark-more-tools branch October 21, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant