Skip to content

Benchmark history never updates on release: benchmark-history.yml triggers on release: published, but releases are published with GITHUB_TOKEN #320

Description

@matt-edmondson

Plan reference

The design spec makes this commitment (https://github.com/ktsu-dev/Semantics/blob/7852eab73d673ab42ff1ad10cb1ad1124b8b0122/docs/superpowers/specs/2026-09-18-strings-paths-benchmarks-design.md#L9-L11):

Every release adds a point to docs/benchmarks/history.json, the chart is redrawn into docs/benchmarks/performance.svg, and the README shows it.

Plan Task 8 (https://github.com/ktsu-dev/Semantics/blob/7852eab73d673ab42ff1ad10cb1ad1124b8b0122/docs/superpowers/plans/2026-09-18-strings-paths-benchmarks.md#L1543-L1548) says: "The workflow itself is proven by the first real dispatch, which is out of scope here."

What exists today

  • .github/workflows/benchmark-history.yml triggers on release: types: [published] and on workflow_dispatch.
  • The Actions API reports zero runs ever for the Benchmark History workflow since it was created on 2026-09-16.
  • In that time at least 8 releases (v5.5.0 through v5.8.3) were published by github-actions[bot]. dotnet.yml passes GH_TOKEN: ${{ github.token }} to ktsubuild.
  • Events caused by GITHUB_TOKEN do not start new workflow runs; only workflow_dispatch and repository_dispatch are exempt. So the release trigger can never fire.

What's missing

The committed histories are frozen:

  • docs/benchmarks/history.json stops at 5.3.2.
  • strings-history.json and paths-history.json stop at 5.3.4, the local seed.

The current version is 5.8.3, so the README charts are stale.

Acceptance criteria

  • The release pipeline triggers the benchmark workflow using any one of these:
    • Have the release job dispatch it explicitly (gh workflow run benchmark-history.yml -f versions=<tag> or a repository_dispatch; GITHUB_TOKEN is allowed to trigger both).
    • Publish releases with a PAT or GitHub App token.
    • Trigger on workflow_run of the .NET workflow and read the new tag.
  • One run (dispatched or release-triggered) backfills 5.8.x points into all three histories.
  • The next release adds a point with no manual step.

Dependencies

None. Related but independent: #303 (SVG/history drift guard).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions