Skip to content

The benchmark-diff workflow has never run; test it end to end on a pull request #1131

Open

Description

@yakew7

Where: .github/workflows/benchmark-diff.yml, CONTRIBUTING.md (Dependency updates).

Current gap: the workflow added for lockfile bumps reruns the full benchmark on a pull request that carries the rerun-benchmark label and posts a comment with scripts/diff_results.py. It has only been validated as YAML. Nobody has run it, and several things can only be learned by running it: whether the refs/pull/N/merge checkout works for a Dependabot branch, whether the GITHUB_TOKEN can comment when the label is added by a maintainer on a Dependabot PR, how long the full benchmark takes on a hosted runner against the 120 minute limit, and whether the comment stays under the 65,536 character limit for the moved-rows section.

Suggested approach: run it on a throwaway pull request (a one-line change to requirements-lock.txt on a branch) with the label, and then through workflow_dispatch. Record the runtime, fix whatever breaks (permissions, comment length, the pip install -e step), and write what was observed in CONTRIBUTING.md. Consider a --limit on the moved-rows list so the comment cannot overflow, and a smoke job that runs it against one small audit on pull requests that touch the workflow.

Done when: a labelled throwaway PR gets the comment, the observed runtime and any limits are documented, and a change to the workflow file is exercised on one audit before it can merge.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions