Skip to content

feat(bench): update the llms-benchmark #650

Description

@using-system

Umbrella issue for a round of updates to the llms-benchmark: its results tables in .llms-benchmark/README.md and the /launch-llms-benchmark command that produces them.

Planned so far:

  • an Effort column right after Model in both results tables; every existing row was measured at medium;
  • /launch-llms-benchmark takes the effort as an optional third argument (medium by default), passed to each CLI's effort flag; model, effort and CLI together identify a row.

More changes to the benchmark will be added here as the work proceeds; the PR that closes this issue lists them all.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestlocalLocal otel-lgtm stackpriority: mediumReal friction, workaround exists

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions