Umbrella issue for a round of updates to the llms-benchmark: its results tables in .llms-benchmark/README.md and the /launch-llms-benchmark command that produces them.
Planned so far:
- an Effort column right after Model in both results tables; every existing row was measured at
medium;
/launch-llms-benchmark takes the effort as an optional third argument (medium by default), passed to each CLI's effort flag; model, effort and CLI together identify a row.
More changes to the benchmark will be added here as the work proceeds; the PR that closes this issue lists them all.
Umbrella issue for a round of updates to the llms-benchmark: its results tables in
.llms-benchmark/README.mdand the/launch-llms-benchmarkcommand that produces them.Planned so far:
medium;/launch-llms-benchmarktakes the effort as an optional third argument (mediumby default), passed to each CLI's effort flag; model, effort and CLI together identify a row.More changes to the benchmark will be added here as the work proceeds; the PR that closes this issue lists them all.