feat(inference): expand lm-eval accuracy configuration - #385
Conversation
3c70617 to
d5af569
Compare
solaiys
left a comment
solaiys
left a comment
There was a problem hiding this comment.
Review: blocking issue
lm-eval 0.4.12's --tasks is a single comma-separated string (type=str, metavar task1,task2). Splatting task_names() turns tasks: ["hellaswag", "gsm8k"] or task: "hellaswag,gsm8k" into --tasks hellaswag gsm8k, which argparse treats as an unknown positional. Pass ",".join(task.task_names()) as one argument (the historical task.task form). The new unit test currently asserts the broken CLI shape.
hnimra-amd
left a comment
hnimra-amd
left a comment
There was a problem hiding this comment.
Not re-filing the --tasks splat. For lm-eval 0.4.12 that matches nargs="+" / SplitArgs, and HarnessCLI inserts run when the first token is --model.
Two schema-default notes below. The device one is the one I would treat as should-fix; num_fewshot is a scoring-contract check.
| num_fewshot: Optional[int] = None | ||
| batch_size: str = "1" | ||
| max_batch_size: Optional[int] = None | ||
| device: Optional[str] = "cuda:0" |
There was a problem hiding this comment.
This default always forwards --device cuda:0 (via _optional_arg when the value is not None). In 0.4.12 --device defaults to unset.
CVS still runs lm_eval as an API client (local-completions / local-chat-completions) on the same node as the server. Pinning that client to GPU 0 is not required for those backends and can contend with the serving job.
Default device to None and omit the flag unless the config sets it.
amd-droy
left a comment
amd-droy
left a comment
There was a problem hiding this comment.
looks good to me. thanks Atul.
Pin the harness contract and expose its supported evaluation controls while retaining existing Atom task configuration.
Keep documented model-argument defaults idempotent while rejecting values that override CVS-owned transport settings.
Keep boolean chat-template settings and upstream seed forms from changing accuracy task semantics.
4faf692 to
b32af43
Compare
4 participants
Summary
Test plan
Made with Cursor