feat(vllm): adopt canonical workload configuration by atnair-amd · Pull Request #386 · ROCm/cvs

atnair-amd · 2026-09-03T22:34:06Z

Summary

Replace the vLLM workload schema with direct server and benchmark parameters plus canonical explicit run cells.
Move static environment to the container, preserve canonical cell identity in reports, and migrate the packaged workload catalog.

Test plan

make fmt-check
make lint
make ut
pytest --collect-only for single and distributed vLLM configs

Made with Cursor

solaiys

Two regressions from the schema move block merge: derived --max-model-len is no longer applied and none of the packaged configs set it, and the ROCm AITER exports were dropped from the serve env script without landing in container.env.

solaiys · 2026-09-08T18:41:33Z

@@ -246,11 +223,6 @@ def _server_argv(self, rank: int) -> list:
            "--port",
            str(self.port_no),
        ]


This used to emit a derived --max-model-len (ISL+OSL, range ratio, prefix pad) whenever serve_args did not pin one. That path is gone, and none of the 28 packaged catalog configs set server_params.max_model_len, so vllm serve now uses each model's full default context (often 128k).

Smoke still injects max_model_len=512, which shows the serve path needs a cap; the benchmark cells do not get one. That can OOM or change KV-cache behavior versus the previous suite.

Fix: restore the derivation when the option is omitted, or put an explicit max_model_len on every packaged server_params (and document it as required). Empty {} sweeps with ISL=1024/OSL=1024 are not a substitute.

solaiys · 2026-09-08T18:41:33Z

@@ -306,9 +280,6 @@ def build_server_cmd(self):
        env_lines = [


The previous job always exported VLLM_USE_AITER_UNIFIED_ATTENTION=1, VLLM_ROCM_USE_AITER_MHA=0, and VLLM_ROCM_USE_AITER_FUSED_MOE_A16W4=1 into the serve env script. This PR removes them and sets server_env = {}, but packaged container.env only has HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE (those already lived on roles.server.env).

The summary says static environment moves to the container; these AITER flags were static and did not move. Every ROCm vLLM cell will now run without them unless the image happens to set them.

Add the three exports to packaged container.env (or keep writing them here). container.env is already wired through ContainerOrchestrator.get_environment().

amd-droy · 2026-09-08T19:50:31Z

    "image": "<changeme>",
+    "env": {
+      "HF_HUB_OFFLINE": "1",
+      "TRANSFORMERS_OFFLINE": "1",


Are these 2 used anywhere? if not, can we drop them - "HF_HUB_OFFLINE": "1","TRANSFORMERS_OFFLINE": "1"?

keeping these in to force local usage since pulling is not tested

amd-droy · 2026-09-08T19:58:10Z

3+ hosts — fails before launch with: “vllm_distributed currently supports exactly two cluster hosts…” - Not for this PR/Release, but later we need to update it for n nodes.

Integrate the updated lm-eval 0.4.12 compatibility changes while preserving stack history.

hnimra-amd

A few contract/docs notes on the new option serializer and cell overrides. Packaged catalog runs still look fine.

hnimra-amd · 2026-09-09T18:06:54Z

-        # auto_map, which transformers refuses to load without trust-remote-code.
-        # Mirror the server's setting so the client can load the same tokenizer.
-        if self.serve_args.get("trust-remote-code") is True:
+        if self.variant.benchmark_params.ignore_eos:


Contract hole, not a packaged-catalog break: --ignore-eos and --trust-remote-code are read from self.variant.benchmark_params (the config defaults), while num_prompts / random_range_ratio / prefix come from the cell benchmark_params passed into VllmJob.

Sweep overrides of those two typed fields are merged in resolved_runs(), then dropped from benchmark_options because they are model fields, so they never reach the bench command. Current catalog JSON sets them on top-level benchmark_params, so those files still work.

If sweeps are meant to override all benchmark_params, please drive both flags from the same resolved map as num_prompts.

hnimra-amd · 2026-09-09T18:06:54Z

+| `ib_netdev` | *(distributed only)* socket interface name for `NCCL_SOCKET_IFNAME` / `GLOO_SOCKET_IFNAME` / `TP_SOCKET_IFNAME`. Must be **UP and hold a routable IPv4 reaching the other node** — check `ip -o -4 addr show`, not just `ip -o link show`. |

 `paths.models_dir` is `/models`, the in-container mount point — it is exported
 as `HF_HUB_CACHE`. When `model.id` is an absolute path under `/models`, vLLM


Leftover model.id here — this should be server_params.model to match the table above and the new schema.

hnimra-amd · 2026-09-09T18:06:54Z

-    Standalone (does not extend BaseVariantConfig) so it can be constructed
-    without the threshold_json field the base requires — absent from unit-test
-    fixtures. Production configs always supply it via substitute_config.
+_BENCHMARK_RESERVED = {


_BENCHMARK_RESERVED (including percentile_metrics / metric_percentiles) is enforced on sweeps only. The same names as extras on benchmark_params still serialize after the harness-owned flags in run_client, so you can get two --percentile-metrics on the bench command.

Low severity — packaged configs do not do this — but please reject those names on benchmark_params extras the same way as on sweeps, since CVS owns percentile reporting.

amd-droy · 2026-09-09T18:23:47Z

  "threshold_json": "mi325x_vllm_gpt-oss-120b_mxfp4_distributed_threshold.json",
+  "_comment_network": "HCA devices may be discovered automatically; ib_netdev selects the distributed socket interface.",
+  "ib_hca_devices": "auto",
+  "ib_netdev": "<changeme>",


this will be going inside container --> env, like this
"container": {
"lifetime": "per_run",
"name": "sglang_container",
"_image_example": "rocm/sgl-dev:v0.5.14-rocm720-mi30x-20260629",
"image": "",
"runtime": {
...
"devices": ["/dev/dri", "/dev/kfd", "/dev/infiniband/rdma_cm"],
"env": {
"NCCL_IB_HCA": " rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7",
"HCA_ID_PREFIX": "",
"NCCL_SOCKET_IFNAME": " eno0",
"GLOO_SOCKET_IFNAME": " eno0",
"NCCL_IB_GID_INDEX": "3",
"NCCL_DEBUG": "ERROR",
...
}
}
}

amd-droy · 2026-09-09T18:27:27Z

+    "env": {
+      "HF_HUB_OFFLINE": "1",
+      "TRANSFORMERS_OFFLINE": "1"
+    },


you can remove this if not used anywhere - "HF_HUB_OFFLINE": "1",
"TRANSFORMERS_OFFLINE": "1"

keeping as it forces offline path

Make server and benchmark settings explicit, use canonical selected run cells, and preserve their identity through execution and reporting.

Keep the full percentile surface harness-owned and remove obsolete server argument plumbing.

Retain per-cell context sizing while allowing explicit overrides and opt-outs. Move static ROCm AITER defaults into the packaged workload configurations.

Replace invalid universal defaults with target-image-supported model profiles.

amd-droy · 2026-09-09T18:40:35Z

+  ],
  "accuracy": {
    "tasks": []
  }


are the accuracy arguments hardcoded inside the libraries? Not needed for this release, but you can bring them here in a later PR.

Repeat list-valued flags, honor cell-level ignore-eos/trust-remote-code, reject reserved extras, drop empty accuracy stubs, and align the README.

Stop encoding unused list serve options as repeated flags. Restore one --flag followed by all values, matching vLLM nargs="+".

Pin packaged distributed configs to rdma0-rdma7 in the container environment and preserve it over generated discovery values.

Make distributed network settings container-owned while retaining legacy top-level fields as external-config fallbacks.

Use unambiguous placeholders and document how RNIC, frontend, and GID selections map to the container environment.

hnimra-amd

lgtm!

approved my Hamna

atnair-amd force-pushed the atnair/vllm-config-refactor branch from 8c7c03c to 8d586f0 Compare September 3, 2026 23:38

atnair-amd self-assigned this Sep 3, 2026

atnair-amd force-pushed the atnair/lm-eval-0-4-12 branch from 3c70617 to d5af569 Compare September 3, 2026 23:48

atnair-amd force-pushed the atnair/vllm-config-refactor branch from 8d586f0 to 29caaab Compare September 3, 2026 23:48

solaiys previously requested changes Sep 8, 2026

View reviewed changes

amd-droy reviewed Sep 8, 2026

View reviewed changes

atnair-amd added a commit that referenced this pull request Sep 8, 2026

Merge PR #385 into PR #386

6b5d913

Integrate the updated lm-eval 0.4.12 compatibility changes while preserving stack history.

atnair-amd requested review from amd-droy and solaiys September 9, 2026 17:38

hnimra-amd reviewed Sep 9, 2026

View reviewed changes

atnair-amd force-pushed the atnair/lm-eval-0-4-12 branch from 4faf692 to b32af43 Compare September 9, 2026 18:25

amd-droy reviewed Sep 9, 2026

View reviewed changes

Atul Nair and others added 4 commits September 9, 2026 11:29

feat(vllm): adopt canonical workload configuration

9ec9beb

Make server and benchmark settings explicit, use canonical selected run cells, and preserve their identity through execution and reporting.

fix(vllm): retain canonical benchmark metrics

c8976f7

Keep the full percentile surface harness-owned and remove obsolete server argument plumbing.

fix(vllm): restore workload runtime defaults

0698ece

Retain per-cell context sizing while allowing explicit overrides and opt-outs. Move static ROCm AITER defaults into the packaged workload configurations.

fix(vllm): scope AITER settings by model

b551428

Replace invalid universal defaults with target-image-supported model profiles.

atnair-amd force-pushed the atnair/vllm-config-refactor branch from b8064cb to b551428 Compare September 9, 2026 18:32

atnair-amd changed the base branch from atnair/lm-eval-0-4-12 to main September 9, 2026 18:32

amd-droy reviewed Sep 9, 2026

View reviewed changes

atnair-amd added 3 commits September 9, 2026 12:17

fix(vllm): apply review follow-ups for CLI lists and cell-map flags

1bd9ff6

Repeat list-valued flags, honor cell-level ignore-eos/trust-remote-code, reject reserved extras, drop empty accuracy stubs, and align the README.

fix(vllm): restore nargs=+ list CLI encoding

d5e0909

Stop encoding unused list serve options as repeated flags. Restore one --flag followed by all values, matching vLLM nargs="+".

fix(vllm): make RDMA device selection explicit

5ee7c36

Pin packaged distributed configs to rdma0-rdma7 in the container environment and preserve it over generated discovery values.

atnair-amd requested review from amd-droy and hnimra-amd September 9, 2026 21:39

atnair-amd added 2 commits September 9, 2026 14:52

fix(vllm): move socket selection into container env

ab83ec6

Make distributed network settings container-owned while retaining legacy top-level fields as external-config fallbacks.

fix(vllm): clarify distributed network placeholders

7239118

Use unambiguous placeholders and document how RNIC, frontend, and GID selections map to the container environment.

hnimra-amd approved these changes Sep 9, 2026

View reviewed changes

atnair-amd merged commit 0ff98eb into main Sep 9, 2026
2 checks passed

cijohnson deleted the atnair/vllm-config-refactor branch September 15, 2026 00:10

		@@ -306,9 +280,6 @@ def build_server_cmd(self):
		env_lines = [

Uh oh!

Conversation

atnair-amd commented Sep 3, 2026

Summary

Test plan

Uh oh!

solaiys left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amd-droy commented Sep 8, 2026

Uh oh!

hnimra-amd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hnimra-amd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants