build: bump vllm to 0.28.0

[yfw]

What does this PR do ?

Bumps vLLM from 0.25.1 to 0.28.0, which moves the whole environment to torch 2.13.0 (torchvision 0.28.0, flashinfer 0.6.16.post3, nvidia-cutlass-dsl 4.6.2, tilelang 0.1.12, llguidance 1.7.x). Still a draft: the first nightly run on both SKUs is summarized below and the fixes it required are in, but a rebuild and re-run are needed to confirm them, and the PR is currently not mergeable against main (needs a rebase; main changed docker/Dockerfile in #4002).

Changes

Bullets are prefixed with the commit that made the change.

Dependencies (pyproject.toml / uv.lock)

• 717ee4a — vLLM 0.25.1 → 0.28.0 from PyPI (default cu130 wheels for both arches), torch 2.11.0 → 2.13.0, torchvision 0.26.0 → 0.28.0, flashinfer-python/-cubin/-jit-cache 0.6.13 → 0.6.16.post3, nvidia-cutlass-dsl[cu13] 4.5.2 → 4.6.2 in the vllm extra, tilelang <0.1.13, llguidance>=1.7.0,<1.8.0. flash-attn switches from the pinned cu13torch2.10 GitHub wheels to flash-attn==2.8.1 (sdist) because no 2.8.x wheel exists for torch ≥ 2.11.
• c759c71 — lockfile regenerated for the Docker uv version.
• 14fb0eb — NEMO_GYM_VLLM_VERSION 0.24.0 → 0.25.1 to match the Gym submodule's vllm==0.25.1 / flashinfer-python==0.6.13 pins. This also fixes the gym image on main, which has failed to build since feat(sc): gate-authoritative token capture via an external TransferQueue sink #3837 (Sept 5) with No solution found on flashinfer-python==0.6.12 vs 0.6.13.
• 5ae0cee — nvidia-cutlass-dsl-libs-base is no longer excluded. The exclusion worked around cutlass#3259 (the 4.5.x libs-base and libs-cu13 wheels overwrite 180 shared files, 99 with different content). 4.6 split the wheels cleanly: libs-base ships the entire Python cutlass DSL and libs-cu13 only the _cutlass_ir.cu13 extension, zero overlapping files (verified by unzipping 4.6.0 and 4.6.2). flashinfer 0.6.16's gdn_prefill.py imports cutlass.cute at module load and vLLM 0.28's qwen_gdn_linear_attn.py imports gdn_prefill eagerly, so with the exclusion in place every Qwen3.5 worker and every Blackwell vLLM worker died with ModuleNotFoundError: No module named 'cutlass.cute' (17 nightly tests plus an external 64-node super run). The mcore extra's own nvidia-cutlass-dsl==4.5.2 pin moves to 4.6.2 so the training venv does not inherit the 4.5.x overlapping pair (flashinfer 0.6.8.post1 accepts >=4.4.2). The only fork still on 4.5.0 is trtllm (via tensorrt-llm), which is unusable on torch 2.13 anyway — see Known issues.
• 5ae0cee — flash-attn source build bounded with MAX_JOBS=4 via [tool.uv.extra-build-variables]. flash-attn's setup.py sizes its parallelism from the host's free memory (not the cgroup) and cpu_count()//2, with --threads 4 per nvcc. Every nemo-ci build attempt on both architectures died with ResourceExhausted after dozens of Killed "$CICC_PATH/cicc" (6 attempts, 22–63 kills each). 4×4 built cleanly on aarch64 (2h47m) and x86 (~2h, alongside TE and flash-mla); the CI images below were produced with the same bound passed as a --build-arg.
• 5ae0cee — ModelOpt c3b913b9 → 613e5e8b. The old vLLM plugin registers a quant module on fused_moe.layer.FusedMoE, which 0.28 removed. ModelOpt swallows that AttributeError inside import_plugin (the trace shows Failed to import modelopt vllm plugin due to: AttributeError(... has no attribute 'FusedMoE')), leaving RowParallelLinear/ColumnParallelLinear/… registered but the module gone from sys.modules; NeMo-RL's explicit re-import in vllm_quant_patch.py re-runs the decorators and every fakequant nvfp4 test died with AssertionError: RowParallelLinear already registered!. 5dde396bd (Aug 4) ported the plugin to RoutedExperts; the pinned commit predates it by 137 commits.
• 5ae0cee — numpy override capped at <2.5 (resolves 2.4.6). vLLM 0.28 pins numba==0.65.0, whose metadata requires numpy<2.5; the numpy>=2.1.0 override forced 2.5.1 past it and numba refused to import in the vLLM venv ("Numba needs NumPy 2.4 or less"). Not fatal by itself — vLLM's optional-module probe catches it — but it left a broken numba in the venv and polluted every trace.
• Lock diff beyond marker churn: numpy 2.5.1 → 2.4.6, cutlass-dsl 4.5.2 dropped, libs-base 4.5.0/4.6.0/4.6.2 and libs-core 4.6.0/4.6.2 added, modelopt 0.46.0.dev86 → 0.47.0rc1.dev36. uv lock --check passes; tests/unit/test_dependency_pins.py passes.

vLLM source patches / worker (patches.py, vllm_worker.py)

• 717ee4a — _patch_vllm_shm_broadcast_bind_retry re-targeted: 0.28 binds the MessageQueue remote socket to port 0 directly (no probe/bind race any more) but ignores VLLM_PORT; the patch restores reserved-band selection with bind retries so engine sockets stay out of the ephemeral range.
• 170beee — mm_device_do_normalize defaults to False. vLLM 0.28 (#50411) calls the HF image processor with do_rescale=False, do_normalize=False and re-applies both on the GPU in the vision tower's dtype. The policy normalizes the same images on the CPU in fp32 through the same processor, and the nightly probability-error checks assume identical inputs on both sides. Verified on CPU with transformers 5.12.1: the fused path is exact in fp32 but differs by up to 0.019 (≈1.7 % of the pixel std) in bf16. Upstream has already shipped one silent-corruption fix for the device path (#55370, encoder cudagraphs). This is a parity measure, not a confirmed fix for the Qwen2.5-VL mismatch below. Opt back in with policy.generation.vllm_kwargs.mm_device_do_normalize=true.

fp8 generation (quantization/fp8.py)

• 8318c80 — make_fp8_moe_kernel lost its layer kwarg in 0.28 (0.25 forwarded it only to the FlashInfer TRTLLM experts). Both call sites updated; the unit test's expected kwargs follow.
• 8318c80 — process_weights_after_loading_kv re-mirrored on 0.28's BaseKVCacheMethod.process_weights_after_loading: the attention layer no longer has calculate_kv_scales (dynamic scales are a KV-cache dtype now, kv_cache_uses_per_token_head_scales), the static-scale branch keys off is_quantized_kv_cache, and the host _k_scale_cpu/_v_scale_cpu copies are refreshed on refit. Parameters are still kept so refit can update them.

Distributed (collectives.py)

• 03b9ee2 — all_gather_into_tensor → all_gather_single (torch 2.13 API).

Known issues not fixed here

Issue	Tests	Status / proposed fix
sglang-kernel ABI vs torch 2.13 — sgl_kernel/.../common_ops.abi3.so: undefined symbol _ZNK2at10TensorBase14const_data_ptr…	all 6 sglang recipes (3 per SKU)	sglang-miles (3003d70f) pins torch==2.11.0 and sglang-kernel==0.4.5; sglang 0.5.19 moved to torch==2.13.0 and sglang-kernel==0.4.6.post1. Needs a sglang-miles rebase; a speculative pin bump is not safe because a failed uv sync --extra sglang fails the entire image build. Note main's sglang tests also fail today on torch.multiprocessing.reductions._rebuild_cuda_tensor_original.
TRT-LLM ABI vs torch 2.13 — tensorrt_llm/libs/libth_common.so: undefined symbol _ZNR5torch7Library4_def… in TrtllmAsyncGenerationWorker.__init__	both GB200 trtllm recipes	No TRT-LLM release supports torch 2.13.0 (1.3.0rc26 requires torch<=2.13.0a0).
Qwen2.5-VL-3B generation/policy mismatch — token_mult_prob_error 198 (DTensor) / 19855 (Megatron) vs 1.02 on main, gen_kl 0.14–0.16 vs 0.0009, reward 0.31/0.30 at step 200 vs 0.93/0.89	vlm_grpo-qwen2.5-vl-3b-instruct-clevr-1n8g-{dtensor2tp1,megatrontp2}.v1	Two training backends agree with each other and disagree with vLLM 0.28, so it is generation-side and image-specific (text recipes and the Qwen2.5-Omni audio recipe have identical probability error to main: 1.008 / 1.004). 170beee removes the one known preprocessing divergence; remaining suspects are #51841 (0.28.0 changed the M-RoPE position H2D copy to per-row transfers) and #52005-class torch.compile miscompiles on torch 2.13 (that fix is in 0.28.0 for the interleaved variant only). First diagnostic on the rebuilt image: rerun with ++policy.generation.vllm_cfg.enforce_eager=True.
w4a16 NVFP4 real-quant refit — ModelOpt layerwise reload is incomplete for 23 layer(s): …routed_experts: 134701312/179601664 elements	grpo-nanov3-30ba3b-4n4g-megatron-qa-nvfp4-w4a16-real (GB200)	Exactly ¾ of each expert tensor arrives, so the streamed w4a16 layout no longer matches 0.28's RoutedExperts. The w4a4 sibling passes. Needs an audit of the real-quant refit against the 0.28 MoE weight layout, probably together with the ModelOpt bump. main Sept 8 ran to completion and failed only on golden metrics.
Two hangs, not root-caused — grpo-nemotron3-super-120BA12B-16n8g-automodel-ep8.v2 (NCCL watchdog timeout, 600 s) and vlm_grpo-qwen3-omni-30ba3b-audiomcq-4n8g-megatron.v1 (gloo recv timeout, 1800 s, inside VllmGenerationWorker.generate)	2	Both pass on main. Re-run on the rebuilt image before digging; the omni recipe also had the broken cutlass/numba probes in its venv.
nixl-cu13 1.3.0 nixl_ep extension built against torch 2.11 — nixl_ep_cpp.cpython-313-…so: undefined symbol _ZN3c104impl3cow23materialize_cow_storage…	none	Non-fatal: vLLM's has_nixl_ep() probe catches it and disables the optional nixl_ep all2all backend, but the traceback appears in every vLLM worker log and nemo-ci's error extractor picked it as the "cause" of seven unrelated failures. Bump nixl to a torch-2.13 build in a follow-up.
grpo-nemotron3-super-120BA12B-8n4g-megatron (GB200) — Unquantized FlashInfer TRTLLM refit does not yet support a co-trained MTP drafter	1	Pre-existing (main Sept 8 identical). Blackwell picks the FlashInfer TRTLLM BF16 MoE backend, and #3545 rejects co-trained MTP on that path. Either add moe_backend: triton to the recipe's vllm_kwargs (supported since 0.25) or extend _weight_update_lifecycle to include the drafter's TRTLLM modules in the layerwise reload. Separate PR.

Notes for reviewers

• The cutlass change removes a [tool.uv] exclusion that was load-bearing at 4.5.x. I verified the 4.5.2 overlap (180 files) and the 4.6.x non-overlap (0 files) directly from the PyPI wheels, and uv pip install of the overlapping 4.5.2 pair does not error — it silently picks a winner per file — which is why the mcore pin had to move to 4.6.2 as well.
• mm_device_do_normalize=False changes a vLLM default for every multimodal recipe. It restores the 0.25 behaviour byte-for-byte on the processor side; the cost is the CPU normalize vLLM's PR was avoiding.
• ModelOpt moves 0.46.0.dev86 → 0.47.0rc1.dev36 (137 + commits). nemo_rl/modelopt/models/generation/vllm_quant_patch.py already guards against double registration of RoutedExperts, and _drop_nonclass_quant_registry_keys becomes a no-op rather than a conflict.
• Unit tests: tests/unit/test_dependency_pins.py passes on the new lock; test_vllm_fp8_quantization.py was updated but could not be executed here (no vLLM in the host venv) — it needs the rebuilt image.
• The infra noise in the first run was large: 11 PR jobs and 24 main jobs died in 7 s at checkout on a corrupted shared-workspace directory (.nemo-ci-triage/...: Bad address), and 3 jobs per side hit the offline Hugging Face cache miss for gemma4-12b / nanov3.5 / qwen3-8-27b. PR-side checkout failures were retried; main's were not, so those tests have no same-day control and Sept 8's nightly is used where it helps.

Test plan

1. Full nightly suite on both SKUs against the PR head, compared per test with the same-day main nightly (RL 74b857c8e, 0 commits behind the merge-base) and Sept 8's (85ec4510c) as fallback. First pass below; second pass after the rebuild with the fixes above.
2. Root-cause every PR-only failure (done for all but the two hangs).
3. DSv3 perf and the disabled.txt qwen3.5 entries: not yet run.

Results — nightly, first pass (PR head 14fb0eb, images built with MAX_JOBS=4/8)

Pipelines: PR H100 nemo-ci#66971980 (tests 67032553) · PR GB200 nemo-ci#66965177 (tests 67018071) · main Sept 9 nemo-ci#66919718 (tests 67009322 H100, 67009271 GB200) · main Sept 8 nemo-ci#66728092. The original combined pipeline nemo-ci#66948077 never produced an image (flash-attn OOM on all six build attempts).

Legend: ✅ pass · ❌ fail · ⏳ running · — not in that run. Snapshot while both PR lanes were still finishing; will be refreshed.

Total PR test jobs: 171 (H100 131, GB200 40). Regressions vs main: 17 · both fail: 31 · PR passes where main failed: 21 · both pass: 80 · still running: 22.

PR fails, main passes (17)

Test	Branch	main Sep 9	main Sep 8	Notes
llm_dapo_gemma4_e2b_it_1n8g_fsdp2_automodel H100	❌ job	✅ job	✅	Fixed (5ae0cee). ModuleNotFoundError: No module named 'cutlass.cute' — nvidia-cutlass-dsl-libs-base was excluded; at 4.6.x it is the wheel that ships the Python DSL, and flashinfer 0.6.16's gdn_prefill imports it at module load.
llm_distillation_nano3_30ba3b_4n4g_megatron_qa_nvfp4_modelopt_spec GB200	❌ job	✅ job	✅	Fixed (5ae0cee). AssertionError: RowParallelLinear already registered! — ModelOpt c3b913b9's vLLM plugin dies on the removed fused_moe.layer.FusedMoE, import_plugin swallows it, and our explicit re-import re-registers the linear classes. Bumped ModelOpt to 613e5e8b.
llm_distillation_qwen3_1_7b_1n8g_megatron_qa_nvfp4 H100	❌ job	✅ job	✅	Fixed (5ae0cee). AssertionError: RowParallelLinear already registered! — ModelOpt c3b913b9's vLLM plugin dies on the removed fused_moe.layer.FusedMoE, import_plugin swallows it, and our explicit re-import re-registers the linear classes. Bumped ModelOpt to 613e5e8b.
llm_grpo_deepscaler_1_5b_8K H100	❌ job	✅ job	❌	Golden-value miss on a marginal bound: max(gen_kl_error) < 0.0005 at 0.00052; main passes. Worth re-checking after 170beee.
llm_grpo_llama3_1_8b_instruct_1n8g_megatron_fp8_rollouts_v3 H100	❌ job	✅ job	✅	Golden-value miss on a marginal bound: ratio_above(token_mult_prob_error, 1.1) < 0.1 at exactly 0.100. Distribution matches main (avg 1.210 vs 1.207, KL 0.0040 vs 0.0039, reward 0.447 vs 0.447; PR max 6.8 vs main 11.8).
llm_grpo_nemotron3_super_120BA12B_16n8g_automodel_ep8_v2 H100	❌ job	✅ job	—	Open — needs re-run on the rebuilt image. NCCL watchdog timeout (ALLREDUCE, 600 s) then actor death; main passes. Not yet root-caused.
llm_grpo_qwen2_5_math_1_5b_instruct_2n8g_fsdp2tp1_sglang H100	❌ job	✅ job	✅	Open — torch 2.13 ABI. sgl_kernel/.../common_ops.abi3.so: undefined symbol _ZNK2at10TensorBase14const_data_ptr.... sglang-miles pins torch==2.11.0 / sglang-kernel==0.4.5; sglang 0.5.19 moved to torch==2.13.0 / sglang-kernel==0.4.6.post1, so this needs a sglang-miles rebase, not a pin bump (a broken uv sync --extra sglang would fail the whole image build).
llm_grpo_qwen3_1_7b_2n4g_fsdp2_trtllm GB200	❌ job	✅ job	✅	Open — torch 2.13 ABI. tensorrt_llm/libs/libth_common.so: undefined symbol _ZNR5torch7Library4_def.... No TRT-LLM release supports torch 2.13.0 yet (1.3.0rc26 requires torch<=2.13.0a0).
llm_grpo_qwen3_5_35ba3b_2n8g_megatron_ep16tp2_fp8 H100	❌ job	✅ job	✅	Fixed (8318c80). TypeError: make_fp8_moe_kernel() got an unexpected keyword argument 'layer' — 0.28 dropped the kwarg.
llm_grpo_qwen3_5_35ba3b_2n8g_megatron_ep16tp2cp2 H100	❌ job	✅ job	✅	Fixed (5ae0cee). ModuleNotFoundError: No module named 'cutlass.cute' — nvidia-cutlass-dsl-libs-base was excluded; at 4.6.x it is the wheel that ships the Python DSL, and flashinfer 0.6.16's gdn_prefill imports it at module load.
llm_grpo_qwen3_5_9b_1n8g_megatron H100	❌ job	✅ job	✅	Fixed (5ae0cee). ModuleNotFoundError: No module named 'cutlass.cute' — nvidia-cutlass-dsl-libs-base was excluded; at 4.6.x it is the wheel that ships the Python DSL, and flashinfer 0.6.16's gdn_prefill imports it at module load.
llm_grpo_qwen3_8b_base_1n8g_fp8_kvcache_megatron H100	❌ job	✅ job	✅	Fixed (8318c80). AttributeError: 'Attention' object has no attribute 'calculate_kv_scales' — vLLM 0.28 removed the attribute; the fp8 KV-cache post-processing is re-mirrored on 0.28's BaseKVCacheMethod.
llm_sft_gpt_oss_20b_1n8g_fsdp8ep8_automodel H100	❌ job	✅ job	✅	Golden-value miss on a marginal bound: grad_norm[50] > 10.0 at 9.98; main's GB200 twin fails the same lower bound at 9.94.
vlm_vlm_grpo_gemma4_e4b_geo3k_1n8g_automodel H100	❌ job	✅ job	❌	Fixed (5ae0cee). ModuleNotFoundError: No module named 'cutlass.cute' — nvidia-cutlass-dsl-libs-base was excluded; at 4.6.x it is the wheel that ships the Python DSL, and flashinfer 0.6.16's gdn_prefill imports it at module load.
vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n4g_megatrontp1_v1 GB200	❌ job	✅ job	✅	Fixed (5ae0cee). ModuleNotFoundError: No module named 'cutlass.cute' — nvidia-cutlass-dsl-libs-base was excluded; at 4.6.x it is the wheel that ships the Python DSL, and flashinfer 0.6.16's gdn_prefill imports it at module load. (Blackwell attention path reaches the same import for every model.)
vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n8g_dtensor2tp1_v1 H100	❌ job	✅ job	✅	Open — generation/policy mismatch. token_mult_prob_error 2e2–2e4 (main: 1.02), gen_kl 0.14–0.16 (main: 0.0009), reward stuck at 0.3 (main: 0.9). Same on DTensor and Megatron, so it is vLLM-side; text and Qwen2.5-Omni audio recipes are unaffected. 170beee restores CPU image normalization as a parity measure (verified: the 0.28 fused GPU path differs by up to 0.019 in bf16). Remaining suspect: #51841's per-row M-RoPE H2D copies (0.28.0) and #52005-class torch.compile miscompiles on torch 2.13. Diagnostic for the rebuilt image: rerun with ++policy.generation.vllm_cfg.enforce_eager=True.
vlm_vlm_grpo_qwen3_omni_30ba3b_audiomcq_4n8g_megatron_v1 H100	❌ job	✅ job	❌	Open — needs re-run on the rebuilt image. gloo recv timeout (1800 s) inside VllmGenerationWorker.generate; main passes. Not yet root-caused.

PR fails, main fails (26)

Test	Branch	main Sep 9	main Sep 8	Notes
llm_dapo_gemma4_12b_it_2n8g_fsdp2_automodel H100	❌ job	❌ job	—	Infra. Hugging Face connection error (offline cache miss); main same.
llm_dapo_nanov3_5_30BA3B_4n8g_automodel H100	❌ job	❌ job	❌	Infra. Hugging Face connection error (offline cache miss); identical on main.
llm_dpo_mistral_nemo_instruct_2407_1n8g_fsdp2tp8_actckpt_long H100	❌ job	❌ job	❌	Pre-existing. Golden-value miss on main too (loss[1] < 0.70 at 0.7087).
llm_dpo_nanov3_30B3AB_1n4g_fsdp4ep4_automodel GB200	❌ job	❌ job	❌	Pre-existing. Step-time bound only (16.6 s vs < 5); same on main.
llm_dpo_qwen2_5_math7b_1n8g_megatron_fused_linear_logprobs H100	❌ job	❌ job	❌	Pre-existing. accuracy[10] >= 0.5 at 0.40625 — bit-identical value on main.
llm_grpo_moonlight_16ba3b_4n4g_megatron GB200	❌ job	❌ job	❌	Fixed (5ae0cee). ModuleNotFoundError: No module named 'cutlass._mlir.dialects' — nvidia-cutlass-dsl-libs-base was excluded; at 4.6.x it is the wheel that ships the Python DSL, and flashinfer 0.6.16's gdn_prefill imports it at module load. main fails later with the pre-existing Moonlight state-dict KeyError.
llm_grpo_moonlight_16ba3b_4n8g_megatron H100	❌ job	❌ job	❌	Pre-existing. Megatron-Bridge KeyError: decoder.layers.N.self_attention.linear_q_proj.layer_norm_weight ... not in state dict; identical on main.
llm_grpo_moonlight_16ba3b_4n8g_megatron_fp8_e2e H100	❌ job	❌ job	❌	Pre-existing. AttributeError: 'DeepseekV3Config' object has no attribute 'qk_head_dim'; identical on main.
llm_grpo_moonlight_16ba3b_4n8g_megatron_tq_simple H100	❌ job	❌ job	❌	Pre-existing. Same Moonlight state-dict KeyError as main.
llm_grpo_nanov3_30ba3b_4n4g_megatron_qa_nvfp4_w4a16_real GB200	❌ job	❌ job	❌	Open — regression in the real-quant refit. ModelOpt layerwise reload is incomplete for 23 layer(s): ...routed_experts: 134701312/179601664 elements — exactly 3/4 of each w4a16 expert tensor arrives, so the streamed layout no longer matches 0.28's RoutedExperts. main Sept 8 ran to completion and failed only on golden metrics.
llm_grpo_nemotron3_super_120BA12B_16n8g_megatron H100	❌ job	❌ job	❌	Pre-existing. AssertionError in MegatronPolicyWorker.get_logprobs; identical on main.
llm_grpo_nemotron3_super_120BA12B_8n4g_megatron GB200	❌ job	⏳ job	❌	Pre-existing. Unquantized FlashInfer TRTLLM refit does not yet support a co-trained MTP drafter (#3545 guard; Blackwell selects the TRTLLM BF16 MoE backend); identical on main Sept 8.
llm_grpo_qwen2_5_0_5b_1n4g_megatron_trtllm_noncolocated_async GB200	❌ job	❌ job	✅	Open — torch 2.13 ABI. tensorrt_llm/libs/libth_common.so: undefined symbol _ZNR5torch7Library4_def.... No TRT-LLM release supports torch 2.13.0 yet (1.3.0rc26 requires torch<=2.13.0a0). main fails later on golden metrics (token_mult_prob_error 20.7).
llm_grpo_qwen2_5_1_5B_4n8g_megatron_yarn_256k H100	❌ job	❌ job	❌	Pre-existing. reward[30] > 0.5 at 0.5; main fails the same check at 0.4375.
llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_fsdp2tp1_sglang GB200	❌ job	⏳ job	❌	Open — torch 2.13 ABI. sgl_kernel/.../common_ops.abi3.so: undefined symbol _ZNK2at10TensorBase14const_data_ptr.... sglang-miles pins torch==2.11.0 / sglang-kernel==0.4.5; sglang 0.5.19 moved to torch==2.13.0 / sglang-kernel==0.4.6.post1, so this needs a sglang-miles rebase, not a pin bump (a broken uv sync --extra sglang would fail the whole image build).
llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_megatrontp1_sglang GB200	❌ job	❌ job	❌	Open — torch 2.13 ABI. sgl_kernel/.../common_ops.abi3.so: undefined symbol _ZNK2at10TensorBase14const_data_ptr.... sglang-miles pins torch==2.11.0 / sglang-kernel==0.4.5; sglang 0.5.19 moved to torch==2.13.0 / sglang-kernel==0.4.6.post1, so this needs a sglang-miles rebase, not a pin bump (a broken uv sync --extra sglang would fail the whole image build). main fails on its own _rebuild_cuda_tensor_original AttributeError.
llm_grpo_qwen2_5_math_1_5b_instruct_2n4g_megatrontp1_sglang_noncolocated GB200	❌ job	⏳ job	✅	Open — torch 2.13 ABI. sgl_kernel/.../common_ops.abi3.so: undefined symbol _ZNK2at10TensorBase14const_data_ptr.... sglang-miles pins torch==2.11.0 / sglang-kernel==0.4.5; sglang 0.5.19 moved to torch==2.13.0 / sglang-kernel==0.4.6.post1, so this needs a sglang-miles rebase, not a pin bump (a broken uv sync --extra sglang would fail the whole image build).
llm_grpo_qwen2_5_math_1_5b_instruct_2n8g_megatrontp1_sglang H100	❌ job	❌ job	❌	Open — torch 2.13 ABI. sgl_kernel/.../common_ops.abi3.so: undefined symbol _ZNK2at10TensorBase14const_data_ptr.... sglang-miles pins torch==2.11.0 / sglang-kernel==0.4.5; sglang 0.5.19 moved to torch==2.13.0 / sglang-kernel==0.4.6.post1, so this needs a sglang-miles rebase, not a pin bump (a broken uv sync --extra sglang would fail the whole image build). main fails on its own torch.multiprocessing.reductions._rebuild_cuda_tensor_original AttributeError.
llm_grpo_qwen3_5_35ba3b_2n8g_automodel_ep16 H100	❌ job	❌ job	❌	Fixed (5ae0cee). ModuleNotFoundError: No module named 'cutlass.cute' — nvidia-cutlass-dsl-libs-base was excluded; at 4.6.x it is the wheel that ships the Python DSL, and flashinfer 0.6.16's gdn_prefill imports it at module load. main fails later with a DeepEP dispatch timeout.
llm_grpo_qwen3_8_27b_2n8g_megatron_tp4pp2cp2 H100	❌ job	❌ job	❌	Infra. OSError: We couldn't connect to 'https://huggingface.co' (model not in the offline cache); identical on main.
llm_ppo_qwen2_5_1_5b_gsm8k_2n8g_megatron_valuetp2sp_dynbatch_noncolocated_async_single_controller H100	❌ job	❌ job	❌	Pre-existing. ValueError: SingleController checkpointing ... requires checkpointing.save_data_plane=true; identical on main.
vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_8n4g_megatron_generation_v1 GB200	❌ job	❌ job	✅	Pre-existing. validation/accuracy[50] > accuracy[0] + 0.01 fails on main Sept 9 too (0.652 vs 0.620); probability error is healthy (1.014).
vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n4g_dtensor2tp1_v1 GB200	❌ job	⏳ job	✅	Fixed (5ae0cee). ModuleNotFoundError: No module named 'cutlass.cute' — nvidia-cutlass-dsl-libs-base was excluded; at 4.6.x it is the wheel that ships the Python DSL, and flashinfer 0.6.16's gdn_prefill imports it at module load. Once past the import, expect the Qwen2.5-VL mismatch below.
vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n8g_megatrontp2_v1 H100	❌ job	❌ job	❌	Open — generation/policy mismatch. token_mult_prob_error 2e2–2e4 (main: 1.02), gen_kl 0.14–0.16 (main: 0.0009), reward stuck at 0.3 (main: 0.9). Same on DTensor and Megatron, so it is vLLM-side; text and Qwen2.5-Omni audio recipes are unaffected. 170beee restores CPU image normalization as a parity measure (verified: the 0.28 fused GPU path differs by up to 0.019 in bf16). Remaining suspect: #51841's per-row M-RoPE H2D copies (0.28.0) and #52005-class torch.compile miscompiles on torch 2.13. Diagnostic for the rebuilt image: rerun with ++policy.generation.vllm_cfg.enforce_eager=True. (main's Megatron variant misses only the 0.9 reward bound at 0.886 with healthy probability error.)
vlm_vlm_grpo_qwen3_5_35ba3b_geo3k_2n8g_automodel_ep16 H100	❌ job	❌ job	❌	Fixed (5ae0cee). ModuleNotFoundError: No module named 'cutlass.cute' — nvidia-cutlass-dsl-libs-base was excluded; at 4.6.x it is the wheel that ships the Python DSL, and flashinfer 0.6.16's gdn_prefill imports it at module load. main fails later with a DeepEP dispatch timeout.
vlm_vlm_grpo_qwen3_5_35ba3b_geo3k_2n8g_automodel_ep16_tq_simple H100	❌ job	❌ job	❌	Fixed (5ae0cee). ModuleNotFoundError: No module named 'cutlass.cute' — nvidia-cutlass-dsl-libs-base was excluded; at 4.6.x it is the wheel that ships the Python DSL, and flashinfer 0.6.16's gdn_prefill imports it at module load. main fails later with a DeepEP dispatch timeout.

PR fails, main had no result (infra) (5)

Test	Branch	main Sep 9	main Sep 8	Notes
llm_dpo_nanov3_30B3AB_1n8g_fsdp8ep8_automodel_v2 H100	❌ job	❌ job	—	Golden-value miss on a marginal bound: loss[11] < 0.57 at 0.5825 and step time 5.03 vs < 5. main's run died at checkout, so no same-day control. main job died at checkout (shared-workspace Bad address), no result.
llm_grpo_nanov3_30ba3b_3n8g_megatron_dynamo_swe1 H100	❌ job	❌ job	❌	Pre-existing. ValueError: NeMo Gym does not support generation backend 'dynamo'; main's run had no gym image and fails too (Sept 8 as well). main has no gym image (Gym pin vs NEMO_GYM_VLLM_VERSION conflict since Sept 5).
llm_grpo_qwen2_5_math_1_5b_instruct_2n8g_megatrontp1_sglang_noncolocated H100	❌ job	❌ job	✅	Open — torch 2.13 ABI. sgl_kernel/.../common_ops.abi3.so: undefined symbol _ZNK2at10TensorBase14const_data_ptr.... sglang-miles pins torch==2.11.0 / sglang-kernel==0.4.5; sglang 0.5.19 moved to torch==2.13.0 / sglang-kernel==0.4.6.post1, so this needs a sglang-miles rebase, not a pin bump (a broken uv sync --extra sglang would fail the whole image build). main job died at checkout (shared-workspace Bad address), no result.
llm_grpo_qwen3_30ba3b_4n8g_megatron_qa_nvfp4 H100	❌ job	❌ job	❌	Fixed (5ae0cee). AssertionError: RowParallelLinear already registered! — ModelOpt c3b913b9's vLLM plugin dies on the removed fused_moe.layer.FusedMoE, import_plugin swallows it, and our explicit re-import re-registers the linear classes. Bumped ModelOpt to 613e5e8b. main job died at checkout (shared-workspace Bad address), no result.
llm_prorlv2_qwen2_5_math_1_5b_instruct_1n8g_fsdp2tp1_v2_tq_mooncake H100	❌ job	❌ job	❌	Open — needs re-run on the rebuilt image. Slurm step hit its time limit; no Python error in the trace. main's run died at checkout. Sept 8 main also failed. main job died at checkout (shared-workspace Bad address), no result.

PR passes, main fails (5)

Test	Branch	main Sep 9	main Sep 8	Notes
llm_grpo_math_qwen3_30ba3b_megatron_tp4_32k H100	✅ job	❌ job	✅
llm_grpo_nanov3_30BA3B_4n4g_megatron_generation_noncolocated_mxfp8_rollouts GB200	✅ job	❌ job	❌
llm_grpo_qwen2_5_32b_32n8g_fsdp2tp8_actckpt_v3 H100	✅ job	❌ job	✅
llm_grpo_qwen3_30ba3b_4n8g_megatron_zmq_deltaweight_noncolocated H100	✅ job	❌ job	❌
llm_sft_gpt_oss_20b_1n4g_fsdp4ep4_automodel GB200	✅ job	❌ job	❌

PR passes, main had no result (infra) (16)

Test	Branch	main Sep 9	main Sep 8	Notes
llm_dpo_llama3_1_8b_instruct_4n8g_megatrontp2pp2_quick H100	✅ job	❌ job	❌	main job died at checkout (shared-workspace Bad address), no result.
llm_grpo_llama3_1_8b_instruct_2n8g_megatron_fp8_e2e H100	✅ job	❌ job	❌	main job died at checkout (shared-workspace Bad address), no result.
llm_grpo_llama3_2_1b_instruct_1n8g_megatron_tq_simple H100	✅ job	❌ job	✅	main job died at checkout (shared-workspace Bad address), no result.
llm_grpo_nanov3_30BA3B_1n8g_fsdp2_tq_mooncake_v2 H100	✅ job	❌ job	—	main job died at checkout (shared-workspace Bad address), no result.
llm_grpo_qwen3_1_7b_1n8g_megatron_eagle3 H100	✅ job	❌ job	✅	main job died at checkout (shared-workspace Bad address), no result.
llm_grpo_qwen3_1_7b_1n8g_megatron_super_swe1 H100	✅ job	❌ job	❌	main has no gym image (Gym pin vs NEMO_GYM_VLLM_VERSION conflict since Sept 5).
llm_grpo_qwen3_1_7b_2n8g_megatron_super_rlhf H100	✅ job	❌ job	❌	main job died at checkout (shared-workspace Bad address), no result.
llm_grpo_qwen3_30ba3b_8n8g_megatron_cp2_r3 H100	✅ job	❌ job	❌	main job died at checkout (shared-workspace Bad address), no result.
llm_mopd_qwen3_1_7b_3n8g_megatron_pack H100	✅ job	❌ job	❌	main has no gym image (Gym pin vs NEMO_GYM_VLLM_VERSION conflict since Sept 5).
llm_mopd_qwen3_1_7b_3n8g_megatron_pack_single_controller H100	✅ job	❌ job	❌	main has no gym image (Gym pin vs NEMO_GYM_VLLM_VERSION conflict since Sept 5).
llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_megatron_valuetp2sp_dynbatch H100	✅ job	❌ job	✅	main job died at checkout (shared-workspace Bad address), no result.
llm_ppo_qwen2_5_1_5b_gsm8k_2n8g_megatron_valuetp2sp_dynbatch_noncolocated_async H100	✅ job	❌ job	✅	main job died at checkout (shared-workspace Bad address), no result.
llm_prorlv2_qwen2_5_math_1_5b_instruct_1n8g_fsdp2tp1_v2 H100	✅ job	❌ job	✅	main job died at checkout (shared-workspace Bad address), no result.
llm_sft_llama3_1_8b_1n8g_fsdp2tp2 H100	✅ job	❌ job	❌	main job died at checkout (shared-workspace Bad address), no result.
llm_sft_llama3_1_8b_1n8g_megatron H100	✅ job	❌ job	❌	main job died at checkout (shared-workspace Bad address), no result.
vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_1n8g_automodel_ep8_v2 H100	✅ job	❌ job	—	main job died at checkout (shared-workspace Bad address), no result.

Still running on the PR (22)

Test	Branch	main Sep 9	main Sep 8	Notes
llm_grpo_deepscaler_1_5b_1n4g_8K GB200	⏳ job	⏳ job	✅
llm_grpo_deepscaler_1_5b_8K_tq_simple H100	⏳ job	❌ job	✅	main job died at checkout (shared-workspace Bad address), no result.
llm_grpo_gemma3_1b_it_1n4g_fsdp2tp1 GB200	⏳ job	✅ job	❌
llm_grpo_gspo_deepscaler_1_5b_8K_tq_simple H100	⏳ job	✅ job	✅
llm_grpo_llama3_1_8b_instruct_2n4g_async_1off_single_controller_streaming2 GB200	⏳ job	✅ job	✅
llm_grpo_llama3_1_8b_instruct_2n4g_fsdp2tp1_noncolocated GB200	⏳ job	✅ job	✅
llm_grpo_llama3_1_8b_instruct_2n8g_fsdp2tp1_noncolocated_tq_simple H100	⏳ job	✅ job	❌
llm_grpo_llama3_2_1b_instruct_1n4g_fsdp2tp1_v3 GB200	⏳ job	✅ job	❌
llm_grpo_llama3_2_1b_instruct_1n4g_megatron GB200	⏳ job	⏳ job	✅
llm_grpo_llama3_2_1b_instruct_1n4g_megatron_generation GB200	⏳ job	⏳ job	✅
llm_grpo_llama3_2_1b_instruct_1n8g_fsdp2tp2_temp0_8_topp0_9_topk50_tq_mooncake H100	⏳ job	❌ job	✅
llm_grpo_moonlight_16b_automodel_1n8g_ep8 H100	⏳ job	✅ job	✅
llm_grpo_nanov3_30BA3B_2n8g_megatron_generation_noncolocated_async_gym H100	⏳ job	✅ job	✅
llm_grpo_nanov3_30BA3B_4n4g_megatron_generation_noncolocated_async_gym GB200	⏳ job	⏳ job	✅
llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_fsdp2tp1_v3 GB200	⏳ job	✅ job	✅
llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_megatron_single_controller_sync GB200	⏳ job	✅ job	✅
llm_grpo_qwen3_1_7b_6n8g_megatron_super_rlvr H100	⏳ job	❌ job	❌	main job died at checkout (shared-workspace Bad address), no result.
llm_grpo_qwen3_30ba3b_thinking_swe1_16n8g_megatron_cp2_r3_async_gym H100	⏳ job	✅ job	❌
llm_grpo_qwen3_5_9b_1n8g_megatron_fp8 H100	⏳ job	✅ job	✅
llm_grpo_qwen3_8b_base_dapo_2n8g_long_megatron_qa_nvfp4_w4a16 H100	⏳ job	❌ job	✅	main job died at checkout (shared-workspace Bad address), no result.
vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_1n8g_megatron_tp8ep8_v1 H100	⏳ job	❌ job	❌	main job died at checkout (shared-workspace Bad address), no result.
vlm_vlm_grpo_nemotron_omni_30ba3b_mmpr_4n8g_megatron_tp8ep16_v1 H100	⏳ job	❌ job	❌	main: AssertionError: defer_fp32_logits must be True if logprob_chunk_size is set.

Both pass: 80 tests. PR passes, main still running: 0.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yfw]

/ok to test 717ee4a

[yfw]

/ok to test 03b9ee2

[yfw]

/ok to test c759c71

[yfw]

/ok to test 14fb0eb

[yfw]

closed in favor of 0.29 bump #4080