build: bump vllm to 0.29.0

[yfw]

What does this PR do ?

Bumps vLLM from 0.26.0 to 0.29.0 (this branch started from 0.25.1; main moved to 0.26.0 with #3566 on Sept 19). The vLLM 0.29.0 wheels are built against torch 2.13, so the vLLM / Megatron / Automodel venvs move with it: torch 2.11.0 → 2.13.0, torchvision 0.26.0 → 0.28.0, triton 3.6.0 → 3.7.1, flashinfer 0.6.14 → 0.6.18, nvidia-cutlass-dsl 4.6.0 → 4.6.2, tilelang 0.1.9 → 0.1.12 in the vLLM venv, llguidance 1.8.0 → 1.7.6 (vLLM 0.29's own <1.8 bound), numpy 2.5.3 → 2.4.6. The sglang and TRT-LLM venvs stay on torch 2.11.0, because their kernels are built against it; torch is now pinned per backend extra and a small IPC-handle shim lets the 2.11 inference venvs open weight handles produced by the 2.13 trainer.

Full nightly suites on H100 (cw_dfw) and GB200 (oci_hsg) were compared recipe by recipe against main's own scheduled nightlies at this PR's merge point. Every regression the bump introduced is fixed on this branch and validated on hardware. The remaining failures are pre-existing on main, infrastructure, or goldens sitting at their bounds, each with the main control that shows it.

Changes

Bullets are prefixed with the commit that made the change.

Dependencies (pyproject.toml / uv.lock)

• 83fa485 / e179799 — vLLM 0.26.0 → 0.29.0 from PyPI (default cu130 wheels for both arches), torch 2.11.0 → 2.13.0, torchvision 0.26.0 → 0.28.0, flashinfer-python/-cubin/-jit-cache 0.6.14 → 0.6.18 (cu130 wheels for both arches, matching the ABI of the 0.29.0 vLLM wheels), nvidia-cutlass-dsl[cu13] 4.6.0 → 4.6.2 in the vllm extra (and 4.5.2 → 4.6.2 in mcore), tilelang <0.1.13 (0.1.9 → 0.1.12 in the vLLM venv), llguidance>=1.7.0,<1.8.0 (1.8.0 → 1.7.6: vLLM's own bound, which main's >=1.3.0 floor bypassed). flash-attn switches from the pinned cu13torch2.10 GitHub wheels to flash-attn==2.8.1 (sdist) because no 2.8.x wheel exists for torch ≥ 2.11.
• e179799 — instanttensor kept to x86_64 via an override. vLLM 0.29 lists instanttensor>=0.1.9 unconditionally; it is the optional load_format="instanttensor" safetensors loader, imported lazily, and NeMo-RL never selects it. 0.2.0 ships aarch64 wheels, so this is a choice to leave an unused dependency out of the GB200 image, not a wheel-availability limit.
• 2e0bdb8 — lockfile regenerated for the Docker uv version.
• 9d8f808 — cutlass-dsl 4.6.2 in every fork that can take it. main already removed the nvidia-cutlass-dsl-libs-base exclusion in feat: add end-to-end TE NVFP4 training with per-token vLLM rollout #3566 (4.6 split the libs-base / libs-cu13 wheels cleanly; the 4.5.x pair overwrote 180 shared files). This PR moves the vllm extra 4.6.0 → 4.6.2 (flashinfer 0.6.18's requirement) and the mcore pin 4.5.2 → 4.6.2 so the training venv does not inherit the overlapping 4.5.x pair, drops feat: add end-to-end TE NVFP4 training with per-token vLLM rollout #3566's quack-kernels==0.6.1 pin (0.29 pins 0.6.4), and keeps a NOTE in pyproject.toml on why the exclusion must not come back. The trtllm fork stays on 4.5.0 via tensorrt-llm.
• 9d8f808 — flash-attn source build bounded with MAX_JOBS=4 via [tool.uv.extra-build-variables]. flash-attn's setup.py sizes its parallelism from the host's free memory (not the cgroup) and cpu_count()//2, with --threads 4 per nvcc; every nemo-ci build attempt on both architectures died with ResourceExhausted after dozens of Killed "$CICC_PATH/cicc". 4×4 builds cleanly on aarch64 (2h47m) and x86 (~2h, alongside TE and flash-mla).
• 9d8f808 — numpy override capped at <2.5 (resolves 2.4.6). vLLM 0.29 pins numba==0.65.0, whose metadata requires numpy<2.5; the numpy>=2.1.0 override forced 2.5.1 past it and numba refused to import in the vLLM venv ("Numba needs NumPy 2.4 or less"). Not fatal by itself — vLLM's optional-module probe catches it — but it left a broken numba in the venv and polluted every trace.
• e991336 — ModelOpt stays at main's 43fd41a58, now with a do-not-move note. The pin must not move past 2ded5aba3 (ray.sub requires submission from NeMo RL home directory, blocking external workflow organization #1550): that commit introduces GroupedQuantizer, which breaks the Megatron fakequant DDP hooks and real-quant dist-ckpt loading (observed on ModelOpt main, 613e5e8b). 43fd41a58 is a release cherry-pick that carries Direct support for megatron bridge checkpoints #2054 (the RoutedExperts vLLM plugin this bump needs; the pre-Direct support for megatron bridge checkpoints #2054 plugin registered on fused_moe.layer.FusedMoE, which 0.29 no longer has) and excludes ray.sub requires submission from NeMo RL home directory, blocking external workflow organization #1550.
• 5200efa — flash-linear-attention / fla-core relocked 0.5.1 → 0.5.2, the version main's lock resolves. fla-core 0.5.1 enabled flash-linear-attention's TileLang kernels wherever tilelang imports (the mcore extra installs it for the fused DSA kernels), including Blackwell, and its TileLang gated-delta-rule backward faults there with CUDA error: misaligned address; 0.5.2 ([Bug] GDN precision error in triton3.5 and h20 fla-org/flash-linear-attention#640) turns TileLang on by default only on Hopper with Triton ≥ 3.4 and keeps Triton elsewhere. The GB200 grpo-qwen3.5-35ba3b-6n4g-async-1off-bf16-trtllm recipe died at step 1 of MegatronPolicyWorker.train() on every run of this branch while main (0.5.2) passes it; it had been mis-cleared against a Sept 10 main control that still carried 0.5.1. This branch's lock had lagged because each main merge kept our uv.lock and re-locked without upgrading transitive packages.
• be440de — every transitive package relocked to at least main's resolution (uv lock --upgrade, with Ray held at main's 2.56.1). About 150 packages had resolved older here than on main for the reason above (pandas 2.3 vs 3.0, starlette, xgrammar, tensordict, nccl4py, cuda-pathfinder, …); 103 move, none of the pinned ones (vLLM, torch per extra, triton, flashinfer, cutlass-dsl, TE, ModelOpt, sglang, TRT-LLM, DeepEP, nixl) change, and the only packages still below main are the deliberate numpy <2.5, llguidance <1.8 and the flash-attn sdist. The nightlies in the Test plan ran on this lock.
• ab6b3f2 — the trtllm venv stays on cuda-bindings / cuda-python 13.3. The relock above moved them to 13.4.x, and cuda-bindings 13.4 removed the reserved field of cudaIpcMemHandle_t that tensorrt_llm 1.3.0rc21's _ipc_utils.py reads while allocating the TP>1 all-reduce fusion workspace; the GB200 grpo-qwen3-1.7b-2n4g-fsdp2-trtllm recipe died at engine start with AttributeError: … has no attribute 'reserved' (450733567) while the TP=1 TRT-LLM recipes passed. The trtllm extra pins both <13.4 (main's versions); the torch-2.13 forks keep 13.4.x, which the vLLM and Megatron recipes passed with.

vLLM source patches / worker (nemo_rl/models/generation/vllm/)

• ae4ebcf — VllmInternalWorkerExtension.synchronize_device renamed to synchronize_sparse_refit_device. vLLM 0.29 adds WorkerBase.synchronize_device ([Bugfix][DP] Synchronize the device on pause completion vllm-project/vllm#52914) and WorkerBase.init_worker asserts that a worker extension never shadows a Worker attribute, so every vLLM engine died at init with AssertionError: Worker class … already has an attribute synchronize_device, which conflicts with the worker extension class. The extension method only drained peers before the sparse-delta refit removed staged batch files; its RPC caller and test follow the rename. An AST scan of both extension classes against 0.29's Worker/WorkerBase shows no other overlap, and a new vllm-marked unit test mirrors vLLM's collision predicate so the next upstream name clash fails in CI rather than on a GPU job.
• ae6507c — ErrorResponse imported from vllm.entrypoints.serve.engine.protocol. vLLM 0.29 moved openai/engine/protocol.py out of the openai package ([Frontend] Move engine/protocol.py out openai folder vllm-project/vllm#54492); VllmAsyncGenerationWorker.post_init_async imported it from the old path when bringing up the OpenAI-compatible server, so every recipe that serves generation over HTTP (Gym, swe1, mopd) died with ModuleNotFoundError: No module named 'vllm.entrypoints.openai.engine'. A new vllm-marked test walks nemo_rl/models/generation/vllm with ast, collects every unguarded from vllm… import statement in the vLLM, Dynamo and ModelOpt generation trees (109 today, mostly lazy, method-local imports no unit test executes) and resolves them against the installed vLLM.
• 710d278 — check_admission proxied through _AsyncLLMHTTPClient. vLLM 0.29's OpenAIServing._preflight calls engine_client.check_admission(n) before every response ([Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management vllm-project/vllm#49445, queue admission control). The async worker's engine-client wrapper exposes an explicit member surface by design, so every /v1/chat/completions 500ed with AttributeError: '_AsyncLLMHTTPClient' object has no attribute 'check_admission' and the Gym-path recipes died on AsyncTrajectoryCollector aborting: … batch-worker failure(s) exceeded max_generation_failures. Forwarded like the other status-only members; unit-tested.
• e1c16e8 — Tied-embedding aliases dropped before load_weights. vLLM 0.29's AutoWeightsLoader (Fix weight tying vllm-project/vllm#51665) skips a tied alias such as lm_head.weight and then asserts that its canonical model.embed_tokens.weight was loaded in the same load_weights call. Refit streams weights in transport-sized batches, so the two routinely land in different calls and every tied-embedding model refit from a DTensor/automodel policy died at its first weight update (ppo_qwen2_5_1_5b_gsm8k_1n8g_automodel_noncolocated{,_async}, dapo_gemma4_e2b_it_1n8g_fsdp2_automodel, vlm_grpo_gemma4_e4b_geo3k_1n8g_automodel) with ValueError: 'lm_head.weight' was skipped because it is tied to 'model.embed_tokens.weight' … was not found in the checkpoint. The alias never loads anything, so it is now dropped up front using vLLM's own _get_tied_embedding_params and the model's hf_to_vllm_mapper (so Gemma-style renamed parameters are covered), on both the batched refit and the native reload_weights path; the MTP drafter still sees the unfiltered stream. Unit-tested; a vllm-marked test pins the helper's existence.
• a419679 — _patch_vllm_ray_executor_v2_tcpstore_port recognises the upstream fix. vLLM 0.29 ships [Bugfix] Avoid TCPStore port collision for co-located non-DP Ray engines vllm-project/vllm#53666 and #50969: RayWorkerProc.create_dist_init_method binds the torch.distributed TCPStore itself on a kernel-assigned port and holds the socket (self._dist_init_store = store) until init_process_group reuses it, so the probe/bind window this patch closed on 0.25 is gone and _select_tcpstore_port no longer exists. Without this change the patch found no anchor and logged "Engines spanning nodes may fail with EADDRINUSE" on every worker start. It now logs at info and leaves the file alone; the port-arithmetic unit tests skip when the installed vLLM carries the upstream marker, and a new test pins the no-warn/no-edit behaviour against a 0.29-style source. The reserved VLLM_PORT band still governs the MessageQueue and API-server ports.
• 83fa485 — _patch_vllm_shm_broadcast_bind_retry re-targeted: vLLM 0.29 binds the MessageQueue remote socket to port 0 directly (no probe/bind race any more) but ignores VLLM_PORT; the patch restores reserved-band selection with bind retries so engine sockets stay out of the ephemeral range.
• f6993c8 — mm_device_do_normalize defaults to False. vLLM 0.29 ([Model] Fused mm preprocess normalisation on the Device vllm-project/vllm#50411) calls the HF image processor with do_rescale=False, do_normalize=False and re-applies both on the GPU in the vision tower's dtype. The policy normalizes the same images on the CPU in fp32 through the same processor, and the nightly probability-error checks assume identical inputs on both sides. Verified on CPU with transformers 5.12.1: the fused path is exact in fp32 but differs by up to 0.019 (≈1.7 % of the pixel std) in bf16; with the default flipped the Qwen2.5-VL clevr DTensor recipe is back at main-level metrics (token_mult_prob_error 1.019, gen_kl 0.0008, vs 198 / 0.14 with on-device normalization). Opt back in with policy.generation.vllm_kwargs.mm_device_do_normalize=true.

fp8 generation (quantization/fp8.py)

• 1734158 — make_fp8_moe_kernel no longer takes a layer kwarg (0.25 forwarded it only to the FlashInfer TRTLLM experts). Both call sites updated; the unit test's expected kwargs follow.
• 1734158 — process_weights_after_loading_kv re-mirrored on 0.29's BaseKVCacheMethod.process_weights_after_loading: the attention layer no longer has calculate_kv_scales (dynamic scales are a KV-cache dtype now, kv_cache_uses_per_token_head_scales), the static-scale branch keys off is_quantized_kv_cache, and the host _k_scale_cpu/_v_scale_cpu copies are refreshed on refit. Parameters are still kept so refit can update them.

ModelOpt fakequant / real-quant refit (nemo_rl/modelopt/models/generation/)

• ee35b44 / f3fea21 — per-expert input_quantizer._amax routed around vLLM's expert loader, with checkpoint names mapped through hf_to_vllm_mapper first. The Megatron side exports one input_quantizer._amax per expert projection (…experts.16.up_proj.input_quantizer._amax); the vLLM ModelOpt MoE module owns a single fused quantizer per projection group (w13_input_quantizer / w2_input_quantizer). vLLM 0.25 loaded these through the model-level parameter dict, which NeMo-RL patches to include quantizer buffers; 0.29's AutoWeightsLoader hands every experts.* name to RoutedExperts.load_weights, which rewrites it with the expert mapping and resolves the result with a single getattr — a dotted buffer name cannot resolve that way and every fakequant nvfp4 MoE refit died with AttributeError: Layer … has no parameter 'w13_input_quantizer._amax'. New vllm_quant_moe_amax.route_moe_input_quantizer_amax applies the model's hf_to_vllm_mapper (Nemotron-H: backbone. → model.) and then the same rewrite vLLM does, walks the dotted path on the expert module, fans the values in with max, and hands the remaining weights to vLLM's loader. Unit-tested (mapped and unmapped cases); validated by the GB200 distillation_nano3_30ba3b_4n4g_megatron_qa_nvfp4_modelopt_spec recipe.

Distributed (collectives.py)

• 3558f0a — all_gather_into_tensor → all_gather_single (torch 2.13 API).

Docker (docker/Dockerfile, 3rdparty/TensorRT-LLM-workspace/_backend.py)

• 40f5bcc / fe6945a — TRT-LLM wheel mirror: skip the copy when it is already there. main's fix(docker): persist the TRT-LLM wheel #4153 mirrors the content-addressed wheel into /opt/trtllm_wheels in the hermetic layer and has the release stage read it from there. This PR's residual is 3rdparty/TensorRT-LLM-workspace/_backend.py not re-copying a mirror that already exists (a stage inheriting the mirror from an earlier layer no longer duplicates it into its own) and the matching Dockerfile comment.

Audio deps (tools/install_audio_deps.sh, docs/guides/grpo-audio-visual.md)

• 2c0ad3c / 72b3b35 — torchcodec==0.11.1 → 0.16.0 in the test-time audio installer. Per the torchcodec compatibility table, 0.11 is built for torch 2.11 only while 0.12+ use the stable ABI for torch ≥ 2.11; the image now ships torch 2.13. 0.16.0 is what Gym's docker/install_codec_deps.sh already installs, so the two venvs agree. torchaudio==2.11.0 stays: it is the final torchaudio release and carries no torch pin. Guide updated to state the exact pins.

Tests (tests/test_suites/)

• 3e3dea6 — grpo-qwen3-30ba3b-4n8g-megatron-qa-nvfp4.sh accepts ModelOpt's Inserted 723 quantizers line as well as 723 TensorQuantizers found in model. The latter comes from mtq.print_quant_summary, which the Megatron quant worker only calls when it quantizes from scratch; when the cached quantized checkpoint (<model>_modelopt_<cfg-hash>, keyed on quant config + calibration settings, not on the ModelOpt version) exists it restores it and only the Inserted … line is printed.

torch per backend extra (pyproject.toml / uv.lock)

• 237a52c / 04b9dc9 — torch is pinned per extra, not globally. The base dependency is a range, torch>=2.11,<2.14 with torchvision>=0.26,<0.29; the vllm, mcore and automodel extras pin torch==2.13.0 / torchvision==0.28.0, and the sglang and trtllm extras pin torch==2.11.0 / torchvision==0.26.0. The extras are declared mutually conflicting under [tool.uv], so uv resolves each one in its own fork and the lockfile carries both torch versions. The driver venv (no extra) resolves 2.13.0 today; the <2.14 ceiling keeps a future relock from silently moving it. This is what makes sglang-kernel 0.4.5 and TRT-LLM usable again: both ship extensions built against torch 2.11 and failed at import on 2.13 (undefined symbol _ZNK2at10TensorBase14const_data_ptr… / _ZNR5torch7Library4_def…). Verified with uv export --frozen --extra <name> per fork and by the sglang / TRT-LLM recipes on both SKUs.
• 94dd20b — CUDA IPC handles from torch 2.13 open in torch 2.11 inference venvs. With the trainer on 2.13 and sglang / TRT-LLM on 2.11, every colocated refit died at the first weight with received sharable handle from a future version of torch that this version does not know how to handle. torch 2.13 ([CUDA] Fix CUDA IPC deserialization mismatch with expandable_segments on FABRIC_HANDLE pytorch/pytorch#179618) bumped the SHAREABLE_HANDLE_VERSION byte in CUDACachingAllocator::shareIpcHandle from 2 to 3 for the expandable-segment header; the 'c' (plain cudaMalloc) payload is byte-identical, and the consumer only checks version <= own. New nemo_rl/utils/cuda_ipc.py::normalize_cuda_ipc_handle rewrites the version byte of 'c' handles to 2 and leaves 'e' handles (whose format did change) and raw 64-byte handles alone; applied in rebuild_cuda_tensor_from_ipc and in the sglang _rebuild_cuda_tensor_modified patch. The shim never consults the consumer: it lowers the version byte of any 'c' handle above 2 and leaves 'e' handles alone (see Notes). 13 unit tests.
• 10e1a86 — DeepSeek V4 refit: model-level post-load hook deferred to the end of the layerwise refit. vLLM 0.29's DeepseekV4ForCausalLM.load_weights now ends with self.process_weights_after_loading(), which recomputes the first layer's hyper-connection broadcast from hc_attn_fn; NeMo-RL streams a refit through buffer-sized load_weights calls while vLLM's layerwise reload still holds that parameter on the meta device, so the 16-node DeepSeek V4 Flash recipe died at its first refit with Cannot copy out of meta tensor. deepseek_v4_fp8.prepare_refit now shadows the hook on the instance, finalize_refit lifts it and runs it once after finalize_layerwise_reload, and restore_refit lifts it on failure too. Unit-tested; see Known issues for the follow-on OOM this uncovered.
• 28beca5 — DeepSeek V4 refit: two more 0.29 changes handled. (a) 0.29 gives DeepseekV4ForCausalLM a packed_modules_mapping in which wkv is a shard of two fused modules (fused_wqa_wkv and fused_wkv_wgate); the fp8 module lookup resolved shards by leaf name alone, sent attn.wkv to a module that does not exist, and refit the weight unquantized. Shard names listed under more than one fused module are now left to the parent-aware DeepSeek V4 remap. (b) 0.29 split the layerwise-reload skip list into SKIP_TENSORS (stay off the meta device) and SKIP_LOAD_TENSORS (loader not wrapped or counted); 0.25.1 did both with one set. The immediately loaded expert tensors were therefore counted, vLLM ran process_weights_after_loading on each expert layer as soon as its last expert arrived, and finalize_refit converted the already kernel-layout weights and scales a second time (verified with a probe: scales doubled twice per refit). prepare_refit registers the names in both sets and returns a per-set SkipNames record. Both were caught by running the vllm-marked unit tests that main added with feat: add DeepSeek V4 Flash GRPO support - Automodel path #3595/fix(vllm): support BF16 TRTLLM NCCL reshard refit #3659 against 0.29; those tests are ported to 0.29's API in the same commit (WeightsMapper.get_rename_mapper, Fp8MoEMethod reading the MoE config, the import-target test skipping try/except ImportError fallbacks).
• 31dc3a8 — torch 2.13 removed the named-tensor API. main's Mooncake TQ checkpoint adapter (feat(data-plane): add Mooncake storage checkpoints #3898) guarded its fast path with produced.names == (None, None); Tensor.names no longer exists on 2.13, so _physical_keys raised AttributeError (caught by test_tq_mooncake_checkpoint.py). The guard now treats a missing attribute as "no named dimensions". The same commit marks the tied-alias unit tests vllm and adds vLLM 0.29's ErrorResponse module to the chat-template wiring test's fake vLLM tree, so the L0 shards' no-vLLM pass over the vLLM test files is green again.
• main merged through 0aa7bf5 (Sept 24; the last merge is ci: Bump Megatron-Bridge to 1f8873bb... (manual) #4139, Megatron-Bridge 1f8873bb / Megatron-LM 6a3660905, which also moves the mcore venv's flashinfer to 0.6.18.post1). Notable merges: chore(deps): bump mooncake and wandb #4179 (mooncake 0.3.13.post1 / wandb 0.30.0), build: exclude PyAV from shipped container #4184 (PyAV exclusion) and feat: add end-to-end TE NVFP4 training with per-token vLLM rollout #3566 (TE NVFP4 end-to-end; on main it bumped vLLM to 0.26.0 / flashinfer 0.6.14 / cutlass-dsl 4.6.0, moved TE to the 2.18.0 PyPI wheels and ModelOpt to 43fd41a58, and refactored the fp8 module lookup into quantization/utils.py). Resolution: this branch's vLLM 0.29.0 / flashinfer 0.6.18 / cutlass-dsl 4.6.2 pins stay (0.29 pins quack-kernels==0.6.4, so feat: add end-to-end TE NVFP4 training with per-token vLLM rollout #3566's 0.6.1 pin is not taken); TE 2.18.0 and ModelOpt 43fd41a58 (a release cherry-pick that carries Direct support for megatron bridge checkpoints #2054 and still excludes ray.sub requires submission from NeMo RL home directory, blocking external workflow organization #1550's GroupedQuantizer) come from main; the ambiguous-shard fix from 28beca5 is ported into the new resolver. feat: add fp32 LM head toggle #4096's nemotron_h.py fp32 LM head patch anchors occur exactly once in 0.29's copy and its anchor tests pass. The Sept 21–22 merges had no conflicts. The Sept 23 merges (refactor(dtensor): remove the DTensor v1 policy worker #4198 removes the DTensor v1 worker and its fsdp extra, dropped here with its torch pin; feat(Automodel): add MiniMax-M3 GRPO training support #4143 MiniMax-M3; feat(telemetry)!: instrument the single-controller path and propagate traces #4052 telemetry; and eleven smaller PRs) conflicted only in pyproject.toml / uv.lock. feat(Automodel): add MiniMax-M3 GRPO training support #4143's _patch_vllm_minimax_m3_topk_buffer_layout is removed rather than carried: it backports vllm-project/vllm@d1a8ba63 onto 0.26.0, its docstring says to drop it at vLLM ≥ 0.27, and 0.29 already has the head-major buf_htk layout. The relock added only feat(research): add flow_grpo, Flow-GRPO for Qwen-Image #3408's flow-grpo research dependencies. nemo-lens follows main to b0f977d with its aiohttp extra.

torch 2.13 / vLLM 0.29 fallout found by the GitHub CICD and the rerun nightly (0cc6fab, 8e97d91)

• 0cc6fab — Async checkpoint daemons join the training TCPStore instead of binding a port. torch 2.13's process-based DCP checkpointer (_async_process_executor) rendezvous its save daemons on a port rank 0 probes with get_free_port(); the port was re-taken before the daemon bound it and every DTensor async save died at the first checkpoint with DistNetworkError … EADDRINUSE (the L1 GRPO resume test, the converter round-trip test, and the llama3.2-1b fsdp2tp2 tq_mooncake nightly recipe at step 10). Surfaced by Automodel 72daceffa making daemon creation synchronous across ranks; main on torch 2.11 does not reach it. init_checkpointer now sets torch's DCP_USE_PREFIX_STORE=1 when MASTER_ADDR/MASTER_PORT are present, so the daemons attach to the process group's store under a prefix. Three unit tests.
• 0cc6fab — nvfp4_pertoken.py (from feat: add end-to-end TE NVFP4 training with per-token vLLM rollout #3566) no longer passes layer= to make_nvfp4_moe_kernel. vLLM 0.29 dropped the parameter (it takes per_token_activation); the new GB200 te_nvfp4_pertoken_quick recipe died with TypeError: unexpected keyword argument 'layer'; a unit test binds the rebuild's kwargs to the installed factory's signature.
• 0cc6fab / 8d1f98a — 2.long_generation_decode_vs_prefill.py gains --gpu-memory-utilization; the L1 test passes 0.7 on GB200 (at 0.8 the FlashInfer TRT-LLM BF16 MoE workspace no longer fit next to 0.29's KV cache) and keeps 0.8 on 80 GiB parts, where 0.7 leaves no KV cache at all. dfc65df adds --max-model-len (the test passes 16384): with chunked prefill off vLLM profiles a full 262k-token prefill; the H100 KV budget was left a 6 GiB margin the L1 shard fell through once, and is 31 GiB now.
• 8e97d91 — test_vllm_http_server ignores the usage.completion_tokens_details field vLLM 0.29 added to UsageInfo.
• af7d9e0 — Routed-experts capture for the per-token NVFP4 MoE goes through the router on 0.29. Router replay worked on 0.26 because NeMo-RL's _patch_vllm_moe_routed_experts_capture fires router.select_experts on the monolithic fused-MoE branch when the router carries a capture callback. 0.29's bind_routed_experts_capturer binds monolithic kernels to the experts object instead (the FlashInfer launch then fills routing_replay_out itself) and rejects kernels without in-kernel capture; the per-token method rebuilds its kernel on every refit, so that binding would be lost after the first weight update and replay would silently see all-zero routes. host_captured_experts_cls wraps the method's experts class to report no in-kernel capture, and _patch_vllm_routed_experts_capture_router_fallback makes the 0.29 binder fall back to router.set_capture_fn for such kernels instead of raising. Kernels that support in-kernel capture keep using it. 7 unit tests (anchor against the installed vLLM, idempotency, fail-closed, an executable check of the patched binder, the wrapper).
• ea439fb — MTP speculative decoding accepted nothing on 0.29. 0.29's default v2 model runner keeps the proposer as model_runner.speculator, not drafter; _get_drafter_model read only drafter, so the co-trained MTP head was never fed on refit and kept its dummy weights: nemotron3-super logged acceptance length 1.00 on every 0.29 lane (main 0.25.1: 4.1), generation fell from ~700 to ~220 tok/s, and the GB200 bf16 300-step release recipe no longer finished a step; no nightly golden reads acceptance. Both names are now resolved, a missing drafter warns once, and a trip-wire reads the installed runners' sources. Validated: H100 16n8g nemotron3-super passes with acceptance 3.0–3.8, ~2000 tok/s (455274835); the GB200 8n4g recipe now stops at NeMo-RL's TRT-LLM/MTP guard as on main.
• a204116 / c42c28b — review rounds 1–2: automodel marks on the new checkpoint tests; a per-token test bound to the installed make_nvfp4_moe_kernel signature; the import-target walk covers the Dynamo and ModelOpt trees; the DeepSeek V4 tests pin vLLM's two skip sets; normalize_cuda_ipc_handle is annotated, rewrites only the verified version 3 and raises on anything newer; test_torch_split_shims_are_still_needed names both torch-split shims for removal once no extra pins torch < 2.13; stale comments, two docs mentions of 0.25.1, and the decode-vs-prefill nvidia-smi fallback corrected.
• Also in the diff: docs/guides/dynamo-generation.md and docs/guides/grpo-audio-visual.md version / pin one-liners, and pyrefly.toml includes for vllm_quant_moe_amax.py and cuda_ipc.py.

Known issues not fixed here

Issue	Recipes	Status / proposed fix
grpo-deepseek-v4-flash-0731-16n8g-automodel-cp8ep128 (H100) — fails on the trainer side, as on main	1	With the 0.29 refit fixes the refit and step-1 generation complete and the run dies in DTensorPolicyWorkerV2.get_logprobs() with DeepEP error: timeout (dispatch CPU), the same point and error as main at this PR's earlier base (446267140); main's Sept 20 / 21 nightlies fail it earlier, at engine start.
grpo-qwen3-30ba3b-4n4g-megatron-te-nvfp4-pertoken-quick (GB200) — resolved; record of a 0.29 behaviour change	1	Recipe landed on Sept 19 (#3566); three 0.29 changes hit it (details in the 0cc6fab and af7d9e0 bullets above): the dropped layer= kwarg; capture bound to the experts object, which this method's per-refit kernel rebuild would silently lose; and, with the in-kernel replay buffer attached (never the case on 0.26), CUDA error: an illegal memory access in the FlashInfer TRT-LLM NVFP4 MoE kernel at the first post-refit generation (450220649). With the capture routed through the router the recipe passes 20/20 (450730474) at main-level goldens with router-replay validation on. main passes it on Sept 20 and aborts on Sept 21 in the HybridEP handle import (see W4A4 below).
Colocated Megatron + sglang dies at the step-10 async checkpoint — AttributeError: module 'torch.multiprocessing.reductions' has no attribute '_rebuild_cuda_tensor_original'	2 (H100 2n8g and GB200 1n4g megatrontp1_sglang)	Pre-existing on main (Sept 20 and 21 fail identically); refits 1–10 through the torch-2.13→2.11 IPC path work. Other sglang recipes pass or match main.
NVFP4 W4A4 real-quant on GB200 — grpo-qwen3-30ba3b-4n4g-megatron-qa-nvfp4-w4a4-real	1	Two failure modes, both seen on main too. (a) Ranks abort at step 1 in DeepEP's HybridEP cuMemImportFromShareableHandle (SIGABRT); main's GB200 nightly aborts identically every day since Sept 21. (b) When it runs, 2/2 steps pass reward and accuracy with js_divergence_error[2] at the 0.007 bound (0.00694 / 0.00718 vs main 0.0039–0.0044). The H100 W4A16 recipe matches main, so (b) is the NVFP4 activation-quant kernels on Blackwell.
nixl-cu13 1.3.0 nixl_ep extension built against torch 2.11 — undefined symbol …materialize_cow_storage…	none	Non-fatal: vLLM's has_nixl_ep() probe disables the optional nixl_ep all2all backend, but the traceback is in every vLLM worker log and nemo-ci's error extractor blames it for unrelated failures. Bump nixl to a torch-2.13 build in a follow-up.
grpo-glm5.2-64n8g-megatron-6K-colocated (H100 release) — AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma'	1	Pre-existing on main since #3566 (Sept 20) moved the Megatron venv's cutlass-dsl to 4.6, which dropped cute.core.ThrMma; the nvidia-cudnn-frontend==1.25.0 override (same on main) still spells it that way in the CuTe DSA kernels Megatron-LM imports. Follow-up PR: bump cudnn-frontend to 1.30.0 (requires cutlass-dsl ≥ 4.6.2).
grpo-nemotron3-super-120BA12B-8n4g-megatron (GB200) — Unquantized FlashInfer TRTLLM refit does not yet support a co-trained MTP drafter	1	Same as main (Sept 20 and 21). Earlier 0.29 lanes passed this guard only because the drafter was not found (the ea439fb bug); with it found again the guard stops the recipe at the first refit as on main (455110829).
mopd-qwen3-1.7b-3n4g-megatron-pack-single-controller-fullvocab (GB200) — never starts: pyxis cannot extract the sandbox squashfs	1	Only USES_SANDBOX=1 recipe in the GB200 nightly; main Sept 20 and 21 fail identically at Slurm step 0. nemo-ci provisioning on oci_hsg. The H100 3n8g twin passes.
Goldens at their bounds	4	llama3_2_1b_instruct_1n4g_fsdp2tp1_v3 (GB200) step time 14.9 / 14.9 / 15.0 s on three runs vs < 14.5 (main 13.7–15.1); the whole gap is the colocated refit (prepare_for_generation 3.7 vs 3.0 s; +0.3 s on the H100 twin), generation at parity. Follow-up: profile the 0.29 refit path. clevr_8n4g_megatron_generation_v1 (GB200) final accuracy 0.59–0.62 vs > 0.6 (main 0.59–0.64). mxfp8_rollouts_flashinfer (GB200) prob-error median / step time over their bounds, main too. sft_gpt_oss_20b (H100) grad_norm[50] > 10.0 at 9.56–10.12 (main 9.83–10.2).
Gym server spin-up flake	3	A Gym server exits during spin-up with no output and the NemoGym actor loses the GCS; main Sept 20 hit it on both mopd recipes. All three passed on retry.

Notes for reviewers

• Two torch versions in one lockfile. Each extra pins its own torch inside a uv conflict fork; uv sync --extra sglang and uv sync --extra trtllm produce torch-2.11 venvs, everything else torch 2.13. The base range torch>=2.11,<2.14 is what the driver venv resolves from; the ceiling is deliberate so a relock cannot move the driver past what the inference venvs can talk to. When sglang-miles / TRT-LLM move to torch 2.13, drop their pins; the trip-wire test names the shims to remove.
• The IPC shim only touches 'c' handles, and never consults the consumer. It lowers the version byte of a version-3 'c' handle to 2 (the one pair whose payload is verified identical) and raises on any newer version; test_torch_split_shims_are_still_needed in test_dependency_pins.py fails the day no extra pins torch below 2.13, naming this shim and _has_no_named_dims for removal. Expandable-segment ('e') handles are left alone: a torch 2.11 consumer (sglang / TRT-LLM, colocated) rejects them, so keep expandable_segments:True off for those two backends; no shipped sglang / TRT-LLM recipe sets it.
• Lock parity with main. Since be440de the lock is at or above main's resolution except for the deliberate pins, so the lock diff exceeds what the dependency bullets suggest.
• Async checkpoints and DCP_USE_PREFIX_STORE. The flag is torch's own opt-in (torch/distributed/checkpoint/_async_process_executor.py); it is set only when the training rendezvous variables are present, and an explicit value in the environment wins. The daemons then need no free port at all, which is also the right behaviour on hosts where the ephemeral range is contended.
• Gym. docker/Dockerfile still forces Gym's server venvs to NEMO_GYM_VLLM_VERSION=0.25.1, which works against this branch (all Gym-path recipes pass). Gym is being bumped separately in build: bump vllm to 0.29.0 Gym#3505; a follow-up RL PR will bump the Gym submodule and that ARG together once it merges.
• instanttensor is the one uv override that drops a dependency on one arch: 0.2.0 ships aarch64 wheels, the skip just keeps an unused loader out of the GB200 image; lifting the marker enables load_format="instanttensor" there.
• mm_device_do_normalize=False flips a default only Qwen2-VL / Qwen2.5-VL opt into in 0.29, restoring the 0.25 processor-side normalize.
• ModelOpt stays at main's 43fd41a58; this PR adds the "do not move past ray.sub requires submission from NeMo RL home directory, blocking external workflow organization #1550" note in pyproject.toml (GroupedQuantizer, see the e991336 bullet).
• The vllm-marked unit suite (462 items) was re-run after every main merge.

Test plan

Nightlies. Full nightly suites on both SKUs on the final lock (be440de; the later commits touch a functional-test script, the per-token NVFP4 capture path, tests, comments, docs, the trtllm venv's cuda-bindings pin (ab6b3f2, validated separately below), and the Sept 23 main merge, which only added research dependencies to the lock). H100 lane 69151691 and GB200 lane 69183689, both complete. Both are compared recipe by recipe with main's Sept 20 scheduled nightly (68861701, RL 880a37a — exactly the merge point of d104085, vLLM 0.26.0) and Sept 21 (69003177, RL 612d527, one commit behind the 094a932 that 51d7ee4 merged). Earlier full passes (fe6945a, 8e97d91) found the DeepSeek V4, torch-2.13 and fla-core items.

Targeted validation of the fixes made after 8e97d91. EADDRINUSE fix, H100 69074478: llama3_2_1b_instruct_1n8g_fsdp2tp2_…_tq_mooncake saves every checkpoint and reaches step 470/500 at the recipe's 180-min limit, exactly where main stops. fla-core relock, GB200 69076114: qwen3_5_35ba3b_6n4g_async_1off_bf16_trtllm trains 16 steps (gen-KL 0.0010–0.0012) where every earlier 0.29 run died at step 1, then hits the oci_hsg Disk quota exceeded at step 17 (450537401). trtllm cuda-bindings pin, GB200 69220858: qwen3_1_7b_2n4g_fsdp2_trtllm passes 10/10 (gen-KL 0.0010, 22 s/step; 451393519). Per-token NVFP4 capture path, GB200 69135677: te_nvfp4_pertoken_quick passes 20/20 with main-level goldens. clevr_8n4g_megatron_generation_v1 rerun (GB200 69074482): 50/50 steps in main's 0.59–0.64 accuracy band.

Release suites and reruns on dfc65df. The 11 "PR fails, main passes" nightly recipes were re-run once the oci_hsg quota was cleared (H100 lane 69699038, GB200 69688016): 7 pass, see that table. The release suites ran on H100 69695985 (17 pass / 11 fail of 28) and GB200 69685830 (11 / 4 of 15), vs main's weekly release runs of Sept 19 (17 / 10 and 10 / 5; vLLM 0.25.1) and Sept 5. Per-recipe table in this comment: 28 pass on both; 11 fail on both with main's signature; 2 have no main control (a new recipe with no cached model; a DeepEP timeout whose main runs bounced at checkout); 2 fail only here: the MTP acceptance regression fixed in ea439fb, and the distillation dynamic-batch validation/accuracy[20] > 0.1 golden: 0.066 / 0.0625 here (one seed) vs main 0.117–0.168 over four runs incl. a main-head control (69775657). The recipe collapses to max-length generations from step 7 on both branches, so the golden grades fully truncated outputs; unresolved, re-evaluate once the recipe's collapse is fixed.

SKU	Recipes	Pass	Fail	Running	PR passes, main fails	PR fails, main passes
H100 (cw_dfw)	134	109	25	0	1	5
GB200 (oci_hsg)	47	32	15	0	1	6

GB200 jobs killed by the full oci_hsg Lustre quota are marked as infrastructure.

GitHub CICD (L0 unit shards incl. the no-vLLM and --vllm-only passes over the vLLM tests, L1 functional tests on both SKUs) is green on dfc65df (run 36000890507) and running on the current head ea439fb (36078855237). On 11bd0e7 two shards were red: H100 L1_Functional_Tests_Other_1 (decode-vs-prefill found no KV cache memory at 0.8; with dfc65df it has 31 GiB) and gb200_L1_Functional_Tests_Megatron_1 (HF 429 on a model download). Nothing this branch pins changed after be440de, so no new full nightly. The one shard the first Sept 23 merge turned red, L0_Unit_Tests_Environments, was main's own breakage from #4198 and is fixed by #4252, which is merged here. One H100 shard, L1_Functional_Tests_Megatron_4, is flaky on this branch and on main's runners: in two of five runs grpo_megatron_generation_colocated.sh trains its 2 steps and the driver then exits non-zero without output; a plain re-run passes, and the GB200 copy of the shard passes every time.

Results — nightly, final lock be440de (both lanes complete)

PR test pipelines: H100 69151691 · GB200 69183689. main controls: the Sept 21 scheduled nightly 69003177 (RL 612d527, one commit behind the 094a932 that 51d7ee4 merged) and the Sept 20 one 68861701 (RL 880a37a, exactly the merge point of d104085; vLLM 0.26.0 on both).

• H100: 109 pass · 25 fail · 0 running
• GB200: 32 pass · 15 fail · 0 running

PR fails, main passes (11) — re-run on dfc65df: 7 pass, 2 goldens at their bound, 2 main-shared HybridEP aborts

Test	PR	main Sept 21	main Sept 20	Notes
llm_grpo_llama3_2_1b_instruct_1n4g_fsdp2tp1_v3 GB200	❌ failed 165 min	✅	✅	❌ Rerun on dfc65df (454617596): 198 min, 500/500 steps, again only mean(total_step_time) < 14.5 misses, at 14.91; third run 455221465: 15.01 (refit phase, see Known issues). Marginal step-time golden. 500/500 steps, correctness checks pass; mean(total_step_time, -6, -1) < 14.5 misses at 14.9–15.1 s on every PR lane (14.92 on 8e97d91, 15.07 on be440de). PR runs 14.8–15.1 s; main 13.9 s on Sept 20 (pass), 13.7–15.1 s over Sept 8–9. About 5 % on a 1-GPU-per-node recipe whose generation share is unchanged; see Known issues.
llm_grpo_nanov3_30BA3B_4n4g_megatron_generation_noncolocated_mxfp8_rollouts GB200	❌ failed 93 min	✅	❌	✅ Passes on the dfc65df rerun (454617598, 90 min). Earlier failure: Single-step prob-error outlier (Megatron generation, MXFP8 rollouts).
llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_megatron_single_controller_sync GB200	❌ failed 377 min	❌	✅	✅ Passes on the dfc65df rerun (454617605, 218 min). Earlier failure: Infra: oci_hsg CI Lustre quota.
llm_grpo_qwen3_1_7b_2n4g_fsdp2_trtllm GB200	❌ failed 251 min	✅	✅	✅ Passes on the dfc65df rerun (454617602, 90 min). Earlier failure: Regression from the transitive lock upgrade (be440de), fixed in the follow-up commit.
llm_grpo_qwen3_30ba3b_4n4g_megatron_qa_nvfp4_w4a4_real GB200	❌ failed 152 min	❌	✅	❌ Rerun on dfc65df (454694390): HybridEP cuMemImportFromShareableHandle abort at step 1 on both attempts (454617603); main's GB200 nightly fails it the same way every day since Sept 21, incl. Sept 24 (454433524). Pre-existing on main. MegatronQuantPolicyWorker ranks abort at step 1 in DeepEP's HybridEP buffer allocation: cuMemImportFromShareableHandle fails in NVLCoordinator::open_handles_from_other_ranks and the process SIGABRTs before any golden. main fails identically on Sept 21 (449569744), Sept 19 (447326502) and Sept 18 (445908944); it passed on Sept 20 and 16; our retry (449824224) aborted the same way. Node/fabric-dependent. When the recipe does run on 0.29 (two earlier samples) it trains 2/2 with reward and accuracy passing and js_divergence_error[2] at the 0.007 bound (0.00694 / 0.00718; main 0.0039–0.0044) — see Known issues. (main Sept 21: Signal/OOM-Kill.)
llm_grpo_qwen3_30ba3b_4n4g_megatron_te_nvfp4_pertoken_quick GB200	❌ failed 246 min	❌	✅	❌ Rerun on dfc65df (454694384): HybridEP cuMemImportFromShareableHandle abort at step 1 on both attempts (454617604); main's GB200 nightly fails it the same way every day since Sept 21, incl. Sept 24 (454433525). Regression (new recipe from #3566); fixed in 0cc6fab + af7d9e0, validated. make_nvfp4_moe_kernel() got an unexpected keyword argument 'layer' (vLLM 0.29 dropped the parameter) is fixed in 0cc6fab. With that fix (450220649) the recipe refits and dies in the first generation with CUDA error: an illegal memory access in the FlashInfer TRT-LLM NVFP4 MoE kernel. 0.29 also changed how routed experts are captured for monolithic kernels (bound to the kernel's experts object, which this method rebuilds on every refit, so router replay would have silently lost its routes after the first refit); af7d9e0 routes the capture through the router hook NeMo-RL already patches in, as on 0.26, so the kernel launches without a replay buffer. With both fixes the recipe passes 20/20 (450730474: gen-KL 0.0083, median prob-error 1.060 — main Sept 20: 0.0082 / 1.058; router replay validation on). main passes on Sept 20 and aborts on Sept 21 in the HybridEP handle import (see W4A4). (main Sept 21: Uncaught Exception.)
llm_grpo_gemma3_1b_it_1n8g_fsdp2tp1_tq_simple H100	❌ failed 141 min	✅	✅	✅ Passes on the dfc65df rerun (454685359, 109 min). Earlier failure: Step-time golden under lane contention.
llm_grpo_llama3_2_1b_instruct_1n8g_megatron_temp0_8_topp0_9_topk50_tq_mooncake H100	❌ failed 184 min	❌	✅	✅ Passes on the dfc65df rerun (454772599, 157 min, 500/500 steps). Earlier failure: Borderline recipe duration (main too).
llm_mopd_qwen3_1_7b_3n8g_megatron_pack H100	❌ failed 48 min	✅	❌	✅ Passes on the dfc65df rerun (454685369, 14 min). Earlier failure: Completed; Slurm TIMEOUT at teardown (launch contention).
llm_sft_gpt_oss_20b_1n8g_fsdp8ep8_automodel H100	❌ failed 42 min	✅	❌	❌ Rerun on dfc65df (454685362): 14 min, every check passes except grad_norm[50] > 10.0 at 9.56. Marginal golden (SFT, no vLLM). grad_norm[50] > 10.0 misses at 9.88; PR runs 9.74–10.12, main 9.83–10.20 (fails Sept 20 at 9.83, passes Sept 21). Inside this recipe's noise on both branches.
llm_sft_qwen3_0_6B_1n8g_megatron_yarn_128k H100	❌ failed 48 min	✅	✅	✅ Passes on the dfc65df rerun (454685368, 14 min). Earlier failure: Completed; Slurm TIMEOUT at teardown (launch contention).

PR fails, main had no usable result (2)

Test	PR	main Sept 21	main Sept 20	Notes
llm_dpo_qwen2_5_math7b_1n8g_megatron_fused_linear_logprobs H100	❌ failed 48 min	❌	❌	Pre-existing on main. DPO, no vLLM; train/accuracy[10] >= 0.5 at 0.406. main bounced at checkout on Sept 20 and 21; fails on Sept 8 and 9.
llm_grpo_nanov3_30ba3b_3n8g_megatron_dynamo_swe1 H100	❌ failed 47 min	❌	❌	Pre-existing on main. ValueError: NeMo Gym does not support generation backend 'dynamo'; main fails whenever it gets past checkout (Sept 8, 9; bounced Sept 20/21).

PR fails, main fails (27)

Test	PR	main Sept 21	main Sept 20	Notes
llm_dpo_nanov3_30B3AB_1n4g_fsdp4ep4_automodel GB200	❌ failed 254 min	❌	❌	Pre-existing on main. DPO, no vLLM. mean(total_step_time, -5, -1) < 5 at 17.3 s; main Sept 20 / 21 fail the same check at 16.5 / 16.1 s plus loss goldens.
llm_grpo_moonlight_16ba3b_4n4g_megatron GB200	❌ failed 29 min	❌	❌	Pre-existing on main. KeyError: decoder.layers.N.self_attention.linear_q_proj.layer_norm_weight from model not in checkpoint at Megatron load; main Sept 20 and 21 fail identically.
llm_grpo_nanov3_30BA3B_4n4g_megatron_generation_colocated_reshard_async_gym GB200	❌ failed 257 min	❌	❌	Pre-existing on main. Megatron generation; Gym rejects the rollouts (pydantic ValidationError … NeMoGymChatCompletionMessageForTraining) until the time limit. main Sept 20 and 21 fail identically. On the be440de lane the node died first (CUDA driver error: uncorrectable NVLink error).
llm_grpo_nanov3_30BA3B_4n4g_megatron_generation_noncolocated_mxfp8_rollouts_flashinfer GB200	❌ failed 256 min	❌	❌	Pre-existing on main. Now trains 10/10 (main's #3630 fixed the earlier abort) and misses two goldens: token_mult_prob_error[10] < 2.0 (a single-step spike to 22.8; median 1.32 passes, gen-KL 0.042 passes) and median(total_step_time) < 48 (57.2 s; 55.9 s on the be440de lane, where the prob-error checks passed). main Sept 20 fails the same recipe on median(token_mult_prob_error) < 1.5 (1.56) and the same step-time bound (55.1 s); Sept 15–19 fail too. Megatron generation with FlashInfer MXFP8 rollouts; vLLM is not in the loop.
llm_grpo_nanov3_30ba3b_4n4g_megatron_qa_nvfp4_w4a16_real GB200	❌ failed 251 min	❌	❌	Pre-existing on main. Completes 1/1 and misses gen_kl_error[1] < 0.003 (0.0116) and max(token_mult_prob_error) < 1.05 (1.094); main Sept 20 misses the same two checks with the same values (0.0112 / 1.094).
llm_grpo_nemotron3_super_120BA12B_8n4g_megatron GB200	❌ failed 44 min	❌	❌	Changed, not vLLM. On main the recipe dies at its first refit (Unquantized FlashInfer TRTLLM refit does not yet support a co-trained MTP drafter; Sept 20 and 21). On 0.29 it refits and trains all 5 steps, then the Megatron dist-ckpt save host-OOMs the 4-GPU nodes. See Known issues.
llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_fsdp2tp1_sglang GB200	❌ failed 170 min	❌	❌	Pre-existing on main. Colocated sglang on GB200 reaches step ~295/450 and hits the 150-min Slurm limit; main Sept 20 and 21 hit the same limit.
llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_megatrontp1_sglang GB200	❌ failed 366 min	❌	❌	Pre-existing on main. Colocated sglang dies at the step-10 async checkpoint on torch.multiprocessing.reductions._rebuild_cuda_tensor_original; main Sept 20/21 fail identically (same step, same attribute) and then sit until the Slurm limit.
llm_mopd_qwen3_1_7b_3n4g_megatron_pack_single_controller_fullvocab GB200	❌ failed 215 min	❌	❌	Infra (main too). Slurm step 0 fails before Ray starts: pyxis failed to create container filesystem extracting the sandbox squashfs on oci_hsg. main Sept 20 and 21 fail identically; the H100 3n8g twin passes.
llm_dapo_nanov3_5_30BA3B_4n8g_automodel H100	❌ failed 12 min	❌	❌	Pre-existing on main. ValueError: Couldn't instantiate the backend tokenizer (HF asset not in the CI cache); main Sept 20 and 21 fail identically.
llm_dpo_mistral_nemo_instruct_2407_1n8g_fsdp2tp8_actckpt_long H100	❌ failed 76 min	❌	❌	Pre-existing on main. DPO, no vLLM; train/loss[1] < 0.70 at 0.7087 — main Sept 20 fails on the identical value (0.7087241); Sept 21 could not submit to Slurm.
llm_dpo_nanov3_30B3AB_1n8g_fsdp8ep8_automodel_v2 H100	❌ failed 57 min	❌	❌	Pre-existing on main. DPO, no vLLM; mean(total_step_time, -5, -1) < 5 at 5.32 s; main Sept 20 fails the same check (5.20 s) plus two loss goldens; Sept 21 could not submit to Slurm.
llm_grpo_deepseek_v4_flash_0731_16n8g_automodel_cp8ep128 H100	❌ failed 37 min	❌	❌	vLLM-side regressions fixed (10e1a86, 28beca5); still fails on the trainer side like main. The refit and step-1 generation complete; the run dies in DTensorPolicyWorkerV2.get_logprobs() with DeepEP error: timeout (dispatch CPU), the same point and error as the main control at this PR's earlier base (446267140). main's Sept 20 / 21 nightlies (vLLM 0.26.0) fail the recipe earlier, at vLLM engine start (RuntimeError: Worker failed with error '' ×11, then the step is cancelled).
llm_grpo_llama3_2_1b_instruct_1n8g_fsdp2tp2_temp0_8_topp0_9_topk50_tq_mooncake H100	❌ failed 215 min	❌	❌	Regression, fixed in 0cc6fab. Dies at the step-10 async checkpoint: torch 2.13's process-based DCP checkpointer rendezvous the save daemons on a get_free_port() probe that is re-taken before the daemon binds it (DistNetworkError … EADDRINUSE). Same crash in the GitHub L1 GRPO checkpoint test and the converter test on this head. init_checkpointer now opts the daemons into DCP_USE_PREFIX_STORE=1 so they join the training TCPStore instead of binding a port. Validated on the fixed head (450067839: no EADDRINUSE, every 10-step checkpoint saved, 470/500 steps until the recipe's 180-min Slurm limit — exactly where main (torch 2.11, never reaches the bug) stops on Sept 20 and 21). What remains is that pre-existing time limit.
llm_grpo_moonlight_16ba3b_4n8g_megatron H100	❌ failed 17 min	❌	❌	Pre-existing on main. KeyError: decoder.layers.N.self_attention.linear_q_proj.layer_norm_weight from model not in checkpoint at Megatron load; main Sept 20 and 21 fail identically.
llm_grpo_moonlight_16ba3b_4n8g_megatron_fp8_e2e H100	❌ failed 12 min	❌	❌	Pre-existing on main. main Sept 20 and 21 fail this recipe at the same point (Megatron load).
llm_grpo_moonlight_16ba3b_4n8g_megatron_tq_simple H100	❌ failed 38 min	❌	❌	Pre-existing on main. KeyError: decoder.layers.N.self_attention.linear_q_proj.layer_norm_weight from model not in checkpoint at Megatron load; main Sept 20 fails identically, Sept 21 bounced at checkout.
llm_grpo_qwen2_5_math_1_5b_instruct_2n8g_megatrontp1_sglang H100	❌ failed 42 min	❌	❌	Pre-existing on main. Colocated sglang dies at the step-10 async checkpoint on torch.multiprocessing.reductions._rebuild_cuda_tensor_original; main Sept 20/21 fail identically (same step, same attribute) and then sit until the Slurm limit.
llm_grpo_qwen3_30ba3b_4n8g_megatron_qa_nvfp4 H100	❌ failed 39 min	❌	❌	Pre-existing on main. Fake-quant NVFP4 ranks abort at step 1 (Fatal Python error: Aborted, cumem_allocator.cpp on main); main Sept 20 fails identically, Sept 21 bounced at checkout.
llm_grpo_qwen3_5_35ba3b_2n8g_automodel_ep16 H100	❌ failed 27 min	❌	❌	Pre-existing on main. RuntimeError: DeepEP error: timeout (dispatch CPU) at step 1; main Sept 20 and 21 fail identically.
llm_grpo_qwen3_8_27b_2n8g_megatron_tp4pp2cp2 H100	❌ failed 17 min	❌	❌	Pre-existing on main. Dies in transformers at model load; main Sept 20 and 21 fail identically.
llm_ppo_qwen2_5_1_5b_gsm8k_2n8g_megatron_valuetp2sp_dynbatch_noncolocated_async_single_controller H100	❌ failed 40 min	❌	❌	Pre-existing on main. Config validation: SingleController checkpointing with a replay-checkpoint-capable sampler requires checkpointing.save_period; main Sept 20 and 21 fail identically.
llm_prorlv2_qwen2_5_math_1_5b_instruct_1n8g_fsdp2tp1_v2_tq_mooncake H100	❌ failed 185 min	❌	❌	Pre-existing on main. Slurm time limit at step 376/450 (refits healthy); main Sept 20 and 21 hit the same limit.
vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_1n8g_megatron_tp8ep8_v1 H100	❌ failed 36 min	❌	❌	Pre-existing on main. CUDA OOM at step 1; main Sept 20 and 21 fail identically.
vlm_vlm_grpo_nemotron_omni_30ba3b_mmpr_4n8g_megatron_tp8ep16_v1 H100	❌ failed 45 min	❌	❌	Pre-existing on main. AssertionError: defer_fp32_logits must be True if logprob_chunk_size is set; main Sept 20 and 21 fail identically.
vlm_vlm_grpo_qwen3_5_35ba3b_geo3k_2n8g_automodel_ep16 H100	❌ failed 43 min	❌	❌	Pre-existing on main. RuntimeError: DeepEP error: timeout (dispatch CPU) at step 1; main Sept 21 (Sept 20 bounced at checkout) fail identically.
vlm_vlm_grpo_qwen3_5_35ba3b_geo3k_2n8g_automodel_ep16_tq_simple H100	❌ failed 22 min	❌	❌	Pre-existing on main. RuntimeError: DeepEP error: timeout (dispatch CPU) at step 1; main Sept 21 (Sept 20 bounced at checkout) fail identically.

PR passes, main fails (2)

• H100 (1): llm_dapo_gemma4_e2b_it_1n8g_fsdp2_automodel
• GB200 (1): llm_sft_gpt_oss_20b_1n4g_fsdp4ep4_automodel

Pass on both PR and main: 139 (H100 108, GB200 31); the list is in a comment below to keep this description under GitHub's size limit.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yfw]

/ok to test 037c678

[yfw]

/ok to test 6ec3dd6

[yfw]

/ok to test be440de

[copy-pr-bot]

/ok to test be440de

@yfw, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

[yfw]

/ok to test be440de

[yfw]

/ok to test 8d1f98a

[yfw]

/ok to test af7d9e0

[yfw]

Recipes that pass on both this PR (final lock be440de: H100 lane 69151691, GB200 lane 69183689) and on main's Sept 20 / 21 nightlies (139):

H100 (108)

• llm_dapo_gemma4_12b_it_2n8g_fsdp2_automodel
• llm_distillation_qwen3_1_7b_1n8g_megatron_qa_nvfp4
• llm_distillation_qwen3_32b_to_1_7b_base_1n8g_fsdp2tp1_v1
• llm_distillation_qwen3_32b_to_1_7b_base_1n8g_megatron_tp2pp2cp2_pack
• llm_dpo_llama3_1_8b_instruct_4n8g_megatrontp2pp2_quick
• llm_dpo_llama3_2_1b_instruct_1n8g_fsdp2tp1_v2
• llm_gdpo_qwen2_5_1_5b_1n8g_gym_multireward
• llm_grpo_cispo_mm1_async_lag1_highoffpolicy_qwen3_30ba3b_3n8g_megatron_cispo
• llm_grpo_deepscaler_1_5b_16K
• llm_grpo_deepscaler_1_5b_24K
• llm_grpo_deepscaler_1_5b_8K
• llm_grpo_deepscaler_1_5b_8K_tq_simple
• llm_grpo_gemma3_1b_it_1n8g_fsdp2tp1
• llm_grpo_gspo_deepscaler_1_5b_8K
• llm_grpo_gspo_deepscaler_1_5b_8K_tq_simple
• llm_grpo_llama3_1_8b_instruct_1n8g_megatron_fp8_rollouts_v3
• llm_grpo_llama3_1_8b_instruct_1n8g_megatron_fp8_rollouts_v3_tq_simple
• llm_grpo_llama3_1_8b_instruct_2n8g_async_1off_single_controller_streaming2
• llm_grpo_llama3_1_8b_instruct_2n8g_fsdp2tp1_noncolocated
• llm_grpo_llama3_1_8b_instruct_2n8g_fsdp2tp1_noncolocated_tq_simple
• llm_grpo_llama3_1_8b_instruct_2n8g_megatron_fp8_e2e
• llm_grpo_llama3_1_8b_instruct_2n8g_megatron_fp8_e2e_tq_mooncake
• llm_grpo_llama3_2_1b_instruct_1n8g_fsdp2tp1_v3
• llm_grpo_llama3_2_1b_instruct_1n8g_fsdp2tp1_v3_tq_simple
• llm_grpo_llama3_2_1b_instruct_1n8g_fsdp2tp2_temp0_8_topp0_9_topk50
• llm_grpo_llama3_2_1b_instruct_1n8g_megatron
• llm_grpo_llama3_2_1b_instruct_1n8g_megatron_generation
• llm_grpo_llama3_2_1b_instruct_1n8g_megatron_generation_tq_mooncake
• llm_grpo_llama3_2_1b_instruct_1n8g_megatron_temp0_8_topp0_9_topk50
• llm_grpo_llama3_2_1b_instruct_1n8g_megatron_tq_simple
• llm_grpo_math_qwen3_30ba3b_megatron_tp4_32k
• llm_grpo_moonlight_16b_automodel_1n8g_ep8
• llm_grpo_nano_v2_12b_1n8g_megatron
• llm_grpo_nano_v2_12b_2n8g_fsdp2tp1
• llm_grpo_nanov3_30BA3B_1n8g_fsdp2_tq_mooncake_v2
• llm_grpo_nanov3_30BA3B_1n8g_fsdp2_v2
• llm_grpo_nanov3_30BA3B_2n8g_fsdp2_lora
• llm_grpo_nanov3_30BA3B_2n8g_megatron_generation_noncolocated_async_gym
• llm_grpo_nanov3_30BA3B_2n8g_megatron_lora
• llm_grpo_nanov3_30BA3B_2n8g_megatron_pack_cp
• llm_grpo_nanov3_30BA3B_2n8g_megatron_pack_cp_tq_simple
• llm_grpo_nemotron3_super_120BA12B_16n8g_automodel_ep8_v2
• llm_grpo_nemotron3_super_120BA12B_16n8g_megatron
• llm_grpo_qwen2_5_1_5B_4n8g_megatron_yarn_256k
• llm_grpo_qwen2_5_32b_32n8g_fsdp2tp8_actckpt_v3
• llm_grpo_qwen2_5_math_1_5b_instruct_1n8g_fsdp2tp1_v3
• llm_grpo_qwen2_5_math_1_5b_instruct_1n8g_fsdp2tp1_v3_tq_simple
• llm_grpo_qwen2_5_math_1_5b_instruct_1n8g_megatron_generation_noncolocated_single_controller_async_lag4
• llm_grpo_qwen2_5_math_1_5b_instruct_1n8g_megatron_generation_single_controller_async_lag4
• llm_grpo_qwen2_5_math_1_5b_instruct_1n8g_megatron_single_controller_sync
• llm_grpo_qwen2_5_math_1_5b_instruct_1n8g_megatron_single_controller_sync_tq_mooncake
• llm_grpo_qwen2_5_math_1_5b_instruct_2n8g_fsdp2tp1_sglang
• llm_grpo_qwen2_5_math_1_5b_instruct_2n8g_megatrontp1_sglang_noncolocated
• llm_grpo_qwen3_1_7b_1n8g_megatron_eagle3
• llm_grpo_qwen3_1_7b_1n8g_megatron_eagle3_tq_mooncake
• llm_grpo_qwen3_1_7b_1n8g_megatron_super_swe1
• llm_grpo_qwen3_1_7b_2n8g_megatron_super_rlhf
• llm_grpo_qwen3_1_7b_6n8g_megatron_super_rlvr
• llm_grpo_qwen3_30ba3b_10n8g_megatron_cp2_r3_async
• llm_grpo_qwen3_30ba3b_10n8g_megatron_cp2_r3_async_single_controller
• llm_grpo_qwen3_30ba3b_2n8g_megatron_fused_linear_logprobs
• llm_grpo_qwen3_30ba3b_4n8g_megatron_zmq_deltaweight_noncolocated
• llm_grpo_qwen3_30ba3b_8n8g_megatron_cp2_r3
• llm_grpo_qwen3_30ba3b_8n8g_megatron_cp2_r3_tq_simple
• llm_grpo_qwen3_30ba3b_thinking_swe1_16n8g_megatron_cp2_r3_async_gym
• llm_grpo_qwen3_5_35ba3b_2n8g_megatron_ep16tp2_fp8
• llm_grpo_qwen3_5_35ba3b_2n8g_megatron_ep16tp2cp2
• llm_grpo_qwen3_5_9b_1n8g_megatron
• llm_grpo_qwen3_5_9b_1n8g_megatron_fp8
• llm_grpo_qwen3_8B_base_1n8g_fsdp2_lora
• llm_grpo_qwen3_8B_base_1n8g_fsdp2_lora_tq_mooncake
• llm_grpo_qwen3_8b_base_1n8g_fp8_kvcache_megatron
• llm_grpo_qwen3_8b_base_1n8g_megatron_lora
• llm_grpo_qwen3_8b_base_dapo_2n8g_long_megatron_qa_nvfp4_w4a16
• llm_mopd_qwen3_1_7b_3n8g_megatron_pack_single_controller
• llm_mopd_qwen3_1_7b_3n8g_megatron_pack_single_controller_fullvocab
• llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_automodel_noncolocated
• llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_automodel_noncolocated_async
• llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_automodel_valuetp2sp
• llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_megatron_valuetp2sp_dynbatch
• llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_megatron_valuetp2sp_pp2cp2_pack
• llm_ppo_qwen2_5_1_5b_gsm8k_2n8g_megatron_valuetp2sp_dynbatch_noncolocated
• llm_ppo_qwen2_5_1_5b_gsm8k_2n8g_megatron_valuetp2sp_dynbatch_noncolocated_async
• llm_prorlv2_qwen2_5_math_1_5b_instruct_1n8g_fsdp2tp1_v2
• llm_sft_llama3_1_8b_1n8g_fsdp2tp1_lora
• llm_sft_llama3_1_8b_1n8g_fsdp2tp2
• llm_sft_llama3_1_8b_1n8g_fsdp2tp4_dynamicbatch
• llm_sft_llama3_1_8b_1n8g_megatron
• llm_sft_llama3_1_8b_1n8g_megatron_lora
• llm_sft_llama3_1_8b_1n8g_megatron_seqpack
• llm_sft_llama3_2_1b_1n8g_fsdp2tp1_v3
• llm_sft_llama3_2_1b_2n8g_hsdp
• llm_sft_nanov3_30BA3B_2n8g_fsdp2
• llm_sft_nanov3_30BA3B_2n8g_fsdp2_lora
• llm_sft_qwen2_5_32b_4n8g_fsdp2tp8sp_actckpt_v3
• llm_sft_qwen2_5_math7b_1n8g_megatron_fused_linear_logprobs
• llm_sft_qwen2_5_math7b_2n8g_megatron
• vlm_vlm_grpo_gemma4_e4b_geo3k_1n8g_automodel
• vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_1n8g_automodel_ep8_v2
• vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_2n8g_megatron_tp8ep8_v1
• vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_2n8g_megatron_tp8ep8_v1_tq_mooncake
• vlm_vlm_grpo_nemotron_omni_30ba3b_mmpr_4n8g_automodel_ep8_v1
• vlm_vlm_grpo_qwen2_5_omni_3b_avqa_1n8g_megatron_v1
• vlm_vlm_grpo_qwen2_5_omni_7b_audiomcq_1n8g_megatron_v1
• vlm_vlm_grpo_qwen2_5_omni_7b_intent_1n8g_megatron_v1
• vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n8g_dtensor2tp1_v1
• vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n8g_megatrontp2_v1
• vlm_vlm_grpo_qwen3_omni_30ba3b_audiomcq_4n8g_megatron_v1

GB200 (31)

• llm_distillation_nano3_30ba3b_4n4g_megatron_qa_nvfp4_modelopt_spec
• llm_distillation_qwen3_32b_to_1_7b_base_1n4g_fsdp2tp1_v1
• llm_distillation_qwen3_32b_to_1_7b_base_1n4g_megatron_tp1pp2cp2_pack
• llm_dpo_llama3_1_8b_instruct_4n4g_megatrontp1pp2_quick
• llm_dpo_llama3_2_1b_instruct_1n4g_fsdp2tp1_v2
• llm_grpo_deepscaler_1_5b_1n4g_8K
• llm_grpo_gemma3_1b_it_1n4g_fsdp2tp1
• llm_grpo_llama3_1_8b_instruct_2n4g_async_1off_single_controller_streaming2
• llm_grpo_llama3_1_8b_instruct_2n4g_fsdp2tp1_noncolocated
• llm_grpo_llama3_2_1b_instruct_1n4g_megatron
• llm_grpo_llama3_2_1b_instruct_1n4g_megatron_generation
• llm_grpo_nano_v2_12b_1n4g_megatron
• llm_grpo_nano_v2_12b_2n4g_fsdp2tp1
• llm_grpo_nanov3_30BA3B_4n4g_megatron_generation_noncolocated_async_gym
• llm_grpo_nanov3_30BA3B_4n4g_megatron_generation_noncolocated_mxfp8_rollouts_packed_refit
• llm_grpo_qwen2_5_0_5b_1n4g_megatron_trtllm_noncolocated_async
• llm_grpo_qwen2_5_7b_instruct_4n4g_fsdp2tp2_v3
• llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_fsdp2tp1_v3
• llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_megatron_generation_noncolocated_single_controller_async_lag4
• llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_megatron_generation_single_controller_async_lag4
• llm_grpo_qwen2_5_math_1_5b_instruct_2n4g_megatrontp1_sglang_noncolocated
• llm_grpo_qwen3_1_7b_1n4g_megatron_eagle3
• llm_grpo_qwen3_5_35ba3b_6n4g_async_1off_bf16_trtllm
• llm_sft_llama3_2_1b_1n4g_fsdp2tp1_v3
• llm_sft_nanov3_30BA3B_2n4g_fsdp2
• llm_sft_nanov3_30BA3B_2n4g_fsdp2_lora
• llm_sft_qwen2_5_math7b_2n4g_megatron
• vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_8n4g_megatron_generation_v1
• vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_8n4g_megatron_single_controller_async_v1
• vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n4g_dtensor2tp1_v1
• vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n4g_megatrontp1_v1

[yfw]

Team review: PR #4080, vLLM 0.26 → 0.29 and torch 2.11 → 2.13

Reviewed at head af7d9e0 by 8 agents (RL codebase, vLLM upstream, torch/packaging, bug finder, design, tests, description/threads, devil's advocate). No correctness bug was found in the code. Inline: one CI-coverage gap that should be fixed here (the three new DCP_USE_PREFIX_STORE tests never run), a handful of low-severity test/doc hardening items with committable blocks, and a description refresh.

Verified against the pinned sources (vLLM v0.29.0 98dff2a8, torch v2.13.0 cf30153c / v2.11.0 70d99e99): all ten source-patch anchors occur exactly once in 0.29 and every patch was executed against a scratch copy of its target file; the TCPStore patch takes the "upstream fixed" branch; all five injected worker extensions are collision-free against Worker/WorkerBase; 239 from vllm... import targets across the repo resolve; check_admission, make_fp8_moe_kernel, make_nvfp4_moe_kernel, _get_tied_embedding_params, SKIP_TENSORS/SKIP_LOAD_TENSORS, the KV-scale mirror and the DeepSeek V4 hook shadow (every exit path) match 0.29 line by line; the tied-alias failure and its fix were reproduced on CPU with real vLLM classes; the amax routing was probed with vLLM's real expert-mapping strings; the CUDA-IPC 'c' payload is byte-identical between the two torch tags and the 2.11 consumer checks version <= own; DCP_USE_PREFIX_STORE and the MASTER_ADDR/MASTER_PORT assert hold in torch 2.13; the lock is below main only for numpy, llguidance and the flash-attn local label (100 packages above, 399 equal).

Local runs (this host has no usable CUDA driver, so GPU-gated tests are unverified). --vllm-only over tests/unit/models/generation in a venv built from this lock's vllm extra: 415 passed, 7 skipped, 8 environment-only failures; every test this PR adds passed. No-vLLM pass (the L0 shards' mode): generation 506 passed, utils 486, data_plane 497. pre-commit run --all-files: clean. 55 mutants were run against the PR's new tests; every pinned fix went red except the vacuities called out inline.

Upstream root causes worth filing (the workarounds are correct, keep them):

• vLLM: DeepseekV4ForCausalLM.load_weights ends with self.process_weights_after_loading() on every call (model.py#L1813-L1816) although the loader already runs the model-level hook (model_loader/utils.py#L145-L146); per-call invocation breaks partial/streamed loads (this PR's Cannot copy out of meta tensor).
• vLLM: AutoWeightsLoader._check_skipped_aliases requires a tied alias and its canonical to arrive in the same load_weights call (utils.py#L442-L470), which no batched weight-update client can guarantee.

FYI, no action. all_gather_single does not exist on torch 2.11, which the sglang/trtllm venvs (also installing nemo_rl) now pin; its only caller BatchedDataDict.all_gather has no production call site, and the rename does remove a per-call FutureWarning on 2.13. check_admission is a no-op today (queue limits default None). The ambiguous-shard detection in quantization/utils.py is generic but only DeepSeek V4 resolves it; four other 0.29 models share the shape and keep the pre-existing bf16-refit fallback (tracking issue at most).

Optional test hardening (verified candidates, not staged): an engine_client member sweep for _AsyncLLMHTTPClient; a WeightsMapper._map_name pin; an IPC tuple-position pin against torch's rebuild_cuda_tensor signature; the mapper hand-off at the tied-alias call sites; RPC caller/callee parity for synchronize_sparse_refit_device; the <= boundary in normalize_cuda_ipc_handle; || true on the nvidia-smi probe in test_decode_vs_prefill.sh; the remaining three injected worker extensions in test_vllm_worker_extension_names.py. Available on request.

Generated by Claude Code

[yfw]

/ok to test e222247

[yfw]

/ok to test ab6b3f2

[yfw]

/ok to test ca9dfe5

[yuki-97]

Thanks for taking this one on — a vLLM 0.26 to 0.29 bump that also drags torch 2.11 to 2.13, while sglang and trtllm have to stay behind on 2.11, is a lot of surface to hold together, and the 33 comments you left walking through your own diff made it reviewable. Two calls stand out. Binding the routed-experts capture to the layer's router rather than to the experts object is right: process_weights_after_loading rebuilds the kernel, and with it the experts, on every refit, so a binding held on the experts would go stale after the first one. And keeping the torch split in two narrow shims — the IPC handle version byte and the named-dims check — rather than branching at each call site means the 2.13 removals stay out of the backends entirely.

Scope: full diff at ca9dfe5, including the merge of main in that commit.

Checked:

• Source patches — every old_snippet re-matched against a freshly fetched v0.29.0, and each patch is pinned by a *_patch_anchor_still_matches_installed_vllm test. The one that no longer matches, _patch_vllm_ray_executor_v2_tcpstore_port, is retired through its upstream_fix_marker now that 0.29 carries the fix, rather than left to no-op.
• Upstream API usage — AsyncLLM.check_admission, WorkerBase.synchronize_device, mm_device_do_normalize, the layer= removals in make_fp8_moe_kernel / make_nvfp4_moe_kernel, and the tuple shape of RoutedExperts.get_expert_mapping.
• Worker extension names against Worker at 0.29 — the intersection is empty, so the synchronize_sparse_refit_device rename closed the only collision.
• Merge resolution — dropping main's MiniMax-M3 topk-buffer patch is correct, since 0.29 ships that transposed form upstream, and nothing else main added to patches.py or vllm_patch_source_utils.py was lost.
• Dependency changes — quack-kernels leaving the vllm extra is covered by vLLM's own direct requirement on it, and the llguidance<1.8.0 ceiling narrows what the trtllm fork resolves to rather than widening it.
• The torch 2.11 / 2.13 split — the IPC handle shim, _has_no_named_dims, and all_gather_single.

Remaining comments are five nits: two on the lifetime of the torch-split shims, one on how the IPC shim should treat a version byte it has not verified, one on the torch floor all_gather_single implies, and one on a shell fallback.

[yfw]

/ok to test 11bd0e7

[yfw]

/ok to test dfc65df

[yuki-97]

LGTM, waiting release test results to see when to merge.

[yfw]

/ok to test ea439fb

[yfw]

Results — release suites on dfc65df (H100 release.txt 28 recipes, GB200 release_gb200.txt 15)

PR release pipelines: H100 69695985 · GB200 69685830. main controls: the weekly release schedule's Sept 19 run 68747432 (RL b7a4d95, vLLM 0.25.1 — the last weekly run before #3566 moved main to 0.26.0) and Sept 5 66361462 (RL 5368eff). One row per recipe (latest attempt shown).

• H100: 17 pass · 11 fail · 0 running
• GB200: 11 pass · 4 fail · 0 running

PR fails, main passes (2)

Test	PR	main Sept 19	main Sept 5	Notes
llm_grpo_nemotron3_super_120BA12B_16n4g_megatron_qa_bf16_300step GB200	❌ failed 110 min	❌	✅	Regression found here, fixed in ea439fb. MTP speculative decoding accepted nothing (mean acceptance length 1.00, 0 accepted tok/s vs main's 4.1 / 530 tok/s), generation ran at ~220 instead of ~700 tok/s and step 1 (32k max tokens) never finished before the job was cancelled externally (uid 146504, both attempts). Cause: the drafter lives under model_runner.speculator on 0.29's v2 runner and was never refit. main itself fails this recipe on a golden. Re-validation of the fix runs on the H100 16n8g nemotron3-super nightly (69725530).
llm_distillation_qwen3_32b_to_4b_base_1n8g_fsdp2tp2_dynamicbatch_v1 H100	❌ failed 63 min	✅	✅	Golden miss, under re-test. 20/20 steps; losses pass (0.472 → 0.184) but validation/accuracy[20] > 0.1 reads 0.066 (0.137 at step 10; main 0.168 Sept 19 / 0.148 Sept 5). The three sibling distillation release recipes pass here (0.129–0.160 on the same check). Second sample: 69724346.

PR fails, main had no usable result (2)

Test	PR	main Sept 19	main Sept 5	Notes
llm_dapo_gemma4_26ba4b_it_4n8g_fsdp2ep16cp2_automodel H100	❌ failed 52 min	❌	—	DeepEP dispatch timeout at step 1, like its Automodel MoE siblings. main's Sept 19 and Sept 5 runs never got past checkout for this recipe, so there is no direct control; the four sibling recipes that did run on main fail the same way. (main Sept 19: died at checkout (shared-workspace Bad address).)
llm_grpo_minimax_m3_32n8g_automodel_cp8ep128_noncolocated H100	❌ failed 28 min	—	—	New recipe, never run on main (added to release.txt on Sept 23, #4143). Dies with We couldn't connect to 'https://huggingface.co': the model is not in the CI offline cache yet. New-test bring-up, not a bump issue.

PR fails, main fails (11)

Test	PR	main Sept 19	main Sept 5	Notes
llm_grpo_dapomath17k_dsv3_32n4g_megatron GB200	❌ failed 238 min	❌	❌	Same as main. Trains 10/10 steps, then the async vLLM engine dies with RuntimeError: Executor failed and the raylet is terminated; main Sept 19 fails with the same Executor failed.
llm_grpo_nemotron3_super_120BA12B_16n4g_megatron_qa_nvfp4_w4a16_real_300step GB200	❌ failed 33 min	❌	❌	Same as main. prepare_refit_info raises No mapping found for the following Megatron parameter(s): mtp.layers.0.…experts.linear_fc1/fc2.weight_quantizer._amax (Megatron-Bridge strict mapping vs ModelOpt quantizer buffers on the MTP experts); byte-identical to main Sept 19 (447605229) and Sept 5.
llm_grpo_nemotron3_super_120BA12B_16n4g_megatron_qa_nvfp4_w4a4_real_300step GB200	❌ failed 148 min	❌	❌	Same mapping error as the W4A16 recipe (No mapping found … weight_quantizer._amax). main fails this recipe too (Sept 19: 24 h limit; Sept 5: get_megatron_calibration_forward_loop requires CP=1).
llm_dapo_mistral_medium_3_5_128b_16n8g_fsdp2_automodel H100	❌ failed 38 min	❌	❌	Same as main. CUDA out of memory on both main release runs and here.
llm_grpo_dapomath17k_dsv3_megatron H100	❌ failed 42 min	❌	❌	Same as main. DTensorPolicyWorkerV2.get_logprobs() dies with DeepEP error: timeout (dispatch CPU) at step 1; main's Sept 19 and Sept 5 release runs fail it identically.
llm_grpo_glm47_flash_4n8g_automodel H100	❌ failed 62 min	❌	❌	Same as main. DTensorPolicyWorkerV2.get_logprobs() dies with DeepEP error: timeout (dispatch CPU) at step 1; main's Sept 19 and Sept 5 release runs fail this Automodel MoE recipe identically.
llm_grpo_glm5_1_64n8g_megatron H100	❌ failed 68 min	❌	❌	Same as main. vLLM worker gloo … Timed out waiting 1800000ms for send operation in the first generation; main Sept 19 and Sept 5 fail identically.
llm_grpo_glm5_2_64n8g_megatron_6K_colocated H100	❌ failed 53 min	❌	❌	Pre-existing on main since #3566 (Sept 20). Megatron logprobs forward imports the cuDNN frontend's DSA CuTe kernels and hits AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma': cudnn-frontend 1.25.0 (global uv override, same on main) predates cutlass-dsl 4.6, which main moved the Megatron venv to. main's Sept 19 run still had cutlass 4.5.2 and failed later (CUDA unknown error). Follow-up PR: cudnn-frontend 1.30.0. See Known issues.
llm_grpo_minimax_m27_dapo_8n8g_automodel H100	❌ failed 51 min	❌	❌	Same as main. DTensorPolicyWorkerV2.get_logprobs() dies with DeepEP error: timeout (dispatch CPU) at step 1; main's Sept 19 and Sept 5 release runs fail this Automodel MoE recipe identically.
llm_grpo_qwen3_5_35ba3b_dapo_4n8g_automodel H100	❌ failed 68 min	❌	❌	Same as main. DTensorPolicyWorkerV2.get_logprobs() dies with DeepEP error: timeout (dispatch CPU) at step 1; main's Sept 19 and Sept 5 release runs fail this Automodel MoE recipe identically.
llm_grpo_qwen3_5_397ba17b_32n8g_megatron_v2 H100	❌ failed 32 min	❌	❌	Same as main. CUDA out of memory during setup on both main release runs and here.

PR passes, main fails (0)

Pass on both PR and main: 28 (H100 17, GB200 11):

H100 (17)

• llm_dapo_gemma4_31b_it_4n8g_fsdp2_automodel
• llm_dapo_qwen2_5_7b_v2
• llm_distillation_qwen3_32b_to_4b_base_2n8g_fsdp2tp2_long_v1
• llm_distillation_qwen3_32b_to_4b_base_2n8g_fsdp2tp2_seqpack_v1
• llm_distillation_qwen3_32b_to_4b_base_2n8g_fsdp2tp8_noncolocated_v1
• llm_dpo_llama3_1_8b_instruct_4n8g_fsdp2tp4
• llm_dpo_llama3_1_8b_instruct_4n8g_megatron_v2
• llm_dpo_llama3_1_8b_tulu3_1n8g_fsdp2tp1
• llm_grpo_gemma3_27b_it_8n8g_fsdp2tp8_actckpt_long
• llm_grpo_gptoss_20b_8n8g_megatron
• llm_grpo_llama3_1_8b_instruct_4n8g_fsdp2tp1_long_v3
• llm_grpo_qwen2_5_32b_32n8g_fsdp2tp8_actckpt_long_v3
• llm_grpo_qwen2_5_7b_instruct_4n8g_megatron
• llm_grpo_qwen3_30ba3b_8n8g_megatron
• llm_ppo_dsr1_7b_math_8n8g_megatron
• llm_sft_llama3_1_70b_8n8g_tp4pp2_long_megatron
• llm_sft_llama3_1_8b_1n8g_fsdp2tp1_long

GB200 (11)

• llm_dapo_qwen2_5_7b_16n4g_fsdp2cp2_v2
• llm_distillation_qwen3_32b_to_4b_base_2n4g_fsdp2tp1_long_v1
• llm_grpo_dapomath17k_nanov3_30BA3B_8n4g_megatron_trtllm
• llm_grpo_gemma3_27b_it_8n4g_fsdp2tp4_actckpt_long
• llm_grpo_gptoss_20b_8n4g_megatron
• llm_grpo_llama3_1_8b_instruct_4n4g_fsdp2tp1_long_v3
• llm_grpo_qwen2_5_32b_32n4g_fsdp2tp4_actckpt_long_v3
• llm_grpo_qwen2_5_7b_instruct_4n4g_megatron
• llm_grpo_qwen3_30ba3b_8n4g_megatron
• llm_sft_llama3_1_70b_8n4g_tp2pp2_long_megatron
• llm_sft_llama3_1_8b_1n4g_fsdp2tp1_long

Wall-clock parity for the 22 recipes that pass on both: PR/main ratio ≤ 1.01 on every H100 recipe (0.64–1.01×), GB200 faster across the board because the Sept 19 run spent hours in the quota-starved snapshot copy; the one slower recipe is gemma3_27b_it_8n4g_fsdp2tp4_actckpt_long at 1.38×.