[Akshay Sonawane (apsonawane)]

Description

Extends com.microsoft::CausalConvWithState and com.microsoft::GatedDeltaNet on the CUDA Execution Provider to support $S \le 8$ speculative decoding, packed-channel / packed-QKV projections, tiled $K=4$ prefill convolution, and compact state_update capture (state_update_capacity in [0, 8]).

Summary of Changes

com.microsoft::CausalConvWithState Extended Projection, Tiled Prefill & Compact state_update

File	Change
onnxruntime/core/graph/contrib_ops/bert_defs.cc	Adds input_channels, input_channel_offset, state_length, state_channels_last, and state_update_capacity (0..8) attributes plus optional Output(2, "state_update") ([B, state_update_capacity, C]), documenting that unused tail rows beyond min(sequence_length, state_update_capacity) are zero-filled
onnxruntime/contrib_ops/cuda/bert/causal_conv_with_state.{h,cc}, causal_conv_with_state_impl.{h,cu}	Implements the extended packed-channel CUDA path with: (1) CausalConvProjectionS2K4Kernel for short decode ($1 \le S \le 8$, kernel_size=4, state_length=4, dilation=1), (2) CausalConvProjectionTiledK4Kernel (8 positions per CTA with sliding registers) for prefill ($S \ge 16$), and (3) a graph-capturable cudaMemset2DAsync zero-fill for unused state_update rows when length < state_update_capacity
onnxruntime/contrib_ops/cpu/bert/causal_conv_with_state.{cc,helper.h}, onnxruntime/contrib_ops/webgpu/bert/causal_conv_with_state.cc	Rejects CUDA-only extended projection attributes/outputs cleanly on CPU and WebGPU

com.microsoft::GatedDeltaNet Packed QKV, TG Gates & $S \le 8$ QwenS2 Decode

File	Change
onnxruntime/core/graph/contrib_ops/bert_defs.cc	Adds TG type constraint so decay and beta can match the query element type (float, float16, bfloat16), and documents gate_activation="softplus" with "qwen" retained as a legacy alias
onnxruntime/contrib_ops/cuda/bert/gated_delta_net.{h,cc}, gated_delta_net_impl.{h,cu}, gated_delta_net_split_impl.cu, gated_delta_net_mma.cuh, gated_delta_net_plan.h	Supports packed QKV without intermediate unpacking on short decode ($S \le 8$), templated GateT for decay/beta, compact state_update_capacity capture (0..8), and the 8-warp QwenS2 recurrent decode specialization (ORT_GDN_S2_SPECIALIZE=1 by default) for single-request $1 \le S \le 8$, 16 Q/K heads, 48 V heads, and head size 128
onnxruntime/contrib_ops/webgpu/bert/gated_delta_net.cc	Supports "softplus" and legacy alias "qwen" for gate_activation

Tests and Documentation

• onnxruntime/test/contrib_ops/causal_conv_with_state_op_test.cc: Covers packed-channel extended history across lengths {1, 2, 3, 5, 8, 9, 16, 17, 24} (exercising both CausalConvProjectionS2K4Kernel and CausalConvProjectionTiledK4Kernel), full and partial channel blocks, both state layouts, present/absent past history, FP32/FP16/BF16, state_update capture verification, and tail zeroing when state_update_capacity > sequence_length.
• onnxruntime/test/contrib_ops/gated_delta_net_op_test.cc: Covers input-type decay/beta (FP16/BF16), QwenS2 ($S \in {1, 2, 4, 8}$) packed-QKV parity, state_update_capacity capture, empty requests in recurrent state, and bit-exact parity between gate_activation="softplus" and "qwen" on CUDA and WebGPU.
• docs/ContribOperators.md, docs/OperatorKernels.md, docs/contrib_ops/cuda/gdn.md: Syncs operator schemas/kernel tables and documents ORT_GDN_S2_SPECIALIZE / ORT_CAUSAL_CONV_S2_SPECIALIZE eligibility and H200 op-level & end-to-end benchmarks.

Performance

Measured on NVIDIA H200 (SM90, CUDA 13.0) at Qwen3.8-Flash-Next RC3 geometry ((B, S, 10240) BF16, state_update_capacity=7):

• GatedDeltaNet (QwenS2): 1.15x–1.25x faster across $S \in {1, 2, 4, 8}$ (18.82 us vs 21.60 us at $B=1, S=8$), saving ~0.10–0.15 ms per DFlash2 $S=8$ verification cycle across 36 layers.
• CausalConvWithState: 1.19x–1.37x faster on short decode ($S \le 8$) and 4.06x faster on prefill at $S=2048$ (55.13 us vs 223.97 us), reducing end-to-end TTFT by 2.9–3.1% at 2K–32K context lengths.

Testing

• Run provider unit tests:
  onnxruntime_provider_test --gtest_filter="CausalConvWithStateTest.*:GatedDeltaNetTest.*:GatedDeltaNetWebGpuTest.*"
• End-to-end verification on qwen_flash_next_int4_int8_rc3_contrib_dflash2 ($S=8$, state_update_capacity=7).

Checklist

• Tests added/updated
• No breaking changes (or documented in description)
• Documentation updated (if applicable)

[Copilot]

🟡 Changes recommended

State initialization, regression expectations, and generated documentation need correction before approval.

3 open issues

• Empty ragged requests return uninitialized state · New
• Regenerate operator and kernel documentation · New
• Reference recurrence omits softplus fused decay · New

What changed in this PR

Extends ONNX Runtime’s GatedDeltaNet and causal convolution operators with CUDA-focused fused input and state handling while preserving default modes.

Changes:

• Adds activation-type GDN gates, direct packed-QKV decoding, and intermediate-state output.
• Adds convolution channel slicing, configurable state layout/history, and first-state output.
• Updates schemas, provider validation, and regression tests.

File	Description
onnxruntime/​test/​contrib_ops/​gated_delta_net_op_test.cc	Adds gate, packed-QKV, and intermediate-state tests.
onnxruntime/​test/​contrib_ops/​causal_conv_with_state_op_test.cc	Tests extended convolution modes and validation.
onnxruntime/​core/​graph/​contrib_ops/​bert_defs.cc	Extends schemas and shape inference.
onnxruntime/​contrib_ops/​webgpu/​bert/​gated_delta_net.cc	Adds alias support and capability guards.
onnxruntime/​contrib_ops/​webgpu/​bert/​causal_conv_with_state.cc	Rejects CUDA-only extensions.
onnxruntime/​contrib_ops/​cuda/​bert/​gated_delta_net.h	Adds intermediate-state configuration.
onnxruntime/​contrib_ops/​cuda/​bert/​gated_delta_net.cc	Handles typed gates, packed inputs, and state outputs.
onnxruntime/​contrib_ops/​cuda/​bert/​gated_delta_net_split_impl.cu	Loads activation-type gates.
onnxruntime/​contrib_ops/​cuda/​bert/​gated_delta_net_plan.h	Renames the fused decay enum.
onnxruntime/​contrib_ops/​cuda/​bert/​gated_delta_net_mma.cuh	Uses the renamed decay mode.
onnxruntime/​contrib_ops/​cuda/​bert/​gated_delta_net_impl.h	Extends kernel arguments and gate loading.
onnxruntime/​contrib_ops/​cuda/​bert/​gated_delta_net_impl.cu	Implements typed gates, packed decoding, and state capture.
onnxruntime/​contrib_ops/​cuda/​bert/​causal_conv_with_state.h	Stores extended convolution attributes.
onnxruntime/​contrib_ops/​cuda/​bert/​causal_conv_with_state.cc	Validates and dispatches extended modes.
onnxruntime/​contrib_ops/​cuda/​bert/​causal_conv_with_state_impl.h	Declares the extended kernel launcher.
onnxruntime/​contrib_ops/​cuda/​bert/​causal_conv_with_state_impl.cu	Implements sliced convolution and state outputs.
onnxruntime/​contrib_ops/​cpu/​bert/​causal_conv_with_state.cc	Rejects CUDA-only extensions.
docs/​ContribOperators.md	Removes a trailing blank line only.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Unresolved optional-output handling, weight-index overflow, and no-exception test failures block approval.

No open issues 🎉

3 previously missed issues

In code that hasn't changed since last review

Prevent 32-bit overflow in weight offset calculation

onnxruntime/​contrib_ops/​cuda/​bert/​causal_conv_with_state_impl.cu:48

The weight offset uses signed 32-bit multiplication. With 65,537 channels, a 32,768-tap kernel, and dilation 1, the extended-path dimension checks pass, but channel 65,536 computes c * kernel_size == 2^31. This overflows the index and can cause an out-of-bounds device read. Cast c to int64_t before multiplication, as the input offsets already do.

Detect optional output presence instead of output count

onnxruntime/​core/​graph/​contrib_ops/​bert_defs.cc:4772

An omitted optional output may be encoded as outputs=["output", "present_state", ""]. Its placeholder still counts toward getNumOutputs(), so this condition rejects a valid default channels-first convolution as requiring extended mode. The CPU constructor, CUDA extended-path selection, and WebGPU output guard repeat the same count-only check. Check actual output presence throughout: use ctx.hasOutput(2) in inference, NodeArg::Exists() where available, and a plugin-compatible presence check for WebGPU (for example, checking Output(2, state_shape) after the shape is known). Add an AddOptionalOutputEdge regression covering these provider paths.

Guard rejection tests against no-exception builds

onnxruntime/​test/​contrib_ops/​causal_conv_with_state_op_test.cc:403

Both new rejection tests can terminate no-exception test runs. With CUDA available, PackedChannelsInvalidConfigurations reaches fail_shape_inference; PackedChannelsRejectsUnsupportedCpu reaches the constructor's ORT_ENFORCE even in CPU-only builds. Under ORT_NO_EXCEPTIONS, these paths abort rather than return a failure that kExpectFailure can check. Wrap both tests (lines 403–441) in #ifndef ORT_NO_EXCEPTIONS, following the existing shape-inference rejection tests below, or add a conditional GTEST_SKIP() before constructing each tester.

3 resolved since last review

• Reference recurrence omits softplus fused decay
• Regenerate operator and kernel documentation
• Empty ragged requests return uninitialized state

_{Review effort: Balanced}

[Akshay Sonawane (apsonawane)]

Added the scoped GDN/convolution decode experiments on top of the current PR head, preserving the existing empty-request and WebGPU review fixes.

Published commit: 62bd923adb13df65127adccd4d7162bb539e41df.

• Opt-in ORT_GDN_S2_SPECIALIZE=1: guarded packed-QKV, single-request/two-token specialization (16 query/key heads, 48 value heads, head size 128).
• Opt-in ORT_CAUSAL_CONV_S2_SPECIALIZE=1: length-two, kernel/state-four, dilation-one specialization.
• ORT_GDN_DECODE_WARPS=2|4|8, default 8.
• Added exact FP16/BF16 packed-QKV parity tests and documented eligibility/fallbacks.
• No multi-draft state outputs, dense-prefill normalization experiments, weights, binaries, or benchmark artifacts were included.

Validation of the publishing sources: all changed CUDA translation units compile. Process-isolated native tests pass with specialization switches both off and on: 6 focused GDN packed/input-type tests and 53 convolution tests per mode. This is an incremental validation against retained build dependencies, not a clean full upstream build.

The broader GatedDeltaNetTest.RecurrentMidStatePreservesEmptyRequests test fails in this incremental build. I also compiled the unmodified PR-head GDN kernel and reproduced an invalid-argument failure (32-wide case). The broader run can poison later CUDA tests, so it is not reported as clean. Clean-build CI is still required; this update does not attempt an unrelated baseline fix.

Historical experimental-package benchmarks improved 8K decode from 213.51 to 228.06 TPS and 16K from 227.85 to 242.60 TPS in a controlled bracket. These numbers are not a re-benchmark of the transplanted PR tip. Full model accuracy remains deferred, and this is not a claim of sustained 250 TPS at 8K.

Companion updates: sparse attention/indexer #33472, MoE/FP8 #33473, and GenAI packaging microsoft/onnxruntime-genai#2759.

[Akshay Sonawane (apsonawane)]

Fresh-build integration validation update (no source changes pushed to this ORT PR in this update):

• Rebuilt ORT from scratch for H200 (sm90a), CUDA provider as a plugin, with FP4 QMoE, FP8/INT4 KV cache, GQA value layout, and FPA-INTB GEMM enabled. onnxruntime_QUICK_BUILD=OFF; no retained build objects were reused.
• Rebuilt GenAI against the fresh ORT headers/libraries, and rebuilt the fold custom library.
• Focused convolution/GDN/fused-tensor/hyper-connection/INT2 tests: 193 passed.
• Sparse/indexer/FP8 suite plus the isolated QSA capacity test: 140 additional passes.
• Known limitations: QSA and CSA MalformedStateLengthsAreClamped tests still fail, including with specialized top-k disabled. These are not being claimed as passing. This is selected-suite validation, not a clean full-suite qualification.
• Selected GenAI Qwen/executor/weight-pool tests: 69 passed, including the opt-in real-GPU mapped-weight test with proper CUDA plugin registration. One additional test is disabled.

The latest fused INT2 model completed one warmup plus three measured requests at each of 2K/8K/16K/32K, generating 512 tokens each. All 16 requests matched the previous-runtime token IDs exactly, with zero generation failures or standard fallbacks. Runtime-map inspection confirmed only the fresh ORT/GenAI libraries were used.

Prompt	Pooled decode TPS	Mean TTFT
2048	269.90	0.584 s
8192	250.17	2.448 s
16384	266.92	5.065 s
32768	264.12	10.578 s

Evidence scope: this is the full dirty experimental source stack containing this PR's optimizations plus other local work, not a standalone build/benchmark of this exact PR head. It is not an isolated speedup attribution, a fresh paired reference comparison, or proof of cross-process sustained 250 TPS. Full model accuracy qualification remains deferred.

The rebuild exposed local expert-cache function-pointer and INT2 direct-link test wiring issues, which were fixed locally. Those expert-cache extensions are not in the published PR and have deliberately not been mixed into this update.

[Copilot]

🟡 Changes recommended

Requested prefix capture can be silently disabled, and valid empty optional convolution outputs are rejected.

2 open issues

• Remove sequence-length veto from uniform capture · New
• Treat empty trailing state output as omitted · New

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Default-enabled tiled convolution and packed Qwen compact capture need targeted regression coverage before merging.

1 open issue

• Add tiled-kernel regression coverage for history and capture · New

1 previously missed issue

In code that hasn't changed since last review

Add compact-capture parity coverage for QwenS2 packed specialization

onnxruntime/​contrib_ops/​cuda/​bert/​gated_delta_net_impl.cu:883

The default-enabled packed QwenS2 specialization lacks compact-capture coverage: TestQwenS2PackedParity uses capacity zero, while the capture regression uses separate Q/K/V and different head counts. Add FP16/BF16 cases with S=8, capacity=7, and full, partial, and inactive capture to validate speculative rollback data. Compare defined captured prefixes against separate-QKV recurrent execution or independent prefix replay, without comparing the unspecified unused suffix.

2 resolved since last review

• Remove sequence-length veto from uniform capture
• Treat empty trailing state output as omitted

_{Review effort: Balanced}