[Akshay Sonawane (apsonawane)]

Description

Optimizes com.microsoft::HyperConnectionLowRankMix on the CUDA Execution Provider for $S \le 8$ speculative decoding (HcDownKernel + HcUpPreKernel fused path) and deterministic prefill (FusedInjectPostGateKernel unfused path):

• 4x2 Fused Row-Group Dispatch for 8-Row Speculative Decode: Uses LaunchFusedRows<T, 4, 2> (4 rows per CTA in HcDownKernel, 2 rows per CTA in HcUpPreKernel) when rows == 8, LaunchFusedRows<T, 2, 2> for rows in 2..7, and LaunchFusedRows<T, 1, 1> for rows == 1.
• Rank-320 Specialization & Register-Spill-Free HcUpPreKernel: Specializes shared-memory and loop bounds for (num_branches=4, rank=320) (HcUpPreKernel<T, ROWS, 4, 320>) and accumulates K-split partial sums directly into va[r] without a float4 pq[8][ROWS] local register array.
• Hopper Programmatic Dependent Launch (ORT_HYPER_CONNECTION_PDL=1 default): Enables cudaLaunchAttributeProgrammaticStreamSerialization between HcDownKernel and HcUpPreKernel on SM90+ (CUDA 12.3+) with __ldcg grid-synchronized reads when weight inputs are constant initializers.
• Deterministic Unfused FusedInjectPostGateKernel with Branch Tiling: Replaces the unfused split-K cuBLAS inject GEMM + PostGateKernel with a deterministic fused inject-GEMV + post-gate kernel (FusedInjectPostGate4RowsKernel for 16-bit num_branches == 4 with 8-byte aligned inject_weight, and FusedInjectPostGateKernel tiled over 8-branch column groups in grid.y to support arbitrary num_branches, including num_branches > 8).

Summary of Changes

CUDA Kernel & Launcher

File	Change
onnxruntime/contrib_ops/cuda/bert/hyper_connection_low_rank_mix_impl.cu	Adds LaunchFusedRows<T, DOWN_ROWS, UP_ROWS> (1x1 for rows == 1, 2x2 for rows in 2..7, 4x2 for rows == 8), HcUpPreKernel<T, ROWS, 4, 320> specialization without local register spills, Hopper PDL launch attributes (ORT_HYPER_CONNECTION_PDL=1), and deterministic FusedInjectPostGate4RowsKernel / branch-tiled FusedInjectPostGateKernel on the unfused path
onnxruntime/contrib_ops/cuda/bert/hyper_connection_low_rank_mix.{cc,impl.h}	Removes the unused inject scratch buffer allocation from the unfused path

Tests and Documentation

• onnxruntime/test/contrib_ops/hyper_connection_low_rank_mix_op_test.cc: Adds FusedDecodeShapeRank320 covering rows in {1, 2, 3, 4, 7, 8} at (branches=4, hidden=2560, rank=320) for BFloat16 and MLFloat16 (with and without fused prologue), and UnfusedMoreThanEightBranches covering num_branches = 9 (FP32, FP16, BF16) and num_branches = 17 (FP16).
• docs/contrib_ops/cuda/hyper_connection_low_rank_mix.md: Documents the fused and unfused paths, row-group dispatch table, ORT_HYPER_CONNECTION_PDL environment variable, and H200 DFlash2 $S=8$ benchmark results.

Performance

Measured on NVIDIA H200 (SM90, Qwen3.8-Flash-Next RC3 INT4 model, DFlash2 $S=8$ decode ms/cycle across 5 repetitions):

Configuration	2K	8K	32K
Default (4x2 rows for $S=8$, ORT_HYPER_CONNECTION_PDL=1)	14.93 ms	15.07 ms	15.30 ms
2x2 rows for $S=8$	15.00 ms	15.11 ms	15.38 ms
ORT_HYPER_CONNECTION_PDL=0	15.09 ms	15.25 ms	15.48 ms

Testing

• Run provider unit tests:
  onnxruntime_provider_test --gtest_filter="HyperConnectionLowRankMixTest.*"
• End-to-end verification on qwen_flash_next_int4_int8_rc3_contrib_dflash2 ($S=8$ DFlash2 speculative decoding).

Checklist

• Tests added/updated
• No breaking changes (or documented in description)
• Documentation updated (if applicable)

[Copilot]

🟡 Changes recommended

The environment-helper include order causes conflicting logging definitions in normal non-plugin CUDA-provider builds.

1 open issue

• Include env_var_utils.h after cuda_common.h · New

What changed in this PR

Adds opt-in single-row work groups for eligible four-row CUDA HyperConnectionLowRankMix decode, supporting the broader speculative-verification integration while retaining the default two-row grouping.

Changes:

• Adds ORT_HC_DECODE_ROWS with supported values 1 and 2.
• Extends rank-320 reference coverage to four-row inputs.
• Documents tuning behavior and separate-process testing for both settings.

File	Description
onnxruntime/​test/​contrib_ops/​hyper_connection_low_rank_mix_op_test.cc	Adds four-row fused-decode reference cases.
onnxruntime/​contrib_ops/​cuda/​bert/​hyper_connection_low_rank_mix_impl.cu	Adds environment-controlled row grouping for eligible four-row decode.
docs/​contrib_ops/​cuda/​hyper_connection_low_rank_mix.md	Documents supported settings and test configurations.

_{Review effort: Balanced}

[github-actions]

You can commit the suggested changes from lintrunner.

[Copilot]

🟡 Changes recommended

Valid inputs with more than eight branches can trigger out-of-bounds shared-memory reads in the replacement post-gate kernel.

2 open issues

• Unfused kernel overruns shared memory above eight branches · New
• Add eight-row dispatch coverage to the fused-row test loop · New

1 resolved since last review

• Include env_var_utils.h after cuda_common.h

_{Review effort: Balanced}