diff --git a/docs/ContribOperators.md b/docs/ContribOperators.md index 2703de7cc1fae..70b798e437eb7 100644 --- a/docs/ContribOperators.md +++ b/docs/ContribOperators.md @@ -36,6 +36,8 @@ Do not modify directly.* * com.microsoft.EngramGate * com.microsoft.ExpandDims * com.microsoft.FastGelu + * com.microsoft.FoldMid2 + * com.microsoft.FoldTail * com.microsoft.FusedConv * com.microsoft.FusedGemm * com.microsoft.FusedMatMul @@ -55,6 +57,7 @@ Do not modify directly.* * com.microsoft.GridSample * com.microsoft.GroupNorm * com.microsoft.GroupQueryAttention + * com.microsoft.HyperConnectionLowRankMix * com.microsoft.HyperConnectionPostMix * com.microsoft.HyperConnectionPreMix * com.microsoft.Inverse @@ -74,6 +77,7 @@ Do not modify directly.* * com.microsoft.MatMulNBitsQkv * com.microsoft.MaxpoolWithMask * com.microsoft.MoE + * com.microsoft.MtpMoe * com.microsoft.MulInteger * com.microsoft.MultiHeadAttention * com.microsoft.MurmurHash3 @@ -894,18 +898,39 @@ This version of the operator has been available since version 1 of the 'com.micr (..., C, H), or flattened shape (..., C * H) when num_branches is specified. The optional scale may have shape (C * H), (C, H), or (H). Arithmetic is performed in float32 and the result is converted to T. + With use_onnx_arithmetic=1, compute FP32 square, mean, epsilon addition, square + root, division, and scale multiplication with separate rounding boundaries. + CUDA reuses the standard fast ReduceMean summation order, including its scalar-result + path; it does not substitute reciprocal square root or change the reduction tree. + For multiple rows, FP32-rounded squaring is performed during the reduction load rather + than materializing a separate FP32 square tensor. + Optional Y_fp32 losslessly widens the already T-rounded Y and requires + use_onnx_arithmetic=1. WebGPU supports only the default mode without Y_fp32. + CUDA-only version 2 can select a contiguous last-axis interval directly from + packed rows using input_start and input_width. It retains the same reduction + order over the selected width and avoids materializing a Slice tensor. #### Version -This version of the operator has been available since version 1 of the 'com.microsoft' operator set. +Version 1 retains the original input layout. Version 2 adds CUDA-only packed-input +selection; CPU and WebGPU implement version 1 only. #### Attributes
epsilon : float
Epsilon added before reciprocal square root.
-
num_branches : int
+
input_start : int (default is 0, version 2)
+
Nonnegative starting offset in the last input dimension. A nonzero offset requires input_width.
+
input_width : int (default is 0, version 2)
+
Width selected before normalization. Zero retains the full input. A positive width +requires use_onnx_arithmetic=1 and num_branches=0, and the selected interval must fit +within the last input dimension.
+
num_branches : int (default is 0)
Number of branches for flattened input. Omit or set to zero for grouped input.
+
use_onnx_arithmetic : int (default is 0)
+
Use explicit FP32 mean/sqrt/divide/multiply arithmetic instead of reciprocal RMS. +Supported values are 0 (default) and 1.
#### Inputs (1 - 2) @@ -921,12 +946,17 @@ This version of the operator has been available since version 1 of the 'com.micr
Y : T
-
RMS-normalized output with the same shape as X.
+
RMS-normalized output with the same shape as X, except that version 2 with +positive input_width replaces the last dimension with input_width.
+
Y_fp32 (optional) : F
+
Float32 widening of the already T-rounded Y. Requires use_onnx_arithmetic=1.
#### Type Constraints
+
F : tensor(float)
+
Widened output is float32.
T : tensor(float), tensor(float16), tensor(bfloat16)
Constrain input and output to floating-point tensors.
M : tensor(float), tensor(float16), tensor(bfloat16)
@@ -1723,6 +1753,12 @@ This version of the operator has been available since version 1 of the 'com.micr auxiliary_key is used as both K and V. DeepSeek-style post-attention inverse/conjugate RoPE and output projection remain the exporter's responsibility. + CUDA also supports INT8 main caches with float32 per-channel key/value scales. Appended rows are + quantized with round-to-nearest-even and saturation to [-128, 127]; attention reads the quantized + representation, including current rows. Auxiliary caches retain the query's floating-point type. + Scales must be positive and finite; requested validation reports invalid scales as status 10, + and CUDA strict validation raises an error. WebGPU supports only query-typed floating caches. + #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1758,7 +1794,7 @@ This version of the operator has been available since version 1 of the 'com.micr
Add a zero-valued sink logit when no explicit head_sink is supplied.
-#### Inputs (11 - 17) +#### Inputs (11 - 20)
query : T
@@ -1767,10 +1803,10 @@ This version of the operator has been available since version 1 of the 'com.micr
Current main key [batch, sequence, kv_num_heads * head_size].
value (optional) : T
Current main value [batch, sequence, kv_num_heads * head_size].
-
past_key (optional) : T
-
Main key cache in BNSH layout.
-
past_value (optional) : T
-
Main value cache in BNSH layout.
+
past_key (optional) : TCache
+
Main key cache in BNSH layout, query-typed or INT8.
+
past_value (optional) : TCache
+
Main value cache in BNSH layout, matching past_key.
auxiliary_key (optional) : T
Read-only auxiliary key sequence in BNSH layout.
auxiliary_value (optional) : T
@@ -1795,17 +1831,25 @@ This version of the operator has been available since version 1 of the 'com.micr
Optional K RMSNorm weight [head_size].
head_sink (optional) : T
Optional sink logit [num_heads]. Participates in the shared softmax denominator in both attention modes.
+
committed_length (optional) : M
+
Optional single-request committed cursor. Inconsistency with sequence metadata produces validation status 9.
+
key_scale (optional) : S
+
Positive finite per-channel INT8 key-cache scales [kv_num_heads * head_size].
+
value_scale (optional) : S
+
Positive finite per-channel INT8 value-cache scales [kv_num_heads * head_size]. Both scales are required for INT8 caches.
-#### Outputs (1 - 3) +#### Outputs (1 - 4)
output : T
Attention output [batch, sequence, num_heads * head_size].
-
present_key (optional) : T
+
present_key (optional) : TCache
Updated main key cache in BNSH layout.
-
present_value (optional) : T
+
present_value (optional) : TCache
Updated main value cache in BNSH layout.
+
validation_status (optional) : M
+
One int32 validation status: zero for valid inputs; 9 for cursor mismatch; 10 for invalid INT8 scales.
#### Type Constraints @@ -1813,6 +1857,10 @@ This version of the operator has been available since version 1 of the 'com.micr
T : tensor(float), tensor(float16), tensor(bfloat16)
Constrain all floating-point inputs and outputs to one element type.
+
TCache : tensor(float), tensor(float16), tensor(bfloat16), tensor(int8)
+
Cache type matches T unless INT8 per-channel scales are supplied.
+
S : tensor(float)
+
INT8 cache scales use float32.
M : tensor(int32)
Constrain selection and sequence metadata to int32.
@@ -2100,6 +2148,106 @@ This version of the operator has been available since version 1 of the 'com.micr +### **com.microsoft.FoldMid2** + + Combines speculative-token acceptance, recurrent-state commit bookkeeping, and head rotary-cache lookup for a two-token verification step. When `ctl[3]` is zero, the operator updates `ctl` and `ctx_commit` in place and writes a result record through the device-accessible pointer and capacity in `ptrs`. + +#### Version + +This version of the operator has been available since version 1 of the 'com.microsoft' operator set. + +#### Inputs + +
+
next_tokens : I64
+
Two predicted token IDs.
+
input_ids : I64
+
Two input token IDs; input_ids[1] is the draft token.
+
status : I32
+
Status values aggregated into the result record.
+
ctl : I64
+
Mutable eight-element control state.
+
ctx_commit : I64
+
Mutable two-element committed context.
+
ctx_a : I64
+
Context selected when the draft is rejected.
+
ctx_b : I64
+
Context selected when the draft is accepted.
+
ptrs : I64
+
Device pointers containing the result-ring address and capacity.
+
cos_cache : B
+
Bfloat16 rotary cosine cache with shape [positions, 32].
+
sin_cache : B
+
Bfloat16 rotary sine cache with shape [positions, 32].
+
+ +#### Outputs + +
+
head_ids : I64
+
Two token IDs consumed by the head.
+
head_c : I32
+
Current head-cache length.
+
slot_ids : I64
+
Two consecutive head-cache slots.
+
seqlens : I32
+
Head sequence length.
+
cos_out : B
+
Two rows of rotary cosine values.
+
sin_out : B
+
Two rows of rotary sine values.
+
sel : I64
+
Selected row: one when the draft is accepted, otherwise zero.
+
+ +#### Type Constraints + +
+
B : tensor(bfloat16)
+
Constrain rotary cache tensors to bfloat16.
+
I32 : tensor(int32)
+
Constrain status and sequence tensors to int32.
+
I64 : tensor(int64)
+
Constrain control and token tensors to int64.
+
+ + +### **com.microsoft.FoldTail** + + Updates the next speculative draft token and aggregate error bit from the head result. When `ctl[3]` is zero, the operator updates the control tensor in place; muted calls return `done=0` without changing the state. + +#### Version + +This version of the operator has been available since version 1 of the 'com.microsoft' operator set. + +#### Inputs + +
+
next_token : I64
+
Next draft token ID.
+
status : I32
+
Head status values.
+
ctl : I64
+
Mutable eight-element control state.
+
+ +#### Outputs + +
+
done : I32
+
One when the control state was updated; zero for muted calls.
+
+ +#### Type Constraints + +
+
I32 : tensor(int32)
+
Constrain status and output tensors to int32.
+
I64 : tensor(int64)
+
Constrain token and control tensors to int64.
+
+ + ### **com.microsoft.FusedConv** The fused convolution operator schema is the same as Conv besides it includes an attribute @@ -3333,6 +3481,94 @@ This version of the operator has been available since version 1 of the 'com.micr +### **com.microsoft.HyperConnectionLowRankMix** + + One hyper-connection block with a low-rank gate network, fused into one operator. C = num_branches streams of + width H are kept as one flattened (..., C * H) tensor. With the optional prologue the operator first applies the + post-mix of the previous block (identity stream mix): + + S[..., c, h] = streams[..., c, h] + prev_post_gate[..., c] * prev_block_output[..., h] + + and otherwise S = streams. It then computes, with `rms` taken per branch over H: + + hn[..., c, h] = S[..., c, h] / sqrt(mean_h(S[..., c, :]^2) + epsilon) * norm_scale[c, h] + d = ScaledSiLU(hn @ down_weight, silu_alpha) (..., R) + g[..., c, h] = sigmoid(d @ up_weight) per stream and feature + pre_output = reduction_scale * sum_c(hn[..., c, h] * g[..., c, h]) (..., H) + post_gate = sigmoid((hn @ inject_weight) / gate_div) * gate_mul (..., C) + + pre_output is the input of the next transformer block and post_gate is prev_post_gate of the next call. + streams_out is S (the streams after the prologue; a copy of streams when there is no prologue). + + Numerics contract. Matrix products and reductions accumulate in float32. The result of every step is rounded + to T once, as the output of the equivalent standard operator would be: + S = cast_T(streams + prev_post_gate * prev_block_output), hn = cast_T(S * inv_rms * norm_scale), + down = cast_T(hn @ down_weight), Z = cast_T(silu_alpha * down), SG = cast_T(sigmoid(Z)), hidden = cast_T(Z * SG), + up = cast_T(hidden @ up_weight), g = cast_T(sigmoid(up)), pre_output = cast_T(reduction_scale * sum_c(hn * g)), + inject = cast_T(hn @ inject_weight), post_gate = cast_T(cast_T(sigmoid(cast_T(inject / gate_div))) * gate_mul). + sigmoid is evaluated in float32 with expf and rounded once to T; it does not reproduce a Sigmoid kernel that + rounds to T after every elementary operation. The accumulation order of the matrix products is fixed, so the + results are deterministic, but they are not bitwise identical to cuBLAS. + +#### Version + +This version of the operator has been available since version 1 of the 'com.microsoft' operator set. + +#### Attributes + +
+
epsilon : float
+
Epsilon added to the mean square of each branch before the reciprocal square root.
+
gate_div : float
+
Divisor applied to the inject logits before the sigmoid. Must not be zero.
+
gate_mul : float
+
Multiplier applied to the sigmoid of the inject logits.
+
num_branches : int (required)
+
Number of residual streams C. The last dimension of streams is C * H.
+
reduction_scale : float
+
Multiplier applied to the sum over branches of pre_output.
+
silu_alpha : float
+
Scale of the ScaledSiLU between the down and up projections.
+
+ +#### Inputs (5 - 7) + +
+
streams : T
+
Flattened streams (..., C * H). With the prologue these are the streams before the previous block's post-mix.
+
norm_scale : T
+
Per-branch RMS norm weight with shape (C * H).
+
down_weight : T
+
Down projection with shape (C * H, R), row-major [K, N].
+
up_weight : T
+
Up projection with shape (R, C * H), row-major [K, N].
+
inject_weight : T
+
Post-gate projection with shape (C * H, C), row-major [K, N].
+
prev_block_output (optional) : T
+
Optional output of the previous block with shape (..., H). Provide together with prev_post_gate.
+
prev_post_gate (optional) : T
+
Optional post-mix gate of the previous block with shape (..., C). Provide together with prev_block_output.
+
+ +#### Outputs (2 - 3) + +
+
pre_output : T
+
Block input with shape (..., H).
+
post_gate : T
+
Post-mix gate for the next call with shape (..., C).
+
streams_out (optional) : T
+
Optional streams after the prologue, with the same shape as streams. Equal to streams when the prologue inputs are absent.
+
+ +#### Type Constraints + +
+
T : tensor(float), tensor(float16), tensor(bfloat16)
+
Constrain all floating-point inputs and outputs to the same type.
+
+ + ### **com.microsoft.HyperConnectionPostMix** Mixes existing streams and injects one branch output: @@ -4500,6 +4736,57 @@ This version of the operator has been available since version 1 of the 'com.micr +### **com.microsoft.MtpMoe** + + Fused bfloat16 NVFP4 mixture-of-experts projection for multi-token prediction. + +#### Version + +This version of the operator has been available since version 1 of the 'com.microsoft' operator set. + +#### Attributes + +
+
gate_up_global_scale : float (required)
+
Global scale applied to the gate/up projection weights.
+
down_global_scale : float (required)
+
Global scale applied to the down projection weights.
+
+ +#### Inputs + +
+
input : T
+
Bfloat16 hidden states with shape [1, rows, 2560].
+
router_logits : T
+
Bfloat16 router logits with shape [1, rows, 512].
+
gate_up_weights : T1
+
Packed NVFP4 gate/up weights with shape [512, 1280, 1440].
+
down_weights : T1
+
Packed NVFP4 down weights with shape [512, 2560, 360].
+
+ +#### Outputs + +
+
output : T
+
Bfloat16 output with the same shape as input.
+
scratch : T2
+
Float scratch buffer used by the fused expert projections.
+
+ +#### Type Constraints + +
+
T : tensor(bfloat16)
+
Constrain activations and output to bfloat16.
+
T1 : tensor(uint8)
+
Constrain packed expert weights to uint8.
+
T2 : tensor(float)
+
Constrain scratch to float.
+
+ + ### **com.microsoft.MulInteger** Performs element-wise binary quantized multiplication (with Numpy-style broadcasting support). @@ -7173,10 +7460,18 @@ This version of the operator has been available since version 1 of the 'com.micr RotaryEmbedding is the implementation of rotary positional embeddings (RoPE). The positions are represented as rotation matrices that are multiplied to query and key before the inner product of query and key is taken. + Version 2 adds optional FP32 arithmetic with activation-type storage. With + use_fp32_arithmetic=1, input and cache values are widened inside the CUDA kernel, + each multiplication and addition is rounded separately in float32, and only the + final result is converted back to the input type. This preserves a + Cast-to-float -> float RotaryEmbedding -> Cast-to-input-type chain. + Version 1 and the default arithmetic are unchanged. #### Version -This version of the operator has been available since version 1 of the 'com.microsoft' operator set. +The latest version is 2 of the 'com.microsoft' operator set. Version 2 is currently +implemented by CUDA only; other providers retain version 1 and cannot silently +execute version 2 with legacy low-precision arithmetic. #### Attributes @@ -7191,6 +7486,10 @@ This version of the operator has been available since version 1 of the 'com.micr
Rotary embedding dimension. Default value is 0.
scale : float
Custom scale will be used if specified. Default value is 1.0
+
use_fp32_arithmetic : int (default is 0; version 2 only)
+
Widen input and cache values, perform separately rounded float32 multiplication +and addition, then round the output once to the input type. Supported values are +0 (legacy arithmetic) and 1.
#### Inputs @@ -8771,4 +9070,3 @@ No versioning maintained for experimental ops.
T : tensor(float)
Constrain input and output types to float32 tensors.
- diff --git a/docs/OperatorKernels.md b/docs/OperatorKernels.md index 9786f3169bac0..6427c68fe789e 100644 --- a/docs/OperatorKernels.md +++ b/docs/OperatorKernels.md @@ -575,7 +575,7 @@ The **OpSet Version** column uses the following notation: |BeamSearch|*in* input_ids:**F**
*in* max_length:**I**
*in* min_length:**I**
*in* num_beams:**I**
*in* num_return_sequences:**I**
*in* length_penalty:**T**
*in* repetition_penalty:**T**
*in* vocab_mask:**M**
*in* prefix_vocab_mask:**M**
*in* attention_mask:**I**
*in* decoder_input_ids:**I**
*in* logits_processor:**I**
*out* sequences:**I**
*out* sequences_scores:**T**
*out* scores:**T**|1+|**T** = tensor(float)| |BiasGelu|*in* A:**T**
*in* B:**T**
*out* C:**T**|1+|**T** = tensor(float)| |BifurcationDetector|*in* src_tokens:**T**
*in* cur_tokens:**T**
*in* prev_suffix_match_idx:**T**
*in* pred_tokens:**T**
*out* tokens:**T**
*out* suffix_match_idx:**T**|1+|**T** = tensor(int64)| -|BranchwiseRMSNorm|*in* X:**T**
*in* scale:**M**
*out* Y:**T**|1+|**M** = tensor(bfloat16), tensor(float), tensor(float16)
**T** = tensor(bfloat16), tensor(float), tensor(float16)| +|BranchwiseRMSNorm|*in* X:**T**
*in* scale:**M**
*out* Y:**T**
*out* Y_fp32:**F**|1|**F** = tensor(float)
**M** = tensor(bfloat16), tensor(float), tensor(float16)
**T** = tensor(bfloat16), tensor(float), tensor(float16)| |CDist|*in* A:**T**
*in* B:**T**
*out* C:**T**|1+|**T** = tensor(double), tensor(float)| |CausalConvWithState|*in* input:**T**
*in* weight:**T**
*in* bias:**T**
*in* past_state:**T**
*out* output:**T**
*out* present_state:**T**|1+|**T** = tensor(float)| |ConvTransposeWithDynamicPads|*in* X:**T**
*in* W:**T**
*in* Pads:**tensor(int64)**
*in* B:**T**
*out* Y:**T**|1+|**T** = tensor(float)| @@ -633,7 +633,7 @@ The **OpSet Version** column uses the following notation: |QuantizeLinear|*in* x:**T1**
*in* y_scale:**T1**
*in* y_zero_point:**T2**
*out* y:**T2**|1+|**T1** = tensor(float)
**T2** = tensor(int16), tensor(int4), tensor(int8), tensor(uint16), tensor(uint4), tensor(uint8)| |QuickGelu|*in* X:**T**
*out* Y:**T**|1+|**T** = tensor(float)| |Range|*in* start:**T**
*in* limit:**T**
*in* delta:**T**
*out* Y:**T**|1+|**T** = tensor(double), tensor(float), tensor(int16), tensor(int32), tensor(int64)| -|RotaryEmbedding|*in* input:**T**
*in* position_ids:**M**
*in* cos_cache:**T**
*in* sin_cache:**T**
*out* output:**T**|1+|**M** = tensor(int64)
**T** = tensor(float), tensor(float16)| +|RotaryEmbedding|*in* input:**T**
*in* position_ids:**M**
*in* cos_cache:**T**
*in* sin_cache:**T**
*out* output:**T**|1|**M** = tensor(int64)
**T** = tensor(float), tensor(float16)| |SampleOp|*in* X:**T**
*out* Y:**T**|1+|**T** = tensor(float)| |ScaledSiLU|*in* X:**T**
*in* scale:**M**
*out* Y:**T**|1+|**M** = tensor(bfloat16), tensor(float), tensor(float16)
**T** = tensor(bfloat16), tensor(float), tensor(float16)| |Sampling|*in* input_ids:**I**
*in* max_length:**I**
*in* min_length:**I**
*in* repetition_penalty:**T**
*in* vocab_mask:**I**
*in* prefix_vocab_mask:**I**
*in* attention_mask:**I**
*in* presence_mask:**I**
*in* seed:**I**
*out* sequences:**I**
*out* filtered_logits:**T**|1+|**T** = tensor(float)| @@ -1087,10 +1087,10 @@ The **OpSet Version** column uses the following notation: |BiasGelu|*in* A:**T**
*in* B:**T**
*out* C:**T**|1+|**T** = tensor(bfloat16), tensor(double), tensor(float), tensor(float16)| |BiasSoftmax|*in* data:**T**
*in* bias:**T**
*out* output:**T**|1+|**T** = tensor(double), tensor(float), tensor(float16)| |BiasSplitGelu|*in* X:**T**
*in* bias:**T**
*out* Y:**T**|1+|**T** = tensor(float), tensor(float16)| -|BranchwiseRMSNorm|*in* X:**T**
*in* scale:**M**
*out* Y:**T**|1+|**M** = tensor(bfloat16), tensor(float), tensor(float16)
**T** = tensor(bfloat16), tensor(float), tensor(float16)| +|BranchwiseRMSNorm|*in* X:**T**
*in* scale:**M**
*out* Y:**T**
*out* Y_fp32:**F**|1, 2|**F** = tensor(float)
**M** = tensor(bfloat16), tensor(float), tensor(float16)
**T** = tensor(bfloat16), tensor(float), tensor(float16)| |BitmaskBiasDropout|*in* data:**T**
*in* bias:**T**
*in* residual:**T**
*in* ratio:**T1**
*in* training_mode:**T2**
*out* output:**T**
*out* mask:**T3**|1+|**T** = tensor(bfloat16), tensor(double), tensor(float), tensor(float16)
**T1** = tensor(bfloat16), tensor(double), tensor(float), tensor(float16)
**T2** = tensor(bool)
**T3** = tensor(uint32)| |BitmaskDropout|*in* data:**T**
*in* ratio:**T1**
*in* training_mode:**T2**
*out* output:**T**
*out* mask:**T3**|1+|**T** = tensor(bfloat16), tensor(double), tensor(float), tensor(float16)
**T1** = tensor(bfloat16), tensor(double), tensor(float), tensor(float16)
**T2** = tensor(bool)
**T3** = tensor(uint32)| -|BranchwiseRMSNorm|*in* X:**T**
*in* scale:**M**
*out* Y:**T**|1+|**M** = tensor(bfloat16), tensor(float), tensor(float16)
**T** = tensor(bfloat16), tensor(float), tensor(float16)| +|BranchwiseRMSNorm|*in* X:**T**
*in* scale:**M**
*out* Y:**T**
*out* Y_fp32:**F**|1, 2|**F** = tensor(float)
**M** = tensor(bfloat16), tensor(float), tensor(float16)
**T** = tensor(bfloat16), tensor(float), tensor(float16)| |CausalConvWithState|*in* input:**T**
*in* weight:**T**
*in* bias:**T**
*in* past_state:**T**
*out* output:**T**
*out* present_state:**T**|1+|**T** = tensor(bfloat16), tensor(float), tensor(float16)| |ComplexMul|*in* A:**T**
*in* B:**T**
*out* C:**T**|1+|**T** = tensor(float), tensor(float16)| |ComplexMulConj|*in* A:**T**
*in* B:**T**
*out* C:**T**|1+|**T** = tensor(float), tensor(float16)| @@ -1105,6 +1105,8 @@ The **OpSet Version** column uses the following notation: |EmbedLayerNormalization|*in* input_ids:**T1**
*in* segment_ids:**T1**
*in* word_embedding:**T**
*in* position_embedding:**T**
*in* segment_embedding:**T**
*in* gamma:**T**
*in* beta:**T**
*in* mask:**T1**
*in* position_ids:**T1**
*out* output:**T**
*out* mask_index:**T1**
*out* embedding_sum:**T**|1+|**T** = tensor(float), tensor(float16)| |EngramGate|*in* key:**T**
*in* query:**T**
*in* value:**T**
*in* key_norm_scale:**T**
*in* query_norm_scale:**T**
*in* conv_norm_scale:**T**
*out* output:**T**
*out* gated_value_normed:**T**|1+|**T** = tensor(bfloat16), tensor(float), tensor(float16)| |FastGelu|*in* X:**T**
*in* bias:**T**
*out* Y:**T**|1+|**T** = tensor(bfloat16), tensor(double), tensor(float), tensor(float16)| +|FoldMid2|*in* next_tokens:**I64**
*in* input_ids:**I64**
*in* status:**I32**
*in* ctl:**I64**
*in* ctx_commit:**I64**
*in* ctx_a:**I64**
*in* ctx_b:**I64**
*in* ptrs:**I64**
*in* cos_cache:**B**
*in* sin_cache:**B**
*out* head_ids:**I64**
*out* head_c:**I32**
*out* slot_ids:**I64**
*out* seqlens:**I32**
*out* cos_out:**B**
*out* sin_out:**B**
*out* sel:**I64**|1|**B** = tensor(bfloat16)
**I32** = tensor(int32)
**I64** = tensor(int64)| +|FoldTail|*in* next_token:**I64**
*in* status:**I32**
*in* ctl:**I64**
*out* done:**I32**|1|**I32** = tensor(int32)
**I64** = tensor(int64)| |FusedConv|*in* X:**T**
*in* W:**T**
*in* B:**T**
*in* Z:**T**
*out* Y:**T**|1+|**T** = tensor(float)| |FusedMatMul|*in* A:**T**
*in* B:**T**
*out* Y:**T**|1+|**T** = tensor(bfloat16), tensor(double), tensor(float), tensor(float16)| |GatedAdd|*in* X:**T**
*in* Y:**T**
*in* gate:**T**
*out* output:**T**|1+|**T** = tensor(bfloat16), tensor(float), tensor(float16)| @@ -1119,6 +1121,7 @@ The **OpSet Version** column uses the following notation: |GridSample|*in* X:**T1**
*in* Grid:**T1**
*out* Y:**T2**|1+|**T1** = tensor(float)
**T2** = tensor(float)| |GroupNorm|*in* X:**T**
*in* gamma:**M**
*in* beta:**M**
*out* Y:**T**|1+|**T** = tensor(float), tensor(float16)| |GroupQueryAttention|*in* query:**T**
*in* key:**T**
*in* value:**T**
*in* past_key:**T_CACHE**
*in* past_value:**T_CACHE**
*in* seqlens_k:**M**
*in* total_sequence_length:**M**
*in* cos_cache:**T**
*in* sin_cache:**T**
*in* position_ids:**tensor(int64)**
*in* attention_bias:**T**
*in* head_sink:**T**
*in* k_scale:**T_KV_SCALE**
*in* v_scale:**T_KV_SCALE**
*in* q_norm_weight:**T**
*in* k_norm_weight:**T**
*out* output:**T**
*out* present_key:**T_CACHE**
*out* present_value:**T_CACHE**
*out* output_qk:**T**|1+|**M** = tensor(int32)
**T** = tensor(bfloat16), tensor(float16)
**T_CACHE** = tensor(bfloat16), tensor(float16), tensor(float8e4m3fn), tensor(int8), tensor(uint8)
**T_KV_SCALE** = tensor(float)| +|HyperConnectionLowRankMix|*in* streams:**T**
*in* norm_scale:**T**
*in* down_weight:**T**
*in* up_weight:**T**
*in* inject_weight:**T**
*in* prev_block_output:**T**
*in* prev_post_gate:**T**
*out* pre_output:**T**
*out* post_gate:**T**
*out* streams_out:**T**|1+|**T** = tensor(bfloat16), tensor(float), tensor(float16)| |HyperConnectionPostMix|*in* streams:**T**
*in* block_output:**T**
*in* post_mix:**M**
*in* stream_mix:**M**
*out* output:**T**|1+|**M** = tensor(bfloat16), tensor(float), tensor(float16)
**T** = tensor(bfloat16), tensor(float), tensor(float16)| |HyperConnectionPreMix|*in* streams:**T**
*in* pre_mix:**M**
*out* output:**T**|1+|**M** = tensor(bfloat16), tensor(float), tensor(float16)
**T** = tensor(bfloat16), tensor(float), tensor(float16)| |Inverse|*in* X:**T**
*out* Y:**T**|1+|**T** = tensor(double), tensor(float), tensor(float16)| @@ -1132,6 +1135,7 @@ The **OpSet Version** column uses the following notation: |MatMulBnb4|*in* A:**T1**
*in* B:**T2**
*in* absmax:**T1**
*out* Y:**T1**|1+|**T1** = tensor(bfloat16), tensor(float), tensor(float16)
**T2** = tensor(uint8)| |MatMulNBits|*in* A:**T1**
*in* B:**T2**
*in* scales:**T1**
*in* zero_points:**T3**
*in* g_idx:**T4**
*in* bias:**T1**
*out* Y:**T1**|1+|**T1** = tensor(bfloat16), tensor(float), tensor(float16)
**T2** = tensor(uint8)
**T3** = tensor(bfloat16), tensor(float), tensor(float16), tensor(uint8)| |MoE|*in* input:**T**
*in* router_probs:**T**
*in* fc1_experts_weights:**T**
*in* fc1_experts_bias:**T**
*in* fc2_experts_weights:**T**
*in* fc2_experts_bias:**T**
*in* fc3_experts_weights:**T**
*in* fc3_experts_bias:**T**
*out* output:**T**|1+|**T** = tensor(bfloat16), tensor(float), tensor(float16)| +|MtpMoe|*in* input:**T**
*in* router_logits:**T**
*in* gate_up_weights:**T1**
*in* down_weights:**T1**
*out* output:**T**
*out* scratch:**T2**|1|**T** = tensor(bfloat16)
**T1** = tensor(uint8)
**T2** = tensor(float)| |MultiHeadAttention|*in* query:**T**
*in* key:**T**
*in* value:**T**
*in* bias:**T**
*in* key_padding_mask:**M**
*in* attention_bias:**T**
*in* past_key:**T**
*in* past_value:**T**
*in* past_sequence_length:**M**
*in* cache_indirection:**M**
*out* output:**T**
*out* present_key:**T**
*out* present_value:**T**
*out* qk:**QK**|1+|**QK** = tensor(bfloat16), tensor(float), tensor(float16)
**T** = tensor(bfloat16), tensor(float), tensor(float16)| |NGramHashMapping|*in* input_ids:**M**
*in* multipliers:**M**
*in* vocab_sizes:**M**
*in* past_ids:**M**
*in* head_offsets:**M**
*in* eos_token_id:**M**
*in* segment_ids:**tensor(int32)**
*out* hash_ids:**M**
*out* present_ids:**M**|1+|**M** = tensor(int32), tensor(int64)| |NGramRepeatBlock|*in* input_ids:**Tid**
*in* scores:**T**
*out* scores_out:**T**|1+|**T** = tensor(float)
**Tid** = tensor(int64)| @@ -1155,7 +1159,7 @@ The **OpSet Version** column uses the following notation: |RemovePadding|*in* input:**T**
*in* sequence_token_count:**M**
*out* output:**T**
*out* token_offset:**M**
*out* cumulated_seq_len:**M**
*out* max_seq_len:**M**|1+|**T** = tensor(float), tensor(float16)| |RestorePadding|*in* input:**T**
*in* token_offset:**M**
*out* output:**T**|1+|**T** = tensor(float), tensor(float16)| |Rfft|*in* X:**T**
*out* Y:**T**|1+|**T** = tensor(double), tensor(float), tensor(float16)| -|RotaryEmbedding|*in* input:**T**
*in* position_ids:**M**
*in* cos_cache:**T**
*in* sin_cache:**T**
*out* output:**T**|1+|**M** = tensor(int64)
**T** = tensor(bfloat16), tensor(float), tensor(float16)| +|RotaryEmbedding|*in* input:**T**
*in* position_ids:**M**
*in* cos_cache:**T**
*in* sin_cache:**T**
*out* output:**T**|1, 2|**M** = tensor(int64)
**T** = tensor(bfloat16), tensor(float), tensor(float16)| |Sampling|*in* input_ids:**I**
*in* max_length:**I**
*in* min_length:**I**
*in* repetition_penalty:**T**
*in* vocab_mask:**I**
*in* prefix_vocab_mask:**I**
*in* attention_mask:**I**
*in* presence_mask:**I**
*in* seed:**I**
*out* sequences:**I**
*out* filtered_logits:**T**|1+|**T** = tensor(float), tensor(float16)| |ScaledSiLU|*in* X:**T**
*in* scale:**M**
*out* Y:**T**|1+|**M** = tensor(bfloat16), tensor(float), tensor(float16)
**T** = tensor(bfloat16), tensor(float), tensor(float16)| |SkipGroupNorm|*in* X:**T**
*in* gamma:**M**
*in* beta:**M**
*in* skip:**T**
*in* bias:**T**
*out* Y:**T**
*out* S:**T**|1+|**T** = tensor(float), tensor(float16)| @@ -1634,7 +1638,7 @@ The **OpSet Version** column uses the following notation: |QLinearSigmoid|*in* X:**T**
*in* X_scale:**tensor(float)**
*in* X_zero_point:**T**
*in* Y_scale:**tensor(float)**
*in* Y_zero_point:**T**
*out* Y:**T**|1+|**T** = tensor(int8), tensor(uint8)| |QuantizeLinear|*in* x:**T1**
*in* y_scale:**T1**
*in* y_zero_point:**T2**
*out* y:**T2**|1+|**T1** = tensor(float), tensor(float16), tensor(int32)
**T2** = tensor(int8), tensor(uint8)| |QuickGelu|*in* X:**T**
*out* Y:**T**|1+|**T** = tensor(float), tensor(float16)| -|RotaryEmbedding|*in* input:**T**
*in* position_ids:**M**
*in* cos_cache:**T**
*in* sin_cache:**T**
*out* output:**T**|1+|**M** = tensor(int64)
**T** = tensor(float), tensor(float16)| +|RotaryEmbedding|*in* input:**T**
*in* position_ids:**M**
*in* cos_cache:**T**
*in* sin_cache:**T**
*out* output:**T**|1|**M** = tensor(int64)
**T** = tensor(float), tensor(float16)| |SkipLayerNormalization|*in* input:**T**
*in* skip:**T**
*in* gamma:**T**
*in* beta:**T**
*in* bias:**T**
*out* output:**T**
*out* mean:**U**
*out* inv_std_var:**U**
*out* input_skip_bias_sum:**T**|1+|**T** = tensor(float), tensor(float16)| |SkipSimplifiedLayerNormalization|*in* input:**T**
*in* skip:**T**
*in* gamma:**T**
*in* bias:**T**
*out* output:**T**
*out* mean:**U**
*out* inv_std_var:**U**
*out* input_skip_bias_sum:**T**|1+|**T** = tensor(float), tensor(float16)| | | diff --git a/onnxruntime/core/providers/cuda/plugin/cuda_kernel_adapter.h b/onnxruntime/core/providers/cuda/plugin/cuda_kernel_adapter.h index cf322fb3615c7..e325bc709ecf4 100644 --- a/onnxruntime/core/providers/cuda/plugin/cuda_kernel_adapter.h +++ b/onnxruntime/core/providers/cuda/plugin/cuda_kernel_adapter.h @@ -1181,6 +1181,7 @@ class CudaKernel : public OpKernel { return device_prop_; } + int GetDeviceId() const { return device_id_; } std::filesystem::path GetSessionModelPath() const { return runtime_config_->GetModelPath(); } // Identifies the session (execution provider instance) that owns this kernel. const void* GetSessionKey() const { return runtime_config_.get(); } @@ -1192,7 +1193,6 @@ class CudaKernel : public OpKernel { bool EnableHostPageableGather() const { return runtime_config_->enable_host_pageable_gather; } bool IsCudaGraphEnabled() const { return runtime_config_->enable_cuda_graph; } bool IsArchAvailable(int arch) const { return GetDeviceProp().major >= arch; } - int GetDeviceId() const { return device_id_; } // Delegate to the base OpKernel::Info() which holds a safe copy of OpKernelInfo. // Do NOT store a reference to the constructor parameter — it becomes dangling. const OpKernelInfo& Info() const { return OpKernel::Info(); }