Repository navigation
Feat(CI): add ci workflows - #8
Closed
real-zhangzhe wants to merge 59 commits into
Closed
real-zhangzhe wants to merge 59 commits into
real-zhangzhe wants to merge 59 commits into
Conversation
Albert/4.3.5 toolkit build See merge request his-devtech/onnxruntime-musa!164
Fix(bert infer RmsNorm bug) See merge request his-devtech/onnxruntime-musa!165
Fix(Dsin splitconcat fusion bug) See merge request his-devtech/onnxruntime-musa!166
Fix(matmul fusion empty concat) See merge request his-devtech/onnxruntime-musa!167
…ze route Concat failed when some of its inputs were produced on the CPU EP and the kernel read them as device memory. Stage those inputs onto the device before the kernel runs, and add a nearest/asymmetric/floor Resize route so models whose Resize nodes previously had no MUSA kernel can run on the plugin EP.
3D convolution (NCDHW) had no MUSA route, so 3D models fell back or failed. Route fp32 NCDHW Conv through muDNN, reusing the grouped/depthwise paths.
Unary kernels read inputs produced on the CPU EP as device pointers, which gave wrong results or page faults. Stage CPU-resident inputs onto the device on the compute stream before launching the kernel.
SiluFusion and Reshape could receive tensors produced on the CPU EP and read them as device memory. Stage such inputs onto the device before use.
ORT inserts MemcpyFromHost/MemcpyToHost at CPU/device boundaries, but the plugin EP had no kernels for them, so host pointers reached device kernels and caused page faults across many models. Register both kernels, running the copy on the plugin's compute stream, and update the CPU-boundary probe tests to expect device inputs. Adds boundary-copy tests and refreshes supported_ops.
Speech models use 1D Conv (NCL), which the EP rejected. Run 1D Conv as a 2D convolution with H=1, reusing the 2D and grouped paths; the output keeps rank 3.
…line unsupported configurations Resize only supported nearest with the scales input. Add linear (2D) and trilinear (3D) modes and the sizes input. Registering Resize made ORT assign every Resize node to MUSA, including configurations the kernel cannot run, which failed at run time. Add a capability predicate, shared with the kernel's own checks (resize_support.h), so unsupported configurations (cubic, antialias, tf_crop_and_resize, ...) stay on the CPU EP.
…D copies on the compute stream With the ORT memory pattern enabled, models failed from the second run on (broadcast with a 0-sized dim, Reshape overflow). Small (<1024 B) pageable host-to-device copies used a blocking memcpy on the null stream, not ordered with the compute stream, so a pending kernel could overwrite the copied shape metadata. Stage small copies through pinned memory and copy them asynchronously on the compute stream. Also declare Slice's starts/ends/axes/ steps as CPU inputs so Shape->Gather->Unsqueeze metadata chains stay on the CPU EP. Adds a discriminating test that fails before the fix and passes after.
Most kernels were registered from opset 13, so models exported at opset 10-12 ran almost entirely on the CPU EP. Following the ORT CUDA EP's version splits, add older-opset registrations for 32 operators whose semantics did not change (elementwise math, Transpose, Reshape, Gather, Concat, Expand, Cast, MatMul, Reduce*, Conv, ...). Fixes the Pow 7-11 type constraint and the Shape opset case found by the new per-opset tests.
Some operators changed semantics before opset 13: Softmax coerces its input to 2D, and Unsqueeze/Squeeze/Split/Clip/Pad/Slice take attributes instead of inputs. Branch on the node's opset in those kernels and register the old ranges, matching the ORT CUDA EP. Keeping Softmax on the CPU had forced large attention tensors across the CPU/device boundary. Widen the upper bounds of Reshape/Transpose/Shape/Squeeze/Unsqueeze to 21 and ReduceMax to 20. Read scalar FLOAT attributes correctly, keep Slice nodes with a negative step on the CPU EP, and keep Resize at opset <= 17.
…s, Gelu; fix empty attribute handles; run fp32 unary ops in single precision Claim Erf from opset 9, add float16 LayerNormalization, AveragePool, Sin, Cos and ONNX Gelu (opset 20-21) kernels. Treat an empty attribute handle from Node_GetAttributeByName (optional attributes with no default, e.g. Transpose perm, ReduceMean axes) as "attribute absent"; it used to crash session creation in GetCapability. Evaluate fp32 unary ops with single- precision functions (erff, sinf, cosf, ...) like the ORT CUDA EP; the shared double-precision path had become several times slower. Adds wide-range precision tests and keeps the boundary-copy tests on an unregistered CPU-side operator.
The fused node reads its Reshape target at runtime and requires it in host memory, but has no KernelDef, so once the Memcpy kernels let this EP claim the CPU->device boundary copies ORT may place that target on the device and the fusion fails with 'requires CPU shape input'. Walk the integer/bool metadata chain that produces the target and keep it off this EP so the tensor stays on the CPU (same effect as ONNX Reshape declaring its shape input as CPUInput).
…ape target The reshape target is built by Shape/Slice/Concat instead of an initializer, exercising the host-memory shape-input path.
…queezeConcat A fused node has no KernelDef, so it cannot declare its shape input as CPU memory. Once this EP claims the CPU->device boundary copies (the Memcpy kernels) ORT places a runtime-computed reshape target on the device and the fusion fails with 'requires CPU shape input'. Keep such a Reshape standalone so its own kernel declares the shape input as CPU memory (as ONNX Reshape does). Reverts the earlier CPU-preferred attempt, which could not stop ORT from copying the fused node's input to the device.
Registering MemcpyFromHost in the plugin EP makes ORT copy CPU-produced scalars onto the device before fused nodes. The raw INT32 mask path in the MHTA SDPA fusion still required its Slice end to be host-resident and rejected the fusion with ORT_NOT_IMPLEMENTED once that copy existed, which turned a recommendation model into a hard error. Accept a device-resident end and read the single int64 back on the compute stream; the copy node precedes this node on the same stream, so a stream synchronize is sufficient. The Slice end comes from shape arithmetic, so cache the resolved value per input-shape set to keep the synchronous read off the steady-state path; a non-shape-derived end is re-read once and the cache refreshed.
With MemcpyFromHost registered in the plugin EP, a CPU-produced repeats tensor is copied onto the device before the fused node runs. TileConcat read it with CopyToHost, which synchronizes the compute stream, so every iteration paid a device-to-host round trip; four recommendation models in the model zoo went from ~0.8-5.8 ms to ~5.5-11 ms. Cache the repeats across runs, but only when the matcher can prove the value is a pure function of fused-input shapes: the repeats chain must consist of shape arithmetic rooted at constants plus Shape ops whose source tensors are inputs of this fusion. The cache key is exactly those shapes, so a repeats that depends on anything else is re-read every run instead of being served stale.
The fused Tile+Concat node cannot declare a CPU input, so once the EP registered MemcpyFromHost ORT copied a runtime Tile repeats tensor onto the device and the kernel read it back with a device-to-host copy plus a compute-stream synchronize on every run. 2d06d07 avoided that by leaving such Tiles unfused, at the cost of a separate Tile kernel. Absorb the shape-only chain that produces repeats into the fusion instead. The matcher takes the chain nodes only when - every node is a shape-only op (Shape, Gather, Unsqueeze, Squeeze, Concat, Add/Sub/Mul/Div/Mod, Cast, Slice, Reshape, Identity, Floor, Ceil, ReduceProd, Constant, ConstantOfShape), - every leaf is a constant (Constant node or initializer) or a Shape source that is already a fused input, - no chain value is a graph output or is read outside the chain, and - Compile() can rebuild the expression from the fused graph. The kernel then evaluates repeats from the shapes of its inputs: no device read and no stream synchronize. Repeats that is a constant or a graph input keeps the previous path, and an expression that fails to evaluate still falls back to reading the tensor when the tensor is an input. The chain is not moved when it has an outside consumer, so such a Tile stays unfused and the original graph is untouched. ORT_MUSA_DISABLE_TILE_CONCAT_FUSION=1 drops the TileConcat match, so one binary can measure the fusion on and off (the H1 attribution lever). DoD: clean build on the test host (SDK 4.3.8) with no new compiler warnings. The built .so sha256 is 9b403064aaa1b4ac0f06dbdb510339724eed01d2db29a668788680b0f6425937. Unit tests are written but not executed: no GPU window was granted this round. H1 (the recommendation-model residual comes from dropping this fusion) remains an unverified hypothesis.
209e500 absorbed the shape-only chain that produces Tile repeats into the Tile+Concat fusion and vetoed the fusion when any chain value had a consumer outside it. In the four recommendation models in the model zoo every such chain value is also read by another branch (MathConcatLog and friends), so all 140/140/136/140 Tiles fell back and the fusion never ran. Absorb nothing: the fused node is Tile + Concat again, the chain stays in the graph, and repeats stays an input of the fused node. The matcher still proves that repeats is a pure function of the shapes of the fused inputs (shape-only whitelist, every Shape source already an input of Tile+Concat, every constant readable) and hands that expression to the fused node, which evaluates it from the input shapes instead of reading the tensor. That keeps the device-to-host copy and the compute-stream synchronize out of the steady state while the tensor remains the fallback. A Tile whose repeats cannot be proven keeps its standalone kernel and becomes an input of the fused Concat; per-Tile fallback replaces rejecting the whole Concat. Compile() only receives the fused subgraph, so it cannot rebuild the expression: GetCapability() stores the proofs in the EP keyed by the fusion boundary (its input and output value names), and Compile() maps the recorded source names onto the input indices of the fused node ORT built. Two vetoes disappear with the absorption: - a chain value consumed outside the fusion no longer vetoes anything, - a chain containing Gather + Reshape no longer vetoes anything. That check kept the fused body away from IsShapeReshapeFusionGraph, a loose presence check (Gather + Concat + Reshape) that Compile() runs before the TileConcat detector: while the chain was absorbed, the fused body carried the chain's Gather and Reshape next to this fusion's Concat and was handed to the ShapeReshape factory. The body now holds only Tile + Concat, so that collision cannot arise. Session creation now prints one MUSA_TILE_CONCAT_REPEATS line per runtime repeats value (shape_expr=1 when the expression arrived, 0 when the fused node will read the tensor) so the verification round can see which path ran. DoD: clean build on the test host (SDK 4.3.8) with no new compiler warnings; the .so sha256 is fc422405abfae83bf8251b812d4063cb6d85ecf40880c2fcf4cbbcd2698cd188. Unit tests are written but not executed: no GPU window this round. The 123 absorbed / 13-17 fallback counts are a static estimate and are not verified on the models. H1 (the recommendation-model residual comes from dropping this fusion) remains an unverified hypothesis.
Drop the starts/ends/axes/steps CPUInput declarations added in fd8ca63. With them, ORT inserts a MemcpyToHost for every Slice execution whose metadata chain runs on the device; inside dien's two GRU Loop bodies this produced about 300 D2H copies per inference (303 vs 3 on master), six per iteration. The Slice kernel already reads those inputs through ReadIntTensors on the compute stream, so this restores the pre-fd8ca63 device-read path while keeping the small-copy ordering fix from that commit. The dynamic-shape metadata and boundary-copy tests, plus zipformer run twice, gate the buffer-reuse failure.
ValueDependsOnlyOnInputShapes required every Shape source to be an input of the fusion. A recommendation model's raw INT32 mask end is Unsqueeze(Gather(Shape( Transpose(k_norm)))) where the Transpose output has no other consumer and its input is the fused K input, so the cache was disabled and the end was read from the device with a stream synchronize on every run (about 0.9 ms). Follow Transpose/Cast/Identity from a Shape source back to the fused inputs that determine its shape. The Shape value is then still a pure function of those input shapes, so the cache key stays sound; no fusion input or output is added and no copy is introduced. Adds a two-shape test that runs the same session with seqlen 5, 4 and 5 again, checks the mask follows K's shape, and a MUSA_MHTA_SLICE_END diagnostic line for verification rounds.
Transpose takes perm as an attribute, not as a second input.
…model onnx.checker requires a shape on every graph output; Y now carries the symbolic [batch, heads, seqlen, head_dim] shape.
…mpile Compile() only receives the fused subgraph, so ValueDependsOnlyOnInputShapes() there could never see the end's producer chain and always returned false for a dynamic end. Every raw-INT32-mask attention node therefore read the end from the device with a stream synchronize on every run (about 0.9 ms on a recommendation model). Prove the dependency in the matcher, which still sees the whole graph, and hand the proven fused-input names to Compile through a boundary-keyed hint map, the same way TileConcat carries its shape expressions. Follow shape-preserving producers (Transpose/Cast/Identity), and accept a value shared with a fused input through another Transpose: both expose its shape up to a fixed permutation. Tests: the dynamic end test now asserts cacheable=1 and keeps running seqlen 5/4/5 in one session; two negative controls (an unrelated shape source, and a producer outside the whitelist) assert cacheable=0 and stay correct.
A fused node cannot declare CPU inputs, so a runtime repeats tensor is copied onto the device and read back with a stream synchronize. 2d06d07 avoided that by leaving such Tiles standalone; on a multi-task recommendation model the 13 remaining Tiles cost about 0.26 ms end to end. With ORT_MUSA_TILE_CONCAT_RUNTIME_REPEATS=1 the matcher keeps those Tiles in the fusion and the kernel reads the shared repeats tensor once per execution (memoized by input index) instead of once per Tile. The switch is off by default so the tree keeps the safe fallback until a measurement round accepts the trade; a unit test covers the outside-source Tile under the switch.
InputIndex takes the end value info, not its name; tile_concat_fusion.cc needs the matcher-utils header for FusionBoundaryKey.
TileConcatFusionKey lives at global scope; the key helper is in musa_ep.
The Slice-end chain (Shape/Gather/Transpose) legitimately stays on the CPU EP; disabling the fallback rejected the session after the matcher had already run.
ORT_MUSA_MHTA_END_TRACE=1 prints hit/miss, key size, resolved value and Sk for the first 64 computes, so a measurement round can tell whether the cache is hit and whether the read is the remaining cost.
The boolean raw-INT32 branch has two construction sites. The bshd-strided-flash one forwarded the matcher's proof; the kBhsd fallback did not, so a recommendation model's fused nodes printed cacheable=1 at Compile and ran with cacheable=0, reading the Slice end from the device every run. The runtime trace added for the diagnosis showed the mismatch (compile cacheable=1, compute cacheable=0).
c4 removed the hard input/output alias so a CPU-produced input could never be read as device memory; every Reshape then allocated a fresh output and copied (about 0.19 ms over 46 Reshapes on a graph model, 0.27 ms on a recommendation model). Advertise a mutable alias instead (KernelDefBuilder_AddInputOutputMutableAliases, the EP API's MayInplace): the allocator reuses the input buffer only when it is safe, Compute() skips the copy when src == dst, and every other case still copies. With MemcpyFromHost registered the kernel only sees device tensors, so the staged-input path keeps c4's safety. Adds a trace line and three tests (reuse, copy with a second consumer, staged CPU-produced input).
The reuse case compared a (3,2) output with a (2,3) expected array, and both trace assertions ran without ORT_MUSA_RESHAPE_TRACE set.
c4 removed the alias before c5 existed. With MemcpyFromHost registered, ORT stages a CPU-produced input onto the device before the kernel runs, so Reshape only sees device tensors and the alias is safe again; Compute() skips the copy when src == dst and keeps the copy on the defensive different-pointer path. The may-inplace route showed no allocator reuse on the measured models.
A graph output does not share a buffer with its input, so the Reshape output must feed another node for the alias to be observable.
…tion standalone cache, cherry-pickable onto ba650d0. Cache successful expression results by repeats input index before source-shape construction; preserve tensor fallback on evaluation failure and reset all cached values on each Compute. The isolated-index apply onto ba650d0 passed with no partner dependency. Stage A static checks pass; build, runtime and performance validation remain pending.
ORT_MUSA_TILE_CONCAT_EXPR_TRACE=1 prints actual evaluator calls and distinct repeats inputs once on every Compute exit. Disabled tracing does not count or allocate distinct-input entries. Keep instrumentation separate from the standalone cache commit for clean performance comparisons. Stage A formatting checks pass; runtime validation remains pending.
Add two Tiles sharing one shape-derived repeats value to the existing shape-chain suite. Reuse one session for batches 3 and 5, compare exactly with CPU, require fused execution and one evaluator call per run when tracing is enabled, and require silence when disabled. Disable runtime CPU retry for the new MUSA session and preserve every existing test and helper. Stage A diff, C++ formatting and Python syntax checks pass; build and full unit tests remain for the Supervisor.
real-zhangzhe
force-pushed
the
albert_dev
branch
from
October 27, 2026 07:25
5ade592 to
f954b50
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
Notes