Consolidate projection packing and lazy weight-layout folding by titaiwangms · Pull Request #754 · onnxruntime/mobius

titaiwangms · 2026-10-02T17:41:50Z

Scope

Consolidate the generic projection-packing and static weight-layout work here.
The independent layout-folding implementation from #788 is included; Gemma4
modeling is kept separately in #787, stacked on this branch.

One canonical float/INT4 QKV projection packer, with pre-mutation guards for
repeated projection sources and shared quantization parameters.
Deferred axis-0 Concat binding and adapter restoration metadata.
Lazy, dtype-preserving Reshape/Transpose folding after weight loading,
including external-source ownership and nested initializer ownership.
Pre-write alias checks covering model, adapter, policy, and MTP artifacts.

The existing float capability gates an additional float projection-packing
stage for both ordinary and arbitrary-mask decoders. The older GQA-input
packing stage remains decoder-only. INT4 projection packing remains
experimental and disabled on every built-in execution provider.

Review follow-up

Includes the adapter hardlink/symlink and repeated-Q/K-source corrections
originally reviewed in #787, plus the captured-parent initializer correction
reviewed in #788. Regression tests check unchanged source data, public adapter
restoration, actual initializer ownership, and persisted payloads.

Qualification limits

Changing GEMM output width can change numerical results. The previously failed
large-dimension CUDA INT4 packing numerical gate remains failed and unwaived;
neither graph validity nor an INT4 storage round-trip establishes numerical
qualification. This PR does not enable a custom GEGLU runtime.

The separate MMLU-Pro500 comparison uses a frozen performance artifact, not a
new export of this PR head. Its eventual score cannot qualify this revision's
full-checkpoint logits or KV-cache parity.

Consolidated-revision validation

Generic packing/folding/package/binding suites plus the 22 unchanged
storage-compatibility regressions: 516 passed, 3 skipped.
Combined modeling-stack non-integration suite: 11,311 passed, 29 failed,
138 skipped. Of the failures, 28 also failed when their exact selectors
were run on the exact base. The additional GGUF file-identity failure did
not reproduce when run separately on either base or head. The suite is
not green; unrelated environment/baseline failures were not changed.
Scoped Ruff check and format check passed. Scoped MyPy remains unclean on
base and head; the two new pass modules have no diagnostics. Four existing
HF generate self-type diagnostics occur on both sides with different
rendered callable signatures and are not new call-site changes.
Two bounded rounds of five-role static review completed. CPU runtime probes
confirmed the nested-scope failure before its fix and validated the repaired
persistence path. This is not CUDA or full-checkpoint qualification.

ID	Severity	Raised by	Finding	Disposition
F1	Major	Code / critical	Nested layout folding exposed root-only Concat mutation	Fixed: scope-local registration, reuse, removal, child cleanup, and sibling-branch persistence regression
F2	Major	Readability	Documentation obscured the new default float stage	Fixed: capability, roles, and float/INT4 qualification are explicit
F3	Major	Lead validation	Folding removed named QMoE/FP8 storage anchors	Fixed: conservative operator-slot storage guards; all 22 unchanged regressions pass

No Critical/Major reviewer finding was overruled. Style-only refactoring was
deferred. Residual non-blocking question: deletion-tree containment for lazy
sources inside replaced MTP directories is not established as a new regression
and is not qualified here. Ordinary-save TOCTOU, custom schemas outside the
emitted operator contracts, CUDA/full-checkpoint parity, and performance remain
excluded.

Complete original captured PR body — historical prior implementation (not validation of the reorganized head)

The historical results in the captured body below describe the prior implementation and are NOT validation of the reorganized head. Older statements about unchanged float behavior are historical and superseded by the new summary above. All original numerical limitations and earlier MMLU/performance evidence are preserved verbatim.

Status: implementation review — experimental opt-in only

This PR implements projection-only INT4 QKV packing as an EP-aware graph pass.
The mechanism is available for implementation review. New same-weight real-model MMLU and generation measurements are recorded below; they do not silently clear the earlier numerical criterion.

Default-disabled for all built-in EPs, including CUDA. Experimental evaluation requires explicitly enabling matmul_nbits_qkv_pack_dtypes through the existing capability API, or invoking the pass directly. The stock CLI --ep cuda retains separate quantized Q/K/V projections. Existing float PackQKV, GQA, and normalization behavior is unaffected.

Representative larger-dimension prefill and cached-decode logits still exceed the proposed rtol=atol=1e-2 parity criterion. Follow-up isolation attributes the tested M=74 projection divergence to shape-dependent CUDA GEMM arithmetic, rather than incorrectly concatenated or differently dequantized weights. This is not a blanket numerical guarantee. Default activation remains blocked on an agreed numerical acceptance contract and checked-in representative larger-dimension CUDA regression coverage.

Changes

An Attention/GQA-anchored pass replaces three compatible MatMulNBits projections with one larger MatMulNBits followed by Split.
Bias, Q/K normalization, RoPE, attention masks, caches, graph outputs, and external consumers retain their existing paths.
Experimental CUDA opt-in scope: FP16, homogeneous INT4, matching canonical blocked layouts and attributes, omitted/zero accuracy level, absent or UINT8 zero-points. All built-in EP defaults leave this pass disabled; incompatible dtypes/layouts remain unchanged even when opted in. Mixed 4/4/8 is not partially packed or promoted.
Parameter Concat nodes retain original checkpoint names before loading; existing post-weight folding materializes the packed initializers. Binding validation follows deferred axis-0 Concats so missing sidecars cannot hide.
Add opset, compatibility, overlap, idempotence, lifecycle, provider-assignment, and numerical tests. Update EP documentation.

This is not a new export path or runtime operator. Projection packing resembles ModelBuilder's packed-projection-plus-Split path for Q/K-normalized models, but the complete attention graphs need not match: this pass leaves Mobius's norm/RoPE paths intact. An aligned-weight ModelBuilder numerical comparison has not been performed.

Validation

Environment: A100-SXM4-80GB, installed ORT 1.30.0 Release build f1d07f0d1, ONNX IR 1.0.0. Tests used worktree-local imports and no checkpoint downloads.

PYTHONPATH=src:. HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 CUDA_VISIBLE_DEVICES=0 \
  python -m pytest -q \
  src/mobius/rewrite_rules/_matmul_nbits_qkv_test.py \
  src/mobius/_component_quantization_loading_test.py \
  tests/ep_optimization_test.py

Original implementation targeted suite: 309 passed. Default-disabled review fix, October 7, 2026: 317 passed, no skips with the same three-file command above.
New default-off graph regressions cover both tiny and K=4096/block=128 decoders, with/without Q/K norm and with/without optimization tracing. All retain separate projections and existing GQA. Packing/lifecycle/numerical tests now explicitly opt in using a non-autouse fixture, preserving their positive fusion coverage. The larger graph regression validates gating, not larger-case numerical parity.
Related float GQA/QKV, Concat/dtype folding, and MatMulNBits regressions: 86 passed, 2 skipped on the first implementation revision; the skips were unavailable qwen3_5_text registry cases.
Ruff, targeted repository lintrunner, and whitespace checks passed. The new pass passed strict MyPy. An unsuppressed scoped baseline comparison had 20 existing errors on base versus 19 on the initial implementation, with no introduced errors; this is not a full clean repository type check.
Small projection tests compare source/packed/reference outputs for prefill and decode, including padded K and odd block counts. Profiling confirms native CUDA MatMulNBits assignment, not CPU fallback.
Small biased full-decoder tests pass prefill and own-cache decode with/without Q/K norm and UINT8 zero-points.
Bit-exact parameter slices, original-name loading, unloaded placeholders, external-data save/ModelPackage.load, missing weights, and 4/4/8 no-op behavior are covered.

CI snapshot on October 2, 2026: 23 successful checks, two cancelled checks (L4 Golden Comparison and L5 Generation E2E), and one skipped check (pinned SmolLM F16 CPU). There were no pending checks at that snapshot. Cancelled/skipped coverage is not counted as passing.

No uncontended model-wide speed improvement is claimed; bounded, shared-GPU measurements are below.

Confirmed larger-dimension limitation

Manual projection reproduction uses the committed _matmul_nbits_qkv_test._model fixture with K=4096, block size 128, output widths (4096, 1024, 1024), weight seed 17, input seed 23, absent/present UINT8 zero-points, and ORT_DISABLE_ALL.

At [1,1,4096], source and packed projection outputs were bit-identical. At [2,37,4096], some K/V elements exceeded rtol=atol=1e-2; Q remained bit-identical. Maximum source/packed absolute difference was 0.0625. The FP32 decoded-weight reference also disagrees with the original source kernel, so that reference does not by itself isolate a packing defect.

A representative offline full decoder used one layer, hidden size 4096, 32 query heads, 8 KV heads, head dimension 128, intermediate size 11008, vocabulary 256, Q/K normalization, block size 128, symmetric INT4, canonical-weight seed 41, and scales in [0.003, 0.01]. Prefill used arange(74).reshape(2,37); decode tokens were [[7],[8]], with each model's own prefill cache. Both exporters folded weights; ORT optimizations were disabled.

CUDA GEMM options	Prefill logits outside tolerance	Decode logits outside tolerance	Maximum absolute difference, prefill / decode
Default	26 / 18,944	7 / 512	0.033203125 / 0.029296875
`ORT_CUDA_GEMM_OPTIONS=2`	0 / 18,944	4 / 512	0.0126953125 / 0.017578125

Full KV tensors stayed within the same tolerance in these runs. Profiling confirmed 14 source / 10 packed native CUDA MatMulNBits events across the two steps. Changing GEMM options changes the observed errors, but does not prove a particular kernel/reduction schedule or resolve decoder parity. No tolerances were loosened.

Follow-up isolation: weights, GEMM arithmetic, and cache propagation

These diagnostic experiments used the installed ORT 1.30 build f1d07f0d1 and its exact source revision f1d07f0d15d91ba4be889b7f0b02f304329bed89. They are separate from the ORT 1.31 real-model evaluation below.

On the published Mobius revision f903b405, exposing intermediate outputs reproduced the original 26 prefill / 7 own-cache-decode logits failures. The first prefill differences occur in K/V projection outputs; Q is bit-identical.
Decode with the same source cache produces bit-identical outputs throughout the tested source/packed graphs. Current decode Q/K/V projections are also bit-identical with each graph's own cache. The tested own-cache decode differences therefore propagate from prefill cache history.
An offline standalone CUDA spike compiled the exact-revision INT4 dequant kernel. At K=4096, block=128, widths (4096,1024,1024), absent/UINT8 zero-points, and M=1/74, corresponding source/packed dequantized FP16 weight sections were byte-identical.
At M=74, a matching half-input cuBLAS cublasGemmEx experiment reproduced both the source and packed ORT projection outputs bit-for-bit, including their differences. This establishes a shape-dependent GEMM arithmetic explanation for those cases, but does not identify the actual cuBLAS tactic or reduction schedule.
The same GEMM mimic does not match the M=1 native ORT path. No conclusion about that path's internal implementation is claimed.
A test-only fixed-K FP32 FMA reference made separate/combined projections bit-identical, but still violated comparisons against the original source runtime. It was not integrated into ORT, is not a production performance implementation, and is not shipped by this PR.
Forcing runtime N-chunks of 1024 made the tested full-decoder source/packed outputs bit-identical, but also changed the original source computation: packed-chunked versus original-default prefill still had 26 logits outside tolerance. This is a diagnostic control, not a proposed workaround.

The standalone spike and larger-case reproductions remain local experiment artifacts, not checked-in portable regression tests. No Mobius compensation, graph-level FP32 promotion, runtime precision override, or norm fusion was added to the implementation.

Same-weight real-model MMLU A/B

Model: Qwen3-30B-A3B, using the cached selected-V/expert-down INT8 RTN artifact associated with microsoft/olive-recipes#637. Original HF snapshot: ad44e777bcd18fa416d9da3bd8f70d33ebb85d39. Default projection/expert precision is symmetric INT4, block size 128; 24 selected layers use INT8 V and expert-down weights; embeddings are INT4 and the untied output head is INT8.

This is not a 4/4/8-versus-8/8/8 quantization comparison. A and B use the same already-quantized weights:

A: the original unpacked graph.
B: that same graph with only this PR's QKV packing pass and post-weight constant folding applied.
24 homogeneous 4/4/4 layers packed, widths 4096+512+512=5120; the other 24 mixed 4/4/8 layers stayed separate.
MatMulNBits: 193 -> 145. Both graphs retain 48 GQA, 48 QMoE, and identical existing norm/RoPE paths. No new Q/K norm fusion.
Persisted packed weights/scales hash-match the canonical source concatenations; other initializer contents are unchanged. The pass is idempotent on the packed graph.

Runtime: A100-SXM4-80GB, the existing isolated ORT 1.31 Release build a11b4e593164c82f7ce95efce3693744103f8176, GenAI 0.16.0.dev0, lm-eval 0.4.12, and the Olive ortgenai backend. A/B loaded identical hashed native libraries. This runtime supports the mixed-width QMoE attributes; the earlier installed ORT 1.30 f1d07f0d1 cannot load either real-model arm and was not used for these scores.

Protocol: 57 MMLU subjects, limit 200 per subject, 9,183 examples per arm, batch 1, max length 4096, task-default zero-shot, no chat template, seeds 0/1234/1234/1234, bootstrap iterations 100000. Offline cached dataset revision: c30699e8356da336a370243923dbaf21066bb9fe. This is a bounded subset, not the complete MMLU test set.

Metric	A: packing off	B: packing on
Correct / 9,183	7,293	7,299
Accuracy	79.4184906893%	79.4838288141%

Net +6 correct / +0.065338 percentage points; no statistical improvement claim.
34 answer flips: 10 correct -> wrong, 16 wrong -> correct, 8 wrong -> a different wrong answer.
All 36,732 per-option request/token records matched across arms, including context, continuation and actual input-token hashes. Task/doc identifiers matched, and all measured option likelihoods were finite.
Maximum absolute option loglikelihood difference: 1.0312480927. Median per-question maximum difference: 0.0468492508.
Top-score ties: A=3, B=6; the existing task's first-index argmax tie rule was retained.

Interpretation: no aggregate MMLU score decrease was observed in this protocol, but numerical outputs and some answers changed. These results do not establish general quality invariance, resolve the earlier pointwise parity failure, or cover other tasks/hardware/runtime builds.

TTFT and decode measurements: shared GPU, descriptive only

Same A/B packages and ORT 1.31 runtime as above. Batch 1, pretokenized identical 32/512/2048-token prompts, greedy 128 new tokens, two warmups plus three measured repeats per shape/arm. All requests generated exactly 128 tokens; decode throughput excludes the first generated token (127 timed decode tokens).

CUDA was synchronized before the timed region, after prompt append, after the first token, and after the final token. Model initialization, tokenization, and Generator construction are excluded from TTFT. Profiling was disabled in latency measurements. A ran before B.

TTFT in milliseconds, median [min, max]:

Prompt tokens	A: packing off	B: packing on
32	68.673 [49.942, 117.461]	64.188 [56.224, 114.593]
512	104.210 [98.392, 151.077]	111.640 [80.860, 150.797]
2048	239.811 [166.061, 245.616]	257.727 [251.707, 258.535]

Decode tokens/s, median [min, max]:

Prompt tokens	A: packing off	B: packing on
32	17.884 [15.762, 31.382]	20.739 [19.258, 21.611]
512	20.024 [18.975, 20.397]	20.100 [19.534, 21.085]
2048	19.476 [19.330, 19.516]	19.730 [19.440, 20.020]

Initialization, measured separately and including the evaluator's three-token probe: A=60.780s, B=62.199s.

Across these three prompt shapes, all corresponding A/B 128-token greedy sequences were identical, including all warmups/repeats; within-arm repeats were identical too. These are bounded timing/generation prompts, not a semantic-quality or HF-reference generation test.

Important limitations: other workloads shared GPU0; observed SM clocks varied between 1155 and 1410 MHz. Only three measured repeats were taken, sequentially by arm. These are descriptive measurements, not an uncontended estimate of packing's speedup. There is no consistent large TTFT improvement across contexts. Both graphs include unchanged QMoE work; no per-op runtime profile was collected, so QMoE is not asserted to be the bottleneck. Fewer projection nodes do not imply proportionally fewer FLOPs or a proportionate whole-model speedup.

Reproducibility and evidence boundaries

All A/B smoke, timing, and MMLU workers completed with exit 0. Original per-example likelihoods/tokens, timing repeats, graph/weight hashes, library hashes, and commands are retained in local experiment artifacts.
These experiment harnesses/results are not checked into this PR. The checked-in unit/lifecycle tests above are directly runnable; the real-model protocol above is specified, but a portable end-to-end reproducer remains work to do.
A/B used matching explicit provider settings, the same native library hashes, and the same configured QMoE scratch cap of 2 GiB. ORT_CUDA_GEMM_OPTIONS was recorded unset. The benchmark provenance mistakenly logged unprefixed aliases for three other controls rather than ORT_FPA_INTB_GEMM, ORT_MATMULNBITS_FORCE_CHUNKED, and ORT_MATMULNBITS_CHUNK_SIZE; historical absence of those actual variables cannot be proven from the saved metadata. No precision/chunk workaround was added by the benchmark scripts, but full environment-override absence is not claimed. Future reproductions must record the actual names.
The standalone ORT diagnostic reference is local and does not constitute a provider integration, runtime fix, or dependency of this PR.
Raw timing and MMLU process wall times are not used as uncontended kernel-performance evidence. No other users' process identities or environment data are published here.

fpA/intB and constant folding

With ORT_FPA_INTB_GEMM=1, exporter fold_constants=False, and runtime ORT_DISABLE_ALL, a packed projection failed because the opt-in CUDA path requires prepacked weight/scales, while deferred Concats produce non-initializer inputs. Source and exporter-folded controls ran.

For the tested K=128, block 128, widths (64,32,32) case, runtime BASIC and ALL optimization folded the Concats into initializers and ran successfully, matching the exporter-folded control. Keep exporter or runtime constant folding enabled for this opt-in path. This evidence is bounded to the tested runtime/configuration, not a guarantee for all kernels.

Review disposition

Five static reviewers reviewed the initial implementation. Minor documentation issues were addressed, the standard-opset guard was added, and CUDA tests now skip only when baseline CUDA cannot initialize; packed-only provider loss still fails. The final bounded hardening diff was reviewed directly by the lead.

The October 5 review requesting default-disabled or explicit experimental opt-in is addressed by leaving every built-in EP's matmul_nbits_qkv_pack_dtypes empty. The pass and existing capability API remain available for explicit experiments. Documentation and regressions reflect that boundary; no tolerance or runtime precision setting was changed. The default-off fix does not resolve F1 below.

ID	Severity	Raised by	Finding	Disposition
F1	Major	Numerical QA / lead acceptance check	Larger-dimension decoder logits do not meet the proposed parity criterion	Confirmed, unresolved. Tested M=74 projection differences were reproduced by shape-dependent GEMM arithmetic; real-model MMLU did not decrease in the measured protocol. Neither clears the pointwise criterion.

No Critical/Major reviewer finding was silently rejected or overruled. The tested M=74 GEMM arithmetic explanation now has executable evidence; the actual selected tactic/reduction schedule remains unidentified. Numerical drift is not automatically classified as a packing-byte defect, nor silently accepted for merge.

Remaining work and reviewer decisions

Keep quantized QKV packing default-disabled and require explicit experimental opt-in.
Check in a portable larger-dimension CUDA regression/reproducer.
Isolate byte-identical dequantization, shape-dependent GEMM arithmetic for M=74, and cache-error propagation for the tested exact runtime.
Identify/cover the remaining native kernel paths and decide representative GPU CI coverage.
Resolve the proposed decoder parity criterion or explicitly agree on a different, justified numerical contract; do not merely loosen tolerance.
Compare matched weights/attributes against ModelBuilder, separating projection packing from attention/norm differences.
Run same-weight real-checkpoint MMLU A/B and bounded 128-token greedy generation comparisons.
Broaden generation/task coverage and add independent reference validation; the timing prompts alone do not establish general quality.
Correct environment provenance and measure performance on an uncontended GPU with more repeated/interleaved trials before making a speedup claim.

Not established: general task/generation quality, other GPU architectures, native block sizes beyond 32/128, dedicated CUDA-graph capture validation, uncontended speedup, robust peak-memory comparisons, or all opt-in runtime configurations. GPU provider libraries were loaded and real inference executed; no node-by-node profile of the real-model run was collected. Pre-RoPE arithmetic outside the bounded matcher and nested attention graphs deliberately remain no-ops. A separate Q/K norm-to-GQA fusion is out of scope and has not been implemented.

Current follow-up validation and CI root cause

Published follow-up c914cc42094baca183dce244e42ed6c11e3c620d. It skips Concat folding if any input is declared in graph inputs across lexical scopes, preserving runtime overrides rather than merely retaining the registry. Root and parent capture default/non-default CPU/ORT tests cover before/after ModelPackage round trips. Two regressions fail with the prior implementation; 78 pass in the current pass files, 225 pass combined, and scoped lint passed.

The initial CI environment used TensorFlow 5.20.2 / Hugging Face Hub 2.3 / httpx 2, then --no-deps forced TensorFlow 5.16.dev0 whose metadata requires huggingface-hub>=1.5,<2; old Hub 1.x supplies the older httpx. This is a dependency-closure violation, not merely missing httpx, and is not changed by this PR. No CI code fix is included and CI green is not claimed. Evidence: pinned TensorFlow setup, Hub v1.5.0 setup.py, and Hub v2.3.0 setup.py. Resolve the pinned TensorFlow dependency normally; use --no-deps only for VibeVoice plus pip-check/import smoke, with no automatic new PR scope. Strict numerical gates remain failed/unwaived. The full five-role, two-round review is already complete; this follow-up is a direct bounded review only.

github-actions · 2026-10-02T17:43:13Z

🏗️ Architecture Diff

Comparing 2e5508f1 → 6ed50516

Model	Sub-model	Changes	Status
bert (feature-extraction)	model	0	⚪
falcon	model	3	🟡
gemma2	model	11	🟡
gemma4 (gemma4)	decoder	11	🟡
gemma4 (gemma4)	embedding	0	⚪
gemma4 (gemma4)	vision_encoder	56	🟡
gemma4_text	model	11	🟡
gpt2	model	0	⚪
llama	model	0	⚪
llama (static-cache)	model	0	⚪
mamba (ssm-text-generation)	model	0	⚪
phi3	model	0	⚪
phi3 (static-cache)	model	0	⚪
qwen	model	0	⚪
qwen (static-cache)	model	0	⚪
qwen2	model	0	⚪
qwen2 (static-cache)	model	0	⚪
qwen2_moe	model	15	🟡
qwen2_moe (static-cache)	model	0	⚪
qwen3	model	0	⚪
qwen3 (static-cache)	model	0	⚪
qwen3_5_moe (hybrid-text-generation)	model	34	🟡
qwen3_5_text (hybrid-text-generation)	model	20	🟡
qwen3_5_vl (hybrid-qwen-vl)	decoder	21	🟡
qwen3_5_vl (hybrid-qwen-vl)	embedding	0	⚪
qwen3_5_vl (hybrid-qwen-vl)	vision_encoder	0	⚪
qwen3_moe	model	19	🟡
qwen3_moe (static-cache)	model	0	⚪
qwen3_next (hybrid-text-generation)	model	56	🟡
t5 (seq2seq)	decoder	0	⚪
t5 (seq2seq)	encoder	0	⚪
whisper (speech-to-text)	decoder	0	⚪
whisper (speech-to-text)	encoder	0	⚪

falcon / model — 3 change(s)

Op summary: 66 → 68 nodes

--- base
+++ head
@@ -1,6 +1,8 @@
 Gather
 Gather
 Gather
+Shape
+ConstantOfShape
 CumSum
 Unsqueeze
 Shape

Added nodes:

+ Shape
+ ConstantOfShape

Connectivity changes:

node[11] Unsqueeze: input_ids [1, 11] → [54, 12]

gemma2 / model — 11 change(s)

Op summary: 105 → 109 nodes

--- base
+++ head
@@ -2,6 +2,8 @@
 Mul
 Gather
 Gather
+Shape
+ConstantOfShape
 CumSum
 Unsqueeze
 Shape
@@ -16,6 +18,8 @@
 Where
 Cast
 Unsqueeze
+Shape
+ConstantOfShape
 CumSum
 Unsqueeze
 Shape

Added nodes:

+ Shape
+ ConstantOfShape
+ Shape
+ ConstantOfShape

Modified attributes:

node[20] Shape: end: 2 → None, start: 1 → None

Connectivity changes:

node[12] Unsqueeze: input_ids [1, 12] → [61, 13]
node[20] Shape: input_ids [0] → [1]
node[36] Add: input_ids [85, 19] → [16, 17]
node[52] Add: input_ids [27, 17] → [103, 25]
node[69] Add: input_ids [120, 34] → [32, 17]
node[85] Add: input_ids [42, 17] → [138, 40]

gemma4 (gemma4) / decoder — 11 change(s)

Op summary: 125 → 129 nodes

--- base
+++ head
@@ -2,6 +2,8 @@
 Gather
 Gather
 Gather
+Shape
+ConstantOfShape
 CumSum
 Unsqueeze
 Shape
@@ -16,6 +18,8 @@
 Where
 Cast
 Unsqueeze
+Shape
+ConstantOfShape
 CumSum
 Unsqueeze
 Shape

Added nodes:

+ Shape
+ ConstantOfShape
+ Shape
+ ConstantOfShape

Modified attributes:

node[20] Shape: end: 2 → None, start: 1 → None

Connectivity changes:

node[12] Unsqueeze: input_ids [1, 12] → [60, 13]
node[20] Shape: input_ids [0] → [1]
node[36] RMSNormalization: input_ids [84, 20] → [0, 17]
node[41] Reshape: input_ids [89, 19] → [89, 21]
node[79] RMSNormalization: input_ids [129, 36] → [129, 34]
node[84] Reshape: input_ids [134, 19] → [134, 21]

gemma4 (gemma4) / vision_encoder — 56 change(s)

Op summary: 176 → 174 nodes

--- base
+++ head
@@ -1,9 +1,7 @@
-Constant
+Cast
+Sub
+Mul
 CastLike
-Constant
-CastLike
-Mul
-Sub
 Transpose
 MatMul
 Constant
@@ -133,11 +131,10 @@
 Equal
 Equal
 And
-Not
 Cast
-CastLike
 Unsqueeze
-Mul
+Constant
+Where
 Constant
 Max
 Constant
@@ -162,12 +159,13 @@
 Add
 Constant
 OneHot
-CastLike
 Transpose
 MatMul
+CastLike
+Cast
 Constant
-CastLike
 Mul
+Cast
 CastLike
 RMSNormalization
 Transpose

Added nodes:

+ Cast
+ Sub
+ Mul
+ Constant
+ Where
+ CastLike
+ Cast
+ Cast

Removed nodes:

- Constant
- Constant
- CastLike
- Mul
- Sub
- Not
- CastLike
- Mul
- CastLike
- CastLike

Modified attributes:

node[8] Constant: value_int: 0 → -1
node[10] Constant: value_int: -1 → 0
node[14] Constant: value_int: 0 → 1
node[16] Constant: value_int: 1 → 0
node[17] Gather: axis: 2 → 0
node[18] Constant: value_int: 0 → 1
node[45] RMSNormalization: stash_type: None → 1
node[50] Constant: value_int: 0 → 1
node[52] Constant: value_int: 1 → 0
node[99] Transpose: perm: [0, 2, 1, 3] → [0, 1, 3, 2]
node[106] Transpose: perm: [0, 2, 1, 3] → [1, 0]
node[128] Constant: value_int: 0 → 1

Connectivity changes:

node[3] CastLike: input_ids [35, 0] → [37, 2]
node[15] Gather: input_ids [46, 47] → [46, 49]
node[17] Gather: input_ids [46, 49] → [2, 51]
node[19] Gather: input_ids [2, 51] → [2, 53]
node[21] Gather: input_ids [2, 53] → [54, 50]
node[35] Transpose: input_ids [11] → [14]
node[36] MatMul: input_ids [67, 68] → [67, 70]
node[37] Transpose: input_ids [12] → [15]
node[38] MatMul: input_ids [67, 70] → [67, 72]
node[41] Reshape: input_ids [69, 14] → [73, 16]
node[45] RMSNormalization: input_ids [75, 16] → [76, 79]
node[56] Gather: input_ids [9, 84] → [11, 88]
node[57] Gather: input_ids [10, 84] → [12, 88]
node[60] Unsqueeze: input_ids [89, 5] → [91, 7]
node[61] Unsqueeze: input_ids [90, 5] → [92, 7]
node[64] Slice: input_ids [77, 18, 19, 20] → [78, 20, 21, 22]
node[65] Slice: input_ids [77, 19, 21, 20] → [78, 21, 23, 22]
node[66] Slice: input_ids [78, 18, 19, 20] → [97, 20, 24, 22]
node[67] Slice: input_ids [78, 19, 21, 20] → [97, 24, 21, 22]
node[98] Transpose: input_ids [115] → [80]
node[99] Transpose: input_ids [130] → [132]
node[106] Transpose: input_ids [138] → [26]
node[110] RMSNormalization: input_ids [142, 25] → [144, 28]
node[129] Gather: input_ids [1, 161] → [1, 163]
node[168] CastLike: input_ids [200, 199] → [33, 202]

Initializer changes:

count 31 → 33; dtype distribution: FLOAT: 22 → 24

gemma4_text / model — 11 change(s)

Op summary: 128 → 132 nodes

--- base
+++ head
@@ -5,6 +5,8 @@
 Gather
 Gather
 Gather
+Shape
+ConstantOfShape
 CumSum
 Unsqueeze
 Shape
@@ -19,6 +21,8 @@
 Where
 Cast
 Unsqueeze
+Shape
+ConstantOfShape
 CumSum
 Unsqueeze
 Shape

Added nodes:

+ Shape
+ ConstantOfShape
+ Shape
+ ConstantOfShape

Modified attributes:

node[23] Shape: end: 2 → None, start: 1 → None

Connectivity changes:

node[15] Unsqueeze: input_ids [1, 13] → [64, 14]
node[23] Shape: input_ids [0] → [1]
node[39] RMSNormalization: input_ids [88, 21] → [52, 18]
node[44] Reshape: input_ids [93, 20] → [93, 22]
node[82] RMSNormalization: input_ids [133, 37] → [133, 35]
node[87] Reshape: input_ids [138, 20] → [138, 22]

qwen2_moe / model — 15 change(s)

Op summary: 214 → 216 nodes

--- base
+++ head
@@ -1,6 +1,8 @@
 Gather
 Gather
 Gather
+Shape
+ConstantOfShape
 CumSum
 Unsqueeze
 Shape

Added nodes:

+ Shape
+ ConstantOfShape

Connectivity changes:

node[11] Unsqueeze: input_ids [1, 11] → [80, 12]
node[54] Mul: input_ids [121, 126] → [115, 126]
node[68] Mul: input_ids [135, 140] → [115, 140]
node[83] Mul: input_ids [150, 155] → [115, 155]
node[98] Mul: input_ids [165, 170] → [115, 170]
node[108] Transpose: input_ids [41] → [40]
node[109] MatMul: input_ids [105, 181] → [180, 181]
node[151] Mul: input_ids [221, 226] → [215, 226]
node[165] Mul: input_ids [235, 240] → [215, 240]
node[180] Mul: input_ids [250, 255] → [215, 255]
node[195] Mul: input_ids [265, 270] → [215, 270]
node[205] Transpose: input_ids [67] → [66]
node[206] MatMul: input_ids [205, 281] → [280, 281]

qwen3_5_moe (hybrid-text-generation) / model — 34 change(s)

Op summary: 269 → 271 nodes

--- base
+++ head
@@ -1,6 +1,8 @@
 Gather
 Gather
 Gather
+Shape
+ConstantOfShape
 CumSum
 Unsqueeze
 Shape

Added nodes:

+ Shape
+ ConstantOfShape

Modified attributes:

node[28] Transpose: perm: [0, 2, 1] → [1, 0]
node[31] Constant: value_float: None → 0.0, value_ints: [128] → None
node[40] Constant: value_ints: [32] → [2, 16]

Connectivity changes:

node[11] Unsqueeze: input_ids [1, 11] → [86, 12]
node[22] Transpose: input_ids [20] → [19]
node[23] MatMul: input_ids [94, 98] → [96, 98]
node[24] Transpose: input_ids [21] → [20]
node[25] MatMul: input_ids [94, 100] → [96, 100]
node[26] Transpose: input_ids [22] → [21]
node[27] MatMul: input_ids [94, 102] → [96, 102]
node[28] Transpose: input_ids [97] → [22]
node[41] Concat: input_ids [95, 116, 119] → [97, 118, 119]
node[69] Cast: input_ids [145] → [26]
node[99] Mul: input_ids [174, 179] → [168, 179]
node[113] Mul: input_ids [188, 193] → [168, 193]
node[128] Mul: input_ids [203, 208] → [168, 208]
node[143] Mul: input_ids [218, 223] → [168, 223]
node[153] Transpose: input_ids [45] → [44]
node[154] MatMul: input_ids [160, 234] → [233, 234]
node[159] Add: input_ids [46, 16] → [227, 239]
node[165] Reshape: input_ids [246, 49] → [245, 48]
node[168] Transpose: input_ids [51] → [50]
node[169] MatMul: input_ids [241, 250] → [243, 250]
node[173] Add: input_ids [54, 16] → [53, 16]
node[174] RMSNormalization: input_ids [252, 255] → [247, 255]
node[205] Mul: input_ids [284, 289] → [278, 289]
node[219] Mul: input_ids [298, 303] → [278, 303]
node[234] Mul: input_ids [313, 318] → [278, 318]
node[249] Mul: input_ids [328, 333] → [278, 333]
node[259] Transpose: input_ids [73] → [72]
node[260] MatMul: input_ids [270, 344] → [343, 344]
node[265] Add: input_ids [74, 16] → [337, 349]

qwen3_5_text (hybrid-text-generation) / model — 20 change(s)

Op summary: 131 → 133 nodes

--- base
+++ head
@@ -1,6 +1,8 @@
 Gather
 Gather
 Gather
+Shape
+ConstantOfShape
 CumSum
 Unsqueeze
 Shape

Added nodes:

+ Shape
+ ConstantOfShape

Modified attributes:

node[28] Transpose: perm: [0, 2, 1] → [1, 0]
node[31] Constant: value_float: None → 0.0, value_ints: [128] → None
node[40] Constant: value_ints: [32] → [2, 16]

Connectivity changes:

node[11] Unsqueeze: input_ids [1, 11] → [58, 12]
node[22] Transpose: input_ids [20] → [19]
node[23] MatMul: input_ids [66, 70] → [68, 70]
node[24] Transpose: input_ids [21] → [20]
node[25] MatMul: input_ids [66, 72] → [68, 72]
node[26] Transpose: input_ids [22] → [21]
node[27] MatMul: input_ids [66, 74] → [68, 74]
node[28] Transpose: input_ids [69] → [22]
node[41] Concat: input_ids [67, 88, 91] → [69, 90, 91]
node[69] Cast: input_ids [117] → [26]
node[96] Reshape: input_ids [148, 35] → [147, 34]
node[99] Transpose: input_ids [37] → [36]
node[100] MatMul: input_ids [143, 152] → [145, 152]
node[104] Add: input_ids [40, 16] → [39, 16]
node[105] RMSNormalization: input_ids [154, 157] → [149, 157]

qwen3_5_vl (hybrid-qwen-vl) / decoder — 21 change(s)

Op summary: 157 → 159 nodes

--- base
+++ head
@@ -26,6 +26,8 @@
 Where
 Where
 Where
+Shape
+ConstantOfShape
 CumSum
 Unsqueeze
 Shape

Added nodes:

+ Shape
+ ConstantOfShape

Modified attributes:

node[53] Transpose: perm: [0, 2, 1] → [1, 0]
node[56] Constant: value_float: None → 0.0, value_ints: [128] → None
node[65] Constant: value_ints: [32] → [2, 16]

Connectivity changes:

node[36] Unsqueeze: input_ids [1, 15] → [88, 16]
node[47] Transpose: input_ids [25] → [24]
node[48] MatMul: input_ids [96, 100] → [98, 100]
node[49] Transpose: input_ids [26] → [25]
node[50] MatMul: input_ids [96, 102] → [98, 102]
node[51] Transpose: input_ids [27] → [26]
node[52] MatMul: input_ids [96, 104] → [98, 104]
node[53] Transpose: input_ids [99] → [27]
node[66] Concat: input_ids [97, 118, 121] → [99, 120, 121]
node[94] Cast: input_ids [147] → [31]
node[116] Add: input_ids [37, 21] → [162, 172]
node[122] Reshape: input_ids [179, 40] → [178, 39]
node[125] Transpose: input_ids [42] → [41]
node[126] MatMul: input_ids [174, 183] → [176, 183]
node[130] Add: input_ids [45, 21] → [44, 21]
node[131] RMSNormalization: input_ids [185, 188] → [180, 188]

qwen3_moe / model — 19 change(s)

Op summary: 194 → 196 nodes

--- base
+++ head
@@ -1,6 +1,8 @@
 Gather
 Gather
 Gather
+Shape
+ConstantOfShape
 CumSum
 Unsqueeze
 Shape

Added nodes:

+ Shape
+ ConstantOfShape

Connectivity changes:

node[11] Unsqueeze: input_ids [1, 11] → [72, 12]
node[20] Transpose: input_ids [17] → [16]
node[21] MatMul: input_ids [79, 82] → [81, 82]
node[22] Transpose: input_ids [18] → [17]
node[23] MatMul: input_ids [79, 84] → [81, 84]
node[57] Mul: input_ids [116, 121] → [110, 121]
node[71] Mul: input_ids [130, 135] → [110, 135]
node[86] Mul: input_ids [145, 150] → [110, 150]
node[101] Mul: input_ids [160, 165] → [110, 165]
node[107] Transpose: input_ids [41] → [40]
node[108] MatMul: input_ids [169, 172] → [171, 172]
node[109] Transpose: input_ids [42] → [41]
node[110] MatMul: input_ids [169, 174] → [171, 174]
node[144] Mul: input_ids [206, 211] → [200, 211]
node[158] Mul: input_ids [220, 225] → [200, 225]
node[173] Mul: input_ids [235, 240] → [200, 240]
node[188] Mul: input_ids [250, 255] → [200, 255]

qwen3_next (hybrid-text-generation) / model — 56 change(s)

Op summary: 565 → 567 nodes

--- base
+++ head
@@ -1,6 +1,8 @@
 Gather
 Gather
 Gather
+Shape
+ConstantOfShape
 CumSum
 Unsqueeze
 Shape

Added nodes:

+ Shape
+ ConstantOfShape

Modified attributes:

node[28] Transpose: perm: [0, 2, 1] → [1, 0]
node[31] Constant: value_float: None → 0.0, value_ints: [128] → None
node[40] Constant: value_ints: [32] → [2, 16]
node[172] Transpose: perm: [0, 2, 1] → [1, 0]
node[175] Constant: value_float: None → 0.0, value_ints: [128] → None
node[184] Constant: value_ints: [32] → [2, 16]
node[316] Transpose: perm: [0, 2, 1] → [1, 0]
node[319] Constant: value_float: None → 0.0, value_ints: [128] → None
node[328] Constant: value_ints: [32] → [2, 16]

Connectivity changes:

node[11] Unsqueeze: input_ids [1, 15] → [145, 16]
node[22] Transpose: input_ids [24] → [23]
node[23] MatMul: input_ids [153, 157] → [155, 157]
node[24] Transpose: input_ids [25] → [24]
node[25] MatMul: input_ids [153, 159] → [155, 159]
node[26] Transpose: input_ids [26] → [25]
node[27] MatMul: input_ids [153, 161] → [155, 161]
node[28] Transpose: input_ids [156] → [26]
node[41] Concat: input_ids [154, 175, 178] → [156, 177, 178]
node[65] Cast: input_ids [200] → [29]
node[155] Transpose: input_ids [48] → [47]
node[156] MatMul: input_ids [215, 295] → [294, 295]
node[161] Add: input_ids [49, 20] → [288, 300]
node[166] Transpose: input_ids [53] → [52]
node[167] MatMul: input_ids [302, 306] → [304, 306]
node[168] Transpose: input_ids [54] → [53]
node[169] MatMul: input_ids [302, 308] → [304, 308]
node[170] Transpose: input_ids [55] → [54]
node[171] MatMul: input_ids [302, 310] → [304, 310]
node[172] Transpose: input_ids [305] → [55]
node[185] Concat: input_ids [303, 324, 327] → [305, 326, 327]
node[209] Cast: input_ids [349] → [57]
node[299] Transpose: input_ids [76] → [75]
node[300] MatMul: input_ids [364, 444] → [443, 444]
node[305] Add: input_ids [77, 20] → [437, 449]
node[310] Transpose: input_ids [81] → [80]
node[311] MatMul: input_ids [451, 455] → [453, 455]
node[312] Transpose: input_ids [82] → [81]
node[313] MatMul: input_ids [451, 457] → [453, 457]
node[314] Transpose: input_ids [83] → [82]
node[315] MatMul: input_ids [451, 459] → [453, 459]
node[316] Transpose: input_ids [454] → [83]
node[329] Concat: input_ids [452, 473, 476] → [454, 475, 476]
node[353] Cast: input_ids [498] → [85]
node[443] Transpose: input_ids [104] → [103]
node[444] MatMul: input_ids [513, 593] → [592, 593]
node[449] Add: input_ids [105, 20] → [586, 598]
node[455] Reshape: input_ids [605, 108] → [604, 107]
node[458] Transpose: input_ids [110] → [109]
node[459] MatMul: input_ids [600, 609] → [602, 609]
node[463] Add: input_ids [113, 20] → [112, 20]
node[464] RMSNormalization: input_ids [611, 614] → [606, 614]
node[555] Transpose: input_ids [132] → [131]
node[556] MatMul: input_ids [629, 709] → [708, 709]
node[561] Add: input_ids [133, 20] → [702, 714]

Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed)

github-actions · 2026-10-02T17:43:48Z

Performance Comparison

Comparing 2e5508f1 → 6ed50516

Model	Metric	Baseline	Current	Delta
bert (feature-extraction)	model_size_bytes	359 KB	359 KB	+0.0%	⚪
bert (feature-extraction)	num_nodes	68	68	+0.0%	⚪
falcon	model_size_bytes	364 KB	364 KB	+0.0%	⚪
falcon	num_nodes	66	68	+3.0%	⚪
gemma2	model_size_bytes	428 KB	428 KB	+0.0%	⚪
gemma2	num_nodes	105	109	+3.8%	⚪
gpt2	model_size_bytes	324 KB	324 KB	+0.0%	⚪
gpt2	num_nodes	54	54	+0.0%	⚪
llama	model_size_bytes	425 KB	425 KB	+0.0%	⚪
llama	num_nodes	60	60	+0.0%	⚪
llama (static-cache)	model_size_bytes	425 KB	425 KB	+0.0%	⚪
llama (static-cache)	num_nodes	56	56	+0.0%	⚪
mamba (ssm-text-generation)	model_size_bytes	296 KB	296 KB	+0.0%	⚪
mamba (ssm-text-generation)	num_nodes	94	94	+0.0%	⚪
phi3	model_size_bytes	421 KB	421 KB	+0.0%	⚪
phi3	num_nodes	58	58	+0.0%	⚪
phi3 (static-cache)	model_size_bytes	421 KB	421 KB	+0.0%	⚪
phi3 (static-cache)	num_nodes	54	54	+0.0%	⚪
qwen2	model_size_bytes	425 KB	425 KB	+0.0%	⚪
qwen2	num_nodes	60	60	+0.0%	⚪
qwen2 (static-cache)	model_size_bytes	425 KB	425 KB	+0.0%	⚪
qwen2 (static-cache)	num_nodes	56	56	+0.0%	⚪
qwen3_5_moe (hybrid-text-generation)	model_size_bytes	506 KB	506 KB	+0.0%	⚪
qwen3_5_moe (hybrid-text-generation)	num_nodes	269	271	+0.7%	⚪
qwen3_5_text (hybrid-text-generation)	model_size_bytes	458 KB	458 KB	+0.0%	⚪
qwen3_5_text (hybrid-text-generation)	num_nodes	131	133	+1.5%	⚪
qwen3_5_vl (hybrid-qwen-vl)	model_size_bytes	977 KB	977 KB	+0.0%	⚪
qwen3_5_vl (hybrid-qwen-vl)	num_nodes	461	463	+0.4%	⚪
t5 (seq2seq)	model_size_bytes	836 KB	836 KB	+0.0%	⚪
t5 (seq2seq)	num_nodes	176	176	+0.0%	⚪
whisper (speech-to-text)	model_size_bytes	1008 KB	1008 KB	+0.0%	⚪
whisper (speech-to-text)	num_nodes	128	128	+0.0%	⚪

No performance regressions.

Copilot

Copilot review overview

🟡 Changes recommended

Automatic enablement exposes existing exports to documented numerical drift before the acceptance contract and representative regression coverage are resolved.

Review effort: Balanced
Findings: 1

Open (1)

Enable weight packing only with an agreed numerical contract · New

What changed in this PR

Adds execution-provider-aware INT4 QKV projection packing to Mobius’s ONNX optimization pipeline.

Changes:

Combines compatible CUDA FP16/INT4 projections while preserving downstream attention paths.
Validates deferred parameter bindings and adds compatibility, lifecycle, and numerical tests.
Documents capability settings and experimental limitations.

File	Description
tests/ep_optimization_test.py	Tests capability gating and decoder packing.
src/mobius/rewrite_rules/_matmul_nbits_qkv.py	Implements compatible projection packing.
src/mobius/rewrite_rules/_matmul_nbits_qkv_test.py	Tests matching, preservation, and numerical behavior.
src/mobius/rewrite_rules/__init__.py	Exports the packing pass.
src/mobius/_optimizations.py	Integrates packing into fusion stages.
src/mobius/_execution_providers.py	Adds and enables the CUDA capability.
src/mobius/_component_quantization.py	Validates bindings through deferred Concats.
src/mobius/_component_quantization_loading_test.py	Tests loading, persistence, and cached decoding.
docs/execution_providers.md	Documents packing behavior and limitations.
docs/ep_quickstart.md	Updates CUDA capability examples.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🔵 Needs a closer look

Larger-dimension CUDA numerical drift and missing portable regression coverage require human acceptance of the experimental merge boundary.

0 open findings

1 resolved since last review

Enable weight packing only with an agreed numerical contract

🧠 Review effort: Balanced

Consolidate generic QKV packing, lazy constant layout folding, adapter aliases, and default-off INT4 behavior with focused tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9a2a5b82-7cd3-4a76-bb94-0998527ea723 Signed-off-by: titaiwang <titaiwang@microsoft.com>

Scope nested Concat folding with child cleanup, document quantized storage-contract eligibility, and record float capability documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9a2a5b82-7cd3-4a76-bb94-0998527ea723 Signed-off-by: titaiwang <titaiwang@microsoft.com>

Copilot

🔵 Needs a closer look

Broad folding changes retain an unresolved runtime-override defect, while default-enabled float packing has numerical-qualification gaps requiring human review.

1 open issue

Reject graph-input initializers to preserve runtime Concat overrides · New

_{Review effort: Balanced}

Captured parent initializers cannot be packed when they are runtime-overridable. Added root and captured runtime/default-override coverage, including public save/load behavior. State: 78 pass, 2 before-fix expected fail, scoped lint passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9a2a5b82-7cd3-4a76-bb94-0998527ea723 Signed-off-by: titaiwang <titaiwang@microsoft.com>

Copilot

🔵 Needs a closer look

Default-enabled float packing lacks target-runtime numerical qualification, and the cross-scope lazy-persistence changes warrant final human review.

No open issues 🎉

1 resolved since last review

Reject graph-input initializers to preserve runtime Concat overrides

_{Review effort: Balanced}

) ## Summary Fix VibeVoice CI dependency installation by resolving the pinned Transformers source together with the editable Mobius project, then installing the pinned VibeVoice source with `--no-deps`. ## Failure and fix The failing install sequence first resolved Transformers 5.20.2 with Hub 2.3.0 and `httpx2`, then force-installed Transformers 5.16.0.dev0 from the combined VibeVoice requirements using `--no-deps`. That left Hub 2.3.0 installed despite the pinned Transformers source requiring `huggingface-hub>=1.5.0,<2.0`. Importing `AutoConfig` then failed when Transformers imported the absent `httpx` module. The split install resolves the pinned Transformers source and its dependencies together with the editable project, installs only VibeVoice with `--no-deps`, and immediately exercises `from transformers import AutoConfig`. It neither blindly adds `httpx` nor changes either source revision. Primary evidence: [pinned Transformers dependencies](https://github.com/huggingface/transformers/blob/640a08a597034221ca1c4fc0c129cf0118179225/setup.py), [upstream VibeVoice metadata](https://github.com/microsoft/VibeVoice/blob/79e516a3e20b599f137c9da03410a2a0b473b63b/pyproject.toml), and the [failed synthetic parity job](https://github.com/onnxruntime/mobius/actions/runs/40044682674/job/120723608401). Both 40-character source pins remain unchanged: - Transformers: `640a08a597034221ca1c4fc0c129cf0118179225` - VibeVoice: `79e516a3e20b599f137c9da03410a2a0b473b63b` All five install steps retain actual dependency resolution and AutoConfig import smoke, and the YAML-structure Python tests were removed following review. ## Validation - `ruff format`, `ruff check`, `ruff format --check`, and `git diff --check`: passed. - Validation log: `ci-vibevoice-deps-validation-v2.log`. A local fresh-venv network resolver run was **not** performed; host CI validation remains pending. No installs, full-model, or GPU runs were performed. VibeVoice's upstream metadata still declares Transformers `<5`. This preserves the existing reference integration's deliberate `--no-deps` exception for VibeVoice; it does not resolve that pre-existing metadata mismatch or apply the exception to Transformers itself. This PR is standalone from main `d69efe03a9f4c1ada744bc31dd01077a710b3c39` and independent of #754 and #787. --------- Signed-off-by: titaiwang <titaiwang@microsoft.com> Copilot-Session: 9a2a5b82-7cd3-4a76-bb94-0998527ea723

justinchuby · 2026-10-26T18:45:12Z


 These are applied during **Stage 2: Fusion** of the optimization pipeline.

+#### Projection packing and numerical limitations


Could you make this section more succinct?

justinchuby · 2026-10-26T18:47:53Z

        folded_count = 0
+        graphs = tuple(model.graphs())
+
+        def retire_sources(


Make be take this out into a root level helper? It seems complicated without much docs

justinchuby · 2026-10-26T18:49:33Z

+QKV_PROJECTION_SOURCES_METADATA_KEY = "mobius.qkv_projection_sources"
+"""Versioned source names and widths retained on projection-packing Split nodes."""
+
+# Transient tensor.meta ownership, never serialized as ONNX metadata.
+LAZY_SOURCE_FILES_META_KEY = "mobius.lazy_source_files"


Use pkg.mobius. for persistent metadata keys. Also document on what LAZY_SOURCE_FILES_META_KEY does?

justinchuby · 2026-10-26T18:52:08Z

        for node in list(model.graph.all_nodes()):
-            if node.op_type != "Transpose":
+            if (
+                node.domain not in ("", "ai.onnx")


Domain is always normalized: https://github.com/onnx/ir-py/blob/1df310ef12ecff98f804911928cb5369280617ae/src/onnx_ir/_core.py#L2190

Suggested change

node.domain not in ("", "ai.onnx")

node.domain != ""

justinchuby · 2026-10-26T18:53:47Z

+                # Preserve legacy root ownership/naming even when the
+                # consumer is in a Loop/If body. Only child-local initializers
+                # need the scoped FoldWeightLayoutPass.
+                or model.graph.initializers.get(inp.name) is not inp


Could you explain this? Not 100% clear from the comment

justinchuby · 2026-10-26T18:59:08Z

+        # Registry identity is authoritative, including parent-owned captures.
+        # Do not infer ownership from the graph containing the consumer node.


Could you clarify?

justinchuby · 2026-10-26T19:00:43Z

+        def owners(value: ir.Value) -> list[ir.Graph]:
+            return [
+                scope
+                for scope in graphs
+                if value.name is not None and scope.initializers.get(value.name) is value
+            ]


I am not entirely sure what this logic is for. Could you explain?

justinchuby · 2026-10-26T19:01:17Z

+FUNCTION_BODY_METADATA_KEY = "mobius.function_body"
+FUNCTION_BODY_METADATA_VALUE = "1"


What does this do?

justinchuby · 2026-10-26T19:02:16Z

+    if (
+        split.domain not in ("", "ai.onnx")
+        or split.op_type != "Split"
+        or split.overload
+        or split.attributes.get_int("axis", 0) not in (-1, 2)
+        or len(split.outputs) != len(sources)
+        or packed is None
+        or projection is None
+        or projection.domain not in ("", "ai.onnx")
+        or projection.op_type != "MatMul"
+        or projection.overload
+        or projection.attributes
+        or len(projection.inputs) != 2
+        or len(projection.outputs) != 1
+        or packed in model.graph.outputs
+        or len(packed.uses()) != 1
+    ):
+        raise ValueError(f"PEFT module {module_key!r} has an unsupported packed projection")


This seems overly complicated. Extract into a named helper? Here and elsewhere

justinchuby · 2026-10-26T19:03:20Z

+
+"""Pack projections, then split before normalization, RoPE, attention, or caches.
+
+Supports distinct Q/K/V or Q/K with raw K reused for V. Parameter Concats defer
+packing until checkpoint binding. No weights are dequantized or requantized.


Also add the "why" in the docstring?

Address Justin's review by moving source retirement and initializer ownership lookup into documented module-level helpers, and naming the topology and loaded-weight validation stages used during packed PEFT restoration. Explain parent-captured initializer ownership, lazy backing-file lifetime, registry-generated function provenance, and why projection packing defers checkpoint binding without changing downstream attention contracts. Use pkg.mobius. for the persistent projection and function-body metadata keys, sharing the function marker between producers and consumers. Keep the old projection metadata readable for previously exported preview packages. Legacy-only function markers conservatively leave the body caller-owned; document and test that specialization is not authorized by those markers. Keep transient lazy-file bookkeeping in .meta rather than serialized metadata_props. Remove redundant ai.onnx checks because IR normalizes it. Shorten the execution-provider documentation while retaining automatic float-stage behavior, experimental INT4 default-off status, the failed numerical gate, and non-transactional save limitations. Validation: 463 focused CPU tests passed, 2 skipped. Scoped Ruff, lintrunner, and whitespace checks passed. MyPy has the same 26 diagnostics as the exact parent revision, with no new diagnostics. No CUDA or full-model numerical requalification is claimed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9a2a5b82-7cd3-4a76-bb94-0998527ea723 Signed-off-by: titaiwang <titaiwang@microsoft.com>

## Organization **Depends on #754.** This PR contains the shared Gemma4 text-only and multimodal modeling changes. Generic projection packing and weight-layout folding live in #754 instead of being duplicated here. Text-only construction already exists on main; this PR corrects and documents the model semantics used by both text and multimodal exports. ## Changes - Preserve HuggingFace arithmetic boundaries in BF16 norms, routing, vision pooling, and expert contributions. - Keep global attention causal. Compose local vision visibility as `(causal OR same-image) AND left-window AND padding`, using physical KV-slot distances even when a padding mask contains holes. - Declare the richer arbitrary-mask contract for vision-overlay exports, so optimization does not collapse it to prefix lengths through GQA fusion. Text-only CUDA exports retain GQA eligibility. - Correct RoPE/shared-KV and quantized expert handling, with explicit experimental custom-GEGLU opt-in and runtime-requirement metadata. - Keep processor-facing vision inputs float32, with internal dtype conversion. ## Review follow-up Includes cached internal-hole mask regressions using the actual optimized CUDA export's bias, a text-only GQA eligibility regression, independent HF mask comparisons, and exception-restoration coverage for the custom-GEGLU flag. The adapter alias and repeated-source packing findings from this PR are handled in its #754 dependency. ## Qualification limits Export support is not released-runtime compatibility or numerical qualification. Custom GEGLU and quantized Gemma4 MoE require the documented experimental runtime; neither is enabled by ordinary CUDA selection. Previously failed strict HF numerical gates remain failed and unwaived. Historical performance and accuracy artifacts are not represented as validation of this reorganized head. The ongoing paired MMLU-Pro500 evaluation uses the frozen optimized text artifact and the pinned decoder-only llama.cpp Q4_K_M artifact, with identical prompts and scoring. ## Consolidated-revision validation - Gemma4 model suite: **118 passed**, including four adversarial cases crossing selected/unselected overflow with zero/unit expert scale. - Independent offline HF attention-mask comparisons: **11 passed**. - Its #754 dependency: **516 passed, 3 skipped**, including all 22 unchanged quantized-storage regressions and nested-scope persistence coverage. - Combined non-integration suite: **11,311 passed, 29 failed, 138 skipped**. Exact-base selector runs reproduce 28 failures; the additional GGUF file-identity failure passes separately on both base and head. This is **not a green full-suite claim**. - Scoped Ruff checks passed. Scoped MyPy remains baseline-unclean; existing HF `generate` self-type diagnostics remain on both base and head. Two bounded five-role review rounds completed. The review's portable-MoE question was verified with the actual dispatch graph: an overflowing **unselected** expert produced 64 NaNs before the correction, versus a selected-only zero reference. An independent Boolean selection mask now selects out unchosen outputs before weighting. The same CPU probe produces 64 finite exact zeros afterward; selected nonfinite outputs still propagate, including with zero learned expert scale. | ID | Severity | Raised by | Finding | Disposition | |---|---|---|---|---| | F4 | Major | Deep / critical / CPU QA | Dense fallback allowed unselected nonfinite experts to contaminate the result | Fixed and CPU-verified; CUDA equivalence not inferred | The dependency's F1/F2/F3 scope, documentation, and storage findings are fixed in #754. No Critical/Major reviewer finding was overruled. Remaining exclusions: full-checkpoint re-export and strict HF logits/KV parity, CUDA qualification of the fallback safeguard, custom-runtime availability, and performance. The failed numerical gates remain unwaived. <details> <summary>Complete original captured PR body — historical prior implementation (not validation of the reorganized head)</summary> The historical results in the captured body below describe the prior implementation and are **NOT validation of the reorganized head**. Older statements about unchanged float behavior are historical and superseded by the new summary above. All original numerical limitations and earlier MMLU/performance evidence are preserved verbatim. ## Summary This draft isolates the Gemma4 runtime work from the unrelated ASR, Qwen4, text-evaluation, and experimental fixed-cache changes. - Align global/local image visibility with the pinned HuggingFace reference: global layers stay causal; local same-image visibility remains subject to the sliding-window left boundary. - Normalize processor pixels in FP32 before casting for the vision projection. The separate vision encoder now accepts FLOAT pixels. - Accumulate vision pooling in FP32, round the pooled mean to the working dtype once, then scale/standardize in FP32. Local window distances use physical slots even with internal padding. - Preserve the learned router scale and apply the hidden-size scalar separately with FP32 opmath and working-dtype rounding. - Adapt the necessary CUDA native MoE/QMoE and projection-packing support to current main without reverting its weight-layout normalization. - Decline float packing when projection weights have outside consumers, and protect lazily restored PEFT weights against destructive same-path/alias saves. ## Compatibility and qualification The vision input dtype change requires rebuilding exports and passing the original FP32 processor pixels, rather than precasting them to BF16. Custom native GEGLU is disabled by default. `MOBIUS_GEMMA4_CUSTOM_GEGLU=1` explicitly opts into the experimental custom CUDA ABI; it is not implied by selecting CUDA. Ordinary float CUDA uses the portable path, while quantized GEGLU experts without the required opt-in fail clearly. Native operators record `mobius.gemma4.required_runtime_feature="custom-cuda-geglu-v1"`; the flag is an assertion about the target runtime, not automatic ABI detection. The native paths require the corresponding ORT runtime/operator support. An exported GQA node does not establish Flash/XQA dispatch, shared KV support, or CUDA Graph capture. **This is not numerical-parity or production approval.** Strict full-vocabulary logits/KV checks still fail at the original `atol=0.01, rtol=0.01`; the previous packing parity gate is not waived. Video and the experimental TensorScatter/SparsePaged conversions are outside this PR. ## Recorded pre-port experiment These results belong to immutable corrected exports built before porting the changes onto the newer main branch, and before the subsequent pool/window, adapter-resave, and runtime-opt-in review fixes. They are not a real-checkpoint re-export/validation of this PR branch. Export graph, source, runtime, and result hashes are retained in the experiment artifacts. Fixed AI2D-200 quantization-selection subset; not an independent full-AI2D evaluation: | Precision | HF reference | Native GenAI | |---|---:|---:| | BF16 | 167/200 | 166/200 | | INT4 | 162/200 | 166/200 | All four runs used matching original FP32 processor tensors and had zero inference failures. Three real-image prompts, each with prefill and two teacher-forced cached decodes, matched all nine first-token argmaxes for each native precision, but failed the strict full-logit/KV gates. Performance on the same frozen exports, batch 1, three single-image prompts of 333/353/356 input tokens: | Variant | TTFT (ms) | Decode (tokens/s) | Sampled device peak (GiB) | |---|---:|---:|---:| | Native BF16 | 127.44 | 78.99 | 56.51 | | Native INT4 | 128.59 | 87.77 | 34.98 | | HF BF16 | 1181.61 | 6.28 | 49.50 | | HF INT4 | 3351.30 | 2.43 | 17.28 | Each case used two discarded warmups and five measured trials, generating one first token and 64 greedy decode tokens. TTFT and throughput are medians of the three case medians. Full-vocabulary diagnostic readbacks were excluded from measured trials; EOS suppression was performance-only. HF used eager experts and SDPA without compilation; its Olive INT4 dequantizes on demand, unlike native QMoE. These are different implementations, not an isolated packing speedup. The selected A100 80GB had no competing compute process, but other GPUs shared the host with text evaluation. Peak memory is a 20ms NVML sample of the whole selected device, not exact model-only allocation. These are short-prompt results, not 2K/8K/16K/32K benchmarks. The experimental ORT 1.32 runtime was built from revision `942356fce6bdc78c016e4b902e49492fdf695453` plus local modifications, not an unmodified upstream release. Its shared-library SHA256 is `3f6b7ff482f8fbd4d3c639e807ed2ec35fa240c88c205dbee7822d7ec497e988`. Both native decoder graphs retained 30 packed-QKV projection records. Diagnostic profiles executed no Constant nodes, but still averaged 126 CUDA Cast nodes per decoder invocation. Constant folding does not justify deleting dynamic precision boundaries; these counts are not per-op GPU timings. ## Isolated-branch validation Against base `d69efe03a9f4c1ada744bc31dd01077a710b3c39`, offline and CPU-only: | Check | Result | |---|---| | Related runtime, package, expert-layout, adapter, and packing regressions | 656 passed, 2 skipped | | GQA rewrite regressions | 32 passed; 4 known offline baseline failures excluded from this invocation | | Public Gemma4 builder ABI | 10 passed | | Actual pinned-HF dynamic/static mask comparisons | 11 passed | | Changed-file Ruff, format, compilation, whitespace | Passed | | Strict mypy with resolved editable ONNX Script imports | Not clean on either revision; zero new normalized diagnostic instances | The four excluded GQA cases were separately reproduced on the exact clean base; unavailable cached remote configurations were not downloaded, and tests were not weakened. The strict typecheck produced 2,798 diagnostics on the head and 2,807 on base with no missing-import diagnostics; this is a baseline-relative result, not clean type approval. Sensitive regressions include BF16 3x3 pooling against the actual HF implementation, padded NaN/+Inf/-Inf handling, physical-window boundaries with internal padding and cached decode, shared-weight packing refusal followed by public attachment of both independent K adapters, source-preserving same-path/hardlink/symlink save rejection, and safe-destination package round-trips. Expanded validation caught and fixed a caller-owned `MatMulNBits` function regression: raw factory results must not claim registry ownership. The existing preservation test now passes unchanged on both base and head, and new tests retain automatic-fallback packing while preserving caller-owned functions/calls. ## Review disposition Two targeted five-reviewer rounds completed. They closed F1-F6 without a remaining grounded Major/Critical in their bounded follow-up. A subsequent adversarial probe promoted the final round's refolding ownership question to F7. The user explicitly approved fixing F7 and running precise validation rather than a third full-team round. | ID | Severity | Raised by | Finding / affected location | Disposition | |---|---|---|---|---| | F1 | Major | Deep | `models/gemma4.py`: premature low-precision pooling | Confirmed; fixed and actual-HF BF16/nonfinite regressions pass; Deep closed | | F2 | Major | Deep | `components/_common.py`: padding compressed local-window distances | Confirmed; fixed; physical-slot prefill/decode boundaries and actual HF masks pass; Deep closed | | F3 | Major | Code | `_qkv_projection_packing.py` / `adapter_io.py`: shared sources broke independent adapter restoration | Confirmed with corrected two-K-target public regression; conservative packing refusal fixes it; Code closed | | F4 | Major | Critical | CUDA implicitly authorized experimental GEGLU | Confirmed; default-off custom-runtime opt-in and persistent requirement metadata; Critical/Integration closed | | F5 | Major | Critical question + QA | Restored lazy slices hid backing files during alias saves | Confirmed; source ownership retained, aliases rejected before writes, safe-output round-trips pass; bounded lifecycle closed | | F6 | Major | Expanded validation | Raw `MatMulNBits` factory incorrectly claimed automatic function ownership | Confirmed by unchanged head/base test; caller-owned factory left unmarked, automatic registry provenance preserved; Code closed | | F7 | Major | Critical question + QA | Refolding replaced tensor wrappers and lost source ownership before first save | Confirmed; lead retains ownership on the in-memory model too; focused regression and unchanged adversarial probe now pass | For F7, the pre-fix generated fixture's external data file changed from 16,384 bytes to zero before a lazy read failed. With the fix, the identical instrument records **zero external reads, zero source write-open requests, and unchanged nonzero sizes/SHA256 digests**. The regression also saves elsewhere and checks exact reloaded payloads. Original model artifacts were not touched. F7's final small fix was not independently re-reviewed by the full team; it has the explicitly approved focused execution evidence. **Corrections/minority record:** Code withdrew its original claim that directly attaching a shared Q remained supported after folding: the exact base requires one initializer consumer. The real two-independent-K-target regression was retained and fixed. Integration withdrew the proposed fused-RoPE restriction for projection-only packing because `MatMul`/`MatMulNBits -> Split` does not match legacy packed-GQA unpacking. No unresolved reviewer objection to those corrections remains. **Attributed open questions:** Deep retains current-revision full-checkpoint logits/KV/packing qualification. Integration retains whether released ORT GenAI versions supply the split decoder's required `input_ids` alongside `inputs_embeds`; the experimental measured runner does not qualify every released runner. Readability's provenance limitation is explicit: the modified runtime patchset and complete experiment records are not included in this repository, so the binary identity is not a reproducible build recipe. **Residual risk / exclusions:** CPU-only branch validation, two skipped tests, four exact-base offline GQA failures, and non-clean baseline typing remain explicit. No new-branch full-checkpoint export, released-runtime qualification, long-context benchmark, CUDA Graph/shared-KV clearance, arbitrary model-replacement metadata guarantee, or concurrent filesystem-race test is claimed. The unselected-expert nonfinite hypothesis was refuted only for its exact FP16 finite-router probe, not universally. ASR, Qwen4, text benchmark implementation, and experimental static/paged-cache work are excluded. **Critical review threat-model appendix:** Inputs include checkpoint/quantization plans, serialized packing metadata and external files, PEFT factors, feature flags, and runtime pixels/positions/IDs/masks. Boundaries are metadata-authorized restoration, lazy backing-file lifetime, and explicit authorization of experimental native operators. Relevant risks are tampering/data integrity and denial of service; no new authentication or elevation path was established. Known source aliases fail before writing; this is not exhaustive transformation or concurrent-filesystem protection. ## Remaining validation - Strict BF16/INT4 full-logit/KV equivalence and packed-projection equivalence. - Real-checkpoint rebuild and revalidation on the main-based PR branch. - Long-context 2K/8K/16K/32K performance; shared/static KV and CUDA Graph evaluation. - Full AI2D and completion of the independent text-only task-quality pilot. </details> ## Current follow-up validation and CI root cause Published follow-up `fd4513b5cc2e4692a2cde51fd8fdd57c4bb0878d`. The test override accepts base forward parameters, including keyword names and optional causal/static-cache arguments, while retaining the isolated MoE probe; 147 model/flag tests and 225 combined tests passed, with scoped lint passed. There are no production forward changes. The restoration assertion is now in `finally`, so `original` is always consumed on either control-flow exit; nested flag contexts/assertions are retained. CodeQL is unavailable locally; no new CodeQL clearance is claimed. The initial CI environment used TensorFlow 5.20.2 / Hugging Face Hub 2.3 / httpx 2, then `--no-deps` forced TensorFlow 5.16.dev0 whose metadata requires `huggingface-hub>=1.5,<2`; old Hub 1.x supplies the older httpx. This is a dependency-closure violation, not merely missing httpx, and is not changed by this PR. No CI code fix is included and CI green is not claimed. Evidence: [pinned TensorFlow setup](https://github.com/onnxruntime/mobius/blob/fd4513b5cc2e4692a2cde51fd8fdd57c4bb0878d/.github/workflows/ci.yml#L80), [Hub v1.5.0 setup.py](https://github.com/huggingface/huggingface_hub/blob/v1.5.0/setup.py), and [Hub v2.3.0 setup.py](https://github.com/huggingface/huggingface_hub/blob/v2.3.0/setup.py). Resolve the pinned TensorFlow dependency normally; use `--no-deps` only for VibeVoice plus pip-check/import smoke, with no automatic new PR scope. Strict numerical gates remain failed/unwaived. The full five-role, two-round review is already complete; this follow-up is a direct bounded review only. --------- Signed-off-by: titaiwang <titaiwang@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9a2a5b82-7cd3-4a76-bb94-0998527ea723

+                    with _flags.override_flags(gemma4_custom_geglu=True):
+                        assert _flags.flags.gemma4_custom_geglu
+                        raise RuntimeError("test restoration")
+                assert not _flags.flags.gemma4_custom_geglu


haozhiii · 2026-10-28T03:35:41Z

@titaiwangms Thanks for the layout-folding pass. I ran into a case that the pass intentionally skips, but it's causing a significant perf issue for us: the fc1 gate/up interleave chain (_interleave_gate_up_rows, Reshape → Transpose → Reshape) on QMoE's fc1_experts_weights, fc1_scales, and fc1_experts_zero_points.

The _moe.py docstring assumes ORT will fold this chain at session load, but ORT skips constant folding for outputs larger than 1 GiB by default. In Step-3.5-Flash INT4, fc1 is 1.41 GiB per layer, so the chain remains in the graph. As a result, CPU QMoE::PrePack never sees a constant and falls back to the slow path.

I tried folding the chain offline, and the result is bit-identical to ORT's fold. This improved decode speed from 0.67 to 7–9 tok/s. EPs that read QMoE initializers directly would also benefit, since they can use the precomputed tensors regardless of size.

Since this chain only rearranges data (no arithmetic or type casts), would it make sense for FoldWeightLayoutPass to fold it for QMoE fc1 inputs? Or is there a reason we need to keep the checkpoint row order explicit, for example, to support adapters or weight overrides?

titaiwangms marked this pull request as ready for review October 5, 2026 22:39

titaiwangms requested review from a team and a balanced review from Copilot October 5, 2026 22:39

Copilot AI reviewed Oct 5, 2026

View reviewed changes

Comment thread src/mobius/_execution_providers.py Outdated

Copilot started reviewing on behalf of titaiwangms October 5, 2026 22:57

View session

titaiwangms requested a balanced review from Copilot October 7, 2026 20:00

Copilot started reviewing on behalf of titaiwangms October 7, 2026 20:01

View session

Copilot AI reviewed Oct 7, 2026

View reviewed changes

titaiwangms and others added 2 commits October 23, 2026 23:33

titaiwangms force-pushed the titaiwang/quantized-qkv-packing branch from f082271 to e01ac8b Compare October 24, 2026 00:22

titaiwangms changed the title ~~Pack compatible INT4 QKV projections for CUDA exports~~ Consolidate projection packing and lazy weight-layout folding Oct 24, 2026

This was referenced Oct 24, 2026

Align shared Gemma4 text and multimodal runtime semantics #787

Merged

Fold static weight layout chains after weight loading #788

Closed

titaiwangms requested a balanced review from Copilot October 24, 2026 01:37

Copilot started reviewing on behalf of titaiwangms October 24, 2026 01:37

View session

Copilot AI reviewed Oct 24, 2026

View reviewed changes

Comment thread src/mobius/_optimizations.py

titaiwangms requested a balanced review from Copilot October 24, 2026 18:44

Copilot started reviewing on behalf of titaiwangms October 24, 2026 18:44

View session

Copilot AI reviewed Oct 24, 2026

View reviewed changes

titaiwangms mentioned this pull request Oct 25, 2026

Fix CI dependency resolution for pinned Transformers and VibeVoice #789

Merged

Merge branch 'main' into titaiwang/quantized-qkv-packing

444cf61

justinchuby reviewed Oct 26, 2026

View reviewed changes

titaiwangms and others added 2 commits October 26, 2026 22:50

github-code-quality Bot found potential problems Oct 27, 2026

View reviewed changes

Comment thread src/mobius/_flags_test.py

with _flags.override_flags(gemma4_custom_geglu=True):

assert _flags.flags.gemma4_custom_geglu

raise RuntimeError("test restoration")

assert not _flags.flags.gemma4_custom_geglu


		These are applied during Stage 2: Fusion of the optimization pipeline.

		#### Projection packing and numerical limitations

		# Registry identity is authoritative, including parent-owned captures.
		# Do not infer ownership from the graph containing the consumer node.

		FUNCTION_BODY_METADATA_KEY = "mobius.function_body"
		FUNCTION_BODY_METADATA_VALUE = "1"

Conversation

titaiwangms commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scope

Review follow-up

Qualification limits

Consolidated-revision validation

Status: implementation review — experimental opt-in only

Changes

Validation

Confirmed larger-dimension limitation

Follow-up isolation: weights, GEMM arithmetic, and cache propagation

Same-weight real-model MMLU A/B

TTFT and decode measurements: shared GPU, descriptive only

Reproducibility and evidence boundaries

fpA/intB and constant folding

Review disposition

Remaining work and reviewer decisions

Current follow-up validation and CI root cause

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🏗️ Architecture Diff

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Comparison

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haozhiii commented Oct 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

titaiwangms commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading