Skip to content

Support per-channel quantized export and configurable MatMulNBits acc… - #794

Open
neilmsft wants to merge 3 commits into
mainfrom
nbudnarain/gemma4-perchannel-accuracy
Open

neilmsft wants to merge 3 commits into
mainfrom
nbudnarain/gemma4-perchannel-accuracy

Conversation

@neilmsft

Copy link
Copy Markdown

Adds support for per channel quantized weight export and configurable MatMulNBits accuracy levels through the builder APIs. Supports explicit accuracy levels from 0 through 4 without changing weight precision, while preserving existing defaults when no override is provided.

@neilmsft
neilmsft requested review from a team and a balanced review from Copilot October 27, 2026 06:26
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Performance Comparison

Comparing 2e5508f1 → 06e00978

Model Metric Baseline Current Delta
bert (feature-extraction) model_size_bytes 359 KB 359 KB +0.0% ⚪
bert (feature-extraction) num_nodes 68 68 +0.0% ⚪
falcon model_size_bytes 364 KB 364 KB +0.0% ⚪
falcon num_nodes 66 66 +0.0% ⚪
gemma2 model_size_bytes 428 KB 428 KB +0.0% ⚪
gemma2 num_nodes 105 105 +0.0% ⚪
gpt2 model_size_bytes 324 KB 324 KB +0.0% ⚪
gpt2 num_nodes 54 54 +0.0% ⚪
llama model_size_bytes 425 KB 425 KB +0.0% ⚪
llama num_nodes 60 60 +0.0% ⚪
llama (static-cache) model_size_bytes 425 KB 425 KB +0.0% ⚪
llama (static-cache) num_nodes 56 56 +0.0% ⚪
mamba (ssm-text-generation) model_size_bytes 296 KB 296 KB +0.0% ⚪
mamba (ssm-text-generation) num_nodes 94 94 +0.0% ⚪
phi3 model_size_bytes 421 KB 421 KB +0.0% ⚪
phi3 num_nodes 58 58 +0.0% ⚪
phi3 (static-cache) model_size_bytes 421 KB 421 KB +0.0% ⚪
phi3 (static-cache) num_nodes 54 54 +0.0% ⚪
qwen2 model_size_bytes 425 KB 425 KB +0.0% ⚪
qwen2 num_nodes 60 60 +0.0% ⚪
qwen2 (static-cache) model_size_bytes 425 KB 425 KB +0.0% ⚪
qwen2 (static-cache) num_nodes 56 56 +0.0% ⚪
qwen3_5_moe (hybrid-text-generation) model_size_bytes 506 KB 506 KB -0.0% ⚪
qwen3_5_moe (hybrid-text-generation) num_nodes 269 265 -1.5% ⚪
qwen3_5_text (hybrid-text-generation) model_size_bytes 458 KB 458 KB -0.0% ⚪
qwen3_5_text (hybrid-text-generation) num_nodes 131 127 -3.1% 🟢
qwen3_5_vl (hybrid-qwen-vl) model_size_bytes 977 KB 977 KB -0.0% ⚪
qwen3_5_vl (hybrid-qwen-vl) num_nodes 461 450 -2.4% 🟢
t5 (seq2seq) model_size_bytes 836 KB 836 KB +0.0% ⚪
t5 (seq2seq) num_nodes 176 176 +0.0% ⚪
whisper (speech-to-text) model_size_bytes 1008 KB 1008 KB +0.0% ⚪
whisper (speech-to-text) num_nodes 128 128 +0.0% ⚪

No performance regressions.

Comment thread src/mobius/_build_context_test.py Fixed
Comment thread src/mobius/_build_context_test.py Fixed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Global per-channel loading and explicit full-channel group sizes still have normalization mismatches that can prevent valid checkpoints from loading.

2 open issues
What changed in this PR

Adds per-channel quantized weight export and configurable MatMulNBits accuracy levels to mobius’s graph-building APIs, preserving existing defaults when no override is supplied.

Changes:

  • Threads accuracy levels 0–4 through core, Transformers, and Diffusers builders.
  • Adds full-channel packed-weight layouts and padding-aware normalization.
  • Extends tests for layouts, numerical parity, option forwarding, and context isolation.
File Description
src/​mobius/​weights/​_codecs.py Passes logical input width to weight normalization.
src/​mobius/​models/​gemma4_test.py Tests per-channel decoder layout isolation.
src/​mobius/​integrations/​transformers/​_builder.py Validates and forwards accuracy overrides.
src/​mobius/​integrations/​transformers/​_builder_test.py Tests forwarding and unsupported-builder rejection.
src/​mobius/​integrations/​diffusers/​_builder.py Forwards accuracy overrides to component builds.
src/​mobius/​integrations/​diffusers/​_builder_test.py Tests accuracy-option forwarding.
src/​mobius/​functions/​matmul_nbits.py Trims packed padding before dequantization.
src/​mobius/​functions/​matmul_nbits_test.py Tests padded-weight numerical correctness.
src/​mobius/​components/​_quantized_linear.py Supports full-channel layouts and accuracy overrides.
src/​mobius/​components/​_quantized_linear_test.py Tests layouts, accuracy attributes, and export parity.
src/​mobius/​_weight_utils.py Adds per-channel packed-weight normalization.
src/​mobius/​_weight_utils_test.py Tests packing preservation and layout validation.
src/​mobius/​_quantization_utils.py Resolves per-channel block size from input width.
src/​mobius/​_component_quantization.py Matches resolved layouts and supplies logical widths.
src/​mobius/​_component_quantization_test.py Tests component-specific per-channel normalization.
src/​mobius/​_builder.py Exposes the core accuracy override.
src/​mobius/​_builder_test.py Tests exported accuracy attributes.
src/​mobius/​_build_context.py Stores and validates scoped accuracy overrides.
src/​mobius/​_build_context_test.py Tests validation, restoration, and concurrent isolation.

Review effort: Balanced

Comment thread src/mobius/_weight_utils.py
Comment thread src/mobius/components/_quantized_linear.py
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

🏗️ Architecture Diff

Comparing 2e5508f1 → 06e00978

Model Sub-model Changes Status
bert (feature-extraction) model 0 ⚪
falcon model 0 ⚪
gemma2 model 0 ⚪
gemma4 (gemma4) decoder 0 ⚪
gemma4 (gemma4) embedding 0 ⚪
gemma4 (gemma4) vision_encoder 0 ⚪
gemma4_text model 0 ⚪
gpt2 model 0 ⚪
llama model 0 ⚪
llama (static-cache) model 0 ⚪
mamba (ssm-text-generation) model 0 ⚪
phi3 model 0 ⚪
phi3 (static-cache) model 0 ⚪
qwen model 0 ⚪
qwen (static-cache) model 0 ⚪
qwen2 model 0 ⚪
qwen2 (static-cache) model 0 ⚪
qwen2_moe model 0 ⚪
qwen2_moe (static-cache) model 0 ⚪
qwen3 model 0 ⚪
qwen3 (static-cache) model 0 ⚪
qwen3_5_moe (hybrid-text-generation) model 33 🔴
qwen3_5_text (hybrid-text-generation) model 33 🔴
qwen3_5_vl (hybrid-qwen-vl) decoder 56 🔴
qwen3_5_vl (hybrid-qwen-vl) embedding 7 🟡
qwen3_5_vl (hybrid-qwen-vl) vision_encoder 35 🟡
qwen3_moe model 0 ⚪
qwen3_moe (static-cache) model 0 ⚪
qwen3_next (hybrid-text-generation) model 1 🔴
t5 (seq2seq) decoder 0 ⚪
t5 (seq2seq) encoder 0 ⚪
whisper (speech-to-text) decoder 0 ⚪
whisper (speech-to-text) encoder 0 ⚪
qwen3_5_moe (hybrid-text-generation) / model — 33 change(s)

Op summary: 269 → 265 nodes

--- base
+++ head
@@ -42,15 +42,11 @@
 Concat
 Reshape
 ReduceSumSquare
-CastLike
-Add
 Sqrt
 Div
 Reshape
 Reshape
 ReduceSumSquare
-CastLike
-Add
 Sqrt
 Div
 Reshape

Removed nodes:

  • - CastLike
  • - Add
  • - CastLike
  • - Add

Connectivity changes:

  • node[4] Unsqueeze: input_ids [79, 11] → [78, 11]
  • node[7] Sub: input_ids [82, 81] → [81, 80]
  • node[8] Slice: input_ids [79, 83, 82, 11] → [78, 82, 81, 11]
  • node[9] Unsqueeze: input_ids [84, 12] → [83, 12]
  • node[10] GreaterOrEqual: input_ids [85, 80] → [84, 79]
  • node[12] Cast: input_ids [87] → [86]
  • node[13] And: input_ids [88, 86] → [87, 85]
  • node[14] Where: input_ids [89, 13, 14] → [88, 13, 14]
  • node[15] Cast: input_ids [90] → [89]
  • node[16] Unsqueeze: input_ids [91, 11] → [90, 11]
  • node[18] RMSNormalization: input_ids [76, 93] → [75, 92]
  • node[19] Shape: input_ids [94] → [93]
  • node[21] MatMul: input_ids [94, 96] → [93, 95]
  • node[23] MatMul: input_ids [94, 98] → [93, 97]
  • node[25] MatMul: input_ids [94, 100] → [93, 99]
  • node[27] MatMul: input_ids [94, 102] → [93, 101]
  • node[28] Transpose: input_ids [97] → [96]
  • node[30] CastLike: input_ids [105, 23] → [104, 23]
  • node[32] Expand: input_ids [106, 107] → [105, 106]
  • node[33] CausalConvWithState: input_ids [104, 23, 108, 3] → [103, 23, 107, 3]
  • node[34] Transpose: input_ids [109] → [108]
  • node[36] Split: input_ids [111, 112] → [110, 111]
  • node[37] Shape: input_ids [94] → [93]
  • node[39] Concat: input_ids [95, 116, 117] → [94, 115, 116]
  • node[41] Concat: input_ids [95, 116, 119] → [94, 115, 118]
  • node[42] Reshape: input_ids [113, 118] → [112, 117]
  • node[43] ReduceSumSquare: input_ids [121, 24] → [120, 24]

Interface changes:

  • output[0]: dtype FLOAT → UNKNOWN; shape ['?', '?', '256'] → []; output[2]: dtype FLOAT → UNKNOWN; output[3]: dtype FLOAT → UNKNOWN; output[4]: dtype FLOAT → UNKNOWN

Initializer changes:

  • count 69 → 68; dtype distribution: FLOAT: 62 → 61
qwen3_5_text (hybrid-text-generation) / model — 33 change(s)

Op summary: 131 → 127 nodes

--- base
+++ head
@@ -42,15 +42,11 @@
 Concat
 Reshape
 ReduceSumSquare
-CastLike
-Add
 Sqrt
 Div
 Reshape
 Reshape
 ReduceSumSquare
-CastLike
-Add
 Sqrt
 Div
 Reshape

Removed nodes:

  • - CastLike
  • - Add
  • - CastLike
  • - Add

Connectivity changes:

  • node[4] Unsqueeze: input_ids [51, 11] → [50, 11]
  • node[7] Sub: input_ids [54, 53] → [53, 52]
  • node[8] Slice: input_ids [51, 55, 54, 11] → [50, 54, 53, 11]
  • node[9] Unsqueeze: input_ids [56, 12] → [55, 12]
  • node[10] GreaterOrEqual: input_ids [57, 52] → [56, 51]
  • node[12] Cast: input_ids [59] → [58]
  • node[13] And: input_ids [60, 58] → [59, 57]
  • node[14] Where: input_ids [61, 13, 14] → [60, 13, 14]
  • node[15] Cast: input_ids [62] → [61]
  • node[16] Unsqueeze: input_ids [63, 11] → [62, 11]
  • node[18] RMSNormalization: input_ids [48, 65] → [47, 64]
  • node[19] Shape: input_ids [66] → [65]
  • node[21] MatMul: input_ids [66, 68] → [65, 67]
  • node[23] MatMul: input_ids [66, 70] → [65, 69]
  • node[25] MatMul: input_ids [66, 72] → [65, 71]
  • node[27] MatMul: input_ids [66, 74] → [65, 73]
  • node[28] Transpose: input_ids [69] → [68]
  • node[30] CastLike: input_ids [77, 23] → [76, 23]
  • node[32] Expand: input_ids [78, 79] → [77, 78]
  • node[33] CausalConvWithState: input_ids [76, 23, 80, 3] → [75, 23, 79, 3]
  • node[34] Transpose: input_ids [81] → [80]
  • node[36] Split: input_ids [83, 84] → [82, 83]
  • node[37] Shape: input_ids [66] → [65]
  • node[39] Concat: input_ids [67, 88, 89] → [66, 87, 88]
  • node[41] Concat: input_ids [67, 88, 91] → [66, 87, 90]
  • node[42] Reshape: input_ids [85, 90] → [84, 89]
  • node[43] ReduceSumSquare: input_ids [93, 24] → [92, 24]

Interface changes:

  • output[0]: dtype FLOAT → UNKNOWN; shape ['?', '?', '256'] → []; output[2]: dtype FLOAT → UNKNOWN; output[3]: dtype FLOAT → UNKNOWN; output[4]: dtype FLOAT → UNKNOWN

Initializer changes:

  • count 41 → 40; dtype distribution: FLOAT: 34 → 33
qwen3_5_vl (hybrid-qwen-vl) / decoder — 56 change(s)

Op summary: 157 → 153 nodes

--- base
+++ head
@@ -67,15 +67,11 @@
 Concat
 Reshape
 ReduceSumSquare
-CastLike
-Add
 Sqrt
 Div
 Reshape
 Reshape
 ReduceSumSquare
-CastLike
-Add
 Sqrt
 Div
 Reshape

Removed nodes:

  • - CastLike
  • - Add
  • - CastLike
  • - Add

Connectivity changes:

  • node[1] Reshape: input_ids [3, 53] → [3, 52]
  • node[2] Transpose: input_ids [54] → [53]
  • node[4] Gather: input_ids [55, 56] → [54, 55]
  • node[6] Slice: input_ids [58, 12, 13, 14] → [57, 12, 13, 14]
  • node[8] Concat: input_ids [60, 59] → [59, 58]
  • node[10] Expand: input_ids [62, 61] → [61, 60]
  • node[11] Reshape: input_ids [63, 61] → [62, 60]
  • node[12] Gather: input_ids [64, 14] → [63, 14]
  • node[13] Squeeze: input_ids [65, 14] → [64, 14]
  • node[14] Gather: input_ids [8, 66] → [8, 65]
  • node[15] Gather: input_ids [9, 66] → [9, 65]
  • node[16] Gather: input_ids [64, 15] → [63, 15]
  • node[17] Squeeze: input_ids [69, 14] → [68, 14]
  • node[18] Gather: input_ids [8, 70] → [8, 69]
  • node[19] Gather: input_ids [9, 70] → [9, 69]
  • node[20] Gather: input_ids [64, 16] → [63, 16]
  • node[21] Squeeze: input_ids [73, 14] → [72, 14]
  • node[22] Gather: input_ids [8, 74] → [8, 73]
  • node[23] Gather: input_ids [9, 74] → [9, 73]
  • node[24] Where: input_ids [10, 71, 67] → [10, 70, 66]
  • node[25] Where: input_ids [11, 75, 77] → [11, 74, 76]
  • node[26] Where: input_ids [10, 72, 68] → [10, 71, 67]
  • node[27] Where: input_ids [11, 76, 79] → [11, 75, 78]
  • node[29] Unsqueeze: input_ids [81, 15] → [80, 15]
  • node[32] Sub: input_ids [84, 83] → [83, 82]
  • node[33] Slice: input_ids [81, 85, 84, 15] → [80, 84, 83, 15]
  • node[34] Unsqueeze: input_ids [86, 16] → [85, 16]
  • node[35] GreaterOrEqual: input_ids [87, 82] → [86, 81]
  • node[37] Cast: input_ids [89] → [88]
  • node[38] And: input_ids [90, 88] → [89, 87]
  • node[39] Where: input_ids [91, 18, 19] → [90, 18, 19]
  • node[40] Cast: input_ids [92] → [91]
  • node[41] Unsqueeze: input_ids [93, 15] → [92, 15]
  • node[43] RMSNormalization: input_ids [0, 95] → [0, 94]
  • node[44] Shape: input_ids [96] → [95]
  • node[46] MatMul: input_ids [96, 98] → [95, 97]
  • node[48] MatMul: input_ids [96, 100] → [95, 99]
  • node[50] MatMul: input_ids [96, 102] → [95, 101]
  • node[52] MatMul: input_ids [96, 104] → [95, 103]
  • node[53] Transpose: input_ids [99] → [98]
  • node[55] CastLike: input_ids [107, 28] → [106, 28]
  • node[57] Expand: input_ids [108, 109] → [107, 108]
  • node[58] CausalConvWithState: input_ids [106, 28, 110, 4] → [105, 28, 109, 4]
  • node[59] Transpose: input_ids [111] → [110]
  • node[61] Split: input_ids [113, 114] → [112, 113]
  • node[62] Shape: input_ids [96] → [95]
  • node[64] Concat: input_ids [97, 118, 119] → [96, 117, 118]
  • node[66] Concat: input_ids [97, 118, 121] → [96, 117, 120]
  • node[67] Reshape: input_ids [115, 120] → [114, 119]
  • node[68] ReduceSumSquare: input_ids [123, 29] → [122, 29]

Interface changes:

  • output[0]: dtype FLOAT → UNKNOWN; shape ['?', '?', '256'] → []; output[2]: dtype FLOAT → UNKNOWN; output[3]: dtype FLOAT → UNKNOWN; output[4]: dtype FLOAT → UNKNOWN

Initializer changes:

  • count 45 → 44; dtype distribution: FLOAT: 33 → 32
qwen3_5_vl (hybrid-qwen-vl) / embedding — 7 change(s)

Op summary: 52 → 46 nodes

--- base
+++ head
@@ -12,20 +12,14 @@
 Gather
 Constant
 Equal
-Constant
-Equal
-Or
 Unsqueeze
 Cast
-Reshape
 Constant
 CumSum
 Constant
 Sub
 Constant
 Clip
-Shape
-Reshape
 CastLike
 Constant
 Shape

Removed nodes:

  • - Constant
  • - Equal
  • - Or
  • - Reshape
  • - Shape
  • - Reshape

Connectivity changes:

  • node[42] Concat: input_ids [46, 47] → [47]
qwen3_5_vl (hybrid-qwen-vl) / vision_encoder — 35 change(s)

Op summary: 252 → 251 nodes

--- base
+++ head
@@ -1,4 +1,3 @@
-Cast
 Reshape
 Conv
 Reshape

Removed nodes:

  • - Cast

Modified attributes:

  • node[93] Constant: value_int: None → 0, value_ints: [1, 0] → None
  • node[127] Constant: value_int: 0 → 1
  • node[202] Constant: value_int: 0 → 1

Connectivity changes:

  • node[10] Mul: input_ids [64, 66] → [61, 66]
  • node[21] Unsqueeze: input_ids [77, 7] → [67, 8]
  • node[39] Mul: input_ids [93, 95] → [90, 95]
  • node[50] Unsqueeze: input_ids [106, 7] → [96, 8]
  • node[60] Gather: input_ids [116, 7] → [115, 8]
  • node[64] Gather: input_ids [12, 119] → [13, 118]
  • node[65] Gather: input_ids [13, 119] → [12, 119]
  • node[66] Gather: input_ids [12, 120] → [13, 119]
  • node[81] Unsqueeze: input_ids [137, 7] → [127, 8]
  • node[103] CastLike: input_ids [125, 160] → [125, 159]
  • node[112] Mul: input_ids [165, 166] → [163, 166]
  • node[119] Mul: input_ids [165, 168] → [163, 168]
  • node[123] Reshape: input_ids [176, 20] → [182, 20]
  • node[130] Unsqueeze: input_ids [190, 8] → [151, 7]
  • node[137] Unsqueeze: input_ids [197, 8] → [196, 7]
  • node[140] CastLike: input_ids [21, 184] → [22, 183]
  • node[143] Unsqueeze: input_ids [203, 23] → [183, 7]
  • node[145] Unsqueeze: input_ids [185, 7] → [158, 7]
  • node[151] Add: input_ids [211, 25] → [88, 211]
  • node[160] Add: input_ids [220, 31] → [212, 220]
  • node[178] CastLike: input_ids [125, 239] → [125, 238]
  • node[187] Mul: input_ids [244, 245] → [242, 245]
  • node[194] Mul: input_ids [244, 247] → [242, 247]
  • node[198] Reshape: input_ids [255, 20] → [261, 20]
  • node[205] Unsqueeze: input_ids [269, 8] → [151, 7]
  • node[212] Unsqueeze: input_ids [276, 8] → [275, 7]
  • node[215] CastLike: input_ids [21, 263] → [22, 262]
  • node[218] Unsqueeze: input_ids [282, 23] → [262, 7]
  • node[220] Unsqueeze: input_ids [264, 7] → [237, 7]
  • node[226] Add: input_ids [290, 44] → [221, 290]
  • node[235] Add: input_ids [299, 50] → [291, 299]
qwen3_next (hybrid-text-generation) / model — 1 change(s)

Op summary: 565 → 565 nodes

No op-sequence changes.

Interface changes:

  • output[0]: dtype FLOAT → UNKNOWN; shape ['?', '?', '256'] → []; output[2]: dtype FLOAT → UNKNOWN; output[4]: dtype FLOAT → UNKNOWN; output[6]: dtype FLOAT → UNKNOWN; output[7]: dtype FLOAT → UNKNOWN; output[8]: dtype FLOAT → UNKNOWN

Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed)

@justinchuby justinchuby left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Could you add the motivation in the PR description? Also cc @titaiwangms for another review.

@titaiwangms

Copy link
Copy Markdown
Contributor

Full review — changes requested

Reviewed commit: 06e009786988f5455575b10a24e02f38e72d7175. Five independent static reviewers plus focused local verification. This review concerns that revision only.

Confirmed findings

F2 — Major: global per-channel GPTQ loading loses logical K
src/mobius/_weight_utils.py:896–906

The global quantization path does not pass in_features or defer normalization to the leaf codec. For a GPTQ down projection with K=33, bits=4, and group_size=-1, five int32 words are interpreted as K=40. Preprocessing produces [64, 1, 20], while the module expects [64, 1, 17]. Reproduced through CausalLMModel.preprocess_weights(), build_from_module(), and apply_weights(), which raises ValueError: Weight shape mismatch. Normalize against each target linear's logical K on the global path as well; do not infer logical K from padded storage.

F3 — Major: explicit group_size=K is accepted but cannot load the same layout as -1
src/mobius/components/_quantized_linear.py:133–148; src/mobius/_weight_utils.py:1494–1508

For K=33, bits=4, QuantizedLinear accepts an explicit full-channel block and allocates 17 bytes. Olive normalization only special-cases group_size=-1; explicit 33 uses floor division and a 16-byte blob, rejecting the valid 17-byte row. For K=group_size=1, it raises ZeroDivisionError. Both failures were reproduced; the equivalent -1 cases succeed. Treat explicit full-channel groups and -1 consistently using padding-aware normalization, or reject the explicit form consistently.

F4 — Major, with reviewer dissent: selected CPU export retains unsupported native block layouts
src/mobius/_optimizations.py:571–576, enabled by src/mobius/components/_quantized_linear.py:133–148

On ONNX Runtime 1.30.0 CPU, native MatMulNBits rejects K=block_size values 7, 96, and 1536. The corresponding explicitly inlined cases all pass. The optimizer checks only the backend-wide supports_matmul_nbits, so CPU/default exports retain the native op without checking layout compatibility. Preserve arbitrary-K graph construction for compatible targets, but lower unsupported layouts or clearly reject incompatible EP/layout combinations before export. CUDA and ORT 1.31 native behavior were not verified.

Dissent: the Critical reviewer ultimately considers this a nonblocking backend-policy question because the constructor documents backend-specific support and the export compatibility promise is not explicit. The Deep reviewer and review lead retain Major specifically for the selected-CPU export path, not because arbitrary-K representation is intrinsically invalid.

F1 — Minor: the new K=1536 parity test fails locally and in CI
src/mobius/functions/matmul_nbits_test.py:206–234

test_inline_trims_single_block_padding[False-1536-1536] fails for 1 of 48 outputs at rtol=atol=1e-5: expected 0.4662437438964844, actual 0.4662666320800781, maximum absolute difference 2.2888184e-05. Reproduced locally on ORT 1.30.0 and observed in the pinned-head Ubuntu CI run 40462101700 on ORT 1.31.1. This establishes a failing test, not weight corruption or a proven numerical implementation defect. Isolate dequantization versus accumulation differences before deciding the appropriate fix; do not simply relax the tolerance without diagnosis.

Findings ledger

ID Severity Raised by Disposition
F2 Major Code / Critical / Integration Confirmed by public build/apply-weights reproduction
F3 Major Deep / Integration Confirmed by K=33 and K=1 normalization reproductions
F4 Major, disputed Critical initially / Deep Confirmed CPU compatibility gap; Critical's later policy-severity dissent preserved above
F1 Minor CI / local verification Confirmed test failure; numerical root cause not established

Verification and remaining questions

Focused local tests: 687 passed, 1 failed, 2 skipped. Additional reproductions covered global logical-K loading, explicit full-channel loading, and native-versus-inline execution. No independent accuracy-override propagation defect was found in the updated Python builders.

Nonblocking questions:

  • Deep / Critical: producer-derived asymmetric GPTQ/AWQ fixtures with distinct channel codes and zero points are still needed to establish real-producer compatibility; no corruption claim is established.
  • Integration: should CLI/auto_export expose the new accuracy override, and is per-channel QMoE intentionally unsupported? These are scope/policy questions, not confirmed regressions.

Not checked: CUDA/OpenVINO execution, real checkpoints, the full test suite, or a full security audit. The Windows GGUF CI failure was not established as caused by this PR. No implementation changes were made.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants