Repository navigation
Conversation
Performance Comparison
|
There was a problem hiding this comment.
🟡 Changes recommended
Global per-channel loading and explicit full-channel group sizes still have normalization mismatches that can prevent valid checkpoints from loading.
2 open issues
What changed in this PR
Adds per-channel quantized weight export and configurable MatMulNBits accuracy levels to mobius’s graph-building APIs, preserving existing defaults when no override is supplied.
Changes:
- Threads accuracy levels 0–4 through core, Transformers, and Diffusers builders.
- Adds full-channel packed-weight layouts and padding-aware normalization.
- Extends tests for layouts, numerical parity, option forwarding, and context isolation.
| File | Description |
|---|---|
src/mobius/weights/_codecs.py |
Passes logical input width to weight normalization. |
src/mobius/models/gemma4_test.py |
Tests per-channel decoder layout isolation. |
src/mobius/integrations/transformers/_builder.py |
Validates and forwards accuracy overrides. |
src/mobius/integrations/transformers/_builder_test.py |
Tests forwarding and unsupported-builder rejection. |
src/mobius/integrations/diffusers/_builder.py |
Forwards accuracy overrides to component builds. |
src/mobius/integrations/diffusers/_builder_test.py |
Tests accuracy-option forwarding. |
src/mobius/functions/matmul_nbits.py |
Trims packed padding before dequantization. |
src/mobius/functions/matmul_nbits_test.py |
Tests padded-weight numerical correctness. |
src/mobius/components/_quantized_linear.py |
Supports full-channel layouts and accuracy overrides. |
src/mobius/components/_quantized_linear_test.py |
Tests layouts, accuracy attributes, and export parity. |
src/mobius/_weight_utils.py |
Adds per-channel packed-weight normalization. |
src/mobius/_weight_utils_test.py |
Tests packing preservation and layout validation. |
src/mobius/_quantization_utils.py |
Resolves per-channel block size from input width. |
src/mobius/_component_quantization.py |
Matches resolved layouts and supplies logical widths. |
src/mobius/_component_quantization_test.py |
Tests component-specific per-channel normalization. |
src/mobius/_builder.py |
Exposes the core accuracy override. |
src/mobius/_builder_test.py |
Tests exported accuracy attributes. |
src/mobius/_build_context.py |
Stores and validates scoped accuracy overrides. |
src/mobius/_build_context_test.py |
Tests validation, restoration, and concurrent isolation. |
Review effort: Balanced
|
|
🏗️ Architecture Diff
qwen3_5_moe (hybrid-text-generation) / model — 33 change(s)Op summary: 269 → 265 nodes --- base
+++ head
@@ -42,15 +42,11 @@
Concat
Reshape
ReduceSumSquare
-CastLike
-Add
Sqrt
Div
Reshape
Reshape
ReduceSumSquare
-CastLike
-Add
Sqrt
Div
ReshapeRemoved nodes:
Connectivity changes:
Interface changes:
Initializer changes:
qwen3_5_text (hybrid-text-generation) / model — 33 change(s)Op summary: 131 → 127 nodes --- base
+++ head
@@ -42,15 +42,11 @@
Concat
Reshape
ReduceSumSquare
-CastLike
-Add
Sqrt
Div
Reshape
Reshape
ReduceSumSquare
-CastLike
-Add
Sqrt
Div
ReshapeRemoved nodes:
Connectivity changes:
Interface changes:
Initializer changes:
qwen3_5_vl (hybrid-qwen-vl) / decoder — 56 change(s)Op summary: 157 → 153 nodes --- base
+++ head
@@ -67,15 +67,11 @@
Concat
Reshape
ReduceSumSquare
-CastLike
-Add
Sqrt
Div
Reshape
Reshape
ReduceSumSquare
-CastLike
-Add
Sqrt
Div
ReshapeRemoved nodes:
Connectivity changes:
Interface changes:
Initializer changes:
qwen3_5_vl (hybrid-qwen-vl) / embedding — 7 change(s)Op summary: 52 → 46 nodes --- base
+++ head
@@ -12,20 +12,14 @@
Gather
Constant
Equal
-Constant
-Equal
-Or
Unsqueeze
Cast
-Reshape
Constant
CumSum
Constant
Sub
Constant
Clip
-Shape
-Reshape
CastLike
Constant
ShapeRemoved nodes:
Connectivity changes:
qwen3_5_vl (hybrid-qwen-vl) / vision_encoder — 35 change(s)Op summary: 252 → 251 nodes --- base
+++ head
@@ -1,4 +1,3 @@
-Cast
Reshape
Conv
ReshapeRemoved nodes:
Modified attributes:
Connectivity changes:
qwen3_next (hybrid-text-generation) / model — 1 change(s)Op summary: 565 → 565 nodes No op-sequence changes. Interface changes:
Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
justinchuby
left a comment
There was a problem hiding this comment.
LGTM. Could you add the motivation in the PR description? Also cc @titaiwangms for another review.
Full review — changes requestedReviewed commit: Confirmed findingsF2 — Major: global per-channel GPTQ loading loses logical K The global quantization path does not pass F3 — Major: explicit For F4 — Major, with reviewer dissent: selected CPU export retains unsupported native block layouts On ONNX Runtime 1.30.0 CPU, native Dissent: the Critical reviewer ultimately considers this a nonblocking backend-policy question because the constructor documents backend-specific support and the export compatibility promise is not explicit. The Deep reviewer and review lead retain Major specifically for the selected-CPU export path, not because arbitrary-K representation is intrinsically invalid. F1 — Minor: the new K=1536 parity test fails locally and in CI
Findings ledger
Verification and remaining questionsFocused local tests: 687 passed, 1 failed, 2 skipped. Additional reproductions covered global logical-K loading, explicit full-channel loading, and native-versus-inline execution. No independent accuracy-override propagation defect was found in the updated Python builders. Nonblocking questions:
Not checked: CUDA/OpenVINO execution, real checkpoints, the full test suite, or a full security audit. The Windows GGUF CI failure was not established as caused by this PR. No implementation changes were made. |

Adds support for per channel quantized weight export and configurable MatMulNBits accuracy levels through the builder APIs. Supports explicit accuracy levels from 0 through 4 without changing weight precision, while preserving existing defaults when no override is provided.