Recurrent Residual Quantization (RRQ) for LLMs - #2308
Open
luoyu-intel wants to merge 44 commits into
Open
luoyu-intel wants to merge 44 commits into
luoyu-intel wants to merge 44 commits into
Conversation
Contributor
Author
Qwen3 Quantization Benchmark: W4A16 vs RRQ (Intel Arc Pro B60 XPU)Test Environment
Test Configurations
Qwen3-0.6B Results (Correct Calibration)
All 4 configurations successfully quantized 196/197 layers (lm_head kept unquantized). Timing ComparisonRTN vs OPT (within same scheme)
W4A16 vs RRQ (within same optimization mode)
Per-Layer Breakdown
Memory Comparison
Qwen3-8B Results (Preliminary — Suboptimal Calibration)
All 4 configurations successfully quantized 252/253 layers (lm_head kept unquantized). Device: xpu:1. 8B Timing Comparison (Preliminary)RTN vs OPT (within same scheme)
W4A16 vs RRQ (within same optimization mode)
8B Per-Layer Breakdown (Preliminary)
8B Memory Comparison (Preliminary)
SummaryQwen3-0.6B (Correct Calibration: nsamples=512, seqlen=2048, bs=8)
Qwen3-8B (Preliminary: nsamples=4, seqlen=64, bs=1)
Conclusions
Appendix: Preliminary 8B Results (nsamples=4, seqlen=64, bs=1)These were the first 8B runs (2026-09-12/13) using suboptimal calibration parameters. They are kept for reference but should be superseded by a proper run with nsamples=512, seqlen=2048, bs=8. Key observations from preliminary data:
|
Contributor
Author
|
implementation of #2300 |
Collaborator
|
/azp run Unit-Test-CUDA-AutoRound |
|
Azure Pipelines successfully started running 1 pipeline(s). |
hshen14
requested review from
lkk12014402,
lvliang-intel,
xin3he and
yiliu30
and
a lite review from Copilot
September 9, 2026 05:50
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
load_rrq_model can silently leave some packed base layers as uninitialized/random nn.Linear weights when residual planes are missing or skipped, which is correctness-critical for inference.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds Recurrent Residual Quantization (RRQ) to AutoRound, enabling a single INT2-base checkpoint plus packed INT2 residual planes to support dynamic 2/4/6/8-bit (and mixed-precision) inference via a new auto_round:rrq residual artifact and corresponding loader/runtime modules.
Changes:
- Introduces RRQ algorithm config + quantizers (RTN and per-plane SignRound tuning) and registers it in the algorithm registry.
- Adds RRQ residual export format (
auto_round:rrq) and inference-time composition (load_rrq_model,RRQLinear, precision switching utilities). - Updates quantized linear kernels to expose
_dequantize()for reuse, plus adds extensive CPU unit tests and user scripts; updates README(+CN) and ignoresrrq_output/.
File summaries
| File | Description |
|---|---|
test/unit/test_cpu/algorithms/test_rrq.py |
Comprehensive CPU unit tests for RRQ config, packing, reconstruction, inference switching, and Phase 2/3 behaviors. |
test_rrq_qwen3_06b.py |
Standalone script to quantize Qwen3-0.6B with RRQ and verify base/residual layout + (optional) load/forward. |
test_rrq_lm_eval.py |
Standalone script to run lm-eval across RRQ bit-widths (base+residual). |
README.md |
Adds RRQ announcement to “What’s New”. |
README_CN.md |
Chinese counterpart update for the RRQ “What’s New” entry. |
auto_round/utils/common.py |
Adds auto_round:rrq to supported formats list. |
auto_round/inference/rrq_model.py |
New loader that merges base + residual artifacts into an RRQ-enabled model by replacing layers with RRQLinear. |
auto_round/inference/rrq_linear.py |
New RRQLinear module and helpers to set uniform or random mixed precision across layers. |
auto_round/inference/backend.py |
Adds RRQ format constant (RRQ_FORMAT). |
auto_round/export/formats/backends/rrq.py |
New OutputFormat backend for auto_round:rrq residual export. |
auto_round/export/formats/backends/__init__.py |
Exposes RRQFormat in backend imports/exports. |
auto_round/export/export_to_mlx/export.py |
Fail-fast guard rejecting RRQ residual models for MLX export. |
auto_round/export/export_to_gguf/conversion/base.py |
Fail-fast guard rejecting RRQ residual models for GGUF export. |
auto_round/export/export_to_autoround/export_to_rrq.py |
Implements RRQ residual serialization + Phase 2 residual generation from base+raw weights. |
auto_round/compressors/model_free.py |
Frees packed shard tensors earlier to improve memory reclamation; tweaks a log message. |
auto_round/cli/algorithms.py |
Improves CLI arg merge logic to avoid mismatching boolean optional arguments with shared dest. |
auto_round/autoround.py |
Forces RRQ to route through calibrated path (disables model-free path) to avoid dropping residual planes. |
auto_round/algorithms/registry.py |
Registers RRQ config/quantizer modules and adds rrq to built-in algorithm order. |
auto_round/algorithms/quantization/rrq/quantizer.py |
Core RRQ quantizers (RTN multi-plane + SignRound per-plane tuning with frozen prefix) and packing logic. |
auto_round/algorithms/quantization/rrq/config.py |
RRQConfig implementation (fixed INT2 planes, tuning params, calibration requirement). |
auto_round/algorithms/quantization/rrq/__init__.py |
RRQ module exports. |
auto_round/__init__.py |
Exposes RRQConfig and lazily exports load_rrq_model / generate_rrq_residual. |
auto_round_extension/torch/qlinear_torch.py |
Fixes device usage in packing; adds _dequantize() helper and adjusts g_idx logic. |
auto_round_extension/torch/qlinear_torch_zp.py |
Adds _dequantize() helper and adjusts g_idx logic (symmetric/GPTQ-style). |
.gitignore |
Ignores rrq_output/ directory. |
Review details
- Files reviewed: 24/25 changed files
- Comments generated: 7
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
hshen14
reviewed
Sep 9, 2026
a32543254
reviewed
Sep 9, 2026
This comment has been minimized.
This comment has been minimized.
wenhuach21
reviewed
Sep 9, 2026
wenhuach21
reviewed
Sep 9, 2026
wenhuach21
reviewed
Sep 9, 2026
This comment has been minimized.
This comment has been minimized.
Contributor
Author
Qwen3 Quantization Benchmark Report (0.6B & 8B)Test Environment
Test Configurations
Key Results
All 4 configurations successfully quantized 196/197 layers (lm_head kept unquantized). Timing ComparisonRTN vs OPT (within same scheme)
W4A16 vs RRQ (within same optimization mode)
Per-Layer Breakdown
Memory Comparison
Qwen3-8B Results
All 4 configurations successfully quantized 252/253 layers (lm_head kept unquantized). Device: xpu:1. 8B Timing ComparisonRTN vs OPT (within same scheme)
W4A16 vs RRQ (within same optimization mode)
Cross-Model RRQ/W4 Ratio
8B Per-Layer Breakdown
8B Memory Comparison
Summary (Qwen3-0.6B)
Summary (Qwen3-8B)
|
…+2+2) Implement RRQ (Recurrent Residual Quantization) algorithm for LLM quantization. Each layer is quantized into 4 planes of INT2 via iterative RTN: - Base plane (plane 0): standard INT2 AutoRound export (auto_round format) - Residual planes (1-3): packed INT2, stored in auto_round:rrq format Key components: - RRQConfig: algorithm config (bits=2, data_type=int, act_bits=16, 4 planes) - RRQRTNQuantizer: iterative RTN quantizer producing packed INT2 planes - RRQFormat: output format backend for residual model export - RRQLinear: inference module with dynamic precision (2/4/6/8-bit) - load_rrq_model: loader combining base + residual into RRQ-enabled model - save_quantized_rrq / save_rrq_base_model: export helpers Fixes: - qlinear_torch.py: self.device -> device param in asym pack path - SUPPORTED_FORMATS: added auto_round:rrq - ModelFreeCompressor: accept auto_round:rrq format - GGUF/MLX export: reject RRQ residual models (fail fast) Validation (Qwen3-0.6B, group_size=128, asym, XPU): - 23/23 unit tests pass - HellaSwag accuracy: 26.5%(2b) -> 35.5%(4b) -> 43.5%(6b) vs fp32 43.5%
… base) Add generate_rrq_residual(base_model_dir, raw_model, output_dir) to generate the 3 RTN INT2 residual planes from an existing INT2 base model + original FP weights, without re-quantizing the base. Supports local dirs and HF model names; validates bits/group_size/sym against the base config; exposed via lazy import from auto_round. Adds 5 unit tests (output structure, residual norm monotonic decrease, config fail-fast, top-level export) and the Phase 1 PR description. All 28 RRQ tests pass.
Add RRQConfig tuning fields and RRQSignRoundQuantizer with four sequential AutoRound sign-SGD rounds. Each round optimizes value_k/min_scale_k/max_scale_k through the STE path while freezing the completed prefix, then exports the existing packed INT2 ABI. Route RRQ OPT configurations through the calibrated compressor, preserve RTN behavior for iters=0, and add Phase 3 tests and Qwen3-0.6B validation documentation. Verified with 31 RRQ tests and a two-iteration Qwen3-0.6B tuning/export/load run.
Fix Phase 3 prefix state so each round freezes the cumulative sum of all previous planes instead of only the immediately preceding plane. Add a reconstruction regression test and expose iters/lr/calibration controls in the Qwen3 RRQ test script. Validated with 32 RRQ tests and corrected OPT-50 versus RTN HellaSwag evaluations.
for more information, see https://pre-commit.ci
…cision
- base plane routed through imatrix-weighted opt-RTN (bit-exact with standard
W2A16 base); residual planes seed scale search via search_optimized_init_scale
- collect per-layer imatrix on both RTN and SignRound paths; need_calib always
- config defaults to SignRound (iters=200); iters=0 selects RTN-only; surface
standard AutoRound knobs; num_residual_planes in {1,3}; disable_opt_rtn kept
as a routing guard
- force RRQ down the regular compressor so all residual planes are retained;
explicit export format overrides a previously resolved format
- add set_rrq_random_residual + load_rrq_model(residual_fraction=...) for
seeded per-layer mixed precision; new unit test
- Auto-set scheme to W2A16 when --format auto_round:rrq is used without --scheme - Force disable_opt_rtn=True in RRQConfig when None (CLI default), preventing the entry from coercing to OptimizedRTNConfig which would drop residual planes
for more information, see https://pre-commit.ci
RRQ is a hidden algorithm (not user-facing), so it shouldn't appear in the builtin algorithm order that defines user-visible enumeration. It remains resolvable by name via _ALG_REGISTRY and _ALIAS_TO_NAME.
…ound:rrq'
Per reviewer feedback, RRQ should use the same quant_method as standard
AutoRound ('auto-round') and differentiate via packing_format instead of
using a separate quant_method. This aligns with the project convention
where quant_method describes the quantization algorithm and packing_format
describes the storage format.
Changes:
- export_to_rrq.py: RRQ_QUANT_METHOD now 'auto-round', added RRQ_PACKING_FORMAT
constant, quantization_config now has both fields
- rrq_model.py: _validate_base_matches_residual checks both quant_method
and packing_format
- GGUF export: check packing_format instead of quant_method
- MLX export: check packing_format instead of quant_method
- Tests updated to verify both fields
for more information, see https://pre-commit.ci
Collaborator
|
/azp run Unit-Test-CUDA-AutoRound |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Critical API, export sequencing, metadata, quantization, and inference issues remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (12)
auto_round/algorithms/quantization/rrq/quantizer.py:446
- This residual packing path assumes
original_weightis[out_features, in_features], but a TransformersConv1Dweight is stored transposed. It constructs the residualQuantLinearusing the raw dimensions, while the standard exporter transposes Conv1D before packing, so the residual qweight shape/semantics will not match the base artifact and can fail inload_rrq_model. Apply the same Conv1D normalization as the standard exporter before calling_pack_plane.
# Residual plane: store packed INT2 (W2A16 layout) so the
# on-disk artifact is a standard single-plane INT2 layout.
in_features = original_weight.shape[1]
qweight, scales, qzeros = self._pack_plane(quantized, scale, zp, bits, group_size, in_features)
auto_round/algorithms/quantization/rrq/quantizer.py:626
- The configured
dynamic_max_gapearly-stop behavior is not implemented here: after tracking the best loss, this branch executespassfor every positive gap. RRQ tuning therefore always runs all iterations even when users request the same stopping criterion supported by SignRound; add the best-iteration tracking and break condition before merging this path.
if not self.not_use_best_mse and 0 < self.dynamic_max_gap:
# Keep the same early-stop contract as SignRound; the default
# -1 disables this path.
pass
auto_round/export/export_to_autoround/export_to_rrq.py:260
save_quantized_rrq()always pops and renames the module buffers in place, butRRQFormat.save_quantized()accepts the standardinplaceargument and does not pass it through. Thereforesave_quantized(..., inplace=False)still mutates the quantized model and can make a subsequent base export lose the residual buffers, violating the exporter contract.
# Build a state dict containing *only* the packed residual planes, and
# rename the in-memory ``rrq_*_k`` buffers to the on-disk ``*_k`` names.
residual_state: dict[str, torch.Tensor] = {}
for name, module in model.named_modules():
if not hasattr(module, "rrq_total_planes"):
continue
for k in range(1, module.rrq_total_planes):
for src, dst in (
(f"rrq_qweight_{k}", f"qweight_{k}"),
(f"rrq_scales_{k}", f"scales_{k}"),
(f"rrq_qzeros_{k}", f"qzeros_{k}"),
):
if src in module._buffers:
module._buffers[dst] = module._buffers.pop(src)
auto_round/export/export_to_autoround/export_to_rrq.py:561
- The incremental generator has the same Conv1D orientation mismatch: raw GPT-2-style weights are
[in, out], whereasQuantLinear.forward(identity).There is[out, in]. The subtraction in the next step will either fail for non-square layers or generate transposed residuals, so this path cannot generate valid residuals for Conv1D-based LLMs.
# Dequant by running forward on identity
identity = torch.eye(in_features, dtype=torch.float32)
with torch.no_grad():
out = ql.forward(identity) # (in_features, out_features)
W_dequant_base = out.T.to(torch.float32) # (out_features, in_features)
auto_round/export/export_to_autoround/export_to_rrq.py:411
- The public
deviceargument is documented as the computation device, but this implementation forces dequantization, identity inputs, quantization, and packing to CPU (ql.to("cpu"),device="cpu"). Passing a GPU/XPU device therefore has no effect and makes residual generation for a large model unnecessarily CPU-bound. Thread the requested device through the per-layer quantization and packing path.
device: Union[str, torch.device] = "cpu",
auto_round/export/export_to_autoround/export_to_rrq.py:266
- If the standard base exporter has already replaced the RRQ layers, this loop finds no
rrq_*buffers, yet the code still writes an artifact. The resulting residual directory looks valid butload_rrq_modellater fails because every base layer is missing residual planes. Reject an empty residual state before serializing it.
# Serialize the residual state dict (safetensors by default, torch otherwise).
_save_state_dict_sharded(residual_state, output_dir, safe_serialization)
auto_round/export/formats/backends/rrq.py:72
inplaceis part of this backend's save contract, but it is never forwarded tosave_quantized_rrq; that helper renames buffers directly on the original model. Consequentlysave_quantized(..., inplace=False)still mutates and removes therrq_*buffers, which can break later exports or use of the source model. Thread the flag through and clone before renaming, or reject unsupportedinplace=Falseexplicitly.
inplace: Whether to modify the model in place.
auto_round/inference/rrq_model.py:304
- Catching every
Exceptionhere turns real load failures (for example missing/corrupt weights, an invalid config, or OOM) into a config-only model with randomly initialized non-quantized parameters.load_rrq_modelcan therefore return a plausible-looking but incorrect model. Restrict the fallback to a known packed-loading incompatibility and otherwise re-raise, or load non-quantized weights through a controlled path.
except Exception: # pragma: no cover - fall back to architecture-only load
logger.warning(
"from_pretrained(%s) failed; loading architecture without weights.",
auto_round/inference/rrq_model.py:324
- Skipping a missing module leaves that packed base layer unreconstructed, but the function only fails when all layers are skipped. The returned model can therefore contain a normal/random
nn.Linearfor one or more checkpoint layers while silently omitting their residuals. Treat any missing eligible module as an incompatible checkpoint and raise instead of continuing.
try:
base_model.get_submodule(layer_name)
except AttributeError:
logger.warning(f"Base module {layer_name!r} not found; skipping.")
continue
test/unit/test_cpu/algorithms/test_rrq.py:1020
set_rrq_bitsonly traversesRRQLinearmodules, butAutoRound.quantize()returns the original quantized linear modules withrrq_*buffers; the conversion toRRQLinearhappens inload_rrq_model. This call therefore only logs a warning and the following supposed 4-bit evaluation still uses the 8-bit/full-plane model, so the regression test does not exercise precision switching.
set_rrq_bits(model, 4)
test/unit/test_cpu/algorithms/test_rrq.py:200
- This test does not exercise the new
AutoRound.__new__format auto-selection at all; it only resolves the registry entry directly. A regression in the constructor/format path would still pass. Exercise the constructor or CLI path and assert both the selected RRQ config and resolved RRQ format.
# Simulate what _CompressorBuilder.__new__ does:
# when alg_configs is None and format is auto_round:rrq, it picks "rrq".
from auto_round.algorithms.registry import resolve_alg_config, resolve_algorithm_names
config = resolve_alg_config("rrq")
test/unit/test_cpu/algorithms/test_rrq.py:198
resolve_algorithm_namesis imported but never used in this test; the unused local import will be flagged by the repository's Ruff/Pyflakes checks. Remove it (or use it in the assertion).
from auto_round.algorithms.registry import resolve_alg_config, resolve_algorithm_names
- Files reviewed: 26/27 changed files
- Comments generated: 11
- Review effort level: Lite
wenhuach21
reviewed
Sep 15, 2026
|
|
||
| * [2026/09] 现在支持在 CUDA 设备上通过 vLLM 和 Transformers 使用 5/6/7-bit WOQ 模型，感谢 Humming Kernel 的支持。 | ||
|
|
||
| * [2026/09] 我们实验性地支持 **递归残差量化（RRQ）**：一种渐进式多精度表示，在标准 INT2 base 之上叠加 INT2 残差平面。单个 checkpoint 即可在加载时选择 2/4/6/8-bit（以及逐层混合精度），无需重新量化：[*论文*](https://arxiv.org/abs/2608.04048)。注意：该功能仍处于实验阶段，尚不支持生产级部署场景。 |
Contributor
There was a problem hiding this comment.
it would be better to generate an example model and detail the quantization cmd and inference code
wenhuach21
reviewed
Sep 15, 2026
|
|
||
| * [2026/09] We now support 5/6/7-bit WOQ models in vLLM and Transformers on CUDA devices, thanks to Humming Kernel. | ||
|
|
||
| * [2026/09] We experimentally support **Recurrent Residual Quantization (RRQ)**, a progressive multi-precision representation that stacks INT2 residual planes on a standard INT2 base. A single checkpoint serves 2/4/6/8-bit — and per-layer mixed precision — selectable at load time without re-quantizing: [*Paper*](https://arxiv.org/abs/2608.04048). Note: this is an experimental feature and production-level deployments are not yet supported. |
Contributor
There was a problem hiding this comment.
I guess users don't care about the details of “a progressive multi-precision representation that stacks INT2 residual planes on a standard INT2 base.” Would simply saying “A single checkpoint supports 2/4/6/8-bit” be enough?
wenhuach21
reviewed
Sep 15, 2026
| args._api_format = args.format if format_was_explicit or args.model_free else None | ||
|
|
||
| # Auto-set scheme to W2A16 for RRQ format if user didn't specify --scheme | ||
| if "auto_round:rrq" in (args.format or "").lower() and not scheme_was_explicit: |
Contributor
There was a problem hiding this comment.
1 This behavior is not exposed in the API. Besides, because 2-bit quantization has a relatively large accuracy drop, so keeping W4 as the default is acceptable.
2 Heng is also consolidating the scheme and AutoScheme APIs, so perhaps users will be able to specify their desired scheme directly in the future.
wenhuach21
reviewed
Sep 15, 2026
| config_factory: Callable[[], object] | None = None, | ||
| summary: str = "", | ||
| alias_factories: dict[str, Callable[[], object]] | None = None, | ||
| hidden: bool = False, |
Contributor
There was a problem hiding this comment.
Could you ask AI to annotate this function, including a description of each argument?
wenhuach21
reviewed
Sep 15, 2026
| # RRQ residual models are not supported by MLX; fail fast so the residual | ||
| # planes are never silently dropped. | ||
| quant_cfg = getattr(getattr(model, "config", None), "quantization_config", None) | ||
| if isinstance(quant_cfg, dict) and quant_cfg.get("packing_format") == "auto_round:rrq": |
Contributor
There was a problem hiding this comment.
These lines of code are not needed, right? Since the format is set to RRQ, it should never reach the MLX path. We should refine this logic in the future to avoid adding this logic to every format if it's needed
wenhuach21
approved these changes
Sep 15, 2026
Conv1D (GPT-2 style) stores weights as (in, out) while the quant helpers and QuantLinear.pack expect (out, in). This fix mirrors the standard WrapperLinear._qdq_weight / QuantLinear.pack path: - transpose weight to (out, in) before quantization - restore native (in, out) orientation when storing base plane - also handle in the standard RTN path (_quantize_layer_via_rtn) Also: - add device check to RRQLinear._get_packed_weight cache hit - fix typo 'auto-round-rrq' -> 'auto_round:rrq' in docstring - add Conv1D regression tests (weight shape, plane shape, recon)
The per-call format parameter in save_quantized() was only applied when self.formats was None. This prevented users from saving a base model in standard auto_round format after an RRQ quantize session had set self.formats to 'auto_round:rrq'. Now an explicitly-passed format always overrides self.formats, enabling the documented two-step RRQ export: 1. save_quantized(format='auto_round') -> base artifact 2. save_quantized(format='auto_round:rrq') -> residual artifact Co-Authored-By: GitHub Copilot <273000991+Copilot@users.noreply.github.com>
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
yiliu30
approved these changes
Sep 19, 2026
…q_model
The compressor's public quantize_and_save(format="auto_round:rrq") path only
invoked RRQFormat.save_quantized -> save_quantized_rrq, which wrote the residual
artifact but never the base model. The documented workaround (call
save_rrq_base_model after save_quantized_rrq) could not be expressed through
the public API: BaseCompressor.save_quantized cached self.formats after the
first save, so a subsequent save_quantized(format="auto_round") silently
re-emitted the residual, and doing base-first would let pack_layer replace the
nn.Linear modules and drop the rrq_* residual buffers.
Fix (approach B — a dedicated export path that preserves both artifacts):
- export_to_rrq.py: add save_rrq_model(), an orchestrator that writes
{output_dir}/residual/ (save_quantized_rrq, buffers still intact)
{output_dir}/base/ (save_rrq_base_model, standard INT2 export)
The residual-first ordering is enforced here, so callers no longer have to
sequence the two low-level helpers.
- export_to_rrq.py: save_rrq_base_model gains tokenizer/processor params and
its docstring documents the ordering constraint as a low-level-helper note.
- backends/rrq.py: RRQFormat.save_quantized now delegates to save_rrq_model so
the quantize_and_save(format="auto_round:rrq") public path produces both
artifacts (base/ + residual/) in one call.
- base.py: add OutputFormat.is_rrq() helper (mirrors is_fake/is_gptq).
- export_to_autoround/__init__.py: export save_rrq_model alongside the others.
- test_rrq.py: add test_save_rrq_model_ordering_residual_before_base to lock in
the residual-before-base invariant.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The default active_bits=8 caused ValueError for checkpoints with fewer than 4 planes (e.g. num_residual_planes=1 → max 4-bit). Now defaults to None, resolved to total_planes * bits from the loaded config. Explicit values exceeding the checkpoint's max are rejected with a clear error.
for more information, see https://pre-commit.ci
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- RRQ: fix active_bits default from 8 to None (auto-detect from checkpoint) - RRQ: remove W2A16 auto-injection in CLI (auto-scheme handles it) - export_to_mlx: remove unreachable RRQ guard (format routing prevents reaching) - export_to_gguf: remove unreachable RRQ guard (same reasoning) - registry: add docstring to register_algorithm() - README: simplify RRQ description, add example commands - tests: add TestResolveActiveBits unit tests for validation logic
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Recurrent Residual Quantization (RRQ) packs a weight into K sequential INT2 planes (1 base + K−1 residuals). A single checkpoint serves 2 / 4 / 6 / 8-bit weight-only precision — switchable at load/runtime — without re-quantizing, and it is selectable per layer for mixed precision.
Each plane is quantized with the same RTN / SignRound machinery as a standard AutoRound W2A16 model, so per-plane quality parity is guaranteed. The residual planes are stored in the stock AutoRound INT2 layout (
qweight/scales/qzerosper plane, indexed*_k), so they reuse existing W2A16QuantLinearpack/forward kernels with no new runtime.API
Design highlights
search_optimized_init_scale. Withiters>0, per-plane sign-SGD tunes each plane against the block calibration loss while the completed prefix is frozen.effective bits = active_planes × 2(base=2, +1=4, +2=6, +3=8).set_rrq_bits/set_rrq_random_residual/load_rrq_model(residual_fraction=...)select precision without re-quantizing.quant_methodstays"auto-round"; RRQ is distinguished bypacking_format = "auto_round:rrq"(consistent with how AutoRound differentiates formats likeauto_round:auto_gptq)._BUILTIN_ALGORITHM_ORDER. Users opt in via--format auto_round:rrqoralg_configs=RRQConfig(...).disable_opt_rtn=Trueis enforced inRRQConfig.check_config()(RTN alone only produces one plane).Changed files
algorithms/quantization/rrq/{__init__,config,quantizer}.py—RRQConfig,RRQRTNQuantizer,RRQSignRoundQuantizer(per-plane sign-SGD);algorithms/registry.pyexport/export_to_autoround/export_to_rrq.py(save_quantized_rrq,save_rrq_base_model,generate_rrq_residual);export/formats/backends/rrq.py(auto_round:rrqOutputFormat);export/__init__.py,export/formats/backends/__init__.pyinference/rrq_linear.py(RRQLinear,set_rrq_bits,set_rrq_random_residual);inference/rrq_model.py(load_rrq_model);inference/backend.py,inference/__init__.py__init__.py,autoround.py,compressors/model_free.py,cli/{algorithms,main}.py,utils/common.pyexport/export_to_gguf/conversion/base.py,export/export_to_mlx/export.py(reject RRQ residual models, no silent drop)auto_round_extension/torch/qlinear_torch{,_zp}.py(asym packself.device→device)test/unit/test_cpu/algorithms/test_rrq.py(45 unit tests, incl. lm-eval accuracy)Validation
pytest test/unit/test_cpu/algorithms/test_rrq.py— 45 passed (config validation, packed-INT2 storage, residual convergence, sym/asym, forward & precision switching, sign-SGD prefix accumulation, export buffer rename, load validation, incremental residual generation, random-residual mixed-precision config, lm-eval accuracy).Backward compatibility & known limitations
load_rrq_modelbehavior (uniformactive_bits) is unchanged whenresidual_fractionis not supplied.act_bits=16); no activation quantization.