[andreyanufr]

Changes

1. LoRA-QAT now support two new modes: CompressWeightsMode.INT3_SYM and CompressWeightsMode.INT2_SYM.

2. New function repack_weights was added to repack 4-8 bit weights to lower bit representation if possible.

Reason for changes

Support of 2-3 bit in compression in QAT and pytorch backend.

Related tickets

CVS-180191

Tests

In progress.

Weight compression - success

Test examples - failure

Conformance WC - success

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate issues remain in low-bit stripping, OpenVINO repacking, and the example workflow.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 · 2 · 2

Open (6)

• Identify zero-point constants by graph role, not shape · New
• Add 2/3-bit DQ packing support or reject validation · New
• Include gate_proj in producer search · New
• Handle meta-device offloaded weights during equalization · New
• Use logger instead of unconditional print calls · New
• Strengthen repacking test with graph/output validation · New

What changed in this PR

Adds INT2/INT3 symmetric LoRA-QAT support and OpenVINO weight repacking.

Changes:

• Adds 2/3-bit quantization mappings and strip validation.
• Implements and exports repack_weights.
• Adds tests, models, and a distillation QAT example with documentation.

File	Summary and review notes
tests/​openvino/​native/​test_model_repacking.py	Adds repacking coverage; assertions should validate graph correctness and numerical equivalence (nit, 2 votes).
tests/​openvino/​native/​models.py	Adds a repacking test model.
src/​nncf/​torch/​function_hook/​strip.py	2/3-bit DQ still uses 4-bit decompression and packing (critical, 3 votes).
src/​nncf/​quantization/​quantize_model.py	Exposes the repacking API.
src/​nncf/​quantization/​algorithms/​weight_compression/​torch_backend.py	Adds INT2/INT3 LoRA quantization mappings.
src/​nncf/​quantization/​algorithms/​weight_compression/​openvino_backend.py	Shape heuristic skips valid signed compressed weights, including 1-D tensors (critical, 2 votes).
src/​nncf/​quantization/​algorithms/​weight_compression/​algorithm.py	Coordinates repacking; direct print calls should use the project logger (nit, 4 votes).
src/​nncf/​quantization/​__init__.py	Exports repack_weights.
src/​nncf/​openvino/​quantization/​quantize_model.py	Provides the OpenVINO repacking implementation.
src/​nncf/​__init__.py	Exports the public API.
examples/​llm_compression/​torch/​distillation_qat_with_lora_2_3_bit/​requirements.txt	Defines example dependencies.
examples/​llm_compression/​torch/​distillation_qat_with_lora_2_3_bit/​README.md	Documents the example workflow.
examples/​llm_compression/​torch/​distillation_qat_with_lora_2_3_bit/​main.py	Omits gate_proj equalization (moderate, 2 votes), skips meta-device weights (moderate, 3 votes), drops partial accumulation gradients (moderate, 1 vote), and does not return its documented metric (nit, 1 vote).
.ci/​cspell_dict.txt	Adds example vocabulary.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[anzr299]

Minor

[Copilot]

🟡 Changes recommended

Repacking edge cases, compression-format handling, incomplete gradient accumulation, and missing behavioral coverage remain unresolved.

8 open findings

• Incomplete microbatch accumulation drops final gradients · New
• Constant tensors with low bit lengths are incorrectly skipped · New
• INT2/INT3 FQ modes incorrectly select LoRA quantizers · New
• Handle meta-device offloaded weights during equalization
• Repacked IR artifact path is documented incorrectly · New
• Return annotation and docstring incorrectly promise a float · New
• Add INT2/INT3 coverage to FQ-LoRA tests · New
• Strengthen repacking test with graph/output validation

4 resolved since last review

• Reject or implement INT2/INT3 support for Torch DQ
• Select signed bit width based on the actual offset range
• Preserve the original Convert element type when replacing it
• Validate microbatch accumulation and flush partial batches

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

Scalar repacking, scoped-format validation, and initial checkpoint persistence have unresolved correctness issues.

3 open issues

• Save equalized weights in the initial checkpoint · New
• Validate scoped quantization modes before inserting FQ · New
• Handle rank-zero scalar constants during repacking · New

8 resolved since last review

• INT2/INT3 FQ modes incorrectly select LoRA quantizers
• Constant tensors with low bit lengths are incorrectly skipped
• Incomplete microbatch accumulation drops final gradients
• Handle meta-device offloaded weights during equalization
• Add INT2/INT3 coverage to FQ-LoRA tests
• Return annotation and docstring incorrectly promise a float
• Repacked IR artifact path is documented incorrectly
• Strengthen repacking test with graph/output validation

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Unresolved graph compatibility and training correctness issues prevent safe approval.

No open issues 🎉

6 previously missed issues

In code that hasn't changed since last review

GPT-2 teacher caching assumes a model attribute

examples/​llm_compression/​torch/​distillation_qad_with_lora_2_3_bit/​main.py:143

The CLI accepts AutoModelForCausalLM models, but GPT-2 exposes its backbone as transformer, not model. Selecting it with --pretrained therefore raises AttributeError while caching teacher states, before compression starts. Use the Transformers base_model accessor so both naming layouts work.

Scheduler miscalculates optimizer steps across epochs

examples/​llm_compression/​torch/​distillation_qad_with_lora_2_3_bit/​main.py:243

The scheduler undercounts updates because run_training carries grad_steps across epochs. For four epochs with five microbatches and accumulation of four, training performs five optimizer steps, but this schedules only four; the final update consequently runs at zero learning rate. Divide after multiplying the epoch and microbatch counts.

Example is missing cross-framework integration coverage

examples/​llm_compression/​torch/​distillation_qad_with_lora_2_3_bit/​main.py:836

The new example is absent from tests/cross_fw/examples/example_scope.json and has no runner in run_example.py, unlike the existing LoRA distillation example. Consequently, the example suite never exercises this training/export pipeline; the added FQ tests do not call it either. Please add small-model integration cases for both bit widths, including the scales-only phase and repacked export.

Matcher repacks boolean destinations into invalid operations

src/​nncf/​quantization/​algorithms/​weight_compression/​openvino_backend.py:162

The matcher also accepts non-weight constants, such as a two-dimensional i8 attention mask containing zeros and ones followed by Convert(boolean). Repacking it constructs Subtract(boolean, boolean), which OpenVINO rejects, so this API fails on an otherwise valid model. Skip boolean destinations before constructing the replacement.

LoRA initialization allocates unnecessary full-size buffers

src/​nncf/​quantization/​algorithms/​weight_compression/​torch_backend.py:336

The dense identity and out-of-place addition allocate two extra full-size buffers during LoRA initialization, including existing 4/8-bit LoRA paths. For an 8192×8192 FP32 matrix, that adds about 512 MiB at the addition. Apply the stabilizer through the reshaped residual's diagonal view instead; this preserves rectangular and batched behavior without those allocations.

Graph conversion lacks support for repacked u3 tensors

src/​nncf/​quantization/​repack_weights.py:44

Calling nncf.repack_weights on its own 3-bit output fails during build_graph: GraphConverter.convert_to_nncf_dtype in src/nncf/openvino/graph/nncf_graph_builder.py:46–73 has no u3 entry and raises NotImplementedError. Add "u3": "int" to that mapping so models containing already-repacked tensors can reach the skip logic.

3 resolved since last review

• Handle rank-zero scalar constants during repacking
• Validate scoped quantization modes before inserting FQ
• Save equalized weights in the initial checkpoint

_{Review effort: Balanced}

[anzr299]

What do you think about moving the helper functions to a utils.py in this directory?