Repository navigation
[OV] Allowed group-size weight compression for INT8 - #4278
Open
AsyaPronina wants to merge 2 commits into
Open
AsyaPronina wants to merge 2 commits into
AsyaPronina wants to merge 2 commits into
Conversation
github-actions
Bot
added
documentation
AsyaPronina
force-pushed
the
allow_int8_group_size
branch
from
October 21, 2026 20:10
1e9c501 to
2033cdc
Compare
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No blocking issues were identified, and focused tests cover INT8 grouping and convolution fallback.
No open issues 🎉
What changed in this PR
Enables group-wise INT8 weight compression for OpenVINO, supporting the accuracy improvement described for LTX-2.3-Diffusers.
Changes:
- Allows INT8 grouping for weights with one reduction axis, retaining per-channel compression for others.
- Rejects group-wise INT8 compression on other backends.
- Adds correctness tests and usage documentation.
| File | Description |
|---|---|
| tests/openvino/native/quantization/test_weights_compression.py | Tests INT8 grouping and per-channel convolution fallback. |
| src/nncf/quantization/quantize_model.py | Enforces the backend restriction and documents grouping. |
| src/nncf/quantization/algorithms/weight_compression/algorithm.py | Enables INT8 grouping and preserves per-channel fallback. |
| docs/usage/post_training_compression/weights_compression/Usage.md | Documents support and adds a group-size example. |
Review effort: Balanced
Open
1 of 3 tasks
| ```python | ||
| from nncf import compress_weights, CompressWeightsMode | ||
|
|
||
| # model is openvino.Model object |
Collaborator
There was a problem hiding this comment.
remove comment from code block
Comment on lines
28
to
+32
| | Compression Mode | Element type | Scale type | Granularity | Supported backends | Description | | ||
| |------------------|--------------|------------|--------------------------|----------------------------------|----------------------------| | ||
| | INT8_ASYM | INT8 | FP16 | Per-channel | OpenVINO, PyTorch, ONNX, TorchFX | [Asymmetric quantization](/docs/usage/Quantization.md#asymmetric-quantization) | | ||
| | INT8_SYM | INT8 | FP16 | Per-channel | OpenVINO, PyTorch, ONNX, TorchFX | [Symmetric quantization](/docs/usage/Quantization.md#symmetric-quantization) | | ||
| | INT8_ASYM, INT8_SYM with `group_size` | INT8 | FP16 | Group-wise | OpenVINO | Weights with a single reduction axis (e.g. MatMul, Gather) are quantized group-wise, the others (e.g. Convolution) per-channel. | |
Collaborator
There was a problem hiding this comment.
Suggested change
| | INT8_ASYM, INT8_SYM with `group_size` | INT8 | FP16 | Group-wise | OpenVINO | Weights with a single reduction axis (e.g. MatMul, Gather) are quantized group-wise, the others (e.g. Convolution) per-channel. | | |
| | Compression Mode | Element type | Scale type | Granularity | Supported backends | Description | | |
| |---------------------|--------------|------------|--------------------------|----------------------------------|----------------------------| | |
| | INT8_ASYM | INT8 | FP16 | Per-channel | OpenVINO, PyTorch, ONNX, TorchFX | [Asymmetric quantization](/docs/usage/Quantization.md#asymmetric-quantization) | | |
| | INT8_SYM | INT8 | FP16 | Per-channel | OpenVINO, PyTorch, ONNX, TorchFX | [Symmetric quantization](/docs/usage/Quantization.md#symmetric-quantization) | | |
| | INT8_ASYM, INT8_SYM | INT8 | FP16 | Group-wise | OpenVINO | Only for weights with a single reduction axis, other per-chanel | |
Please the table alignment
| return all_weight_params | ||
| # Group-wise quantization needs a single reduction axis. The other weights (e.g. of convolutions) keep | ||
| # the initial config from `_get_backup_config`: per-channel quantization in the same INT8 mode. | ||
| return [wp for wp in all_weight_params if len(wp.reduction_axes) == 1] |
Collaborator
There was a problem hiding this comment.
-1 for reduction axies is defined group_size, there is no need to filter it and select backup mode
Just set -1 in place where crteates WeightCompressionParameters
dont need to modify _get_ratio_defining_params and _get_backup_config function
| :type ratio: float | ||
| :param group_size: number of weights (e.g. 128) in the channel dimension that share quantization parameters (scale). | ||
| The value -1 means no grouping. | ||
| The value -1 means no grouping. INT8 modes are per-channel (-1) by default; with a group size (OpenVINO |
Collaborator
There was a problem hiding this comment.
The docstrings do not include information about mods or any specific cases. FIt should be describved in documantaiton.
| compressed_model = compress_weights(model, mode=mode, group_size=group_size) | ||
|
|
||
| nodes = {op.get_friendly_name(): op for op in compressed_model.get_ops()} | ||
| is_asym = mode == CompressWeightsMode.INT8_ASYM |
Collaborator
There was a problem hiding this comment.
Look at how to collected reference data for scoped condig, instead of use many asserts
And add test to validate ScopedConfig with same configurations
AlexanderDokuchaev
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Group-size parameter is not prohibited now with INT8 weight compression mode.
Reason for changes
The change is required because LTX-2.3-Diffusers compressed in INT8_SYM group_size==32 mode shows better accuracy.
Related tickets
Tests