Repository navigation
Aanuf/2 3 bit training adjust lora params - #4287
Open
andreyanufr wants to merge 63 commits into
Open
andreyanufr wants to merge 63 commits into
andreyanufr wants to merge 63 commits into
Conversation
2) Removed 6bit from repacking.
…4164) ### Changes Add TorchAO to GPTQModel Test Requirement and pin it to `0.17.0` ### Reason for changes It was installed transitively before with torch. Recent release of torchao(`0.18.0`) was being installed this way which caused unexpected error. (Failing action: https://github.com/openvinotoolkit/nncf/actions/runs/30928990933/job/92221632392) ### Related tickets <!--- Post the numerical ID of the ticket, if available --> ### Tests <!--- How was the correctness of changes tested and whether new tests were added --> [Test examples](https://github.com/openvinotoolkit/nncf/actions/runs/30984266172) - success
…o an/fx/conformance/fix
2) Added optional possibility to save torch checkpoint with dequantized weights.
2) Added README and requirements.txt
2) Removed debug prints.
2) Added arg for deterministic cuda execution. This arg is optional because it can slow down the training.
2) Added test for 2-3 bit LoRA.
2) Expand FQ_LORA test for case 2 and 3 bit.
…modes before inserting FQ' Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…training_adjust_lora_params
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The fallback rejects valid grouped weights and still permits rank errors when all compressed weights use INT8 backup precision.
2 open issues
What changed in this PR
This PR adds automatic backup compression for Torch weights whose dimensions cannot support the requested LoRA rank, aiming to prevent rank errors during quantization-aware training and distillation.
Changes:
- Track the compression backend and filter rank-incompatible weights out of primary compression candidates.
- Add parameterized tests for both LoRA formats and unchanged DQ behavior.
| File | Description |
|---|---|
tests/torch/function_hook/quantization/test_fq_lora.py |
Tests rank filtering across compression formats. |
src/nncf/quantization/algorithms/weight_compression/algorithm.py |
Adds backend-aware LoRA rank filtering before primary configuration assignment. |
Review effort: Balanced
| ratio_defining_params, group_size_values | ||
| ) | ||
|
|
||
| ratio_defining_params = self._handle_lora_rank_compatible_params_fallback(ratio_defining_params) |
…aining_adjust_lora_params
…ouped weight trailing dimensions' Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Assign a backup mode for layers where at least one dimension is smaller than
lora_rank— for cases using the Torch backend and QAD/QAT.Reason for changes
Some models contain layers where the output dimension is smaller than the
lora_rankvalue (for example,model.layers.*.mlp.shared_expert_gate.weighthas a dimension of[1, 2048]), which triggers an exception. This issue can be resolved either by excluding such layers from the scope—a capability not currently supported in existing training pipelines—or automatically, which is what this PR achieves.Related tickets
Tests