Repository navigation
Torch strip register parametrization and export image-text-to-text models after training. - #4288
Open
andreyanufr wants to merge 15 commits into
Open
andreyanufr wants to merge 15 commits into
andreyanufr wants to merge 15 commits into
Conversation
…4164) ### Changes Add TorchAO to GPTQModel Test Requirement and pin it to `0.17.0` ### Reason for changes It was installed transitively before with torch. Recent release of torchao(`0.18.0`) was being installed this way which caused unexpected error. (Failing action: https://github.com/openvinotoolkit/nncf/actions/runs/30928990933/job/92221632392) ### Related tickets <!--- Post the numerical ID of the ticket, if available --> ### Tests <!--- How was the correctness of changes tested and whether new tests were added --> [Test examples](https://github.com/openvinotoolkit/nncf/actions/runs/30984266172) - success
…o an/fx/conformance/fix
github-actions
Bot
added
NNCF PT
2) Changed export of torch models to ov in QAD.
2) Added test for new strip.
andreyanufr
changed the title
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Export failures can silently reuse stale IR, and missing AWQ activation-scaling hooks can produce incorrect outputs, so exported models may not match the restored checkpoint.
5 open issues
What changed in this PR
Adds parametrization-based weight decompression to NNCF’s PyTorch stripping API and extends the 2/3-bit QAD example to export supported image-text-to-text models to OpenVINO.
Changes:
- Introduces
StripFormat.RPto attach decompressors directly to module weights. - Adds multimodal checkpoint restoration, language-model repacking, and INT8 compression of other model components.
- Adds tests for RP inference, tied weights, in-place stripping, and models without compression hooks.
| File | Description |
|---|---|
tests/torch/function_hook/quantization/strip/test_strip_parametrization.py |
Adds RP stripping and output-equivalence tests. |
src/nncf/torch/function_hook/strip.py |
Implements parametrization-based weight decompression. |
src/nncf/parameters.py |
Exposes and documents StripFormat.RP. |
examples/llm_compression/torch/distillation_qad_with_lora_2_3_bit/main.py |
Extends QAD export and repacking to supported multimodal models. |
Review effort: Balanced
Comment on lines
+335
to
+337
| language_model = extract_causal_lm_from_visual(model) | ||
| language_model = load_checkpoint(language_model, ckpt_file) | ||
| nncf.strip(language_model, do_copy=False, strip_format=StripFormat.RP) |
Comment on lines
+344
to
+347
| try: | ||
| export_from_model(model, ir_dir, device="cpu") | ||
| except Exception as e: | ||
| print(f"Failed to export the model or part of the model, but repacking will still proceed: {e}") |
|
|
||
| if is_visual: | ||
| model = AutoModelForMultimodalLM.from_pretrained(pretrained, device_map="cpu", torch_dtype=torch.float32) | ||
| if "gemma" in pretrained.lower(): |
| targets = [ | ||
| (module, attr_name, decompressor_per_param[param]) | ||
| for module in model.modules() | ||
| for attr_name, param in module.named_parameters(recurse=False) |
Comment on lines
+54
to
+57
| CompressWeightsMode.INT4_SYM, | ||
| CompressWeightsMode.INT4_ASYM, | ||
| CompressWeightsMode.INT8_SYM, | ||
| CompressWeightsMode.INT8_ASYM, |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Added new option for representation compressed torch model before conversion to OV format. register_parametrization was used.
Changed the export of torch models after training in QAD for 2,3 bits: it is possible to convert some image-text-to-text models after QAD now.