Repository navigation
Consider offline static weight-layout folding separately - #801
Draft
titaiwangms wants to merge 1 commit into
Draft
titaiwangms wants to merge 1 commit into
titaiwangms wants to merge 1 commit into
Conversation
Preserve the original implementation in an isolated draft so exporter-versus-runtime design choices can be discussed independently of #754. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9a2a5b82-7cd3-4a76-bb94-0998527ea723 Signed-off-by: titaiwang <titaiwang@microsoft.com>
3 of 9 tasks
Performance Comparison
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status: design discussion / not ready to merge
Extract the existing generic static weight-layout folding implementation from
#754 into a separate draft. This preserves the implementation and its regression
coverage while we reconsider whether these deployment transformations belong in
Mobius or in ONNX Runtime graph optimization / operator prepacking.
Stacked on
titaiwang/quantized-qkv-packing(#754) so the diff contains only theextracted work. #787 has already merged into that branch; its modeling changes
are not being reverted or duplicated here. This supersedes the closed #788.
Extracted implementation
FoldWeightLayoutPass: lazy, dtype-preserving static Reshape/Transpose folding.ownership, lazy backing-file tracking, and dead-source cleanup.
The draft restores the implementation from the pre-split #754 head
6ed50516a7c15af00a1018418cd982a05c5310cdwithout expanding its eligibility.The draft's tracked tree is identical to that pre-split tree.
Open design decision
Offline folding can simplify deployment, but also changes stored weight layouts
and obscures their relationship to the source checkpoint. We want to retain
explicit graph transformations and checkpoint-compatible weight storage where
useful for inspection, reloading, and adapters.
ORT-side folding or QMoE prepacking is an alternative to investigate without
simply removing runtime constant-folding memory limits. This draft does not
implement an ORT change, select a final design, or add a new public opt-in API.
Its existing automatic post-weight integration is preserved for discussion, not
an agreement to enable that behavior in the main projection-packing PR.
In particular, MoE/QMoE and QDQ storage-layout paths remain protected as before.
This does not implement the requested large QMoE fc1 gate/up interleave
folding or claim to resolve the performance report in #754.
Validation
CPU/offline, worktree-local imports:
Result: 554 passed. Scoped Ruff check/format and whitespace checks passed.
This is extraction regression coverage, not a new whole-model numerical or
performance qualification. Existing failed/unwaived numerical gates are
unchanged. No CUDA benchmark, runtime rebuild, or change to the running paired
MMLU evaluation is included.