Fix out-of-bounds read for unscattered dims in multi-dim reduce-scatter lowering - #1463
Open
EylonKrause wants to merge 1 commit into
Open
Fix out-of-bounds read for unscattered dims in multi-dim reduce-scatter lowering#1463EylonKrause wants to merge 1 commit into
EylonKrause wants to merge 1 commit into
Conversation
…er lowering rewriteReduceScatterCombiningMultipleDims splits each scattered dimension into (factor, quotient). The branch handling a non-scattered dimension pushes the original size but is missing a `continue`, so execution falls through to `scatteredFactors[std::distance(scatteredDims.begin(), it)]` with `it == end()`, indexing one past the end of scatteredFactors (its size equals scatteredDims.size()). It then pushes a bogus factor sub-dimension, corrupts the transpose permutation via factorDimIndices, and computes `inputShape[i] % factor` on garbage (SIGFPE if 0); an assertions build aborts on the SmallVector bounds check. Only reachable with combine-multi-dimension-reduce-scatter and a reduce-scatter that scatters >= 2 dims while leaving >= 1 unscattered; the existing multi-dim test scatters all dims, so this path was untested. Add the missing `continue`, matching the sibling all-to-all routine.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
-sdy-convert-global-to-local='combine-multi-dimension-reduce-scatter=true'reads past the end of aSmallVector(and can divide by a garbage factor) when lowering areduce_scatterthat scatters two or more dimensions but leaves at least one dimension unscattered.Root cause
In
ReduceScatterOpPattern::rewriteReduceScatterCombiningMultipleDims(convert_global_to_local.cc), the loop that splits each scattered dimension into(factor, quotient)handles a non-scattered dimension but forgets tocontinue:When dimension
iis not scattered,it == scatteredDims.end(), sostd::distance(begin, end) == scatteredDims.size(), andscatteredFactors[scatteredDims.size()]is one past the end (scatteredFactors.size() == scatteredDims.size()). Execution then pushes a bogus factor sub-dimension intosplitShape, records it infactorDimIndices(corrupting the subsequent transpose permutation), and computesinputShape[i] % factoron the garbage value (a SIGFPE if it happens to be 0). Under an assertions-enabled build (standard for MLIR) theSmallVector::operator[]bounds assert aborts.The sibling all-to-all combining routine has the
continue; this is an omission in the reduce-scatter adaptation. The only existing multi-dimension test happens to scatter all dimensions, so the unscattered-dimension path is untested.Fix
Add the missing
continueso a non-scattered dimension keeps its original size and does not emit a factor sub-dimension. This is a no-op for the all-dimensions-scattered case.Test
sdy_reduce_scatter_partial_dims.mlir: a rank-3reduce_scatterthat scatters dimensions 0 and 1 but leaves dimension 2 unscattered now lowers to a singlestablehlo.reduce_scatterinstead of reading out of bounds.Disclosure: this contribution was authored with an AI coding assistant (Claude) and reviewed before submission.