Skip to content

Avoid materializing unused Megatron forward outputs - #2403

Draft
kalectory wants to merge 1 commit into
NovaSky-AI:mainfrom
kalectory:neil/megatron-collection-outputs
Draft

kalectory wants to merge 1 commit into
NovaSky-AI:mainfrom
kalectory:neil/megatron-collection-outputs

Conversation

@kalectory

Copy link
Copy Markdown
Contributor

Megatron policy and reference forward passes return token arrays only from collection ranks instead of materializing outputs that dispatch immediately discards.

WorkerDispatch.forward retrieves every rank's WorkerOutput, then WorkerOutput.cat selects only collection ranks. With context parallelism and long sequences, the unused Python float lists still incur serialization, transfer, deserialization, and garbage collection. Runtime stack samples showed the head deserializing these results with active Python GC after worker GPU computation ended.

The five-line change uses the existing is_collection_dp_rank() predicate after _forward_logprobs finishes. Every rank still executes the same distributed forward; dispatch still waits for every result and propagates worker exceptions. Collection-rank values are unchanged. No retries, timeouts, or numerical computation changes.

Validation: formatting and secret hooks passed. In a separate process with the deployed Megatron dependencies, the old policy method failed the non-collection materialization check and all eight candidate cases passed (policy/reference × collection/CP/TP/PP rank). The check executed the candidate methods and committed test function directly with mocked forward results; it did not initialize Ray or run GPU collectives. Full pytest/GPU validation and an end-to-end performance measurement remain pending. Not deployed to the active run.

Signed-off-by: Neil Kale <263453039+kalectory@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant