Repository navigation
Stabilize medium CUDA BF16 attention and prefix gradients - #54
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Use a bounded tiled CUDA BF16 attention path for 257–8191 tokens, preserve the state boundary in independent full-row execution, and accumulate RMSNorm in double under CUDA BF16 so the same prefix does not change with outer row count. Medium K/V gradients retain the original double cache dtype. The derivative version advances to
cuda-bf16-canonical-v3, so incompatible BF16 resumes fail closed.Evidence
Pinned actual-weight generated requests on RTX 3060 at the unchanged 70% process cap:
The portable receipt gives source, request, checkpoint and private receipt hashes. This repairs a medium BF16 numerical gap for #3. Full long-context parity, calibrated serving and representative quality remain open; #3 and #4 should not close from this PR.