Skip to content

Stabilize medium CUDA BF16 attention and prefix gradients - #54

Merged
CompleteDotTech merged 2 commits into
mainfrom
feat/medium-bf16-canonical
Sep 29, 2026
Merged

CompleteDotTech merged 2 commits into
mainfrom
feat/medium-bf16-canonical

Conversation

@CompleteDotTech

Copy link
Copy Markdown
Collaborator

Scope

Use a bounded tiled CUDA BF16 attention path for 257–8191 tokens, preserve the state boundary in independent full-row execution, and accumulate RMSNorm in double under CUDA BF16 so the same prefix does not change with outer row count. Medium K/V gradients retain the original double cache dtype. The derivative version advances to cuda-bf16-canonical-v3, so incompatible BF16 resumes fail closed.

Evidence

Pinned actual-weight generated requests on RTX 3060 at the unchanged 70% process cap:

  • 13-token state, branch totals 97/583/113/977: batched, serial and full logits exact; all 100 gradients pass 2e-5, worst tolerance ratio 0.007795.
  • 613-token state, branch totals 697/1183/713/1577: exact logits; all 100 gradients pass, worst ratio 0.009824.
  • One KV group's 8,191-token tiled forward/backward completed with finite gradients and 472,698,368 peak allocated bytes. This is a kernel resource check, not a full-model parity claim.
  • Pinned 62-token FP32 Transformers hidden oracle passed. Five focused native CUDA tests passed, including a direct double-cache-gradient test; CPU suite 858 passed/31 skipped; compileall, critical Ruff, and staged source audit passed.

The portable receipt gives source, request, checkpoint and private receipt hashes. This repairs a medium BF16 numerical gap for #3. Full long-context parity, calibrated serving and representative quality remain open; #3 and #4 should not close from this PR.

@CompleteDotTech
CompleteDotTech marked this pull request as ready for review September 29, 2026 14:27
@CompleteDotTech
CompleteDotTech merged commit 9504b6e into main Sep 29, 2026
5 checks passed
@CompleteDotTech
CompleteDotTech deleted the feat/medium-bf16-canonical branch September 29, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant