Skip to content

TrainerRank reusable-cache admission can leave insufficient physical CUDA memory for backward library allocations #870

Description

@bradhilton

Type: confirmed backward allocation failure with a controlled experiment-local mitigation; generic policy remains unresolved.
Agent owner: Schulman, related to #848.

A rank-1 Qwen3.6-35B-A3B eight-pair dynamics batch on one H200 passed its forwards, then failed in micro_batch_loss.backward with CUBLAS_STATUS_ALLOC_FAILED from cublasCreate(handle). GeneratorExit was secondary iterator teardown. The exact internal cuBLAS request size was not measured.

At the first gradient admission, live allocated/reserved memory was about 74.662/147.394 decimal GB with only 0.034734 GB physically free. The planner treated reusable allocator slack as available; its 3% capacity subtraction did not physically reserve CUDA memory for external library allocations. At the caller boundary, an unconditional torch.cuda.empty_cache() released 42.754638 GB without changing live allocation. The matched release-only replay completed all five backward waves and the optimizer; the observe-only replay failed. Pre-intervention plans/budgets and corresponding forward counters matched exactly.

The qualified mitigation is experiment-local cache release at the start of every gradient caller, after receiving its microbatch. It is not equivalent to a conditional 3% policy, a pre-yield trainer hook, or handle warmup. Later nonzero-generator-gradient updates also succeeded, but larger traces and full validation remain unqualified. Cold forward estimate underprediction is a separate problem.

Investigate a bounded generic physical-headroom/library-allocation contract with mixed-grad plans, device/stream behavior and normal downstream workloads. Prefer trainer_rank-local work; no art.megatron change is proposed. Do not silently globalize the workaround based on one fixture.

Evidence: /home/brad/.local/share/schulman/retail49-memory-failure-20260909/MORNING-ALLOCATOR.md and replay-{observe,release}-comparison/. Local Caladan mitigation 431e424 and timing-only follow-up 177a9fce. Raw failures and cleanup receipts are preserved.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions