Type: confirmed backward allocation failure with a controlled experiment-local mitigation; generic policy remains unresolved.
Agent owner: Schulman, related to #848.
A rank-1 Qwen3.6-35B-A3B eight-pair dynamics batch on one H200 passed its forwards, then failed in micro_batch_loss.backward with CUBLAS_STATUS_ALLOC_FAILED from cublasCreate(handle). GeneratorExit was secondary iterator teardown. The exact internal cuBLAS request size was not measured.
At the first gradient admission, live allocated/reserved memory was about 74.662/147.394 decimal GB with only 0.034734 GB physically free. The planner treated reusable allocator slack as available; its 3% capacity subtraction did not physically reserve CUDA memory for external library allocations. At the caller boundary, an unconditional torch.cuda.empty_cache() released 42.754638 GB without changing live allocation. The matched release-only replay completed all five backward waves and the optimizer; the observe-only replay failed. Pre-intervention plans/budgets and corresponding forward counters matched exactly.
The qualified mitigation is experiment-local cache release at the start of every gradient caller, after receiving its microbatch. It is not equivalent to a conditional 3% policy, a pre-yield trainer hook, or handle warmup. Later nonzero-generator-gradient updates also succeeded, but larger traces and full validation remain unqualified. Cold forward estimate underprediction is a separate problem.
Investigate a bounded generic physical-headroom/library-allocation contract with mixed-grad plans, device/stream behavior and normal downstream workloads. Prefer trainer_rank-local work; no art.megatron change is proposed. Do not silently globalize the workaround based on one fixture.
Evidence: /home/brad/.local/share/schulman/retail49-memory-failure-20260909/MORNING-ALLOCATOR.md and replay-{observe,release}-comparison/. Local Caladan mitigation 431e424 and timing-only follow-up 177a9fce. Raw failures and cleanup receipts are preserved.
Type: confirmed backward allocation failure with a controlled experiment-local mitigation; generic policy remains unresolved.
Agent owner: Schulman, related to #848.
A rank-1 Qwen3.6-35B-A3B eight-pair dynamics batch on one H200 passed its forwards, then failed in micro_batch_loss.backward with CUBLAS_STATUS_ALLOC_FAILED from cublasCreate(handle). GeneratorExit was secondary iterator teardown. The exact internal cuBLAS request size was not measured.
At the first gradient admission, live allocated/reserved memory was about 74.662/147.394 decimal GB with only 0.034734 GB physically free. The planner treated reusable allocator slack as available; its 3% capacity subtraction did not physically reserve CUDA memory for external library allocations. At the caller boundary, an unconditional torch.cuda.empty_cache() released 42.754638 GB without changing live allocation. The matched release-only replay completed all five backward waves and the optimizer; the observe-only replay failed. Pre-intervention plans/budgets and corresponding forward counters matched exactly.
The qualified mitigation is experiment-local cache release at the start of every gradient caller, after receiving its microbatch. It is not equivalent to a conditional 3% policy, a pre-yield trainer hook, or handle warmup. Later nonzero-generator-gradient updates also succeeded, but larger traces and full validation remain unqualified. Cold forward estimate underprediction is a separate problem.
Investigate a bounded generic physical-headroom/library-allocation contract with mixed-grad plans, device/stream behavior and normal downstream workloads. Prefer trainer_rank-local work; no art.megatron change is proposed. Do not silently globalize the workaround based on one fixture.
Evidence: /home/brad/.local/share/schulman/retail49-memory-failure-20260909/MORNING-ALLOCATOR.md and replay-{observe,release}-comparison/. Local Caladan mitigation 431e424 and timing-only follow-up 177a9fce. Raw failures and cleanup receipts are preserved.