Skip to content

Fix stale free state handling in custom allocator for CUDA and HIP - #1377

Open
bwlang wants to merge 1 commit into
3dem:masterfrom
nebiolabs:fix/gpu-allocator-stale-free-state
Open

bwlang wants to merge 1 commit into
3dem:masterfrom
nebiolabs:fix/gpu-allocator-stale-free-state

Conversation

@bwlang

@bwlang bwlang commented Oct 7, 2026

Copy link
Copy Markdown

without this fix i was seeing GPU runs crashing with errors like these:

  • "illegal memory access" errors inside RELION's GPU allocator (custom_allocator.cuh)
  • segfaults in CudaCustomAllocator::_free
  • an out-of-memory report asking for 564 TB
  • "No orientation was found as better than any other"
  • NVIDIA Xid 31 and Xid 13 ("Out Of Range Address") in the kernel log, on two different GPU

with this fix it's working.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant