Cuda 3 device binding - #87
Draft
max-models wants to merge 3 commits into
Draft
max-models wants to merge 3 commits into
max-models wants to merge 3 commits into
Conversation
Make feectools work when cunumpy's backend is CuPy, without any device kernels yet: kernels that stay on the host still copy their arrays. - Wrap all Pyccel kernels (stencil, B-splines, field evaluation, DOF kernels) in cunumpy.PyccelKernel, so they accept CuPy arrays. - Keep host-only metadata on NumPy: MPI/index bookkeeping in ddm (cart, partition, petsc) and fem.partitioning, Kronecker solver sizes, and index arithmetic with Python ints (compute_diag_len, math.prod). - Stage data for host-only libraries by array, not by global backend: LAPACK/SuperLU direct solvers, SciPy FFT, SciPy sparse products. - Fix calls that ran host kernels on device arrays: the second stencil2coo call in StencilMatrix.tosparse and the conjugate transpose. - Vectorize the construction of the 1D collocation matrices in the global projectors (element-wise indexing was a device round trip per entry: 334 s of a 348 s Derham setup on the GPU). - GMRES: take real scalars from CuPy views before modifying them. - Fix StencilMatrix._update_ghost_regions_serial: the ghost region is pads * shifts wide (wrong whenever shifts > 1, on both backends). - Tests: work with CuPy arrays; skip PETSc tests without petsc4py. Serial tests pass on both backends (core, ddm, fem, linalg). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Allow MPI on the CuPy backend and make it correct with device buffers (requires a CUDA-aware MPI library). - feectools.ddm.mpi no longer disables MPI when ARRAY_BACKEND=cupy; the segfaults it guarded against come from MPI libraries that are not CUDA-aware. - Call cunumpy.synchronize_for_mpi before every MPI call on device buffers: CuPy kernels run asynchronously and MPI does not know about CUDA streams, so a buffer still being written would be sent silently wrong. Covers the blocking, non-blocking and interface data exchangers, the Allreduce in StencilVectorSpace.inner and the Alltoallv calls of the parallel Kronecker solver. Requires cunumpy >= 0.3.0. - Fix CuPy incompatibilities reached only by the MPI tests: xp.dot/vdot on .flat iterators in StencilInterfaceMatrix._dot and the pure-Python inner product, and test_cart_1d assigning Python lists to CuPy arrays. - Add test_mpi_device.py: distributed results against global references, and that the exchangers synchronize before MPI. With 2 MPI ranks and a CUDA-aware Open MPI, all MPI tests in ddm and linalg pass on both backends; serial tests are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the initialization that always used GPU 0 by cunumpy.bind_local_device(): each process uses GPU local_rank % device_count, chosen from the node-local rank that the MPI launcher exports, and its CUDA context is created before MPI is initialized (as CUDA-aware MPI requires). No-op on the NumPy backend. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 1, 2026
Merged
Draft
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Each MPI rank now uses its own GPU instead of always GPU 0.
Changes
feectools.ddm.cart: the CUDA setup that always picked GPU 0 is replaced bycunumpy.bind_local_device(). Each process uses GPUlocal_rank % device_count, wherelocal_rankis the rank within the node that the MPI launcher exports.feectools.ddmrunsddm/__init__.pyfirst, which importscartfirst, so the GPU is chosen beforefeectools.ddm.mpistarts MPI, whatever the user imports first.ddm/tests/test_device_binding.py: checks that the current device islocal_rank % device_countaftercartis imported. It is skipped without CuPy or a GPU.Stack
This is step 3 of 4 for CUDA support. The PRs are stacked, and each one targets
devel-tiny, so the diff here also contains the earlier steps. Review only commit3a11ad1in this PR, and merge them in order:🤖 Generated with Claude Code