Cuda 4 device kernels - #88
Draft
max-models wants to merge 4 commits into
Draft
max-models wants to merge 4 commits into
max-models wants to merge 4 commits into
Conversation
Make feectools work when cunumpy's backend is CuPy, without any device kernels yet: kernels that stay on the host still copy their arrays. - Wrap all Pyccel kernels (stencil, B-splines, field evaluation, DOF kernels) in cunumpy.PyccelKernel, so they accept CuPy arrays. - Keep host-only metadata on NumPy: MPI/index bookkeeping in ddm (cart, partition, petsc) and fem.partitioning, Kronecker solver sizes, and index arithmetic with Python ints (compute_diag_len, math.prod). - Stage data for host-only libraries by array, not by global backend: LAPACK/SuperLU direct solvers, SciPy FFT, SciPy sparse products. - Fix calls that ran host kernels on device arrays: the second stencil2coo call in StencilMatrix.tosparse and the conjugate transpose. - Vectorize the construction of the 1D collocation matrices in the global projectors (element-wise indexing was a device round trip per entry: 334 s of a 348 s Derham setup on the GPU). - GMRES: take real scalars from CuPy views before modifying them. - Fix StencilMatrix._update_ghost_regions_serial: the ghost region is pads * shifts wide (wrong whenever shifts > 1, on both backends). - Tests: work with CuPy arrays; skip PETSc tests without petsc4py. Serial tests pass on both backends (core, ddm, fem, linalg). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Allow MPI on the CuPy backend and make it correct with device buffers (requires a CUDA-aware MPI library). - feectools.ddm.mpi no longer disables MPI when ARRAY_BACKEND=cupy; the segfaults it guarded against come from MPI libraries that are not CUDA-aware. - Call cunumpy.synchronize_for_mpi before every MPI call on device buffers: CuPy kernels run asynchronously and MPI does not know about CUDA streams, so a buffer still being written would be sent silently wrong. Covers the blocking, non-blocking and interface data exchangers, the Allreduce in StencilVectorSpace.inner and the Alltoallv calls of the parallel Kronecker solver. Requires cunumpy >= 0.3.0. - Fix CuPy incompatibilities reached only by the MPI tests: xp.dot/vdot on .flat iterators in StencilInterfaceMatrix._dot and the pure-Python inner product, and test_cart_1d assigning Python lists to CuPy arrays. - Add test_mpi_device.py: distributed results against global references, and that the exchangers synchronize before MPI. With 2 MPI ranks and a CUDA-aware Open MPI, all MPI tests in ddm and linalg pass on both backends; serial tests are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the initialization that always used GPU 0 by cunumpy.bind_local_device(): each process uses GPU local_rank % device_count, chosen from the node-local rank that the MPI launcher exports, and its CUDA context is created before MPI is initialized (as CUDA-aware MPI requires). No-op on the NumPy backend. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run the main stencil operations on the GPU when their data is there, instead of copying it to the host for the Pyccel kernels: - StencilMatrix.dot: CUDA matvec (feectools.linalg.kernels.device_matvec), one thread per output point, generated per dimension (1-3) and dtype (float64, complex128) and cached with cunumpy.CudaKernelVariants. - StencilMatrix.transpose: CUDA transpose for 3D real matrices (device_transpose, a cunumpy.CudaKernel). - StencilVectorSpace.inner: reduction on the device (returns a NumPy scalar in the serial case, as the host kernel does). - StencilVectorSpace.axpy: scaled add on the device. Other cases (other dtypes, dimensions, backends) keep the host path. Adds test_device_matvec.py. A 64x64x32 matvec with degree 3 takes 0.5 ms instead of 165 ms on an H100. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 1, 2026
Merged
Draft
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The main stencil operations now run on the GPU when their data is already there. Before, the data was copied to the host for the Pyccel kernels. A 64×64×32 matvec with degree 3 takes 0.5 ms instead of 165 ms on an H100.
Changes
StencilMatrix.dot: a CUDA matvec infeectools/linalg/kernels/device_matvec.py, one thread per output point. It is generated for each dimension (1–3) and dtype (float64,complex128) and cached withcunumpy.CudaKernelVariants. It is used only when the matrix, the input vector and the output vector are all on the device and the matrix uses the precompiled kernel arguments.StencilMatrix.transpose: a CUDA transpose for 3D real matrices inkernels/device_transpose.py, usingcunumpy.CudaKernel.StencilVectorSpace.inner: the reduction runs on the device. In the serial case it still returns a NumPy scalar, as the host kernel does.StencilVectorSpace.axpy: a scaled add on the device, including the interface data.linalg/tests/test_device_matvec.py.Notes for review
_device_matvec_args()caches its result on the matrix, andset_backend()doesn't clear that cache. If a matrix switches to another backend after its firstdot, it keeps using the device kernel. The result is still correct, but the backend choice is ignored.Stack
This is step 4 of 4 for CUDA support. The PRs are stacked, and each one targets
devel-tiny, so the diff here also contains the earlier steps. Review only commit1253695in this PR, and merge them in order:🤖 Generated with Claude Code