Add a linux-arm64-cuda prebuilt (NVIDIA Grace-class: DGX Spark GB10, GH200, Jetson Thor)

[hassanhabib]

Thanks for node-llama-cpp — it's the backbone of our cross-platform host.

On an NVIDIA DGX Spark, CUDA is fully installed and the driver is loaded, but the linux-arm64 prebuilt has no CUDA support compiled in, so inference falls back to CPU on the Grace cores and the Blackwell GPU sits idle.

Environment

OS:     Ubuntu 24.04.5 LTS (arm64), DGX OS, kernel 7.0.0-1019-nvidia
Node:   22.23.2 (arm64)
node-llama-cpp: 3.21.1 (prebuilt binaries v0.4.0)
GPU:    NVIDIA GB10, compute capability 12.1 (sm_121)
Driver: 580.178.04, CUDA 13.0
CUDA target: /usr/local/cuda/targets/sbsa-linux  (standard arm64 server CUDA, not Tegra)
RAM:    121 GB unified LPDDR5X (CPU+GPU coherent)

npx node-llama-cpp inspect gpu

OS: Ubuntu 24.04.5 LTS (arm64)
Node: 22.23.2 (arm64)

node-llama-cpp: 3.21.1
Prebuilt binaries: v0.4.0

CUDA: CUDA is detected, but using it failed
Vulkan: Vulkan is detected, but using it failed

CPU model: unknown
Math cores: 20
Used RAM: 6.32% (7.7GB/121.69GB)
Free RAM: 93.67% (113.99GB/121.69GB)

What's published

linux-x64, linux-x64-cuda, linux-x64-vulkan, linux-arm64, linux-armv7l, linux-riscv64, mac-arm64-metal, mac-x64, win-x64, win-x64-cuda, win-x64-vulkan, win-arm64.

There's a CUDA variant for x64 but not for arm64, so @node-llama-cpp/linux-arm64 installs cleanly and then runs CPU-only on hardware bought specifically for GPU inference.

Why this cell may be worth adding now

arm64 + CUDA on Linux used to be datacenter-only, but DGX Spark puts a Grace-Blackwell part on desks at a consumer-ish price, and GH200 / GB200 / Jetson Thor share the same shape. GitHub now offers free hosted ubuntu-24.04-arm runners, and the CUDA sbsa toolkit installs on them normally — so the matrix cell looks buildable in CI without self-hosted hardware.

One detail if you do add it

On GB10, nvidia-smi --query-gpu=memory.total,memory.free,utilization.gpu --format=csv,noheader,nounits returns literally:

[N/A], [N/A], 0

and nvidia-smi shows Memory-Usage: Not Supported, because CPU and GPU share one coherent pool rather than the GPU having its own. Anything parsing VRAM gets NaN (or 0 via systeminformation) on these parts. That reading means "unified memory", not "probe failed" — we hit exactly this in our own memory accounting and had to special-case it.

Happy to test any build on real GB10 hardware, and to share what we learn if we end up doing a source build with CMAKE_CUDA_ARCHITECTURES=121 in the meantime.

[hassanhabib]

Following up with what actually worked, in case it saves someone else the week, and to correct the last line of my own issue above.

I suggested doing the source build with CMAKE_CUDA_ARCHITECTURES=121. Don't. That route produces a perfectly good binary that the application then ignores without a word.

The trap

Custom cmake options are part of a build's identity, so passing the architecture as NODE_LLAMA_CPP_CMAKE_OPTION_CMAKE_CUDA_ARCHITECTURES=121 lands the output in:

node_modules/node-llama-cpp/llama/localBuilds/linux-arm64-cuda-<hash>/

A plain getLlama() asks for a build with no custom options, computes a different hash, finds nothing that matches, and falls back to the CPU prebuilt. The build is green, inspect gpu still reports the same CUDA line, and the GPU stays idle. There is no error anywhere in the chain, because every component did exactly what it was told.

CUDAARCHS is a standard CMake environment variable, so it reaches nvcc without joining the identity:

# On DGX OS the toolkit ships but is not on PATH, hence the explicit CUDACXX.
CUDA_PATH=/usr/local/cuda \
CUDACXX=/usr/local/cuda/bin/nvcc \
CUDAARCHS=121 \
  npx --no node-llama-cpp source download --gpu cuda

Verify by directory name, not by exit code

This is the part worth automating, because the failure mode is silence:

ls node_modules/node-llama-cpp/llama/localBuilds
# linux-arm64-cuda          <- discoverable, getLlama() resolves to cuda/localBuild
# linux-arm64-cuda-a1b2c3   <- built fine, silently ignored

We now fail our CI build when the output lands in a hash-suffixed directory. A compiled-but-ignored binary is worse than a failed compile, since it ships.

Two notes for Electron consumers

Source builds are disabled by default in a packaged app, so the build has to happen before packaging and the result has to survive asar. On the CLI side we do it in postinstall against the host's own CUDA, which also means we distribute no NVIDIA binaries.

The build tree is about 457 MB, nearly all of it llama.cpp sources, CMake state and standalone tools. Keeping only Release/ and buildDone.status takes it to roughly 52 MB, and the pruned tree still resolves to cuda/localBuild and still infers. Verified on GB10.

Verified on

DGX Spark, NVIDIA GB10, compute capability 12.1
Ubuntu 24.04.5 LTS (aarch64), DGX OS
CUDA 13.0.88, cmake 3.28.3, g++ 13.3.0, Node 22.23.2
node-llama-cpp 3.21.x, built from source

Loading a 82.75 GB gpt-oss-120b takes about 46 seconds and serves fine, and a smaller dense model roughly doubles its output rate against CPU-only on the same box. The unified-memory reporting quirk in the original post is unchanged and still worth handling on these parts.

None of this replaces the prebuilt. A source build asks every host for a toolchain and several minutes, which is a lot to ask of someone who just wants to install a package. The request in the issue stands, and I am happy to test any linux-arm64-cuda build on real GB10 hardware whenever it is useful.

[giladgd]

I'd love to add prebuilt CUDA binaries for arm64, but I don't have a machine to test it on to validate that I actually got things right, and for running tests when things change to ensure everything still works as it should.

After you build the CUDA binaries on arm64, what does npx node-llama-cpp inspect gpu show? Does it detect the unified memory correctly?

[hassanhabib]

I'd love to add prebuilt CUDA binaries for arm64, but I don't have a machine to test it on to validate that I actually got things right, and for running tests when things change to ensure everything still works as it should.

After you build the CUDA binaries on arm64, what does npx node-llama-cpp inspect gpu show? Does it detect the unified memory correctly?

@giladgd It detects unified memory correctly, with no changes on your side. Fresh run on the GB10 just now, node-llama-cpp 3.21.1 built from source with CUDAARCHS=121:

OS: Ubuntu 24.04.5 LTS (arm64)
Node: 22.23.2 (arm64)

node-llama-cpp: 3.21.1
Prebuilt binaries: v0.4.0
Cloned source: v0.4.0

CUDA: available
Vulkan: Vulkan is detected, but using it failed

CUDA device: NVIDIA GB10
CUDA used VRAM: 28.7% (34.93GB/121.69GB)
CUDA free VRAM: 71.29% (86.76GB/121.69GB)
CUDA unified memory: 121.69GB (100%)

CPU model: unknown
Math cores: 20
Used RAM: 44.98% (54.75GB/121.69GB)
Free RAM: 55.01% (66.94GB/121.69GB)
Wired RAM: 0.03% (37.89MB/121.69GB)
Used swap: 21.2% (3.39GB/16GB)
Max swap size: 16GB
mmap: supported

Before the CUDA build that first line read CUDA: CUDA is detected, but using it failed, which is the whole difference.

Against the API, unifiedSize is populated and the documented unifiedSize > 0 check works:

gpu         : cuda | buildType: localBuild
device      : NVIDIA GB10
vram.total  : 121.69 GB
vram.used   : 34.93 GB
vram.free   : 86.75 GB
unifiedSize : 121.69 GB   unified detected: true
ram.total   : 121.69 GB

vram.total, ram.total and unifiedSize all agree at 121.69 GB, which is right for a coherent pool.

Worth knowing that you get this right where the obvious probe does not. On the same box:

$ nvidia-smi --query-gpu=name,compute_cap,memory.total,memory.free,utilization.gpu --format=csv,noheader
NVIDIA GB10, 12.1, [N/A], [N/A], 0 %

and nvidia-smi prints Memory-Usage: Not Supported. Anything parsing that gets NaN or zero. Your CUDA path reports the real pool.

Kernels compute correctly too, not just load. GLM 4.7 Flash Q8_0, 31.8 GB, all layers offloaded:

buildType   : localBuild
loadSeconds : 17.3
prompt      : "In one sentence: what is 17 times 23?"
answer      : "17 times 23 is 391."

Two things on your actual blocker, since they need different answers:

0/ Validating you got it right. I have the hardware and I am glad to run anything you want against a branch or a candidate binary: inspect gpu, the test suite, a specific model, whatever is useful. Give me a branch name and I will run it and paste the output.

1/ Catching regressions when things change. GitHub's free ubuntu-24.04-arm runners build this fine, the CUDA sbsa toolkit installs normally, and we build our own arm64 CUDA binary that way on every release. What they cannot do is execute anything, since there is no NVIDIA GPU on them. So hosted CI can protect you from the build breaking, but only real hardware can tell you the kernels still work. If a self-hosted runner on a GB10 would be useful to you, say so and I will look into what that would take.

One unrelated observation while I was in there: Vulkan still reports Vulkan is detected, but using it failed on this machine. I have not looked into it and it does not matter to me, but you may want it as a separate issue rather than buried here.