ci: give the owned-CUDA worker a Windows build gate that can reach nvcc

[Qiiks]

Problem

synapse-worker-cuda — the owned-CUDA, VRAM-resident embedding worker — has never had a Windows build gate, and its build script cannot reach nvcc on MSVC.

crates/synapse-engine-cuda/build.rs hard-wired the compiler path as bin/nvcc. On Windows the binary is nvcc.exe; cc::Build reports the missing path as a failed tool rather than falling back, so every --features cuda build on Windows dies before compiling a single kernel. Two follow-on defects sit behind that one:

• -Xcompiler=-fPIC is forwarded unconditionally. Position-independent host code is an ELF concern; MSVC's cl rejects the unknown option and nvcc surfaces it as a build failure.
• Link search covers only lib/x64. CUDA 13's redist archives place cublas.lib beside its DLL in bin/x64 (the 12.x layout keeps it under lib/x64), so a 13.x toolkit does not resolve -lcublas.

Neither crate is in SYNAPSE_CRATES, so nothing in CI compiles or lints them on any platform. That gap is not theoretical: synapse-worker-cuda fails cargo clippy --all-targets -- -D warnings on Windows today, because the Unix handshake helpers (read_json_frame, validate_ack, WorkerHelloAck) are dead code in the cfg(windows) build. It has never been caught because it has never been gated.

Change

crates/synapse-engine-cuda/build.rs — resolve nvcc.exe on Windows (CUDACXX still wins when set); gate -fPIC to non-Windows hosts; search bin/x64 and bin in addition to lib/x64 on Windows. The PTX distribution contract is unchanged: virtual arch compute_75 only, no sm_* SASS image. (Turing/7.5 remains CUDA 13's lowest supported arch — 13.0 removed pre-7.5 offline compilation, not 7.5 itself.)

crates/synapse-worker-cuda/src/main.rs — cfg(unix) on the Unix-only handshake imports and helper; bail! fully qualified so the import is no longer Windows-dead. Behaviour is unchanged on every platform; the change is cfg-correctness only.

.github/workflows/tests.yml — add windows-owned-cuda-manual, mirroring the structure of linux-llama-cuda-manual and windows-llama-vulkan-manual, and widen SYNAPSE_CRATES with the two owned-CUDA crates so the fmt/clippy gates start covering them.

Toolkit assembly

Hosted Windows runners have no nvcc, and the repository's existing Linux lane cannot use apt on Windows. The gate assembles a toolkit from NVIDIA's own per-component redist archives — pinned version + SHA-256, no installer, no driver, nothing written outside $RUNNER_TEMP:

component	version	size
cuda_nvcc	13.2.78	31.3 MB
cuda_crt	13.2.78	0.1 MB
libnvvm	13.2.78	55.2 MB
cuda_cccl	13.2.75	3.5 MB
cuda_cudart	13.2.75	3.1 MB
libcublas	13.4.0.1	388.2 MB

Six components are what the kernel sources actually need: cuda_family_common.cuh includes cublasLt.h, cuda_fp16.h, cuda_runtime.h, and crt/math_functions.h pulls the cccl/crt header chains; nvcc needs cicc/ptxas (nvcc + libnvvm) and crt headers; the link flags name cuda/cublasLt/cublas/cudart.

The build step enters the MSVC developer environment first (vswhere → Enter-VsDevShell): nvcc drives cl.exe by name, and it is not on PATH on a clean runner — reproduced locally as nvcc fatal : Cannot find compiler 'cl.exe' in PATH. CUDA 13's redist archives also keep the runtime DLLs in bin/x64 rather than bin, so both directories are exported.

CUDA 13, deliberately. 13.x is the line whose runtime sonames (cudart64_13.dll, cublas64_13.dll, cublasLt64_13.dll) llama.cpp's Windows CUDA builds already ship — so this worker drops into an existing CUDA-13 deployment instead of dragging a second, conflicting runtime beside it. The engine's declared floor (driver API ≥ 12040) is satisfied: a 13.x-built binary runs against any 580-or-newer driver.

Anti-hollow-green assertion

Building is not the same as building with the backend compiled in. --features cuda is the only thing that links cudart/cublas, and on Windows those are load-time imports: the exe cannot start when they are absent from PATH (STATUS_DLL_NOT_FOUND, 0xC0000135) even though its --version path never calls into them. The gate exploits that as a dependency-free discriminator:

• run off-PATH → must fail 0xC0000135 ⇒ the CUDA backend is genuinely baked in;
• run on-PATH → must exit 0 ⇒ the resolved DLL set is complete.

A worker that silently compiled without CUDA prints --version in both cases, and the step refuses that. Evidence files follow the existing nonmac-build-gates-v1 contract (execution_status, binary SHA-256, skipped_is_not_pass=true, manual_gate=mandatory); the produced exe ships as a CI artifact.

Verification

Windows x64, MSVC 2022 Build Tools, redist-assembled CUDA toolkit (same archive set and digests as the gate):

• cargo fmt --all --check → clean.
• cargo clippy -p synapse-engine-cuda -p synapse-worker-cuda --all-targets -- -D warnings → clean (fails before this change: 3 dead-code/unused-import errors).
• cargo test -p synapse-engine-cuda -p synapse-worker-cuda → 7 + 3 pass, 0 fail.
• Workflow YAML parses; job graph test, linux-llama-cuda-manual, windows-llama-vulkan-manual, windows-owned-cuda-manual.
• cuda_cudart 12.6.68 and 13.2.75 Windows archives downloaded and SHA-256-verified against NVIDIA's manifest; libcublas/libnvvm/cuda_crt 13.2.1 layouts confirmed from their central directories by ranged reads (contents quoted above: bin/x64/cublas64_13.dll, lib/x64/cublas.lib, nvvm/bin/cicc.exe, include/crt/math_functions.h).

Not verified locally — superseded: the --features cuda compile and runtime probe were since executed end-to-end on a local Windows box (RTX 4050, driver 610.88, MSVC BuildTools) with the exact recipe this gate automates. See the "Local end-to-end proof" comment on this PR for the build log, the off-PATH/on-PATH import-probe results, and a real Qwen3-Embedding-0.6B safetensors load + embed round trip. The first hosted dispatch still re-checks the redist pinning independently.

Steady-state from that run (per-shape wall timing; item counts reconstructed from the engine's own shape logs): owned-CUDA f16 at 1024 × ~24-token items = 646.6 items/s sustained, ~95% of the 683/s stage-timer GPU floor, vs llama.cpp CUDA Q8_0 (same GPU, HTTP endpoint) at 188 items/s at the matched 1024-item shape — ~3.4× faster while retaining f16. VRAM is stated as a delta over this box's ~2 GiB system baseline: +75 MiB persistent weights at load, +2,665 MiB at the 1024-batch steady state (two runs: +2,650 / +2,665). Cold load 26–41 s, dominated by the 1.19 GB sha256 verify plus BF16→F32 host conversion. Method, caveats (transport differs; the 5-round 256× window includes one-time graph capture), and raw logs are in the proof comment.

Deliberately not in this PR: release assets, installer inventory, config auto-wiring. The test lane's widened clippy set is the only push-triggered change; everything CUDA is workflow_dispatch-gated.

Why the release asset and ck setup synapse install are deliberately excluded

The tempting follow-up — add ck-synapse-worker-cuda to the windows-x64 release matrix and to Subc's component_binaries_for_target — would publish a worker that cannot run, and break installs for most Windows users. Three independent blockers, each verified in source at 5680bd7 / 0ed4dcb5:

1. The worker is not self-contained, and the archive contract has one binary. build.rs links cudart/cublas/cublasLt dynamically; only nvcuda.dll ships with the driver. Subc's ReleaseArtifactSource::install copies only the named candidate out of each zip, so packaged DLLs are discarded. Worse, Acceptance::RunsAndReports executes <destination> --version before writing configuration — and a PE with unresolved load-time imports fails before executing a single instruction. The install would refuse, aborting the whole ck setup synapse component on every Windows machine without the toolkit. Sidecar-file delivery needs an installer-contract change in subconscious, designed together with a per-target asset split (the shared non-Darwin match arm currently would also make Linux and Windows-Arm64 read release-incomplete).
2. Nothing resolves the worker binary by default. load_worker_backend_blocking requires the per-model preload worker_bin or SYNAPSE_OWNED_CUDA_WORKER_BIN; installing the exe beside ck-synapse.exe does not make the module use it. A default needs either a Subc config-writer emit (modules.synapse.env, precedent: claustrum's CK_MASTER_KEY_PATH) or a new sibling-of-exe convention — which sits on the same code path as the llama worker and must stay gated to owned-CUDA.
3. The hardware floor is env-only, with no producer. ensure_owned_cuda_floor() reads SYNAPSE_CUDA_DRIVER_API/CUDA_DRIVER_API and SYNAPSE_CUDA_COMPUTE_CAPABILITY/CUDA_COMPUTE_CAPABILITY and returns HardwareUnavailable when either is absent — refusing before the worker spawns. There is no probe, CI step, or installer that sets them on any platform. The engine already has the primitives to derive both (cuInit + device attributes, device_meets_floor); wiring a real probe into the module's load path is the honest fix, and it is its own change.

Landing the asset + inventory pair without those would be a hollow-green release: an artifact that passes --version acceptance on some machines and hard-fails the installer on the rest. This PR delivers the gate that makes the binary buildable and provably CUDA-backed; the distribution shape should be designed as a unit against the three facts above.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[cubic-dev-ai]

All reported issues were addressed across 3 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Qiiks]

Local end-to-end proof (upgrades the "not verified locally" caveat)

Self-correction note: an earlier version of this comment understated owned-CUDA throughput ~16× — the probe's item counter summed n per round while each round embeds n × 16 items. Everything below is from the fixed probe: per-shape wall timing, and item counts that agree with the engine's own shape logs.

The author machine turned out to have everything needed to run this lane by hand — RTX 4050 Laptop (driver 610.88, CC 8.9), MSVC 2022 BuildTools, and the CUDA 13 runtime DLLs already on disk from a llama.cpp deployment. So the exact recipe this gate automates was executed locally, and the full loop works:

1. Redist assembly — the six pinned archives (nvcc/crt/nvvm/cccl/cudart 13.2.x, libcublas 13.4.0.1) downloaded, SHA-256-verified against the pins in this PR, merged into one root the same way the gate does.

2. Build — cargo build -p synapse-worker-cuda --no-default-features --features cuda --release under vcvars64 + CUDA_PATH/CUDA_HOME pointed at the assembled root:

Compiling synapse-engine-cuda v0.0.0
Compiling synapse-worker-cuda v0.1.0-alpha.2
warning: linker stdout: LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs
Finished `release` profile [optimized] target(s) in 27.78s

This also confirmed, by reproduction, the two P1s fixed in 465072c: without the dev shell nvcc dies with Cannot find compiler 'cl.exe' in PATH, and the archive-flatten bug makes bin\nvcc.exe unresolvable.

3. Import probe — the same off-PATH/on-PATH discriminator the gate asserts, run against the locally built exe:

OFF-PATH  rc = -1073741515 (0xC0000135 STATUS_DLL_NOT_FOUND)   → CUDA backend baked in
SIDECAR   rc = 0  (DLLs co-located next to the exe)            → loader satisfied by sidecars
ON-PATH   rc = 0  stdout: ck-synapse-worker-cuda 0.1.0-alpha.2

The sidecar row is direct evidence for blocker #1 in the description: the worker runs from co-located cudart64_13.dll/cublas64_13.dll/cublasLt64_13.dll — which today's installer discards.

4. Real work, not just --version — a throwaway driver replicated the module side of the protocol (named-pipe server, accept_worker_handshake, LOAD with safetensors-package + digest, EMBED_BATCH over frames) against the built worker, loading the genuine Qwen/Qwen3-Embedding-0.6B BF16 safetensors (1,191,586,416 bytes, sha256 0437e45c…23fd) with a real HF tokenizer:

LOADED ref=owned-cuda:qwen3-0.6b:0 dims=1024 worker_cold_ms=25562
CUDA Qwen3 persistent weights: layers=28 dtype=f16 accum=fp32 norm_params=fp32
CUDA Qwen3 shape 16x24:   arena=20747264   captured_exact=true launches=534
CUDA Qwen3 shape 256x24:  arena=331879424  captured_exact=true launches=534
CUDA Qwen3 shape 1024x24: arena=1327502336 captured_exact=true launches=534
per-shape: 256x24 1280 items in 2.79s = 459.5 items/s | 1024x24 87040 items in 134.61s = 646.6 items/s
GPU memory at load: 1897 -> 1972 MiB (delta +75 MiB)
steady state: worker RSS 1494.8 MB, GPU 4562 MiB used (system-wide; +2665 MiB over baseline)

The model runs: Qwen3-0.6B f16, CUDA graphs captured per shape (captured_exact=true), PTX JIT'd onto consumer hardware from the compute_75 virtual arch, embeddings returned and frame-decoded (1024-dim, normalized).

Throughput against the llama.cpp CUDA lane (Q8_0, same GPU, --n-gpu-layers all, HTTP embedding endpoint), matched item counts per request (~19–24 tokens/item; both sides full client↔server round trip, both post-warmup):

batch shape	owned-CUDA f16 (pipe)	GPU-only floor (stage timers)	llama.cpp Q8_0 (HTTP)
256 × ~24 tok	459 items/s (5 rounds, includes the shape's one-time graph capture)	667/s	190 items/s
1024 × ~24 tok	646.6 items/s (85 rounds sustained)	683/s	188 items/s

The owned lane is ~3.4× faster at the large sustained shape while carrying f16 instead of Q8_0, and it runs ~95% GPU-bound (646.6 measured vs 683 GPU floor). At 256 the measured 459 vs 667 floor is the one-time capture inside a 5-round window, not steady state. Transports differ (named pipe vs HTTP); the per-item small-batch path (~16 items) is unmeasured.

Three observations for whoever picks up the distribution follow-ups:

• Cold load 26–41 s across three runs on a laptop 4050 — dominated by the sha256 verify of the 1.19 GB file plus the BF16→F32 host conversion before upload. The module's worker load timeout must accommodate this on first load.
• VRAM attribution: nvidia-smi memory.used is system-wide; this box runs at a ~1.9–2.2 GiB baseline (overlay/chrome/Discord). The worker's own delta is the honest figure: +75 MiB persistent at load, +2,665 MiB at steady state with the 1024×24 activation arena (1,327 MB) + CUDA context/graphs + host f32 staging.
• Per-core profile: during the sustained loop ck-synapse-worker-cuda.exe held ~92% of one core (driver/launch thread — the worker receives prepared ids, tokenization is the caller's) and ~1.5 GB RSS.

Fixes from the review round are in 465072c (vcvars dev shell, archive flatten, bin\x64 DLL path — the third found by this local run, not visible in the diff alone).

[synapse-alfonso]

Reviewed, and I verified each claim against master rather than taking the description on trust. They all hold: build.rs:35 hard-wires bin/nvcc, line 40 forwards -Xcompiler=-fPIC unconditionally, line 56 searches only lib/x64, neither CUDA crate appears in SYNAPSE_CRATES, and validate_ack is defined ungated while used only inside the cfg(unix) arm — so it is genuinely dead on Windows. Thank you for the write-up; the reasoning about why the release asset and installer wiring are excluded is the right call, and the three blockers you list match what I see in the source.

CI is red on one thing, and it is mine rather than yours:

train precondition failed: path-dependent if at job 'windows-owned-cuda-manual':
github.event_name == 'workflow_dispatch'

scripts/check-train-preconditions.sh keeps an explicit allow-list of jobs that are deliberately dispatch-only. The gate exists because a job that skips on push still reports a green check, so a new one has to be registered on purpose rather than appearing silently. Your job is correct; it is simply not registered yet. Adding it to ALLOWED_JOB_CONDITIONS alongside the two existing gates clears it:

    "windows-owned-cuda-manual": (
        "github.event_name == 'workflow_dispatch'",
        "The owned-CUDA Windows build is an explicit manual gate, not a push-triggered train gate.",
    ),

Worth knowing about the run: Windows passed, including the widened clippy set — that is the real confirmation of your cfg(unix) fix, since those lints had never executed anywhere. Linux is still unverified: the precondition step runs before clippy, so it aborted before reaching the lint. That is the one open question on my side, and I cannot answer it locally (cross-compiling to Linux from macOS dies in ring's build script). Pushing the allow-list entry will let CI answer it.

Two smaller notes, neither blocking:

• The off-PATH 0xC0000135 discriminator is a good way to prove the backend is linked without needing a GPU. One caveat worth a comment in the step: it depends on cudart/cublas staying load-time imports. If a future change moves them to delay-load, the probe would exit 0 and the step would read that as "compiled CPU-only" — a false failure rather than a false pass, so it fails safe, but the next person will want to know why.
• 480 MB of redist per dispatch is fine for a manual gate. If this lane ever becomes frequent, libcublas at 388 MB is the obvious thing to cache.

I will re-run CI once the allow-list entry is in.

[Qiiks]

Allow-list entry is in (eb79421), plus the delay-load caveat comment on the probe step, worded as you suggested. The precondition script passes locally against the current tests.yml with the new job registered (rc=0).

Re the caching note — agreed, left as-is for a manual lane; the entry point when it matters is obvious.

Two corrections to my previous comment, since I'd posted it before finishing the measurement work:

1. The throughput comparison was wrong in both directions of the error. The probe's item counter summed n per round while each round embeds n × 16 items (the engine's 256x24/1024x24 shape logs give the true counts) — so the old "28.9 items/s" understated reality 16×, and the resulting "llama.cpp is faster" conclusion flips. With per-shape timing added to the probe and llama.cpp re-measured at matched batch shapes on the same GPU: owned-CUDA f16 sustains 646.6 items/s at 1024×24-token items (~95% of the 683/s stage-timer GPU floor; 459.5 items/s at the 256 shape, which includes that shape's one-time graph capture inside a 5-round window) vs llama.cpp Q8_0's 188 items/s. The proof comment now carries the corrected table and caveats.
2. VRAM attribution: nvidia-smi memory.used is system-wide and this box runs at a ~2 GiB baseline (overlay/chrome/Discord). The worker's honest footprint is the delta — +75 MiB persistent at load, +2,650–2,665 MiB at the 1024-batch steady state (two runs) — which the proof comment now states; "4,801 MiB used" was never the model's number.

Both are reflected in the description's Verification section.

[synapse-alfonso]

Verified end to end and merged.

The one thing neither you nor PR CI could exercise from a fork is the gate itself, since workflow_dispatch only runs from branches in the base repo. I pushed your head to verify/pr10-owned-cuda here and dispatched it, so the workflow ran once before landing rather than after:

windows-owned-cuda-manual passed on its first dispatch. The six pinned redist archives resolved and matched their digests, the vswhere/Enter-VsDevShell entry gave nvcc its cl.exe, the --features cuda build completed, and the import probe behaved exactly as designed — off-PATH 0xC0000135, on-PATH exit 0. Your redist pinning is correct as written.

Coverage on the rest:

• Linux and Windows CI green with the widened SYNAPSE_CRATES, so the cfg(unix) fix is confirmed on the platform that motivated it.
• macOS is the platform CI does not cover for these crates, so I ran it locally: clippy --all-targets -D warnings clean, 7 + 3 tests pass.
• Your allow-list entry is exactly the registration I asked for, with no change to the checker's logic. I mutation-proved both refusal arms still fire afterwards: altering the registered condition, and making the job push-triggered in the workflow, each still refuse by name.
• The new job mirrors the two existing manual gates in token shape and checkout count, so it adds no privilege surface.

On your throughput correction — the self-catch is appreciated, and the corrected figure cross-checks against measurements you had no way to see. Our own owned-CUDA Qwen3-Embedding-0.6B run on an RTX 4090 recorded 63,330 tok/s, against llama.cpp CUDA f16 GGUF at 9,205 on the same box. Your 646.6 items/s at ~24 tokens/item is ~15,500 tok/s on a laptop 4050, which is close to the 4090 figure scaled by hardware class. The ratio gap (your 3.4x against our 6.9x) is explained by the baseline: you measured llama.cpp at Q8_0 where we measured f16 GGUF, and quantized formats do relatively better on a bandwidth-bound engine — the same effect we saw on RDNA3, where Q6_K beat f16. So your number is independent corroboration on a third GPU we had never measured, and I would not have had that without the comment.

One unrelated finding the dispatch surfaced, recorded so it is not mistaken for yours: linux-llama-cuda-manual failed in the same run, at "Provision pinned NVIDIA CUDA toolkit". That job is byte-identical between master and your branch, so it is a pre-existing break on master that has been invisible because manual gates only run when dispatched and nobody had dispatched one. I am tracking it separately. My first guess — that NVIDIA had pruned the pinned 12.6.1-1 deb — is wrong; it is still in their index, so I am pulling the log rather than guessing again.

Thanks for this. The gap it closes is real: two crates that nothing compiled on any platform, which is how synapse-worker-cuda came to fail clippy on Windows without anyone noticing.