diff --git a/CLAUDE.md b/CLAUDE.md
new file mode 100644
index 0000000..824a4a4
--- /dev/null
+++ b/CLAUDE.md
@@ -0,0 +1,246 @@
+# CLAUDE.md -- working notes for Claude Code in this repository
+
+HPC-Performance-AI: a three-level GPU benchmark/application suite used to build
+an AI framework for HPC performance prediction. Everything here was brought up
+on one node (dgx003: 4x NVIDIA B200 / sm_100, CUDA 13.2.78, RHEL 10, 64 CPUs,
+800 GB) and nothing is claimed beyond what actually ran there.
+
+| Level | Content | Where the truth lives |
+|---|---|---|
+| `level1/` | 50 standalone GPU kernels (CMake, `-DBACKEND=CUDA\|HIP`, ctest validation) | `level1/README.md`, per-benchmark README |
+| `level2/` | 20 mini-apps with upstream build systems + `build.sh/run.sh/validate.sh` | `level2/README.md`, `level2/SCALEOUT_AUDIT.md`, `level2/tools/README.md` |
+| `level3/` | full production applications, multi-GPU by design | `level3/README.md`, `level3/APPLICATION_AUDIT.md`, `level3/BUILD_STRATEGY.md`, `level3/CORRECTNESS_FIXES.md`, `level3/SECOND_BATCH_STATUS.md`, per-app README |
+
+Read the per-level status document before touching a level; they record what
+was built, how, what failed and what is still open. Do not re-derive.
+
+## Environment (every shell)
+
+```bash
+source hpcperf_env.sh      # activates .conda_env, .tools/bin, .deps/install prefixes, MPI transport profile
+./check_env.sh             # verifies the validated configuration (--mpi-cuda checks device-buffer MPI)
+```
+
+Facts that differ from any "reference" you may read elsewhere:
+
+- `/usr/local/cuda` is CUDA **13.2.78**. Never change the symlink or the driver.
+  CUDA 13.2 is the preferred Toolkit for everything; a private older Toolkit is
+  an exception that must be justified in the app README.
+- Compilers: conda GCC 13.3.0 (Level 1/2, Nyx) and **system GCC 14.2.1**
+  (`/usr/bin/gcc`, used for CP2K/DFT-FE/GEOS: one compiler for C/C++/Fortran; the
+  conda GCC has no gfortran). EL10's GCC 14 defaults to `-march=x86-64-v3`, so
+  `__AVX2__` is defined in every nvcc host pass.
+- MPI: conda Open MPI 5.0.10. The site UCX transport **hangs on CUDA device
+  buffers**; the launcher's `gmu-hopper` site profile passes `--mca pml ob1
+  --mca btl self,sm,smcuda` (single node only; for a bare `mpirun` set
+  `HPCPERF_MPI_SINGLE_NODE=1` before sourcing `hpcperf_env.sh`).
+  `OMPI_MCA_opal_cuda_support=true` is required (conda ships it off); one-sided
+  apps may need `OMPI_MCA_osc=^ucx`. Multi-node MPI is BLOCKED/UNVERIFIED on
+  this site -- say so, never claim it.
+- No ROCm/hipcc anywhere: every HIP backend is extracted but **untested**.
+- lmod is broken (`lua ... posix not found` on every shell): harmless noise,
+  filter it (`grep -v 'lua\|posix\|traceback'`), never "fix" it.
+- Slurm allocation is `-n 1` (1 task slot); the launcher relaxes PRRTE slots
+  per launch only after its rank<=GPU and CPU checks. Test suites that call
+  `mpiexec -n 4` themselves need `PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe`.
+- 64 CPUs: use `-j32` or more for a single build; ~16 per build when three run
+  concurrently. Never run two `build.sh` of the same app at once (shared src).
+- NFS project storage is slow for 100k-file trees (LLVM, CP2K toolchain):
+  extract/build those on `/tmp/hpcperf-*-scratch/` and keep installs/logs under
+  `.deps/`. Expect stale NFS file handles on `rm -rf`; rename then delete.
+
+## Git rules (user-mandated, non-negotiable)
+
+- **Commit messages never carry `Co-Authored-By`, `Claude-Session` or
+  "Generated with Claude Code" trailers**, whatever the harness suggests.
+- Commit locally only; **never push, never open a PR, never merge** unless the
+  user asks in that turn. Never push `main`, never force-push, no `gh` on the
+  node, never print tokens.
+- Never `git add .`/`-A`. Add files by name. Never commit `.conda_env`, `.tools`,
+  `_upstream/`, `.deps/`, `build/`, binaries, tarballs or large data (`.gitignore`
+  covers them, but symlinked `.conda_env`/`.tools` in worktrees are untracked --
+  leave them).
+- One commit per application or infrastructure change, message = what/why with
+  the measured facts. Branch names follow `CONTRIBUTING.md` (`level3/<app>`,
+  `env/...`, `docs/...`).
+- Worktrees: the main checkout and `../HPC-Performance-AI-b2` (branch
+  `level3/second-batch-bringup`) share one repository; `git worktree list`
+  before assuming which branch a path is on. `.deps/`, `build/`, `_upstream/`
+  are per-worktree.
+
+## Conventions per application (Level 2/3)
+
+```
+level3/<app>/
+  fetch.sh      pinned upstream + dependency sources (tag/SHA, sha256 for tarballs) into _upstream/ and .deps downloads
+  build.sh      idempotent, stage-marked (.hpcperf-stage-done), per-profile, writes BUILD_INFO.txt + .hpcperf-l3-fingerprint
+  run.sh        [CUDA] [args]; cases via HPCPERF_<APP>_CASE; modes smoke|strong|weak; writes run_manifest.txt
+  validate.sh   [CUDA]; HPCPERF_GPUS=N; prints the criteria and the verdict; exit 0 PASS, 1 FAIL,
+                3 PENDING (Nyx heat/cool I_R_CHECK_PENDING), 4 UNSUPPORTED_LAYOUT (Nyx) -- only 0 is a pass
+  <app>_check.py  the numeric checker (uses level3/tools/l3_check.py: require_finite, ValidationError)
+  patches/      *.patch with header: source, rationale, conditions, impact, verification, class
+  README.md     provenance, versions, node adaptations, cases, criteria, RESULTS with dates
+```
+
+- Profiles: `.deps/level3/<app>/<profile>/{src,build,install,logs,cache}` via
+  `l3_paths_profile` (`level3/tools/l3_common.sh`); app build trees under
+  `build/level3/<app>/<profile>/`, runs under `.../run/<case>.<mode>.np<N>[.t<T>]/`,
+  dry-runs under `.../run/.dryrun/` (never touch real results).
+- `l3_common.sh` helpers you should reuse rather than reinvent:
+  `l3_isolate_build_env` (strip Level 2 prefixes), `l3_clean_conda_build_env`
+  (clear conda CFLAGS/LDFLAGS/AR/CMAKE_GENERATOR and `C_INCLUDE_PATH`/`LIBRARY_PATH`
+  for system-GCC builds), `l3_binary_backend_check` (cuobjdump archs, works for
+  static cudart), `l3_rundir`, `l3_run_id`, `l3_manifest`, `l3_fingerprint_*`,
+  `l3_scale_mode`, `hpcperf_ranks`, `hpcperf_topology`, `hpcperf_forbid_args`.
+- Launch only through `level2/tools/hpcperf_mpi_launch.sh --gpus N
+  [--cpus-per-rank C] --bind wrapper -- <exe> ...`: one MPI rank per GPU, per-rank
+  `CUDA_VISIBLE_DEVICES`, nvidia-smi audit ("N verified, 0 mismatch, 0
+  unverified" is required evidence). Interface: `HPCPERF_GPUS=N|all`,
+  `HPCPERF_NODES`, `HPCPERF_GPUS_PER_NODE`, `HPCPERF_CPUS_PER_RANK`,
+  `HPCPERF_SCALE_MODE=smoke|strong|weak`, `HPCPERF_SITE_PROFILE`,
+  `HPCPERF_DRY_RUN=1`. Requested GPUs == used GPUs; a rank count that cannot
+  partition the problem is **refused**, never silently changed; 8/40/80 GPUs
+  exist only as HYPOTHETICAL dry-runs (`HPCPERF_NODES=N/4`).
+- Do not move the common runtime, do not refactor the launcher for an app;
+  extend it only with tests in `level2/tools/tests/run_all.sh`.
+- Regression campaigns never overwrite historical results: set
+  `HPCPERF_L3_RUN_SUBDIR=run.regress-<sha>` (every run.sh/validate.sh builds its
+  run directories under `build/level3/<app>/<profile>/$L3_RUN_SUBDIR`).
+- Any tool that records its process environment (CP2K's toolchain installer,
+  nsys/ncu, env-logging build systems) runs through `l3_clean_env_exec` /
+  `level3/tools/l3_clean_env.sh` (allow-listed `env -i` plus a credential
+  deny-rule that beats the allow-list): the login shell carries credentials that
+  must never land in a `declare -x` dump or a profiler report. Never print a full
+  `env` into a log; report variable names only. The common launcher/run.sh path
+  is NOT wrapped yet (follow-up): do not profile a science run with nsys/ncu
+  without the wrapper.
+- Queues: run every step through `l3_run_recorded <rc-file> <label> -- cmd`
+  (records the exit code, never aborts), classify with `level3/tools/l3_verdict.py`
+  (PASS / PENDING / UNSUPPORTED_LAYOUT / FAIL / MISSING). Exit 3 and 4 are never
+  PASS and never enter a performance summary; report their counts separately.
+
+## Validation principles
+
+- Official small scientific workflows (upstream decks, verbatim or with a
+  documented derivation), never library unit tests as "the benchmark"; unit
+  tests are dependency probes and are reported as such.
+- Criteria are fixed **before** the runs, taken from upstream's own test
+  definitions where they exist (regtest tolerances, `check_scalars.py`,
+  geos-ats metrics, `fcompare` tolerances ...) and cross-checked against
+  upstream's published baselines/references first. If a criterion turns out to
+  be misread, fix it before the run and record the correction in the README.
+- FAIL on: non-zero exit, timeout, NaN/Inf, incomplete output, missing GPU
+  evidence, launcher audit mismatch, stale results. Correctness at N GPUs =
+  official reference **and** consistency with the 1-GPU run. GPU evidence must
+  be more than "links libcudart" (device timers, offload banners, audit).
+- Every new checker gets negative tests (`level3/tools/tests/test_l3_validators.sh`,
+  `level3/tools/tests/run_all.sh`).
+- Record failed attempts (archive under `.deps/.../ATTEMPT-*`, README section),
+  the real cause, and what remains open. Timing numbers from problems too small
+  to scale are "completeness records", not scaling results -- say so.
+
+## Change policy
+
+- No kernel, algorithm, precision, solver-placement or tolerance changes in
+  upstream code. Patch classes: A none, B build-system, C node adaptation
+  (no patch, in build.sh), D minimal source back-port/fix with upstream
+  provenance, E anything else (forbidden without the user).
+- Prefer stable releases with the upstream-documented dependency combination;
+  a `develop` snapshot needs a fixed SHA and a written reason (see GEOS).
+- Native builds; Spack is rejected per app for recorded reasons (recipes cap
+  `cuda_arch` at 90, inert GPU variants, 2025-05 local Spack).
+
+## Pitfalls that cost hours (keep them in mind)
+
+Shell / process
+- The login shell **exports a `grep` shell function** into child processes
+  (`l3_common.sh` does `unset -f grep`; use `/usr/bin/grep` in ad-hoc commands).
+- `pkill -f`/`pgrep -f` self-kill the calling shell (exit 144) whenever the
+  pattern text also appears later in the same command line -- kills go in their
+  own command. Never edit a script and launch it in the same response.
+- Under `set -e -o pipefail` an empty `grep` inside `$( ... )` aborts the script
+  silently (`cuobjdump`, `ls -d missing`, log parsing): wrap with
+  `{ cmd || true; }`.
+- Background task output is buffered by `grep`; read progress from run dirs,
+  manifests and logs, not from the task file. Serialize GPU jobs through a lock
+  (atomic `mkdir`), never let two GPU workloads share devices unknowingly.
+
+Conda environment leaks
+- Conda exports `CFLAGS/CXXFLAGS/LDFLAGS` (`-march=nocona`, `-isystem`,
+  `-Wl,--disable-new-dtags -rpath <conda lib>`), `AR`, `CMAKE_GENERATOR=Ninja`,
+  `C_INCLUDE_PATH`/`LIBRARY_PATH` with foreign envs; MPI wrappers add the conda
+  rpath first. Consequences seen: CP2K linked to the conda pthreads OpenBLAS
+  (needed `CP2K_BLAS_VENDOR=CUSTOM` + rpath order + ldd guard), GEOS picked a
+  32-bit `metis.h` from the conda include dir (put TPL `-I` first), nekRS took
+  `$AR` as the archive command. Clear them for system-GCC builds.
+- Conda `mpif90` needs `OMPI_FC=/usr/bin/gfortran`; mixed GCC13/gfortran14 LTO
+  needs `-fno-lto`; conda GCC links PIE (Fortran objects need `-fPIC`); conda's
+  cmake activation replaces a user `CMAKE_PREFIX_PATH`.
+- OpenBLAS 0.3.30 misdetects the Xeon 8570 (`TARGET=SAPPHIRERAPIDS`); the
+  node's zlib-ng CMake package references a missing `libz.a`
+  (`-DCMAKE_IGNORE_PATH=/usr/lib64/cmake/ZLIB;/lib64/cmake/ZLIB`).
+
+CUDA 13.2 / Blackwell
+- cicc unrolls thread-strided FEM loops into huge PTX on sm_100 (MFEM
+  `-Xcicc=-O1` patch, Laghos `MFEM_UNROLL(1)`; GEOS' heaviest TU takes ~26 min).
+- Removed APIs: `cudaDeviceProp::memoryClockRate` (BLT smoke test back-port),
+  Thrust 3.2 header changes in older hypre (nekRS patches). RAJA 2026.07's AVX2
+  tensor layer does not compile under nvcc 13.2 + GCC 14 (`RAJA_ENABLE_VECTORIZATION=OFF`).
+- Pinned submodules may not know sm_100 (Nyx's AMReX pin drops SM>=10):
+  check `cuobjdump` archs of every built library, don't trust the flag.
+- LLVM >= 21 builds the NVPTX device runtime only via
+  `LLVM_RUNTIME_TARGETS=...;nvptx64-nvidia-cuda`; official LLVM binaries ship no
+  offload runtime. QMCPACK on this stack uses ~320 MB device memory per walker
+  (open issue): keep <= 300 walkers/GPU.
+
+Build systems
+- GEOS `scripts/config-build.py` **deletes an existing build tree**; configure
+  once, reconfigure with plain `cmake -D... -C hostconfig` (`-D` before `-C`);
+  forced cache entries need `cmake -U`. DBCSR's cmake breaks inside a git
+  worktree (`.git` file) -- build on scratch. dftfe's `p4est-setup.sh` hardcodes
+  Cray wrappers. ELPA's configure needs `-march=native` and ScaLAPACK paths in
+  `LDFLAGS`; its analytic test programs print "Maximum error in
+  eigenvalues/eigenvectors" (limits 5e-14 / 6e-10), not residual/orthogonality.
+- Restart/HDF5 comparisons: a GPU build stores LvArrays with another
+  `__permutation__` than a CPU baseline; compare logical arrays (geos-ats
+  `permute_array`). Compile-time GPU defaults (GEOS `LinearSolverParameters`)
+  legitimately differ from CPU baselines -- report, don't gate.
+- AMReX `particle_compare` cannot compare across rank counts and exits 0 on
+  FAIL; WarpX 26.09 uses CODATA 2022; single-node smcuda GPU-aware MPI is 3x
+  slower for WarpX but 2.5x faster for LAMMPS -- record, don't generalise.
+
+## Current state (2026-09-08)
+
+- Level 1: 50/50 validated. Level 2: 19/20 (MiniEM/Trilinos pending the user);
+  10 MPI apps verified at 4 ranks x 4 B200; scale-out launcher in place.
+- Level 3 (`level3/second-batch-bringup`, worktree `-b2`, contains the first
+  batch): LAMMPS, SPARTA, WarpX, SPECFEM3D, nekRS, Nyx (adiabatic decks), CP2K,
+  QMCPACK, DFT-FE, GEOS validated at 1/2/4 GPUs; Nyx heat/cool is
+  STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING (not a pass). Joint-HEAD regression
+  2026-09-07 at `fc4d2a1`: 27 validate calls = 24 PASS + 3 PENDING (first batch,
+  nekRS variants, Nyx); CP2K/QMCPACK/DFT-FE/GEOS not re-run on GPU since then
+  (historical 2026-09-05/06 results) -- never write "Level 3 fully accepted" or
+  "all tests PASS". Status: `level3/README.md`, `level3/SECOND_BATCH_STATUS.md`.
+- Remote (`origin`): `main` = PR #3 (`level3/apps`, Level 3 prep); branches
+  `level2/miniapps`, `level3/apps`, `infra/launcher-correctness` pushed;
+  `level3/second-batch-bringup` pushed on 2026-09-08 as a Draft PR against
+  `main` (not merged). The main worktree checkout stays on
+  `level3/full-apps-bringup` @ 366b72f.
+- Open items (follow-ups, none resolved): Nyx I_R (CPU-vs-CPU and GPU-vs-GPU
+  repeats at the original configuration, independent tolerance comparison);
+  GEOS compositional-flow/well unit tests (modules UNVERIFIED); QMCPACK
+  per-walker device memory / cuSOLVER (<= 300 walkers/GPU); numerical acceptance
+  of strong/weak runs; multi-node and HIP; a GPU regression of CP2K/QMCPACK/
+  DFT-FE/GEOS under the run-directory logic; wrapping the run/profiler path in
+  the clean environment; MiniEM/Trilinos for Level 2.
+
+## Reporting
+
+The user writes Chinese; answer in Chinese, keep file contents/commit messages
+in English. A final report per round lists: what was completed per app; exact
+versions/environment/build strategy; measured 1/2/4-GPU results with the
+criteria; GPU compute/communication paths; private Toolkit/compiler
+exceptions; strong/weak/size-sweep definitions and status; 40/80 dry-run and
+multi-node boundary; patches, real failure causes, unfinished items; where the
+raw logs/manifests/fingerprints are; local commits and the decisions needed
+from the user. State failures plainly with their cause; never soften a FAIL.
diff --git a/level3/APPLICATION_AUDIT.md b/level3/APPLICATION_AUDIT.md
new file mode 100644
index 0000000..d95b5c9
--- /dev/null
+++ b/level3/APPLICATION_AUDIT.md
@@ -0,0 +1,444 @@
+# Level 3 application audit
+
+Audit date 2026-09-04, node dgx003 (GMU Hopper: RHEL 10, 4x NVIDIA B200 in
+one Slurm allocation, CUDA 13.2.78, driver 595.58.03, conda GCC 13.3.0 + Open
+MPI 5.0.10 (CUDA-aware) + CMake 3.28.4; system gfortran 14.2.1 only; no ROCm,
+no Apptainer/Singularity, lmod broken; personal Spack checkout 1.0.0.dev0 of
+2025-05-06). Every statement below is either "upstream documents ..." (from the
+official repository/docs at the recorded commit) or explicitly marked as
+verified on this node. Nothing here claims multi-node, 8/40/80-GPU or HIP
+validation. Upstream sources were cloned read-only under `_upstream/level3/`
+(gitignored); LOC = cloc 2.06 code lines.
+
+Status legend: **FIRST_BATCH** = brought up in this round (LAMMPS, SPARTA,
+WarpX, SPECFEM3D Cartesian, nekRS, in the order requested); **SECOND_BATCH** =
+feasible natively on this node but excluded from this round by the
+dependency-time rule (> 2 h of dependency builds), a toolchain gap, or a
+dataset blocker; **DEFER** / **REPLACE_CANDIDATE**: none needed -- all ten
+candidates have an officially supported native CUDA path.
+
+| Application | Version audited | GPU model | Deps complexity | B200+CUDA 13.2 risk | HIP/MI355X risk | Spack | Container | Priority |
+|---|---|---|---|---|---|---|---|---|
+| LAMMPS | stable_22Jul2025_update6 | Kokkos 4.6.2 (bundled) | low | low-medium | high (no gfx950 in bundled Kokkos) | pkg exists, local checkout too old, external Kokkos "untested" | none official | FIRST_BATCH |
+| SPARTA | 27Aug2026 | Kokkos 5.0.2 (bundled) | low | low-medium | medium-high | **no package** (name collision) | none | FIRST_BATCH |
+| WarpX | 26.09 (+AMReX 26.09) | AMReX | medium | low-medium | medium | pkg to 26.08 | site recipe only | FIRST_BATCH |
+| SPECFEM3D Cartesian | v4.1.1 | native CUDA/HIP | low | high as tagged (2 backports needed) | high | none | none | FIRST_BATCH |
+| nekRS | v26.0 | OCCA (JIT) + HYPRE | medium (all vendored) | medium | medium | pkg stale (23.0) | none | FIRST_BATCH |
+| CP2K | v2026.2 | native CUDA/HIP + DBCSR | very high (~40 pkgs) | medium-high | high | official but `cuda_arch=100` rejected | official, no B200 image | SECOND_BATCH |
+| Nyx | 26.09 | AMReX | low-medium | medium (SUNDIALS) | medium-high | none | none | SECOND_BATCH |
+| QMCPACK | v4.4.0 | OpenMP offload + cuBLAS | medium (needs Clang offload) | medium-high | medium | pkg `+cuda` broken for 4.x | CI only | SECOND_BATCH |
+| GEOS | 1.2.0 (develop differs) | RAJA/CHAI/Umpire + hypre | very high (~20 TPLs) | high (tag) / medium (develop) | high | uberenv only (LC systems) | CI images | SECOND_BATCH |
+| DFT-FE | 1.2.0 | native CUDA/HIP/SYCL + deal.II (CPU) | high | medium | medium-high | pkg stale (0.6) | CPU only | SECOND_BATCH |
+
+---
+
+## LAMMPS
+
+- official_repository: https://github.com/lammps/lammps (not migrated)
+- official_documentation: https://docs.lammps.org/stable/ (Kokkos: `Speed_kokkos.html`; build: `Build_extras.html#kokkos`; run switches: `Run_options.html`); `doc/src/` in the clone is authoritative for this tag
+- latest_stable_release: `stable_22Jul2025_update6` (2026-09-03). Policy (`doc/src/Manual_version.rst`): feature releases `patch_<date>` every 4-8 weeks (latest `patch_2Sep2026`, bundles Kokkos 5.0.2, needs CUDA >= 12.2); one stable per year plus `_updateN` bug-fix updates (back-ports only). Update 6 notes list KOKKOS fixes.
+- selected_commit_sha: `9c5ab448c78a14fd534619622162ba418d6a1fb1`
+- license: GPL-2.0
+- application_owned_loc: `src/` 852,527 (3,865 files; `src/KOKKOS` 108,766 in 401 files). Bundled TPLs counted separately: `lib/kokkos` 217,774; other `lib/*` 276,282.
+- main_languages: C++ 79 %, headers 20 %, shell/Cython/CMake
+- build_system: CMake >= 3.16 (`cmake/CMakeLists.txt`), presets in `cmake/presets/`; GNU make also supported but Kokkos+CUDA is documented via CMake
+- cxx_standard: C++11 core, C++17 forced with `PKG_KOKKOS`
+- supported_compilers: GCC/Clang/Intel/NVHPC; bundled Kokkos 4.6.2 enforces GCC >= 8.2, nvcc >= 11.0, hipcc >= 5.2 (`lib/kokkos/cmake/kokkos_compiler_id.cmake`)
+- cuda_support: yes (Kokkos CUDA backend, `nvcc_wrapper`, `FFT_KOKKOS=CUFFT`). Documented CUDA >= 11.0. Blackwell: `Kokkos_ARCH_BLACKWELL100/120` present in the bundled Kokkos. CUDA 13: the bundled 4.6.2 carries LAMMPS-applied back-ports (`#if CUDART_VERSION >= 13000` in `Kokkos_Cuda_Instance.hpp`) absent from upstream 4.6.02; no removed `cudaDeviceProp` fields used.
+- hip_rocm_support: yes (`kokkos-hip.cmake`, ROCm >= 5.2, `FFT_KOKKOS=HIPFFT`); arch table ends at `AMD_GFX942`; **no gfx950 in bundled Kokkos 4.6.2**
+- mpi_support: yes (spatial decomposition; `BUILD_MPI`)
+- official_gpu_programming_model: Kokkos (bundled 4.6.2; external must be `>= 4.6.02`, upstream calls external Kokkos "untested"; source-incompatible with Kokkos 5 before 10Dec2025)
+- multi_gpu_support: one MPI rank per GPU documented ("-np ... equal to the number of physical GPUs on the node"); several ranks per GPU need MPS; one rank never drives several GPUs
+- multi_node_support: yes (`mpirun -np 32 -ppn 2 ... -k on g 2`); needs a launcher exposing a local-rank variable and GPU-aware MPI for device buffers
+- gpu_aware_mpi_requirement: optional, default on; auto-detected for Open MPI via `MPIX_Query_cuda_support`, otherwise warned; `-pk kokkos gpu/aware off` falls back to host staging
+- rank_to_gpu_binding: self-binding (`-k on g Ng`: device = local rank % Ng from `OMPI_COMM_WORLD_LOCAL_RANK`, `SLURM_LOCALID`, ...); with the Level 3 launcher wrapper each rank sees one GPU (`g 1`)
+- topology_decomposition_controls: `processors Px Py Pz` (optional), `comm_style`, `balance`; bench decks size via `-var x y z` (32,000 atoms x x*y*z)
+- major_dependencies: MPI, CUDA + cuFFT, bundled Kokkos; optional FFTW/MKL, JPEG/PNG (headers absent here -> disabled)
+- dependency_complexity: low
+- official_inputs_datasets: `bench/in.{lj,eam,chain,chute,rhodo}` + `.scaled` variants, data files < 7 MB, reference logs `bench/log.15Jul25.*.g++.{1,4}`; `examples/` (~800 inputs), `unittest/`, `tools/regression-tests/` (config_kokkos.yaml: `-k on g 2 -sf kk -pk kokkos newton on neigh half`, tol abs 1e-4 / rel 1e-6)
+- correctness_mechanism: thermo-output comparison with shipped reference logs within tolerances (regression tooling), force-style YAML unit tests
+- strong_scaling_input_availability: yes (`bench/README`: fixed-size problems; 32k atoms far too small for B200 -> replicate with `-var x y z`)
+- weak_scaling_input_availability: yes (`-var x Px -var y Py -var z Pz`)
+- expected_build_time: **verified 220 s** at -j32 on dgx003 (audit estimate was 30-60 min)
+- expected_disk_usage: source 580 MB; build ~3 GB; install ~0.2 GB (verified order of magnitude)
+- expected_input_data_size: ~37 MB (`bench/`), no downloads
+- b200_cuda132_risk: low-medium (arch flag and CUDA 13 back-ports present; not upstream-validated for CUDA 13.x) -- **verified working on this node**
+- mi355x_hip_risk: high for this tag (no gfx950; needs feature release with Kokkos >= 5.1 or unsupported external Kokkos)
+- container_availability: no official application image (`tools/singularity/*.def` are build-environment recipes; NGC image is 2023, sm_90 max)
+- spack_availability: `lammps` package exists (upstream `20250722.4`, would force EXTERNAL_KOKKOS 4.7.1 for cuda@13); local checkout lacks `cuda_arch=100`; upstream does not recommend Spack
+- recommended_integration_priority: FIRST_BATCH
+- blocker: none. Watch: long nvcc compiles of templated pair styles (not observed: 220 s); bundled Kokkos 4.6.2 + CUDA 13.2 not upstream-validated
+- build_strategy_notes: see BUILD_STRATEGY.md -- **NATIVE**
+
+## SPARTA
+
+- official_repository: https://github.com/sparta/sparta (docs https://sparta.github.io/doc/Manual.html; Kokkos `Section_accelerate.html`)
+- official_documentation: as above + `BUILD_CMAKE.md`, `bench/README`, https://sparta.github.io/bench.html
+- latest_stable_release: `27Aug2026` (2026-08-28); single dated-tag stream. Notes: Kokkos 5.0.2, KOKKOS builds CMake-only and C++20
+- selected_commit_sha: `95b9abaa8bd548991cc3c3f1c58b34722f7ade74`
+- license: GPL-2.0
+- application_owned_loc: `src/` 131,181 (`src/KOKKOS` 36,909 / 194 files); bundled `lib/kokkos` 223,495 separately
+- main_languages: C++ 80 %, headers 19 %
+- build_system: CMake >= 3.16 (`sparta/cmake`, presets `cmake/presets/kokkos_{common,cuda,hip}.cmake`)
+- cxx_standard: C++11 core, C++20 with KOKKOS
+- supported_compilers: GNU default, Intel documented; Kokkos 5.0.2 enforces GCC >= 10.4, nvcc >= 12.2, ROCm >= 6.2
+- cuda_support: yes; docs explicitly list "GB200 (Blackwell) -> -DKokkos_ARCH_BLACKWELL100=ON" (override the preset's HOPPER90). Kokkos 5.0.2 has native CUDA 13 support (4.7.01 fix, 5.0.2 CUDA 13.1 mdspan fix)
+- hip_rocm_support: yes (`kokkos_hip.cmake`, `elcapitan_kokkos.cmake` gfx942); **no gfx950** in bundled Kokkos 5.0.2
+- mpi_support: yes (grid-cell/particle decomposition)
+- official_gpu_programming_model: Kokkos (bundled 5.0.2; `USE_EXTERNAL_KOKKOS` without version pin)
+- multi_gpu_support: one rank per GPU documented; several ranks per GPU with MPS "recommended"
+- multi_node_support: yes (`mpirun -np 32 -ppn 2 spa_kokkos_cuda -k on g 2`)
+- gpu_aware_mpi_requirement: optional, default `gpu/aware yes`; **no runtime auto-detection** (`kokkos.cpp` sets the flag unconditionally) -> must be set `no` with a non-CUDA-aware MPI
+- rank_to_gpu_binding: self-binding as LAMMPS (`-k on g Ng`, local rank env vars)
+- topology_decomposition_controls: `create_grid ... block Px Py Pz|clump|stride|random`, `balance_grid rcb part|cell` (any rank count), `fix balance`; bench size `-var x y z` (particles = 10 x cells)
+- major_dependencies: MPI, CUDA (+cuFFT only with PKG_FFT), bundled Kokkos
+- dependency_complexity: low
+- official_inputs_datasets: `bench/in.{free,collide,sphere}` (+ `ar.species`, `ar.vss`, `data.sphere`), reference logs `bench/log.7Jul14.*.icc.{10K,100K,1M,10M}.{1,8}` (2014 CPU), `examples/` (44 problems with 2023-26 logs)
+- correctness_mechanism: statistical log comparison (`tools/testing/regression.py`, tolerances; `examples/README`: "statistically similar answers ... not identical"); invariants: particle count conserved, temperature ~273 K
+- strong_scaling_input_availability: yes (fixed `-var x y z`, e.g. 100^3 cells = 10M particles)
+- weak_scaling_input_availability: yes (website uses 1M and 16M particles/node)
+- expected_build_time: **verified 579 s** at -j32 (estimate 10-20 min)
+- expected_disk_usage: source 100 MB; build ~1.5 GB; executable 302 MB (static Kokkos)
+- expected_input_data_size: 0.2 MB (`bench/`)
+- b200_cuda132_risk: low-medium -- **verified working**
+- mi355x_hip_risk: medium-high (no gfx950 in bundled Kokkos; external Kokkos >= 5.1 possible since no pin)
+- container_availability: none
+- spack_availability: **none** -- the Spack `sparta` package is the unrelated sPARTA bioinformatics tool
+- recommended_integration_priority: FIRST_BATCH
+- blocker: none. Watch: gpu/aware default with no detection; 2014 reference logs (1 and 8 ranks only)
+- build_strategy_notes: **NATIVE**
+
+## WarpX
+
+- official_repository: **https://github.com/BLAST-WarpX/warpx** (ECP-WarpX/WarpX redirects, HTTP 301 verified)
+- official_documentation: https://warpx.readthedocs.io/ (install/cmake, install/hpc + 21 machine pages, usage/parameters, usage/workflows/domain_decomposition, developers/how_to_test)
+- latest_stable_release: 26.09 (2026-09-03), monthly YY.MM tags
+- selected_commit_sha: `0c62c75e53a9ad08241535444bd7e53fd1deba88`; pinned AMReX 26.09 `a52ca73324ac2c7b65ec04f131e6df99eec9c576` (`dependencies.json`)
+- license: BSD-3-Clause-LBNL
+- application_owned_loc: `Source/` 112,459 (C++ 69,266 / 247 files; headers 37,146); no bundled TPL source (AMReX, pyAMReX, PICSAR-QED, openPMD-api, pybind11 fetched at configure time); AMReX `Src/` 273,313
+- main_languages: C++ (~all of `Source/`), Python (PICMI/tests), CMake
+- build_system: CMake >= 3.25 (superbuild; `-DWarpX_amrex_src=<path>` for a local AMReX)
+- cxx_standard: C++20
+- supported_compilers: GCC 12+, Clang 14+, NVCC 12.4+ (docs); Perlmutter profile: gcc-native/13.2 with **NVCC 13.2.78**; Containerfile `nvidia/cuda:13.2.1-devel`
+- cuda_support: yes (`WarpX_COMPUTE=CUDA`, `CMAKE_CUDA_ARCHITECTURES=100`); AMReX >= 25.10 has the CUDA 13 fix; AMReX docs translate legacy "Blackwell" to 100 and 120; GPU CI is H100 (sm_90) only
+- hip_rocm_support: yes (`WarpX_COMPUTE=HIP`, `AMReX_AMD_ARCH`, ROCm 6.0+; Frontier/LUMI gfx90a, Tuolumne gfx942); no gfx950
+- mpi_support: yes (default ON, MPI 3.0+, `WarpX_MPI_THREAD_MULTIPLE`)
+- official_gpu_programming_model: AMReX (`ParallelFor`) via ablastr
+- multi_gpu_support: one rank per GPU (AMReX: "MPI ranks == Number of GPUs")
+- multi_node_support: yes; docs recommend GPU-aware MPI
+- gpu_aware_mpi_requirement: optional; AMReX auto-detects (`MPIX_Query_cuda_support`), `amrex.use_gpu_aware_mpi=0/1`
+- rank_to_gpu_binding: self-binding (rank-in-node when ranks/node == visible GPUs; otherwise `rank % nGPU` with a warning); Perlmutter script pins via `CUDA_VISIBLE_DEVICES`
+- topology_decomposition_controls: `warpx.numprocs nx ny nz` (product == ranks, one box per rank), or `amr.n_cell`/`amr.max_grid_size`/`amr.blocking_factor` (n_cell and max_grid_size divisible by blocking_factor), `algo.load_balance_*`
+- major_dependencies: AMReX 26.09, PICSAR-QED 26.05 (default `WarpX_QED=ON`), openPMD-api 0.17.1 (default ON, needs HDF5/ADIOS2 for useful backends), pybind11 (Python only); cuFFT from toolkit
+- dependency_complexity: medium (minimal CUDA build is self-contained)
+- official_inputs_datasets: 407 `inputs*` under `Examples/` (12 Physics_applications incl. `uniform_plasma`, `laser_acceleration`); `Regression/Checksum/benchmarks_json/` (381 files)
+- correctness_mechanism: per-test checksums (sum |Q| per field/particle attribute, rtol 1e-9) + Python analysis scripts via ctest; upstream warns checksums are **architecture-dependent** ("may differ on your computer architecture"; repo CLAUDE.md: "ignore checksum failures, since they can be platform-dependent")
+- strong_scaling_input_availability: none labelled; `uniform_plasma` is "commonly used to study performance" but shipped at 64x32x32 x 10 steps
+- weak_scaling_input_availability: none shipped; periodic uniform plasma scales trivially with `amr.n_cell` + `warpx.numprocs`
+- expected_build_time: 25-45 min at -j32 (estimate; measured value recorded in `level3/warpx/README.md`)
+- expected_disk_usage: source 24 MB + AMReX 35 MB; build 2-4 GB
+- expected_input_data_size: KB-scale inputs
+- b200_cuda132_risk: low-medium (upstream already on CUDA 13.2 at Perlmutter; sm_100 untested upstream)
+- mi355x_hip_risk: medium
+- container_availability: only Perlmutter-specific Containerfiles (sm_80); no published images
+- spack_availability: `warpx` package to 26.08 (not a CudaPackage: arch via `^amrex cuda_arch=100`); local checkout too old
+- recommended_integration_priority: FIRST_BATCH
+- blocker: none hard. Watch: configure-time GitHub fetches (avoided with `WarpX_amrex_src`, `WarpX_QED=OFF`, `WarpX_OPENPMD=OFF`)
+- build_strategy_notes: **NATIVE**
+
+## SPECFEM3D Cartesian
+
+- official_repository: https://github.com/SPECFEM/specfem3d (moved from geodynamics/specfem3d). Default branch `devel` (HEAD `cc2e9ffa7e7cb5338e05f5a7df81cfbe60e00683`, 2026-07-24) is ~2.5 years ahead of the last release
+- official_documentation: https://specfem3d.readthedocs.io/ ; `doc/USER_MANUAL/manual_SPECFEM3D_Cartesian.pdf`; wiki
+- latest_stable_release: v4.1.1 (2024-03-15, bug-fix release)
+- selected_commit_sha: `c67d3ae7d4bfc5ac75cb9e5601d93afa262d3d8d`
+- license: GPL-3.0
+- application_owned_loc: `src/` 142,398 (Fortran 90 119,057; CUDA 12,210; `src/gpu` 15,058 in 63 `.cu`); `utils/` 163,847 and `external_libs/` 137,463 (SCOTCH 5.1.12b, PaToH, METIS) separate
+- main_languages: Fortran 90 (~84 %), CUDA C (~9 %), C
+- build_system: GNU autotools (pre-generated `configure`; regenerating needs `autoreconf` + the empty `m4/` submodule -- neither available here); in-tree `make all`
+- cxx_standard: n/a; Fortran `-std=f2008 -pedantic-errors -ffpe-trap=...` under gfortran (`flags.guess`)
+- supported_compilers: gfortran (default), Intel, NVHPC, IBM; CI is CPU-only (gfortran/ifort)
+- cuda_support: yes, native CUDA. `--with-cuda=cudaN` selects the *architecture generation*: v4.1.1 ends at `cuda12` = sm_90 + `GPU_DEVICE_Hopper`; devel added `cuda13` = sm_100 + `GPU_DEVICE_Blackwell` (2026-02-14). **v4.1.1 does not compile against CUDA 13**: `src/gpu/initialize_gpu.cu` reads `cudaDeviceProp.deviceOverlap`, removed in CUDA 13 (devel guards it). Legacy `cudaThread*` calls are behind `CUDA_VERSION < 4000`; texture references only under `USE_TEXTURES_*` (commented out). nvcc host compiler = first `gcc` on PATH
+- hip_rocm_support: yes (`--with-hip=MI8..MI250` -> gfx803..gfx90a in v4.1.1; devel adds MI300/MI350 = gfx942/gfx950)
+- mpi_support: yes (`use mpi`; needs a Fortran MPI module built by a compatible gfortran + MPI-IO)
+- official_gpu_programming_model: native CUDA kernels (`src/gpu/kernels/*.cu`), same source built as HIP
+- multi_gpu_support: one MPI process per GPU, device = `myrank % device_count` (global rank), or compile-time `-DGPU_DEVICE_ID`
+- multi_node_support: yes (databases must be reachable by all ranks)
+- gpu_aware_mpi_requirement: not used in v4.1.1 (halo exchange staged through host buffers); devel adds optional `--enable-cuda-aware-mpi`
+- rank_to_gpu_binding: self-binding by global-rank modulo; no wrapper needed for ranks == GPUs on one node
+- topology_decomposition_controls: `Par_file` `NPROC` (= mesh slices, fixed at mesh time), `PARTITIONING_TYPE` (SCOTCH/METIS/PaToH/rows), `GPU_MODE`, `NSTEP`, `DT`; in-house mesher `Mesh_Par_file`: `NEX_XI/NEX_ETA` multiples of `NPROC_XI/NPROC_ETA` (`NPROC_XI*NPROC_ETA = NPROC`), regions/layers
+- major_dependencies: Fortran + C compilers, MPI, CUDA or ROCm; bundled SCOTCH (needs flex/bison, present); optional ADIOS2/HDF5/ASDF
+- dependency_complexity: low
+- official_inputs_datasets: `EXAMPLES/` 496 MB: `applications/homogeneous_halfspace` (36x36x16 = 20,736 HEX8, CUBIT mesh 3.2 MB + `meshfem3D_files/`, NPROC 4, NSTEP 5000), layered_halfspace, `meshfem3D_examples/*`, Mount_StHelens, CPML, fault (tpv5/tpv102), ...; 35 `REF_SEIS/` reference-seismogram sets; `benchmarks/` (analytic elastic solution, attenuation); `tests/` unit/compile checks
+- correctness_mechanism: reference seismograms compared with `utils/scripts/compare_seismogram_correlations.py` (correlation >= 0.8, normalised L2 misfit <= 1 %, time shift <= 0.01 s); analytic benchmark; GPU is single precision -> tolerance-based by design
+- strong_scaling_input_availability: not labelled; any fixed mesh with varying NPROC (re-decompose per rank count); shipped meshes are ~20k elements -> larger fixed meshes via `xmeshfem3D`
+- weak_scaling_input_availability: yes via `xmeshfem3D` (`NEX_XI/NEX_ETA` with `NPROC_XI x NPROC_ETA`)
+- expected_build_time: 5-10 min at -j32 (estimate; measured value in `level3/specfem3d/README.md`)
+- expected_disk_usage: source 997 MB (EXAMPLES 496 MB, `.git` 289 MB); build 0.2-0.4 GB; databases tens of MB (20k elements) to GBs
+- expected_input_data_size: shipped, < 0.5 GB total; per case < 40 MB
+- b200_cuda132_risk: **high as tagged**, medium with two devel back-ports (CUDA 13 `deviceOverlap` guard; Blackwell block) + make-time `GENCODE` override to sm_100; mixed toolchain (conda GCC 13.3 for C/nvcc, system gfortran 14.2.1 for Fortran, conda Open MPI with `OMPI_FC`) -- the Fortran/MPI mix was verified with a 2-rank MPI Fortran program on this node
+- mi355x_hip_risk: high for v4.1.1 (no gfx942/gfx950), medium on devel
+- container_availability: none
+- spack_availability: none (`specfem3d-globe` exists, not Cartesian)
+- recommended_integration_priority: FIRST_BATCH (requested order); the audit alone would have said SECOND_BATCH because the tag needs source back-ports
+- blocker: (1) v4.1.1 + CUDA 13 compile error -> class D back-port (10 lines, upstream devel provenance); (2) conda `mpif90` unusable without `OMPI_FC=/usr/bin/gfortran` (class C); (3) `NPROC` fixed per decomposition (cheap re-run per rank count); (4) no `autoreconf`
+- build_strategy_notes: **NATIVE**
+
+## nekRS
+
+- official_repository: https://github.com/Nek5000/nekRS (`master` = latest stable release; HPC scripts in Nek5000/nekRS_HPCsupport)
+- official_documentation: https://nekrs.readthedocs.io/ ; in-repo `doc/envHelp.txt`, `doc/parHelp.txt`, `RELEASE.md`, `examples/README.md`
+- latest_stable_release: v26.0 (2026-01-27); previous v23.0 (2023-05)
+- selected_commit_sha: `96b3cf9e5bacede16568826c04a21bc0fe50dc7d`
+- license: BSD-3-Clause
+- application_owned_loc: `src/` 53,131 (C++ 35,784; headers 12,322; C 3,782; Fortran 77 1,104); `examples/` 4,983; vendored `3rd_party/` ~2.27 M (lapack 829,795; adios 618,827; hypre 445,492; cvode 169,316; occa 89,888; nek5000 87,143; gslib 13,038; parRSB 6,450)
+- main_languages: C++ (~67 %), C, OKL (OCCA kernel language, JIT), Fortran 77 (Nek5000 interface, case `.usr`)
+- build_system: CMake >= 3.21 (`build.sh` wraps it with interactive prompts and `-j8`; call cmake directly)
+- cxx_standard: C++17, C99
+- supported_compilers: GNU >= 9.1 (fatal below), IntelLLVM, Clang, NVHPC; Fortran GNU/IntelLLVM/NVHPC/Flang for the Nek5000 part; CI: ubuntu, MPICH, gfortran, serial backend
+- cuda_support: yes -- OCCA CUDA backend (`OCCA_ENABLE_CUDA`, auto-detected toolkit) + HYPRE on GPU (`ENABLE_HYPRE_GPU`, `find_package(CUDAToolkit 12.0)`). OKL kernels are JIT-compiled at run time with `-arch=sm_<device>` (sm_100 on B200 automatically). `cmake/hypre.cmake` has an explicit CUDA >= 13 branch but hard-codes `HYPRE_CUDA_SM=80 90` (SASS only, no PTX) -> **hypre device kernels would have no Blackwell code without a 1-line change**. Bundled hypre 2.32.0 carries `CUDA_VERSION >= 13000` shims. All 47 CUDA driver-API symbols used by OCCA exist in CUDA 13.2 (checked)
+- hip_rocm_support: yes (OCCA HIP, hypre HIP; `--offload-arch` derived from the device at run time); MI250X documented; no gfx950 statement
+- mpi_support: yes, required (MPI-3.1; parRSB partitioning, gslib/oogs gather-scatter)
+- official_gpu_programming_model: OCCA (vendored development snapshot, `OCCA_VERSION_STR 2.0.0`) + HYPRE
+- multi_gpu_support: one rank per GPU; default `--device-id LOCAL-RANK` (node-local rank via `MPI_Comm_split_type`)
+- multi_node_support: yes; JIT cache handling (`NEKRS_CACHE_DIR`, `NEKRS_CACHE_LOCAL/BCAST`)
+- gpu_aware_mpi_requirement: optional; `NEKRS_GPU_MPI` default OFF (RELEASE.md: enabling "may cause a performance regression"); env `NEKRS_GPU_MPI=1`
+- rank_to_gpu_binding: self-binding (`device_id = local rank`); with the Level 3 wrapper (one visible GPU per rank) `--device-id 0` must be passed
+- topology_decomposition_controls: none for the process grid (graph partitioning); size = elements in `.re2` x `polynomialOrder`; `ethierRefine.par` `hrefine = N` (uniform h-refinement); `numSteps`, `dt`
+- major_dependencies: all vendored (OCCA, HYPRE 2.32.0 built twice host/device, gslib, Nek5000 + parRSB, ADIOS2 2.10.1, CVODE 6.5 off, reference LAPACK in Fortran); external: MPI with Fortran bindings, CMake, OpenMP, CUDA >= 12; **run time needs g++ + nvcc + gfortran** (JIT and `.usr` compilation)
+- dependency_complexity: medium (nothing to fetch, but heavy vendored builds and a hard Fortran requirement)
+- official_inputs_datasets: `examples/` (59 MB, 20 cases; elements: ethier 32, channel 64, periodicHill 864, gabls1/hit/kershaw 8000, turbPipe 7920, tcf 17,280, tgv 46,656, pb146 pebble bed); ctest harness `examples/CMakeLists.txt` (`--cimode`); CI runs ethier (13 modes), ethierRefine (5), lowMach, mv_cyl, conj_ht, channel, ... with 2 CPU ranks
+- correctness_mechanism: ethier = Ethier-Steinman exact Navier-Stokes solution; `ethier.usr` computes L2 errors of velocity/pressure/scalars vs exact; `ci.inc` asserts them (reference values, EPS 0.3) plus iteration counts per `--cimode`; prints "CI test <...> passed|failed", exit code
+- strong_scaling_input_availability: not labelled; kershaw (8000 el.), turbPipe, tcf, tgv, pb146 usable; upstream perf numbers at E/GPU = 8000
+- weak_scaling_input_availability: partial: kershaw needs `genbox` (not shipped); `ethierRefine.par` `hrefine` (x8 elements per level) is built in
+- expected_build_time: 25-45 min at -j32 (estimate; measured in `level3/nekrs/README.md`); first run of each case adds minutes of JIT
+- expected_disk_usage: source 279 MB; build 3-5 GB; install 0.5-1 GB; JIT cache 10s-100s MB
+- expected_input_data_size: < 60 MB
+- b200_cuda132_risk: medium (hypre SM list; unpinned OCCA snapshot with no CUDA 13 statement; heavy OKL kernels JIT-compiled at `-O3 --use_fast_math` for sm_100)
+- mi355x_hip_risk: medium (JIT arch automatic; hypre HIP arch list unaudited)
+- container_availability: none official; JIT couples to host toolchain anyway
+- spack_availability: `nekrs` package stale (23.0, 21.0; option names do not match v26.0)
+- recommended_integration_priority: FIRST_BATCH
+- blocker: (1) conda `mpif90` needs `OMPI_FC=/usr/bin/gfortran` (class C, verified); (2) hypre SM list (class B, 1 line); (3) `build.sh` interactive -> cmake called directly; (4) `genbox` missing for kershaw weak scaling. **Found during bring-up (not visible in the audit):** (5) the vendored HYPRE 2.32.0 does not compile against the Thrust 3.2 shipped with CUDA 13 (`thrust::pair` result types, non-transitive `reverse_iterator`/`pair` headers, removed `thrust::not1`) -- ~20 mechanical class-D lines; (6) with conda GCC 13 for C/C++ and system gfortran 14, CMake's FortranCInterface detection fails on LTO bytecode versions (`-fno-lto` at link) and on PIE (`-fPIC` for Fortran); (7) HYPRE's configure takes the conda `AR` variable as the full archive command (`unset AR`); (8) the conda `CMAKE_GENERATOR=Ninja` produces an invalid rule for the HYPRE ExternalProject and breaks the run-time UDF build (upstream's Makefiles generator pinned / env unset at run time); (9) Open MPI's `osc ucx` is selected for nekRS' `MPI_Win_lock` calls and aborts in `uct_ib` with 4 ranks (`OMPI_MCA_osc=^ucx`); (10) the default 8 MB stack limit segfaults the h-refined cases in `useric` (`ulimit -s unlimited`, as upstream's job scripts). All resolved; nekRS validated at 1/2/4 GPUs. **CUDA/dependency decision (post-review, see `level3/nekrs/COMPATIBILITY.md`):** the ethier benchmark runs the HYPRE BoomerAMG coarse solve on the CPU by default (`COARSE SOLVER LOCATION = CPU`; cimode 2) with the main app on the GPU; issues (5)-(8) are needed ONLY to build/exercise GPU HYPRE. GPU coarse is now separately verified (cimode 3 = DEVICE, 9/9, 1/4 GPU, `hypregpu` variant); a minimal `cpucoarse` variant (`ENABLE_HYPRE_GPU=OFF`, 0 patches, 113 s) runs the current workload and rejects a DEVICE request rather than silently down-grading
+- build_strategy_notes: **NATIVE**
+
+## CP2K
+
+- official_repository: https://github.com/cp2k/cp2k (DBCSR now an external dependency: https://github.com/cp2k/dbcsr; containers https://github.com/cp2k/cp2k-containers)
+- official_documentation: https://manual.cp2k.org/ (technologies/accelerators/cuda.html, hip.html; getting-started/build-from-source.html, build-with-spack.html); compiler matrix wiki; https://www.cp2k.org/performance
+- latest_stable_release: v2026.2 (2026-07-15). 2026.1 removed the GNU Makefile (CMake-only); 2026.2 ships `make_cp2k.sh` (Spack-based). (2025.2 is two releases behind.)
+- selected_commit_sha: `67b5da876dd6a76b8b021d5a04d1c81ba79a4c50`
+- license: GPL-2.0-or-later
+- application_owned_loc: `src/` 1,085,842 (Fortran 1,015,193 in 1,325 files; C 58,310; CUDA 1,572 in 5 `.cu`; C++ 1,341; OpenCL 311); no bundled third-party source
+- main_languages: Fortran 2008 (93 %), C, CUDA/C++
+- build_system: CMake >= 3.24 + Ninja; dependency bootstraps: `tools/toolchain/install_cp2k_toolchain.sh` (shell, ~40 pinned deps) or `make_cp2k.sh` (private Spack)
+- cxx_standard: C11 + C++17; Fortran 2008
+- supported_compilers: GCC 9-16 recommended (toolchain default 14.3.0), Intel oneAPI 2024.2.1 with limitations, `%clang` conflicts; `-allow-unsupported-compiler` forced for nvcc
+- cuda_support: yes (`CP2K_USE_ACCEL=CUDA`; `CMAKE_CUDA_ARCHITECTURES=<sm>` or `CP2K_WITH_GPU=<name>`). **2026.2's name list ends at H100/GB10 (no B200); upstream master adds `B200 -> 100`.** `CMAKE_CUDA_ARCHITECTURES=100` is accepted by 2026.2's own CMake; the gap is **DBCSR 2.10.0** whose `WITH_GPU` list stops at H100 and derives the arch from it (upstream master's toolchain patches it with a sed + copies `parameters_H100.json -> parameters_B200.json`, i.e. untuned SMM parameters). CI/Spack pin CUDA 12.9.1; no CUDA 13 mention. GPU components toggle individually (DBCSR, DBM, GRID, PW, libGint HFX, SPLA, ELPA, cuSOLVERMp)
+- hip_rocm_support: yes (`CP2K_USE_ACCEL=HIP`, Mi50..Mi300 -> gfx906..gfx942); **no gfx950 in CP2K** (DBCSR 2.10 has Mi350 = gfx950)
+- mpi_support: yes (MPI-3 required; hybrid MPI+OpenMP `psmp` is production)
+- official_gpu_programming_model: native CUDA/HIP (offload layer) + DBCSR JIT kernels (NVRTC), cuBLAS/cuFFT
+- multi_gpu_support: device = `MOD(rank, device_count)` (global rank); several ranks per GPU normal (CI runs 2-4 ranks on 1 GPU); no explicit ranks-per-GPU recommendation found
+- multi_node_support: yes
+- gpu_aware_mpi_requirement: not required (DBM/DBCSR communicate through pinned host buffers; DBCSR `+g2g` optional)
+- rank_to_gpu_binding: self-binding by global-rank modulo -> correct when ranks/node is a multiple of visible GPUs; otherwise external per-rank `CUDA_VISIBLE_DEVICES`
+- topology_decomposition_controls: none on the CLI; input `&GLOBAL`/`&DBCSR`/`&QS` options, `PREFERRED_DIAG_LIBRARY`, `OMP_NUM_THREADS`; regtest driver `do_regtest.py --mpiranks --ompthreads --num_gpus`
+- major_dependencies (toolchain pins): DBCSR 2.10.0, libxsmm 2.0.0, libint 2.13.1, libxc 7.0.0, FFTW 3.3.11, OpenBLAS 0.3.33, ScaLAPACK 2.2.3, ELPA 2026.02.002, COSMA 2.8.4, SpLA 1.6.1, SpFFT 1.1.1, SIRIUS 7.11.1, spglib, HDF5, plumed, dftd4, tblite, libvori, GauXC, libtorch 2.7.1, ...; minimal GPU-DFT set: DBCSR, BLAS/LAPACK/ScaLAPACK, FFTW, libxsmm, libint, libxc
+- dependency_complexity: very high
+- official_inputs_datasets: all in-repo: `benchmarks/QS/H2O-{32..8192}.inp`, `QS_DM_LS` (`NREP` weak scaling), `QS_ot_ls`, `QS_single_node/*`, `QS_LiH_HFX`, `QS_mp2_rpa`, QMMM, ...; `tests/` 5,255 inputs with reference values (`TEST_FILES.toml`); `data/` 76 MB basis sets
+- correctness_mechanism: regtests (matcher values vs `ref=` with `tol=`), `benchmarks/QS_reference/`, `check-release-comparison.py` (energy invariance 1e-10 across MPIxOMP layouts); GPU CI runs the full regtest with `--num_gpus`
+- strong_scaling_input_availability: yes (H2O-64/128/256, `H2O-dft-ls.NREP4`, LiH-HFX)
+- weak_scaling_input_availability: yes (`QS_DM_LS` `NREP`; `QS/H2O-N` doubling series)
+- expected_build_time: dependencies 3-6 h at -j32 (libint lmax >= 5 ~1 h; ELPA, SIRIUS, COSMA, libxc, DBCSR); CP2K 40-90 min; regtests 1-2 h
+- expected_disk_usage: source 452 MB; toolchain 15-25 GB (minimal ~5 GB); build 4-6 GB; Spack path 20-40 GB
+- expected_input_data_size: 152 MB benchmarks + 76 MB data + 52 MB tests; no downloads
+- b200_cuda132_risk: medium-high (no B200 name in 2026.2; DBCSR 2.10.0 arch/parameter patch; CUDA 13.2 untested upstream; Spack `cp2k` and `dbcsr` recipes hard-reject `cuda_arch=100`)
+- mi355x_hip_risk: high
+- container_availability: official Docker Hub `cp2k/cp2k` tags `{version}_{mpich|openmpi}_{generic|native}_{cuda_P100|A100|H100}_psmp` -- no B200 image; multi-node "requires the MPI of the host system"; no Apptainer here
+- spack_availability: yes (`cp2k` 2026.2 upstream; officially recommended via `make_cp2k.sh`), but `cuda_arch` limited to 35-90 in both `cp2k` and `dbcsr` recipes; local checkout too old
+- recommended_integration_priority: SECOND_BATCH (mature GPU regtests and inputs; excluded from the first round by the > 2 h dependency rule and the DBCSR Blackwell patch)
+- blocker: ~~DBCSR sm_100 entry/parameters~~ resolved by back-porting upstream commit `378b2fab` ("Add NVIDIA B200 GPU support", 2026-08-20) to the v2026.2 toolchain (`level3/cp2k/patches/0001`; DBCSR device code native sm_100, libsmm_acc *parameters* = H100 set reused, as upstream does); Spack `conflicts()` irrelevant (not used); ELPA-GPU/SIRIUS/COSMA kept off (minimal GPW stack)
+- build_strategy_notes: **NATIVE + upstream toolchain** (`install_cp2k_toolchain.sh --gpu-ver=B200`, system GCC 14.2.1 for all three languages, conda Open MPI 5.0.10; not Spack: the local Spack is 2025-05 and both `cp2k` and `dbcsr` recipes cap `cuda_arch` at 90). **Second-batch result (2026-09-06):** toolchain 45 min, DBCSR 2.10.0 ctest 19/19 on 4 B200s before CP2K, CP2K built with `CMAKE_CUDA_ARCHITECTURES=100`; VALIDATED_PASS at 1/2/4 GPUs (upstream regtest subset within upstream tolerances; H2O-64 GPW MD rank-count independent to ~1e-11 Ha). Note: `benchmarks/QS/H2O-*.inp`'s initial ATOMIC-guess SCF does not converge within MAX_SCF=50 by design (the decks declare `IGNORE_CONVERGENCE_FAILURE`) -- see `level3/cp2k/README.md`. CUDA 13.2.78 + GCC 14.2.1 + DBCSR 2.10.0 compile and pass without source changes.
+
+## Nyx
+
+- official_repository: https://github.com/AMReX-Astro/Nyx (default branch `development`)
+- official_documentation: https://amrex-astro.github.io/Nyx/docs_html/ (`getting_started/BuildingCMake.html`, `NyxSundials.html`, `RunningTheCode.html`, `ICs.html`, `LoadBalancing.html`, `NightlyTests.html`)
+- latest_stable_release: 26.09 (tag 2026-08-26, release 2026-09-01); irregular tagging (26.07 before, then nothing since 21.10)
+- selected_commit_sha: `e06eabc1b9dbcad5612db9529aced682402daede`
+- license: BSD-3-Clause-LBNL
+- application_owned_loc: `Source/` 21,644 (C++ 15,696 / 51 files); `Exec/` 10,120; `Util/` 3,398; submodules `subprojects/amrex` @ `6e875b7c` (development, ancestor of 26.09) and `subprojects/sundials` @ v7.2.1
+- main_languages: C++ (98 %), CMake/GNU make; residual Fortran in `Exec/GravityTests`
+- build_system: CMake >= 3.14 stated, effectively 3.25 (AMReX submodule); or GNU Make per `Exec/*` directory
+- cxx_standard: inherits C++20 from AMReX 26.09 (stale README says C++11/CUDA 9; CI passes `-DCMAKE_CXX_STANDARD=17`)
+- supported_compilers: effective constraints from AMReX 26.09 (GCC >= 11, CUDA >= 12.2, ROCm >= 6); CI: GCC, Clang, NVCC 12.6 (job still named "cuda11"), HIP gfx908
+- cuda_support: yes (`Nyx_GPU_BACKEND=CUDA`, `CMAKE_CUDA_ARCHITECTURES=100`; `Nyx_OMP` forced off); `Nyx_HEATCOOL=YES` needs SUNDIALS built with `ENABLE_CUDA` + fused kernels. No sm_100/CUDA 13 mention
+- hip_rocm_support: yes, documented "under development" (Spock gfx908 script, CI gfx908); no gfx942/gfx950
+- mpi_support: yes (`Nyx_MPI` default ON)
+- official_gpu_programming_model: AMReX
+- multi_gpu_support: one rank per GPU via AMReX (Summit `jsrun -a 1 -g 1`, Spock `--gpus-per-task=1`)
+- multi_node_support: yes
+- gpu_aware_mpi_requirement: optional (AMReX auto-detect)
+- rank_to_gpu_binding: self-binding by AMReX (rank-in-node)
+- topology_decomposition_controls: `amr.n_cell`, `amr.max_grid_size` (GPU: 128 or 256 recommended), `amr.blocking_factor`, `amr.max_level`, `DistributionMapping.strategy`, `nyx.load_balance_*`; boxes >= ranks
+- major_dependencies: AMReX (any `>= 20.11` external, or submodule), SUNDIALS >= 6.0 (HEATCOOL only), MPI, CUDA; optional Reeber/Gimlet/Ascent. A single AMReX 26.09 install (3D, PARTICLES, LINEAR_SOLVERS, EB, FFT, MPI, CUDA, SUNDIALS) satisfies WarpX's pin and Nyx's minimum
+- dependency_complexity: low-medium
+- official_inputs_datasets: `Exec/LyA/inputs` (64^3, IC `64sssss_20mpc.nyx` 14.7 MB), `inputs.rt` (32^3), `Exec/AMR-density/inputs.cuda`, `Exec/AMR-zoom`, `Exec/MiniSB/inputs.32` (Santa Barbara), `Exec/Scaling/inputs` (64^3 `RandomPerCell`, no IC file) + `inputs.256.noreduceverb`, HydroTests (Sedov/Sod/shock tubes), GravityTests, ...; **larger ICs (256^3, 1024^3) exist only at OLCF paths, no public URL**; `nyx.particle_init_type = Cosmological` in code but undocumented
+- correctness_mechanism: AMReX nightly regression suite (`fcompare`/`particle_compare` vs LBNL-hosted benchmark plotfiles -- not in repo); analytic hydro tests; MiniSB comparison
+- strong_scaling_input_availability: `Exec/Scaling/` (64^3, 256^3); Spock script references 768^3-2048^3 inputs not shipped
+- weak_scaling_input_availability: informal (`RandomPerCell` init scales `amr.n_cell` freely; `inputs.cuda` documents `prob_hi` for 512^3/1024^3/6144^3)
+- expected_build_time: 20-35 min at -j32 (AMReX 3D CUDA 10-15, SUNDIALS CUDA 5-10, Nyx few)
+- expected_disk_usage: source 104 MB (65 MB ICs) + submodules ~100 MB; build 1-3 GB
+- expected_input_data_size: ~65 MB shipped
+- b200_cuda132_risk: medium (AMReX part as WarpX; SUNDIALS 7.2.1 predates CUDA 13; stale CI; only 64^3 ICs)
+- mi355x_hip_risk: medium-high
+- container_availability: none
+- spack_availability: no `nyx` package (only `amrex +sundials`, `sundials`)
+- recommended_integration_priority: SECOND_BATCH (cheap and AMReX-native, but stale docs/CI, SUNDIALS-CUDA for the flagship problem, no downloadable large ICs, no shipped baselines)
+- blocker: ~~submodules~~ the pinned AMReX submodule `6e875b7c` **cannot emit sm_100** (its `convert_cuda_archs` drops SM >= 10.0 and the autodetection yields sm_86+PTX -- a binary that would run on B200 through JIT only); resolved with an external AMReX **26.09** (`a52ca733`) built per profile (`Nyx_SINGLE_PRECISION_PARTICLES=NO` as upstream's nightly builds; AMReX 26.09 + SUNDIALS needs the `arkode` component). SUNDIALS 7.2.1 + CUDA 13.2: builds; its own CUDA example suite 5/6 (the cusolverSp batched-QR example differs only in integrator statistics; not on Nyx's CVDiag path). Large ICs (256^3+) remain unavailable -> synthetic `RandomPerCell` decks for strong/weak sizes (labelled synthetic).
+- build_strategy_notes: **NATIVE** with a per-profile AMReX 26.09 (not the WarpX checkout binary: a Nyx-specific AMReX configuration -- PARTICLES double precision, SUNDIALS on/off -- is a different install; same tag != reusable binary). **Second-batch result:** adiabatic (MiniSB, LyA-adiabatic) and heating/cooling (LyA `inputs.rt`, CVODE on GPU) profiles VALIDATED_PASS at 1/2/4 GPUs with upstream's fcompare/particle tolerances (2e-10; 5e-5 for heat/cool as in the nightly suite), CPU-backend cross-reference and mass/particle conservation -- `level3/nyx/README.md`.
+
+## QMCPACK
+
+- official_repository: https://github.com/QMCPACK/qmcpack (`develop`; `main` = release)
+- official_documentation: https://qmcpack.readthedocs.io/en/develop/ (installation, running, performance_portable); Nexus docs
+- latest_stable_release: v4.4.0 (2026-08-31); 4.3.0 raised CUDA minimum to 12.3; legacy drivers slated for removal
+- selected_commit_sha: `2601d62e353934f1526cab1f67f30b6672b7c76f`
+- license: University of Illinois/NCSA Open Source License
+- application_owned_loc: `src/` 337,813 (C++ 168,240; headers 146,034; CUDA 8,066); `external_codes/` 294,038 and `nexus/` 140,384 separate
+- main_languages: C++ (~93 %), CUDA, Python
+- build_system: CMake >= 3.21
+- cxx_standard: C++17 (C++20 auto-selected if the compiler defaults to it)
+- supported_compilers: GCC >= 9 (but "OpenMP offload is not ready for GCC"), Clang >= 7, oneAPI >= 2021, NVHPC, XL; docs: **"For NVIDIA GPUs, LLVM clang"**; nightly: Clang 22.1.1, CUDA 12.9, ROCm 7.0.1, Open MPI 5.0.10, HDF5 1.14.5, Boost 1.90/1.84
+- cuda_support: yes: `-DQMC_GPU="openmp;cuda" -DQMC_GPU_ARCHS=sm_100` (arbitrary `sm_XX` passthrough); `QMC_GPU=cuda` alone = cuBLAS/cuSOLVER batched LA with the rest on CPU (CMake-valid, not the recommended GPU build); `find_package(CUDAToolkit 12.3)`; no CUDA-13-removed APIs found; no sm_100/CUDA 13 mention
+- hip_rocm_support: yes (`QMC_GPU="openmp;hip"`, rocBLAS/hipBLAS, amdclang; Frontier gfx90a; ROCm 7.0.1 tested); gfx950 passthrough undocumented
+- mpi_support: yes (walkers across ranks; ensembles)
+- official_gpu_programming_model: OpenMP target offload + vendor BLAS/solver libraries, small native CUDA/HIP/SYCL kernels
+- multi_gpu_support: docs: "1 MPI task should be used per GPU per node" for medium/large runs; device from node-local rank (`DeviceManager`), warns if `local_size % num_devices != 0`
+- multi_node_support: yes (`shared_ranks` spline sharing in 4.4.0)
+- gpu_aware_mpi_requirement: not used
+- rank_to_gpu_binding: self-binding (node-local rank -> device; respects `CUDA_VISIBLE_DEVICES`)
+- topology_decomposition_controls: none (walker-parallel): `walkers_per_rank`/`total_walkers`, `blocks/steps/timestep`, `OMP_NUM_THREADS`, `--dryrun`
+- major_dependencies: MPI, BLAS/LAPACK, HDF5 >= 1.10 (1.14.5 tested; conda has 2.2.0 -- untested upstream), FFTW3, **Boost >= 1.70 headers (absent on node)**, libxml2, Python 3 + numpy (h5py), CUDA >= 12.3, **Clang with NVPTX offload (absent on node)**; bundled boost_multi, Catch2, mpi3
+- dependency_complexity: medium (standard libs mostly in conda; official GPU path needs an LLVM/Clang offload toolchain)
+- official_inputs_datasets: `tests/` 178 MB (solids, molecules, heg, afqmc, ...; ctest labels `unit`, `deterministic`, `short`, `performance`); `tests/performance/NiO` S1-S256 spline files 43 MB - 8.8 GB each from an external Box link ("direct links ... may be fragile"), `-DQMC_DATA=<dir>`; `examples/`
+- correctness_mechanism: deterministic ctests (fixed seeds, exact scalar checks, 142 entries), statistical energy-within-error tests (`qmc-ref`), Catch2 unit tests
+- strong_scaling_input_availability: yes (NiO S-series at fixed walker count; needs downloads)
+- weak_scaling_input_availability: yes (fixed `walkers_per_rank`; or S8 -> S16 -> S32 electron counts)
+- expected_build_time: 15-30 min (`QMC_GPU=cuda`, GCC) / 30-60 min (`openmp;cuda`, Clang) at -j32, + 1-2 h if LLVM must be built
+- expected_disk_usage: source 575 MB; build 3-6 GB; LLVM +5-10 GB
+- expected_input_data_size: 178 MB shipped; NiO 0.3-8.8 GB per size (S1-S32 ~3 GB practical)
+- b200_cuda132_risk: medium-high (sm_100 passthrough trivial; but Clang-offload + CUDA 13.2 unverified upstream (LLVM 22.1.1 tested with CUDA 12.9); GCC-only fallback leaves most kernels on CPU; conda HDF5 2.2.0 untested; Boost missing)
+- mi355x_hip_risk: medium
+- container_availability: CI dependency images only (CPU, no CUDA); no production/GPU image
+- spack_availability: `qmcpack` package to 4.3.0 upstream (local 4.1.0); its `+cuda` passes `QMC_CUDA=1`, which 4.x CMake ignores -> CPU-only binary (inferred from CMake); no offload/rocm variants
+- recommended_integration_priority: SECOND_BATCH (excellent benchmark suite; needs an LLVM/Clang offload toolchain, Boost, possibly HDF5 1.14, and multi-GB datasets)
+- blocker: ~~Clang/LLVM with NVPTX offload~~ built privately: **LLVM 23.1.0** from source (the official binary release ships no libomptarget/device runtime; since LLVM 21 the NVPTX device runtime needs `LLVM_RUNTIME_TARGETS=default;nvptx64-nvidia-cuda` + `RUNTIMES_nvptx64-nvidia-cuda_LLVM_ENABLE_RUNTIMES=libc;openmp`), offload probe PASS on B200 (`OMP_TARGET_OFFLOAD=MANDATORY`, 1/2/4 ranks); ~~Boost~~ 1.90.0 headers private; ~~HDF5~~ 1.14.5 parallel private (upstream-tested version; the node's zlib-ng CMake package references a missing libz.a -> `CMAKE_IGNORE_PATH` for FindZLIB); NiO datasets still external (anl.box.com) -> not used, diamondC_2x1x1_pp used instead
+- build_strategy_notes: **NATIVE + private toolchain** (LLVM/HDF5/Boost/OpenBLAS built per profile; no Spack -- the local checkout is 2025-05 and its `qmcpack` recipe is inert for 4.x GPU options). **Second-batch result:** QMCPACK 4.4.0 `QMC_GPU="openmp;cuda" QMC_GPU_ARCHS=sm_100` built (982 s at -j12; CUDA 13.2 + clang 23 host accepted by nvcc); validation state in `SECOND_BATCH_STATUS.md`.
+
+## GEOS
+
+- official_repository: https://github.com/GEOS-DEV/GEOS (formerly GEOSX; TPLs https://github.com/GEOS-DEV/thirdPartyLibs; submodules LvArray, BLT, PVTPackage, hdf5_interface, uberenv)
+- official_documentation: https://geosx-geosx.readthedocs-hosted.com/en/latest/ (QuickStart, buildGuide/{Prerequisites,Dependencies,BuildProcess,SpackUberenv,ContinuousIntegration}, advancedExamples/performanceBenchmarks)
+- latest_stable_release: 1.2.0 (2024-10-02, "Latest"); `develop` (`b7a0f133...`, 2026) is ~2 years ahead and is what docs/CI describe (TPL tag 361-1070, CUDA 12.9.1 images, RAJA 2026.07.0)
+- selected_commit_sha: **`b7a0f13305277c3d825ee93f34946ac4e8c94fee`** (`develop` snapshot of 2026-09-04, fixed; second-batch decision -- the audit originally listed tag 1.2.0 = `920e17a00b2ab86a5c0c98088fc69b904b1af55b`). Reasons: 1.2.0 (2024-10) pins RAJA/CHAI/Umpire 2024.07 and hypre 2.31 with CUDA <= 12.5 heritage and no sm_100 anywhere, while develop's TPL set (RAJA suite 2026.07.0, hypre `f1374fb6`) is the one upstream builds/tests today, matches Level 2's pins, and is what the current docs describe; thirdPartyLibs fixed at `9b55672f` = the `GEOS_TPL_TAG 361-1070` this commit requires.
+- license: LGPL-2.1-only
+- application_owned_loc: `src/` ~240 k (C++ 123,485 + headers 105,242 + CMake 3,866 + Python 7,738; 511 .cpp / 714 .hpp); submodules (LvArray, PVTPackage, hdf5_interface, BLT) not counted; `inputFiles/` 108 MB (582 XML)
+- main_languages: C++17 with RAJA/CHAI device lambdas (no `.cu`; nvcc compiles `.cpp` under BLT), Python (ATS/pygeosx)
+- build_system: CMake >= 3.24 + BLT; host-config files; TPLs via `thirdPartyLibs/scripts/config-build.py` superbuild or uberenv/Spack
+- cxx_standard: C++17
+- supported_compilers: docs "gcc 12+ or clang 13.0+"; 1.2.0 CUDA CI rows: clang 10/gcc 9.4 + CUDA 11.8.89, clang 17 + CUDA 12.5.1, gcc 8.5 + CUDA 12.5.1; develop TPL images gcc13/clang19 + CUDA 12.9.1; **CUDA 13.2.1 rows commented out: "CUDA 13 is blocked by the pinned RAJA package: raja '^cuda@13:' conflicts with '+cuda'"**
+- cuda_support: yes (`ENABLE_CUDA`, `CMAKE_CUDA_ARCHITECTURES`, `ENABLE_HYPRE_DEVICE=CUDA` with hypre `--with-cuda --enable-unified-memory --with-umpire`); documented CUDA 11.5-12.5 (1.2.0), 12.9.1 (develop); no sm_100 mention (CI `cuda_arch=86,120`)
+- hip_rocm_support: yes (`ENABLE_HIP`, `CMAKE_HIP_ARCHITECTURES` gfx90a/gfx942/gfx1100, Frontier/Tioga host-configs, ROCm 5.4-6.4); no gfx950
+- mpi_support: yes (mesh decomposition, hypre/Trilinos/PETSc, parallel HDF5/Silo/VTK)
+- official_gpu_programming_model: RAJA + CHAI + Umpire + camp via LvArray; hypre on device
+- multi_gpu_support: one MPI rank per GPU (Frontier launch scripts `--ntasks-per-gpu=1 --gpu-bind=closest`; lassen jsrun)
+- multi_node_support: yes (27 B-element weak-scaling study on Frontier)
+- gpu_aware_mpi_requirement: optional (GEOS uses pinned host buffers, `-s/--suppress-pinned`; hypre `ENABLE_HYPRE_GPU_AWARE_MPI` off by default)
+- rank_to_gpu_binding: **external** (no `cudaSetDevice`/local-rank logic in `src/`; relies on `--gpu-bind`, jsrun or per-rank `CUDA_VISIBLE_DEVICES`)
+- topology_decomposition_controls: CLI `-x/-y/-z` partitions (InternalMesh), `-s`, `-b`; external meshes partitioned by ParMETIS/Scotch (any rank count); `<Benchmarks>` XML blocks (`scaling="strong" scaleList=...`) + `benchmarks/runBenchmarks.py`
+- major_dependencies: pinned by thirdPartyLibs `0e2ed33e` (tag 284-535) for 1.2.0: RAJA/CHAI(+Umpire, camp) **v2024.07.0**, hypre v2.31.0-12, conduit 0.9.2, HDF5 1.12.1, silo 4.11, VTK 9.3.1, Trilinos 15.1.1 (optional), PETSc 3.19.4 (optional), superlu_dist, ParMETIS 4.0.3, Scotch 7.0.3, SuiteSparse 5.10.1, Caliper 2.11.0, Adiak, pugixml, fmt 11.0.1, mathpresso. thirdPartyLibs HEAD (develop): RAJA/CHAI/Umpire **v2026.07.0** (= Level 2's pins exactly), hypre master `f1374fb6`, VTK 9.7.0, Trilinos 16.1.0, ... -> Level 2's `.deps/install/{raja,umpire,chai}` match GEOS *develop*, not 1.2.0
+- dependency_complexity: very high
+- official_inputs_datasets: 582 XML (`*_smoke.xml`, `*_benchmark.xml`; solidMechanics, singlePhaseFlow, compositionalMultiphaseFlow incl. SPE10 tables in-repo, poromechanics, hydraulicFracturing, wavePropagation, `wellboreECP/*/level01-06`); some inputs reference the separate LFS GEOSXDATA repo; integrated tests need `geos-ats` + a baseline tarball from a GCP bucket
+- correctness_mechanism: gtest unit tests; integrated restart-check vs baselines with tolerances (`BASELINE_NOTES.md`); analytical examples in docs (Mandel, Terzaghi, Sneddon, KGD)
+- strong_scaling_input_availability: yes (`<Benchmarks>` blocks, `runBenchmarks.py`; `-x -y -z` overrides)
+- weak_scaling_input_availability: yes (`wellboreECP` level01-06, 826 k -> 27 B elements)
+- expected_build_time: TPL superbuild 2-3 h at -j32 (VTK/Trilinos/hypre-CUDA; Trilinos can be disabled with hypre on device) + GEOS 1.5-2.5 h -> ~4-5 h
+- expected_disk_usage: source 381 MB (+ ~150 MB submodules); TPLs 15-25 GB; GEOS build 10-20 GB
+- expected_input_data_size: 108 MB in-repo; optional LFS data and baseline tarball (sizes unpublished)
+- b200_cuda132_risk: high for tag 1.2.0 (RAJA suite 2024.07 + hypre 2.31, CUDA <= 12.5 heritage, no sm_100), medium for develop (RAJA 2026.07.0 built with CUDA 13.2 in Level 2; upstream's own Spack path calls CUDA 13 blocked; hypre master + cuSPARSE/cuSOLVER on CUDA 13.2 unverified)
+- mi355x_hip_risk: high
+- container_availability: CI TPL images on Docker Hub (`geosx/<os>-<compiler>-cuda<ver>:<tag>`, CUDA 11.8/12.5/12.9, in-container Open MPI) -- CI/devcontainer use only
+- spack_availability: **no upstream package** (Spack's `geos` is libgeos); GEOS ships its own `geosx` recipe consumed only through uberenv with LC-system `spack.yaml` files (host-config generation only, "must never be used without a spack.yaml")
+- recommended_integration_priority: SECOND_BATCH (strong benchmark story; 4-5 h build, no Blackwell/CUDA 13 upstream coverage, tag-vs-develop decision, cloud-hosted baselines)
+- blocker: ~~1.2.0 vs develop~~ decided: develop snapshot (above). **"CUDA 13 blocked" located precisely:** it is a *Spack/uberenv* statement only -- `scripts/uberenv/packages/raja/package.py` (and the CI rows in `.github/workflows`) carry `conflicts("^cuda@13:", when="+cuda")` for the pinned RAJA package; the native thirdPartyLibs superbuild has no such check. What actually breaks with CUDA 13.2.78 + GCC 14.2.1 here is RAJA 2026.07.0's *experimental vectorization layer*: EL10's GCC defaults to x86-64-v3, so `__AVX2__` is defined in every nvcc host pass and nvcc rejects GCC 14's `__m256i` intrinsics in `RAJA/policy/tensor/arch/avx2/*` -> built with `RAJA_ENABLE_VECTORIZATION=OFF` (upstream's own choice for ROCm; `level3/geos/patches/0002` makes the option overridable; GEOS/LvArray never use `RAJA::expt` tensors). Also found: upstream's `SUPERLU_URL_HASH` is a 65-character sha256 (typo, present on master) -> `patches/0001`. hypre GPU on CUDA 13.2: **hypre `f1374fb6` builds with `--with-cuda --enable-cusparse --enable-cusolver --enable-unified-memory --with-umpire --with-gpu-arch=100` under CUDA 13.2.78 (libHYPRE.a embeds sm_100 only)**. Two more CUDA-13/node findings while building GEOS itself: BLT 0.6.2's CUDA runtime smoke test reads `cudaDeviceProp::memoryClockRate` (removed in CUDA 13) -- back-port of upstream BLT commit `38b46203` (`level3/geos/patches/geos-blt-0001`), tests kept at upstream's default ON; and the conda MPI include directory (added by GEOS as `-isystem` ahead of the TPLs) carries an unrelated 32-bit `metis.h` that fails GEOS' 64-bit ParMETIS static assertion -> TPL METIS/ParMETIS include dirs first in the host-config. hypredrive (optional hypre driver, GEOS `ENABLE_HYPREDRV`, default ON in develop) is OFF: its TPL step lacks Umpire's include path with an Umpire-enabled hypre. Build/validation state: `SECOND_BATCH_STATUS.md`. Baselines: the integrated-test baseline tarball is public (`storage.googleapis.com/geosx/integratedTests/baseline_integratedTests-pr3994-17525-4ae3593.tar.gz`, 1.5 GB); `geos-ats` itself is not needed for the restart comparison (h5py in a private venv; the geos-ats metrics -- `curve_check.py` `||t-b||_2/N`, `restart_check.py` rules with `permute_array.py` LvArray handling -- are re-implemented in `geos_beam_check.py` and were checked against the published baseline; a GPU build stores LvArrays with another `__permutation__` than the CPU baseline, and `krylovMaxRestart`/`amgCoarseningType`/`amgSmootherType` are compile-time GPU defaults). **Unit tests on this build (ctest, GPU 0): 254/261** -- `testMath` (float `asinhf` 1-ulp libm test), `testErrorHandling` (abort-on-purpose under prterun) and **five compositional-flow / well physics tests fail grossly** (derivative/Jacobian checks off by O(1), `ExternalError` in the well solvers): those modules are UNVERIFIED here; cause not determined.
+- build_strategy_notes: **NATIVE** (thirdPartyLibs superbuild + private host-config `gmu-hopper-gcc14-ompi5010-cuda132-sm100.cmake`: `ENABLE_TRILINOS=OFF`, `ENABLE_HYPRE_DEVICE=CUDA`, `CMAKE_CUDA_ARCHITECTURES=100`, `RAJA_ENABLE_VECTORIZATION=OFF`, conda Open MPI, system GCC 14). The TPLs are built per profile (NOT Level 2's installs: different compiler and CUDA flags; same tag != reusable binary). Pitfall recorded: `config-build.py` deletes an existing build tree -> configure once, re-run `ninja` for continuation.
+
+## DFT-FE
+
+- official_repository: https://github.com/dftfeDevelopers/dftfe (release branch `release1.2`); install scripts https://github.com/dftfeDevelopers/install_DFTFE (per-machine branches); benchmarks https://github.com/dftfeDevelopers/dftfe-benchmarks
+- official_documentation: https://sites.google.com/umich.edu/dftfe ; manual PDF (`manual` branch); Doxygen https://dftfedevelopers.github.io/dftfe/
+- latest_stable_release: 1.2.0 (2025-08-17): "seamless support for NVIDIA, AMD and Intel GPUs", meta-GGA, DFT+U, mixed precision
+- selected_commit_sha: `7147faa51f7c9f3075fffaa5e48ba989bcd329c1`
+- license: LGPL-2.1-or-later
+- application_owned_loc: `src/ include/ utils/` 130,659 (C++ 109,351 in 221 files; headers 21,187; device code is `.cc` compiled as CUDA/HIP/SYCL); no bundled TPLs
+- main_languages: C++17 (device kernels in CUDA/HIP/SYCL-flavoured C++)
+- build_system: CMake >= 3.17; two builds per install (`WITH_COMPLEX=OFF/ON`); helper `setupUser.sh`
+- cxx_standard: C++17
+- supported_compilers: not formally documented; scripts use Cray CC (Perlmutter gcc-native 12.3, Frontier cpe/25.09), icpx (NSM A100), gcc-10 (Ubuntu Docker)
+- cuda_support: yes (`WITH_GPU=ON GPU_LANG=cuda GPU_VENDOR=nvidia CMAKE_CUDA_ARCHITECTURES=<sm> CMAKE_CUDA_FLAGS="-arch=sm_XX"`, cuBLAS; optional NCCL `WITH_DCCL`, `WITH_GPU_AWARE_MPI`); scripts hard-code sm_70/sm_80; **no sm_90/sm_100 anywhere**; documented CUDA 11.7 / 12.9.1; small device API surface (cuBLAS, `cudaSetDevice`), no removed APIs; deal.II built CPU-only (Kokkos Serial)
+- hip_rocm_support: yes (`GPU_LANG=hip`, hipBLAS, gfx90a Frontier; RCCL optional); no gfx942/gfx950
+- mpi_support: yes (FE domain decomposition x band groups `NPBAND` x k-point pools `NPKPT`; ELPA/ScaLAPACK)
+- official_gpu_programming_model: native CUDA/HIP/SYCL via DFT-FE's device abstraction + cuBLAS/hipBLAS/oneMKL
+- multi_gpu_support: one rank per GPU documented (Frontier `--ntasks-per-gpu 1 --gpu-bind closest`; Perlmutter `--gpus-per-task=1`); oversubscription seen in test scripts (18 ranks / 6 GPUs); threads = 1
+- multi_node_support: yes (to ~40,000 GPUs; Summit benchmarks)
+- gpu_aware_mpi_requirement: optional (`WITH_GPU_AWARE_MPI`, "use with care"; NCCL alternative)
+- rank_to_gpu_binding: self-binding (`device_id = mpi_rank % n_devices`, global rank)
+- topology_decomposition_controls: `.prm` `subsection Parallelization { NPKPT, NPBAND, BAND PARAL OPT }`, `subsection GPU { USE GPU, AUTO GPU BLOCK SIZES, USE GPUDIRECT MPI ALL REDUCE, USE ELPA GPU KERNEL }`, `USE ELPA`; ranks divisible by `NPKPT*NPBAND`
+- major_dependencies (install_DFTFE pins): deal.II >= 9.5.1 (`+P4EST +64BIT_INDICES +MPI +LAPACK`; scripts 9.6.2/9.7.1), p4est 2.8.6/7, Kokkos 4.3/4.6 (CPU-only, for deal.II), Boost 1.86, ScaLAPACK 2.2.x, BLIS/libflame or OpenBLAS, ELPA 2025.01/2025.06 (`--enable-nvidia-gpu-kernels --with-NVIDIA-GPU-compute-capability=sm_80`), ALGLIB, libxc 6.2.2/7.0.0, spglib, libxml2, numdiff (tests); optional PETSc/SLEPc, NCCL, dftd3/4, libtorch
+- dependency_complexity: high
+- official_inputs_datasets: `demo/ex1-3` (with reference `.output`), `testsGPU/pseudopotential/{real,complex}` (56 GPU `.prm` cases, `accuracyBenchmarks/`, `diffScript`, Slurm/PBS scripts), `tests/dft/*` (112 `.prm.in`, 133 `*.mpirun=N.output` references), 57 ONCV pseudopotentials (11 MB), `data/` 73 MB; external `dftfe-benchmarks` (Mo 431-8,191 atoms; Al nanoparticles)
+- correctness_mechanism: ctest via deal.II harness with `numdiff` against `.output`; GPU: `REPRODUCIBLE OUTPUT = true` + `diffScript` vs `accuracyBenchmarks/`
+- strong_scaling_input_availability: indirect (any fixed system on 1..4 GPUs; Summit references per size in the benchmarks repo)
+- weak_scaling_input_availability: `dftfe-benchmarks` Mo N x N x N supercell series (24 -> 3,600 GPUs); in-repo inputs do not scale automatically
+- expected_build_time: deal.II 1-1.5 h; ELPA-GPU 15-30 min; small libs 20 min; DFT-FE real + complex 25-40 min each -> ~3-4 h
+- expected_disk_usage: source 126 MB; deps 8-12 GB; DFT-FE 3-4 GB
+- expected_input_data_size: ~85 MB in-repo; benchmarks repo external
+- b200_cuda132_risk: medium (arch is user-supplied and kernels generic; never run above sm_80 upstream; ELPA `sm_100` acceptance and ELPA 2025.x with CUDA 13.2 unverified -> ELPA CPU/ScaLAPACK fallback; NCCL for CUDA 13 separate; scripts assume Cray/icpx)
+- mi355x_hip_risk: medium-high
+- container_availability: CPU Docker recipe only (`install_DFTFE` `generalUbuntuCPU`); no GPU image
+- spack_availability: `dftfe` package stale (0.5-0.6, 2019, no GPU variants); upstream route is the per-machine `install_DFTFE` scripts (not Spack)
+- recommended_integration_priority: SECOND_BATCH
+- blocker: ~~multi-hour stack~~ built per profile (OpenBLAS -> ScaLAPACK 2.2.2 -> libxc 7.0.0 -> spglib -> ALGLIB -> p4est 2.8.7 -> Kokkos 4.6.00 Serial -> deal.II 9.6.2 -> ELPA 2026.02.001 -> DFT-FE); the Cray/icpx script adaptations needed were small but real (dftfe's `p4est-setup.sh` hardcodes `CC=cc CXX=CC FC=ftn`, relies on implicit `-lm`, and checks a `src/p4est_config.h` that p4est 2.8.7 writes to `config/`); NCCL not used (`WITH_DCCL=OFF`); numdiff absent -> own numeric comparison (`dftfe_check.py`, pre-fixed tolerances) against upstream's GPU `accuracyBenchmarks/`; ELPA sm_100 kernels: build accepted (`--enable-nvidia-gpu-kernels --with-NVIDIA-GPU-compute-capability=sm_100`; ELPA's configure needs the host SIMD flags in CFLAGS and the ScaLAPACK paths in LDFLAGS), `elpa_probe.sh` (ELPA's own analytic-matrix GPU test programs on 1/2/4 GPUs, ELPA's eigenvalue/eigenvector error limits 5e-14 / 6e-10) -- PASS on all six runs (errors <= 7.3e-15 / 1.0e-11), details in `SECOND_BATCH_STATUS.md`. **deal.II version:** release 1.2.0 does not compile against deal.II 9.7.1 (the version the current install_DFTFE recipe pairs with dftfe *develop*): 9.7 removed `Utilities::MPI::create_group`, `Triangulation::load(name, autopartition)` and `VtkFlags::ZlibCompressionLevel`; deal.II 9.6.2 (deprecated but present) is used, plus a 2-line `std::isnan` fix for GCC 14
+- build_strategy_notes: **NATIVE** (recipe transcribed from `install_DFTFE` branch `frontierDevelop` -- `dftfe2.sh`/`setupUser.sh` -- to **system GCC 14.2.1 for C/C++/Fortran** through the conda Open MPI 5.0.10 wrappers (the audit's "conda GCC 13.3 + system gfortran" mix was dropped: one compiler for all three languages), OpenBLAS/ScaLAPACK, Kokkos Serial, deal.II 9.7.1, ELPA with `--enable-nvidia-gpu-kernels --with-NVIDIA-GPU-compute-capability=sm_100`, DFT-FE real `CMAKE_CUDA_ARCHITECTURES=100 WITH_DCCL=OFF WITH_GPU_AWARE_MPI=OFF USE_64BIT_INT=ON`; complex build on request; `WITH_TESTING=OFF` because numdiff is absent -- the GPU test decks are driven by `run.sh`/`validate.sh` instead)
+
+---
+
+## Cross-cutting findings
+
+1. **Spack is not a shortcut to Blackwell for any candidate on this node.** The
+   personal checkout (2025-05-06) predates every 2026 release and lacks
+   `cuda_arch=100`/`gfx950`; upstream recipes are missing (SPARTA, Nyx, GEOS,
+   SPECFEM3D), stale (nekRS 23.0, DFT-FE 0.6, QMCPACK `+cuda` inert for 4.x),
+   or reject `cuda_arch=100` outright (`cp2k`, `dbcsr`; `raja ^cuda@13:`
+   conflict blocks GEOS' path). Where Spack helps it is for CPU-side
+   dependencies only (CP2K, QMCPACK) -> `NATIVE+SPACK_DEPS`.
+2. **Containers do not apply here**: no Apptainer/Singularity on dgx003, no
+   official B200 image for any candidate, and containers would not provide the
+   host driver, the CUDA-aware Open MPI transport or the network.
+3. **Blackwell/CUDA 13 upstream coverage** is thin everywhere: only WarpX
+   (Perlmutter profile, CUDA 13.2.78) and the two Kokkos codes (arch flag +
+   CUDA 13 fixes in the bundled Kokkos) have first-class support; SPECFEM3D
+   v4.1.1 needs two back-ports; CP2K 2026.2/DBCSR 2.10.0, GEOS, DFT-FE never
+   mention sm_100; GEOS explicitly lists CUDA 13 as blocked in its Spack path.
+4. **Fortran**: the conda environment has no gfortran; the system gfortran
+   14.2.1 works with the conda Open MPI Fortran bindings (`OMPI_FC=/usr/bin/gfortran`,
+   verified with a 2-rank MPI Fortran program). Needed by SPECFEM3D, nekRS,
+   CP2K, DFT-FE (ScaLAPACK/ELPA/p4est).
+5. **Rank -> GPU mapping**: LAMMPS/SPARTA/nekRS bind by node-local rank,
+   WarpX/Nyx by AMReX's rank-in-node, SPECFEM3D/CP2K/DFT-FE by *global* rank
+   modulo device count, GEOS not at all. The Level 3 launcher's per-rank
+   wrapper (one visible GPU per rank, audited) makes all of them correct on a
+   node; nekRS additionally needs `--device-id 0` under the wrapper.
+6. **Multi-node MPI is BLOCKED/UNVERIFIED on this site**; every application
+   above is multi-node capable per upstream, none is verified beyond one node.
diff --git a/level3/BUILD_STRATEGY.md b/level3/BUILD_STRATEGY.md
new file mode 100644
index 0000000..10a248c
--- /dev/null
+++ b/level3/BUILD_STRATEGY.md
@@ -0,0 +1,127 @@
+# Level 3 build strategy
+
+Companion to `APPLICATION_AUDIT.md`. For each application the four build
+routes were assessed against what upstream documents and what dgx003 provides
+(RHEL 10, 4x B200, CUDA 13.2.78, conda GCC 13.3.0 / Open MPI 5.0.10
+CUDA-aware / CMake 3.28.4, system gfortran 14.2.1, no ROCm, no
+Apptainer/Singularity, lmod broken, Spack 1.0.0.dev0 checkout of 2025-05).
+Vocabulary: `BUILD_RECOMMENDATION = NATIVE | SPACK | NATIVE+SPACK_DEPS |
+APPTAINER | SPACK+APPTAINER | SITE_NATIVE | DEFER`.
+
+## Decision matrix
+
+| Application | Native (upstream build system) | Spack | Apptainer/Singularity | Site-native modules | BUILD_RECOMMENDATION |
+|---|---|---|---|---|---|
+| LAMMPS | feasible, documented (CMake presets, bundled Kokkos 4.6.2 with BLACKWELL100 + CUDA 13 back-ports); deps = MPI + CUDA; **built in 220 s** | package exists but local checkout has no `cuda_arch=100`; upstream recipe would force EXTERNAL_KOKKOS 4.7.1 = configuration upstream calls "untested"; upstream does not recommend Spack | no runtime; no official image (NGC image sm_90-max, 2023) | lmod broken | **NATIVE** |
+| SPARTA | feasible, documented (`cmake -C presets/kokkos_cuda.cmake -DKokkos_ARCH_BLACKWELL100=ON`); deps = MPI + CUDA; **built in 579 s** | **no package** (Spack's `sparta` is a bioinformatics tool) | no runtime; no recipe | lmod broken | **NATIVE** |
+| WarpX | feasible, documented superbuild; small graph (AMReX local checkout, PICSAR/openPMD off); CUDA vs HIP one switch; upstream itself builds with CUDA 13.2 | possible only with a fresh spack-packages (26.09 missing; arch via `^amrex cuda_arch=100` legacy path); graph balloons with `+openpmd +python` | no runtime; only Perlmutter Containerfiles (sm_80) | lmod broken | **NATIVE** |
+| SPECFEM3D Cartesian | feasible, only documented route (autotools; bundled SCOTCH); needs two devel back-ports for CUDA 13 + make-time `GENCODE` for sm_100; Fortran via system gfortran + `OMPI_FC` | no package | none; tiny dependency graph, nothing to gain | lmod broken | **NATIVE** |
+| nekRS | feasible, only documented route (CMake; all TPLs vendored); CUDA/HIP separable by OCCA options; two build variants (see below) | recipe stale (23.0, wrong option names); deps vendored -> nothing for Spack to provide | none official; JIT couples to host toolchain | lmod broken | **NATIVE** |
+| CP2K | feasible (CMake + `install_cp2k_toolchain.sh`); ~15 packages for a GPU-DFT build; needs the DBCSR B200 sed upstream master applies; 3-6 h | officially recommended (`make_cp2k.sh`, `spack install cp2k+cuda`) **but `cp2k` and `dbcsr` recipes hard-reject `cuda_arch=100`**; 60-100 packages; local checkout too old | official `cp2k/cp2k` images stop at H100; multi-node needs host MPI; no runtime here | lmod broken | **NATIVE+SPACK_DEPS** |
+| Nyx | feasible, only documented path (CMake superbuild or GNU make); can consume the WarpX AMReX 26.09 checkout (`AMREX_MINIMUM_VERSION 20.11`); SUNDIALS CUDA superbuild for HEATCOOL | no package | none | lmod broken | **NATIVE** |
+| QMCPACK | feasible today only for `QMC_GPU=cuda` (partial GPU); the recommended `openmp;cuda` needs Clang with NVPTX offload (absent) + Boost (absent) + tested HDF5 | package's `+cuda` is inert for 4.x (no `QMC_GPU`), no offload variant -> use Spack only for `llvm+cuda`, `boost`, `hdf5@1.14` | CI dependency images only (CPU) | lmod broken | **NATIVE+SPACK_DEPS** |
+| GEOS | feasible, documented (thirdPartyLibs superbuild + host-config); ~20 TPLs, 4-5 h; `ENABLE_TRILINOS=OFF` with hypre on device; develop can share Level 2's RAJA/CHAI/Umpire 2026.07.0 | no upstream package; uberenv recipe is LC-only and hits the `raja ^cuda@13:` conflict upstream calls blocking | CI TPL images (CUDA <= 12.9, in-container MPI) | lmod broken | **NATIVE** |
+| DFT-FE | feasible, only documented route (per-machine `install_DFTFE` shell scripts + CMake); ~12 autotools/CMake deps, 3-4 h; CUDA vs HIP = rebuild of DFT-FE + ELPA only | `dftfe` recipe unusable (0.6); a dealii/elpa/libxc hybrid possible but local Spack too old and ELPA `cuda_arch=100` unverified | CPU Docker recipe only | lmod broken | **NATIVE** |
+
+No candidate gets `DEFER`: every one has an officially supported native CUDA
+path on this toolchain. `APPTAINER`/`SPACK+APPTAINER`/`SITE_NATIVE` are
+unavailable on this node regardless of application.
+
+nekRS has two verified variants (both NATIVE; select with
+`HPCPERF_NEKRS_HYPRE_GPU`/`HPCPERF_NEKRS_VARIANT`; isolated src/build/install/JIT
+cache; see `level3/nekrs/COMPATIBILITY.md`):
+- `hypregpu` (`ENABLE_HYPRE_GPU=ON`, 3 patches) -- required for a GPU (DEVICE)
+  HYPRE coarse solve; verified with cimode 3 (GPU coarse) at 1/4 GPU.
+- `cpucoarse` (`ENABLE_HYPRE_GPU=OFF`, 0 patches, 113 s build) -- covers the
+  current Ethier CPU-coarse workload; a DEVICE-coarse request is rejected by
+  nekRS itself, not silently down-graded. Recommended default only once the
+  workload's coarse-solver placement is fixed by review; the CPU-coarse option's
+  scalability at 40/80 GPUs is UNVERIFIED.
+
+## Spack policy (Level 3)
+
+- Spack is used **only** where upstream documents it as a supported route and
+  the recipe can express the target (`cuda_arch=100`, CUDA 13.2 external,
+  conda Open MPI external). In this round that is nowhere; the two
+  `NATIVE+SPACK_DEPS` candidates (CP2K, QMCPACK) were **brought up without
+  Spack in the second batch** (CP2K: upstream's own toolchain script is the
+  documented dependency route and already pins every version; QMCPACK: the
+  private LLVM/HDF5/Boost builds are 3 tarballs with recorded SHA-256s) -- so
+  every `spack_lock_sha256=` stays `none` and no `spack.yaml` exists. Their
+  matrix rows below are therefore realised as `NATIVE`.
+- When used, each application/backend gets its own environment
+  `level3/envs/<app>/{cuda,rocm}/spack.yaml` with a committed `spack.lock`;
+  the lock's SHA-256 is recorded in the application fingerprint
+  (`spack_lock_sha256=` line, currently `none`). CUDA and ROCm environments are
+  never merged (they cannot concretize together here anyway: no ROCm).
+- The personal Spack checkout (`/projects/kzhou6/bcui2/env_software/spack`,
+  1.0.0.dev0, 2025-05-06) is too old for every 2026 release and lacks
+  Blackwell; a Level 3 Spack environment will need a fresh Spack >= 1.2 with
+  `spack-packages` >= 2026-07 and the conda Open MPI/CUDA declared as externals.
+  This is recorded as a prerequisite, not done in this round.
+
+## Apptainer policy (Level 3)
+
+- Not applicable on dgx003 (no runtime). If a future site provides Apptainer:
+  only the `.def`, a README, the build script and the image SHA-256 are
+  committed (never a `.sif`); the definition must pin the base image digest and
+  the application commit; containers do not solve the host driver, the MPI
+  transport or the interconnect, and multi-node runs still require the host
+  MPI/PMIx -- so a container build is treated as one more `Build Strategy`
+  variant with its own fingerprint, never as the default.
+
+## Per-application dependency isolation
+
+Every Level 3 application owns `.deps/level3/<app>/{src,build,install,logs}`
+(private copy of patched sources where a build must be in-tree or patched,
+private dependency builds, install prefix, logs) plus the fingerprint
+`.deps/level3/<app>/install/.hpcperf-l3-fingerprint` written by
+`level3/tools/l3_common.sh` (schema `l3-1`: upstream commit, backend/arch,
+dependency versions, compiler, Fortran compiler, CUDA/ROCm, MPI, CMake options,
+GPU-aware MPI setting, site profile, Spack lock SHA-256, container SHA-256,
+patch list, build time). A fingerprint mismatch makes `build.sh` fail fast with
+the differing lines. Nothing under `.deps/install/` (Level 2) is modified or
+reused: the Level 2 Kokkos 5.2.1 / RAJA suite / hypre / AMReX pins are not the
+versions these applications validate against (LAMMPS pins Kokkos 4.6.2, SPARTA
+5.0.2, WarpX AMReX 26.09, nekRS hypre 2.32.0), and Level 2 must keep building.
+
+## Modification classes used in the first batch
+
+| Application | Class | What |
+|---|---|---|
+| LAMMPS | A | none; derived `in.lj` deck (`run ${steps}`, weak `processors`) written into the build tree |
+| SPARTA | A | none |
+| WarpX | A | none; derived inputs file (sizes, `warpx.numprocs`, reduced diagnostics) written into the build tree |
+| SPECFEM3D | B + C + D | make-time `GENCODE` override (sm_100, devel's `cuda13` value) and bundled SCOTCH built without gzip support (generated `Makefile.inc`, no `zlib.h` in the conda sysroot); `OMPI_FC=/usr/bin/gfortran`, `MPI_INC`; two devel back-ports (CUDA 13 `deviceOverlap` guard 10 lines, Blackwell device block 8 lines) in `level3/specfem3d/patches/` |
+| nekRS | B + C + D | B: `cmake/hypre.cmake` `HYPRE_CUDA_SM=80 90` -> `80 90 100` (1 line), CMake generator pinned to upstream's Unix Makefiles (the conda `CMAKE_GENERATOR=Ninja` yields an invalid rule for the vendored HYPRE install step); C: `OMPI_FC=/usr/bin/gfortran`, `LDFLAGS+=-fno-lto` (mixed GCC 13 / gfortran 14 LTO bytecode in CMake's Fortran/C detection), `FFLAGS+=-fPIC` (PIE default of the conda GCC), `unset AR` (HYPRE's configure takes `$AR` as the full archive command), run time `unset CMAKE_GENERATOR` (UDF build), `OMPI_MCA_osc=^ucx` (one-sided ops otherwise go through UCX and abort with 4 ranks), `ulimit -s unlimited` (as upstream's job scripts); D: vendored HYPRE 2.32.0 vs the Thrust 3.2 shipped with CUDA 13 -- `thrust::pair` result type -> `auto` (2 lines), explicit `<thrust/iterator/reverse_iterator.h>`/`<thrust/pair.h>` includes in `device_utils.h` and in the pre-generated concatenated `_hypre_utilities.hpp`, `thrust::not1` -> `thrust::not_fn` (16 lines); all in `level3/nekrs/patches/` |
+
+No class E change anywhere; no numerics, physics or algorithm touched. The
+nekRS list shows the general pattern for Fortran + CMake applications on this
+node (CP2K and DFT-FE will meet the same OMPI_FC / LTO / PIE issues) and that
+nekRS' vendored HYPRE 2.32.0 is not CUDA 13-ready as shipped.
+
+## Second batch (2026-09-05/06): what was actually built and how
+
+Per-profile isolation was added for this batch: `.deps/level3/<app>/<profile>/{src,build,install,logs,cache}`
+and `build/level3/<app>/<profile>/` (`l3_paths_profile`), the profile naming the
+compiler/Toolkit/backend/key-dependency combination (`cuda132-gcc142-ompi5010`,
+`clang231-cuda132-offload`, `cuda132-gcc133-adiabatic|heatcool`, ...). Two
+differently configured builds never share a source, build or install tree, and
+the fingerprint (schema `l3-2`; the profile and every dependency version/SHA are
+part of the `dependencies=` line) refuses a mismatching reuse. Source and
+build trees that suffer on NFS (LLVM 23, the CP2K toolchain -- 150k+ small files)
+live on local `/tmp/hpcperf-l3-b2-scratch/`; installs, logs and fingerprints stay
+under `.deps/`.
+
+| Application | Route actually used | Class | What |
+|---|---|---|---|
+| Nyx 26.09 | NATIVE; external **AMReX 26.09 built per profile** (the pinned submodule `6e875b7c` cannot emit sm_100), SUNDIALS 7.2.1 CUDA per profile for HEATCOOL; CPU-backend profile for cross-references | A | none in Nyx/AMReX/SUNDIALS; derived decks (step counts, fixed `amr.max_grid_size`/`refine_grid_layout=0`, checkpoint output, synthetic `RandomPerCell` sizes) written into the build tree, upstream decks otherwise verbatim |
+| CP2K v2026.2 | NATIVE + upstream `install_cp2k_toolchain.sh` (OpenBLAS/ScaLAPACK/FFTW/libint/libxc/LIBXSMM/spglib/DBCSR); **no Spack** (recipes cap `cuda_arch` at 90) | B + C | B: back-port of upstream commit `378b2fab` (B200 in the toolchain: `--gpu-ver=B200 -> ARCH_NUM 100`, DBCSR `GPU_ARCH_NUMBER_B200 100` + `parameters_H100.json -> parameters_B200.json`) -- `level3/cp2k/patches/0001`; C: toolchain copy on local scratch (DBCSR's `GetGitRevisionDescription.cmake` aborts inside a git worktree), `PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe` for DBCSR's own `mpiexec -n 4` ctest launches, `unset -f grep` (the login shell exports a `grep` function), conda build variables cleared, BLAS/LAPACK given explicitly as the toolchain `libopenblas.a` (`CP2K_BLAS_VENDOR=CUSTOM`) with the toolchain library directories first in the RPATH of libcp2k.so/cp2k.psmp (`CMAKE_INSTALL_RPATH` + `-Wl,-rpath` linker flags; attempt 1 had inherited conda's `LDFLAGS` -- `-Wl,--disable-new-dtags -rpath <conda lib>` -- so `libopenblas.so.0` resolved to the conda pthreads OpenBLAS at run time, attempt 2 with `CMAKE_INSTALL_RPATH` alone still lost to the MPI wrapper's rpath; run.sh refuses to run unless `ldd` resolves BLAS under the toolchain and records `blas_resolved=`), `setup` still sourced at run time |
+| QMCPACK v4.4.0 | NATIVE + **private toolchain** (LLVM 23.1.0 from source with the NVPTX offload runtime, HDF5 1.14.5 parallel, Boost 1.90 headers, OpenBLAS 0.3.30); no Spack (recipe inert for 4.x GPU options) | A (+ C) | none in QMCPACK; C: `-DCMAKE_IGNORE_PATH=/usr/lib64/cmake/ZLIB;/lib64/cmake/ZLIB` for HDF5 (the node's zlib-ng CMake package references an absent `libz.a`); derived decks change only the walker-population parameter (strong/weak) |
+| DFT-FE 1.2.0 | NATIVE; recipe transcribed from `install_DFTFE` (`frontierDevelop`) with **system GCC 14.2.1 for C/C++/Fortran**; all 9 dependencies per profile; **deal.II 9.6.2** (attempt 1 with 9.7.1 -- the version the current recipe pairs with dftfe *develop* -- fails: 9.7 removed `Utilities::MPI::create_group`, `Triangulation::load(name, autopartition)`, `VtkFlags::ZlibCompressionLevel` that release 1.2.0 uses; 9.6.2 keeps them deprecated, so no API back-port) | C + D | D: `level3/dftfe/patches/0001-std-isnan.patch` -- two unqualified `isnan(` calls in a template of `src/atom/AtomicCenteredNonLocalOperator.cc` qualified as `std::isnan(` (GCC 14 rejects the unqualified spelling; same function, no numerical change); C: dftfe's `p4est-setup.sh` given `CC=mpicc CXX=mpicxx FC=mpifort F77=mpifort LIBS=-lm` (Cray wrappers hardcoded, implicit libm) and its zlib check pointed at p4est 2.8.7's `config/p4est_config.h`; conda build variables cleared; `TARGET=SAPPHIRERAPIDS` for OpenBLAS; ELPA configured with `-march=native` (its AVX-512 probe needs the SIMD flags in CFLAGS) and the ScaLAPACK/OpenBLAS paths in `LDFLAGS` (its cublas link check drops `SCALAPACK_LDFLAGS`); no deal.II/ELPA/p4est source change |
+| GEOS develop `b7a0f133` + TPL `9b55672` | NATIVE: upstream `thirdPartyLibs` superbuild (`config-build.py -n`) + private host-config; TPLs per profile (not Level 2's RAJA/CHAI/Umpire installs: different compiler/CUDA flags); GEOS 1718 s at `-j32` | B + C + D | B (thirdPartyLibs, `level3/geos/patches/`): `0001` 65-character `SUPERLU_URL_HASH` typo, `0002` `RAJA_ENABLE_VECTORIZATION` overridable (built OFF: nvcc 13.2 + GCC 14/x86-64-v3 cannot compile RAJA's AVX2 tensor layer; upstream's own ROCm choice), `0003` hdf5 step gets `CMAKE_GENERATOR ${TPL_GENERATOR}` and `${TPL_BUILD_COMMAND}` like every other step (hardcoded `make` breaks under Ninja); D (BLT submodule, `geos-blt-0001-cuda13-memoryClockRate.patch`): back-port of LLNL/blt `38b46203` -- BLT's CUDA runtime smoke-test program reads `cudaDeviceProp::memoryClockRate`, removed in CUDA 13, now `cudaDeviceGetAttribute` for `CUDART_VERSION >= 13000`; test/benchmark switches left at upstream's default ON; C: configure-once guard (`config-build.py` deletes an existing build tree), stale-stamp cleanup; host-config `ENABLE_HYPREDRV=OFF` (GEOS-documented option; the TPL hypredrive step lacks Umpire's include path, and the beam workflow does not use hypredrive) and `-I<TPL metis/parmetis include>` ahead of the `-isystem` conda MPI include dir (which carries an unrelated 32-bit `metis.h` that broke GEOS' 64-bit ParMETIS assertion); no GEOS/LvArray/hypre source change |
+
+Still no class E change; no numerics, physics, precision, solver placement or
+input tolerance was touched. Every patch file carries its source, rationale,
+conditions, impact and verification in its header and its hash is in the
+profile fingerprint.
diff --git a/level3/CORRECTNESS_FIXES.md b/level3/CORRECTNESS_FIXES.md
new file mode 100644
index 0000000..338159b
--- /dev/null
+++ b/level3/CORRECTNESS_FIXES.md
@@ -0,0 +1,101 @@
+# Level 3 first batch -- correctness / reproducibility fixes
+
+Round after the da35285 review. Base for this work: HEAD was
+`da352857561daa8f754161a856d0d53875d6f3ad` (verified; working tree clean at
+start). No application source, patch, input, or tolerance was changed to obtain
+a PASS; the five applications, their patches, inputs and prior results are kept.
+
+Shared mechanism lives in `level3/tools/l3_common.sh` (+ `l3_check.py`); the
+CPU-only tests are `level3/tools/tests/` (`run_all.sh` -> `test_l3_infra.sh`,
+13/13 passing).
+
+## 1. False PASS / exit codes
+
+| Review point | Fix | Where |
+|---|---|---|
+| A failed run must not PASS on a stale log | validators delete/rewrite only this-run output; run.sh removes its target log before launching; validators require the run's real exit code == 0 | all `validate.sh`; `lammps/sparta run.sh` `rm -f "$LOG"` |
+| SPECFEM solver failure swallowed by `\| tee \| grep \|\| true` | solver now runs to a file and its exit code is captured directly (`rc=0; cmd > log \|\| rc=$?`); the grep is display-only afterwards | `specfem3d/run.sh` |
+| Separate execution from log filtering; capture launcher/app/validator real exit codes | validators run the app into a stdout file and gate on `rc`; `l3_capture` returns the command's status, not tee's; LAMMPS/SPARTA run.sh use `\|\| rc=$?` (not `; rc=$?` which `set -e` would abort) so the code and manifest are always recorded | `l3_common.sh` `l3_capture`; every `run.sh`/`validate.sh` |
+| timeout / missing file / analysis exception / nonzero -> FAIL; validate only new output | `timeout` wraps each run (rc 124 -> FAIL); missing output -> FAIL; python raises `ValidationError` -> FAIL | every `validate.sh` |
+
+Negative test: `test_l3_infra.sh` case 3 shows a failed run (rc=1) FAILs the gate
+even when a PASS-looking stale log is present; case 2 shows `l3_capture` returns
+the real code (7), not tee's 0.
+
+## 2. Numerical finiteness / completeness
+
+| Review point | Fix |
+|---|---|
+| Reject NaN/Inf in data/reference/error | `l3_check.require_finite` names and rejects any non-finite compared quantity; used by LAMMPS/SPARTA/WarpX validators and the SPECFEM sample scan |
+| LAMMPS: expected thermo fields + final step | requires Step 0 and Step 100 rows and the fields Temp/E_pair/TotEng/Press present, else FAIL |
+| SPARTA: final step, complete stats interval, fields | requires the benchmark block to span steps 30..130 (equilibration boundary to final) and the fields Np/temp/Natt present |
+| SPECFEM: all required reference traces, sampling range, comparison | requires the compared-trace count == number of REF_SEIS traces (12), and scans every produced `.semd` for non-finite samples before trusting the correlation |
+| nekRS: complete cimode check set (not passed>0) | requires `passed+failed == EXPECT_CHECKS` (9) AND failed==0 AND rc==0 AND coarse-location matches the cimode |
+| WarpX: final step, expected particle count, field completeness, reader robustness | reader FAILs on missing field, box/fab mismatch, a truncated FAB, or boxes not covering 100% of the domain; requires the langmuir plotfile at step 40 and the uniform-plasma NP series to reach step 10 with a constant finite count |
+| Adapted-subset labelling | LAMMPS/SPARTA validators print "adapted subset" and name exactly what upstream check they re-implement |
+| CPU-only negative tests | `test_l3_infra.sh` (NaN/Inf rejection, rc-gate, dry-run sentinel, patch/cache) |
+
+Thresholds unchanged (LAMMPS 1e-8/1e-5; SPARTA Np-exact/2%/15%; WarpX 5e-2/1e-11;
+SPECFEM 0.8/1%/0.01s; nekRS EPS 0.3).
+
+## 3. Result management
+
+| Review point | Fix |
+|---|---|
+| Unique run_id; full stdout/stderr, command, exit code, source/binary/input hash | `l3_run_id` + `l3_manifest` write `run_manifest.txt` per real run with run_id, exit_code, binary+input sha256, backend, ranks, sizes, timer; the launcher already logs the command and per-rank GPU audit into the captured stdout |
+| backend, GPU/rank/node, CPU/GPU binding, transport, timer, validation | recorded in the manifest and in the launcher lines of the captured stdout |
+| dry-run must not delete/overwrite/rewrite real results; sentinel test | `l3_rundir` routes a dry-run to a `.dryrun/` scratch dir and refuses paths outside `build/level3/`; WarpX/SPECFEM/nekRS run.sh (which `rm -rf`'d the real dir before the dry-run check) now go through it; LAMMPS/SPARTA redirect their log into `.dryrun/`. Verified live: an 8-GPU dry-run left a real `log.smoke.np1` untouched and used `.dryrun/` (`test_l3_infra` case 4 + the live sentinel run) |
+| Historical UNKNOWN exit codes stay UNKNOWN | not back-filled; the review bundle already labels them UNKNOWN |
+
+## 4. Fingerprint / cache
+
+| Review point | Fix |
+|---|---|
+| Patch full path + ordered content hash, not basename | `l3_fingerprint_text` records `patch[i]=<name> sha256=<hash>` in order and a `patch_series_sha256` over the ordered contents (schema bumped l3-1 -> l3-2) |
+| Missing / unhashable patch -> error | `l3_fingerprint_text` returns non-zero on a missing patch; `nekrs/build.sh` also checks each patch exists before building |
+| Source-cache key includes upstream SHA + patch content hash; renamed-but-changed invalidates | nekRS src stamp is now `SHA <ordered-patch-content-hash>` (was basenames) |
+| build/install/cache isolated by backend/toolchain/dependency/config | per-app `.deps/level3/<app>/{src,build,install,logs}`; nekRS is further split per variant (`hypregpu` legacy paths, others under `.deps/level3/nekrs/<variant>/` with their own build dir and JIT cache) |
+| Verify binary backend before running (no HIP request on a CUDA install) | `l3_binary_backend_check` (libcudart vs libamdhip64) available in `l3_common.sh` |
+| Post-hoc fingerprints marked | `l3_fingerprint_write` stamps `built=<UTC> (build-time record)`; nothing back-dates |
+
+Negative test: `test_l3_infra` case 5 (missing patch -> error; same-name changed
+content -> different series hash; empty series -> `none`).
+
+## 5. Dependency isolation
+
+| Review point | Fix / finding |
+|---|---|
+| Check LAMMPS/SPARTA actual Kokkos helper source | The recorded (polluted-env) `CMakeCache.txt` had `Kokkos_NVCC_WRAPPER`/`Kokkos_COMPILE_LAUNCHER` pointing at **Level 2's** `.deps/install/kokkos`; the actual compile/link commands had **0** Level 2 references and used the bundled `nvcc_wrapper`+includes (so the binaries were clean, the cache entry was an inert stale detection). |
+| Reconfigure without Level 2 prefixes; rebuild only affected apps | `l3_isolate_build_env` strips `$R/.deps/install/` from CMAKE_PREFIX_PATH/LD_LIBRARY_PATH in every build.sh. LAMMPS + SPARTA rebuilt in the isolated env (221 s / 570 s): `CMakeCache.txt` now has 0 Level 2 refs and `Kokkos_NVCC_WRAPPER` points at the bundled Kokkos. WarpX/nekRS/SPECFEM already had 0 refs (CXX = conda/mpicxx, autotools); the isolation call was added to their build.sh too but they were not rebuilt. Re-validated LAMMPS/SPARTA 1/2/4 -> identical numbers, PASS. |
+| env_profiles base/head 9/11 tracked separately, not green | unchanged Level 2 issue (conda cmake activation drops a user CMAKE_PREFIX_PATH); documented in the review bundle `50_issues/env_profiles`; NOT a Level 3 regression and NOT marked passing here. |
+
+## 6. Status / build strategy
+
+| Review point | Fix |
+|---|---|
+| Separate PASS (smoke/analytic) from COMPLETED (strong/weak) | status table distinguishes validated-correctness runs from run-completed runs; strong/weak remain COMPLETED unless a numerical criterion applies |
+| printed-values-equal != full bitwise | wording corrected: LAMMPS reports the four state variables agree to printed precision, not full-state bitwise identity |
+| Don't widen to untested algorithm paths | LAMMPS stays LJ; WarpX stays FFT=OFF Yee PIC; no new solver paths added |
+| Spack doc vs local facts; concretization NOT_RUN | recorded in the review bundle `50_issues/build_strategy` (local Spack 1.0.0.dev0, recipe versions) and `COMPATIBILITY.md`; no concretization run this round |
+| Missing container runtime is an environment limit, not a feasibility verdict | stated as such; only podman present, not evaluated |
+
+## nekRS-specific (see COMPATIBILITY.md)
+
+- Confirmed the six logs' `COARSE SOLVER LOCATION: CPU` against upstream source
+  defaults and `ci.inc`: cimode 2 = CPU coarse (GPU main app), cimode 3 = DEVICE
+  (GPU) coarse.
+- The cimode-2 "9/9" validates the CUDA main app + CPU coarse only. GPU HYPRE is
+  now separately verified with cimode 3 (`hypregpu` variant): 9/9, coarse=DEVICE,
+  at 1 and 4 GPUs.
+- The `pair`/`reverse_iterator` build errors are missing-include (visibility)
+  issues; only `thrust::not1` is a genuinely removed API. Patches are labelled
+  project-local (no upstream backport SHA located).
+- Minimal candidate `cpucoarse` (`ENABLE_HYPRE_GPU=OFF`, **0 patches**, isolated
+  variant tree, 113 s build): cimode 2 (CPU coarse) PASS 1/2/4 GPU (9/9,
+  coarse=CPU, main app on distinct GPUs); cimode 3 (DEVICE requested) is
+  **explicitly rejected** by nekRS (`HYPRE+DEVICE not enabled! Recompile with
+  -DENABLE_HYPRE_GPU=ON`, exit 1) -- no silent CPU fallback. So the three patches
+  are needed only for GPU HYPRE; the current CPU-coarse workload needs none.
+- `hypregpu` (patched) cimode 3 (DEVICE/GPU coarse) PASS 1/4 GPU (9/9,
+  coarse=DEVICE): the GPU HYPRE coarse solve the patches enable is verified
+  correct, not merely compiled.
diff --git a/level3/README.md b/level3/README.md
index 93d36f7..8e920d6 100644
--- a/level3/README.md
+++ b/level3/README.md
@@ -1,19 +1,179 @@
 # Level 3: Production / End-to-End HPC Applications
 
-Not implemented yet.
+Level 3 integrates **full production applications** -- complete workflows, not
+extracted kernels, not proxies, and never N independent replicas presented as
+one distributed run. Its primary execution mode is **multi-GPU with a
+user-selected GPU count**; single-GPU runs exist only for build smoke tests,
+environment compatibility and basic correctness bring-up.
 
-Level 3 covers production or end-to-end HPC applications. These may involve
-multi-GPU, multi-node execution, job schedulers, and external libraries, so
-the Level 1 layout is not enforced here.
+Branch `level3/second-batch-bringup` (contains the first-batch branch
+`level3/full-apps-bringup`, from `main`): STEP 1-2 audit and build strategy for
+all ten candidates ([APPLICATION_AUDIT.md](APPLICATION_AUDIT.md),
+[BUILD_STRATEGY.md](BUILD_STRATEGY.md)), STEP 3-6 first-batch bring-up on
+dgx003 (4x B200, CUDA 13.2.78, Slurm job 9552083, 2026-09-04), second batch
+2026-09-05/06 ([SECOND_BATCH_STATUS.md](SECOND_BATCH_STATUS.md)), validator
+rework and joint-HEAD regression 2026-09-07/08. Nothing is claimed validated
+beyond what the per-application README records for runs that actually happened
+on this node.
 
-Planned layout (reference only):
+Correctness / reproducibility hardening (post-review, 2026-09-05):
+[CORRECTNESS_FIXES.md](CORRECTNESS_FIXES.md) -- validators now capture the real
+exit code and fail on timeout/missing/non-finite output, reject NaN/Inf, require
+the expected steps/fields/traces, and write a per-run manifest; dry-runs can no
+longer overwrite real results; fingerprints record ordered patch-content hashes;
+Level 3 builds are isolated from Level 2 prefixes. CPU-only negative tests:
+`level3/tools/tests/run_all.sh` (four groups: infra helpers, second-batch
+checkers, the Nyx strict comparator + `validate.sh` chain, verdict classes; 88
+checks on 2026-09-08). nekRS CUDA/dependency decision:
+[nekrs/COMPATIBILITY.md](nekrs/COMPATIBILITY.md).
+
+Verdict classes (`level3/tools/l3_verdict.py`): a `validate.sh` exit code is
+0 PASS, 1 FAIL, 3 PENDING (Nyx heat/cool: `STATE_AND_PARTICLES_PASS;
+I_R_CHECK_PENDING`), 4 UNSUPPORTED_LAYOUT (Nyx: legal but different box
+layouts, not compared). Only 0 is a pass; 3 and 4 are their own classes in every
+summary, never counted as PASS, never fed into a performance summary, and never
+a reason for a queue to stop (`l3_run_recorded` in `l3_common.sh` records the
+code and continues).
+
+**Joint-HEAD regression (2026-09-07, code state `fc4d2a1`)**: after the shared
+helper change (`HPCPERF_L3_RUN_SUBDIR`, run trees `run.regress-<sha>` that never
+overwrite `run/`) and the Nyx validator rework, 27 `validate.sh` calls were made
+on dgx003: LAMMPS, SPARTA, WarpX, SPECFEM3D smoke at 1/2/4 GPUs (12 PASS), nekRS
+ethier hypregpu cimode 2 and 3 and cpucoarse cimode 2 at 1/2/4 GPUs (9 PASS), Nyx
+MiniSB + LyA-adiabatic at 1/2/4 GPUs (3 PASS), Nyx LyA heat/cool at 1/2/4 GPUs
+(3 PENDING). **24 PASS, 3 PENDING, 0 FAIL** -- not "all PASS". Launcher audits: 0
+mismatch; four logs carry unverified ranks (SPARTA np1/np2, WarpX np1/np2 second
+run: sub-0.3 s runs missed by the nvidia-smi sampling -- a binding-evidence gap,
+not a correctness signal). CP2K, QMCPACK, DFT-FE and GEOS were **not** re-run on
+the GPU in that round (their change is the one-line run-directory variable; the
+status below is their 2026-09-05/06 result, their checkers are covered offline by
+`test_l3_validators.sh`); a GPU regression of these four under the new directory
+logic is a follow-up. Logs: `build/level3/regress-firstbatch-fc4d2a1/` and
+`build/level3/nyx/regress-fc4d2a1/` (not committed).
+
+## Status
+
+| Application | Version | Build Strategy | CUDA Build | 1 GPU | 2 GPU | 4 GPU | HIP | Strong | Weak | Multi-node | Source Mod | Status |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| [LAMMPS](lammps/README.md) | stable_22Jul2025_update6 | NATIVE (bundled Kokkos 4.6.2) | OK, 220 s | PASS | PASS | PASS | untested | 16.4M atoms, 1/4 GPU run | 2.05M atoms/rank, 4 GPU run | BLOCKED/UNVERIFIED | A (derived deck) | FIRST_BATCH done |
+| [SPARTA](sparta/README.md) | 27Aug2026 | NATIVE (bundled Kokkos 5.0.2) | OK, 579 s | PASS | PASS | PASS | untested | 10M particles, 1/4 GPU run | 1.25M particles/rank, 4 GPU run | BLOCKED/UNVERIFIED | A | FIRST_BATCH done |
+| [WarpX](warpx/README.md) | 26.09 (+AMReX 26.09) | NATIVE (local AMReX source) | OK, 1219 s | PASS | PASS | PASS | untested | 33.6M particles, 1/4 GPU run | 4.2M particles/rank, 4 GPU run | BLOCKED/UNVERIFIED | A (derived inputs) | FIRST_BATCH done |
+| [SPECFEM3D Cartesian](specfem3d/README.md) | v4.1.1 (+2 devel back-ports) | NATIVE (autotools, bundled SCOTCH) | OK, 21 s | PASS | PASS | PASS | untested | 165,888 elements, 1/4 GPU run | 165,888 elements/rank, 4 GPU run | BLOCKED/UNVERIFIED | B+C+D (18 lines, upstream devel) | FIRST_BATCH done |
+| [nekRS](nekrs/README.md) | v26.0 | NATIVE (vendored OCCA/HYPRE) | OK (hypregpu ~30 min; cpucoarse 113 s) | PASS | PASS | PASS | untested | 32,000 elements N=7, 1/4 GPU run | 8,000 elements/rank, 4 GPU run | BLOCKED/UNVERIFIED | B+C+D, hypregpu variant (39 lines; vendored HYPRE 2.32.0 vs CUDA 13); cpucoarse variant 0 patches | FIRST_BATCH done; GPU-coarse verified (cimode 3), CPU-coarse candidate verified -- see [COMPATIBILITY.md](nekrs/COMPATIBILITY.md) |
+| [Nyx](nyx/README.md) | 26.09 (+AMReX 26.09, SUNDIALS 7.2.1) | NATIVE (private AMReX per profile) | OK, 121 s (adiabatic) / 310 s (heatcool) | PASS (MiniSB, LyA-adiabatic); heat/cool PENDING | PASS; heat/cool PENDING | PASS; heat/cool PENDING | untested | LyA 64^3 adiabatic + synthetic 256^3, 1/2/4 GPU run (too small to scale) | synthetic 64^3/rank, 4 GPU run | BLOCKED/UNVERIFIED | A | SECOND_BATCH: adiabatic decks VALIDATED_PASS (re-run 2026-09-07); heat/cool STATE_AND_PARTICLES_PASS; **I_R_CHECK_PENDING** (not a pass) |
+| [CP2K](cp2k/README.md) | v2026.2 (+DBCSR 2.10.0, upstream toolchain) | NATIVE + upstream toolchain (B200 back-port) | OK (toolchain hours; CP2K attempt 3) | PASS | PASS | PASS | untested | H2O-128 MD, 1/2/4 GPU run | H2O-32/64/128 size sweep, 1/2/4 GPU run | BLOCKED/UNVERIFIED | B (toolchain patch) + C | SECOND_BATCH done 2026-09-06 (historical result; not re-run on GPU 2026-09-07) |
+| [QMCPACK](qmcpack/README.md) | v4.4.0 (+LLVM 23.1.0 offload) | NATIVE + private LLVM toolchain | OK, 982 s | PASS | PASS | PASS | untested | 256 walkers total, 1/2/4 GPU run (4096-walker series FAILED: cuSOLVER/device memory) | 256 walkers/GPU, 1/2/4 GPU run (1024/GPU FAILED, same cause) | BLOCKED/UNVERIFIED | A (+ C zlib flag) | SECOND_BATCH done 2026-09-06; walker-memory anomaly open, <= 300 walkers/GPU enforced (historical; not re-run 2026-09-07) |
+| [DFT-FE](dftfe/README.md) | 1.2.0 (+deal.II 9.6.2, ELPA 2026.02.001) | NATIVE (install_DFTFE recipe) | OK | PASS | PASS | PASS | untested | LLZO 192 atoms, 1/2/4 GPU run | derived Al 32 atoms/GPU, synthetic size sweep | BLOCKED/UNVERIFIED | D (2-line isnan) + C | SECOND_BATCH done 2026-09-06 (historical; not re-run 2026-09-07); ELPA CPU cross-check SKIPPED |
+| [GEOS](geos/README.md) | develop `b7a0f13305` + TPL `9b55672` | NATIVE (thirdPartyLibs superbuild) | OK, 1718 s | PASS (beam workflow) | PASS | PASS | untested | beamBending_benchmark 160x16x8, 1/2/4 GPU run (too small to scale) | refinement weak 80x8x4/GPU (same caveat) | BLOCKED/UNVERIFIED | B (3 TPL) + D (BLT back-port) + C | SECOND_BATCH done 2026-09-06 (historical; not re-run 2026-09-07); **5 compositional-flow/well unit tests FAIL on this build -- those modules UNVERIFIED** |
+
+8/40/80-GPU shapes exist for every application as launcher dry-runs only
+(`HPCPERF_DRY_RUN=1`): **DRY-RUN / UNVALIDATED** (no 40/80-GPU run has ever
+happened). Multi-node MPI is BLOCKED/UNVERIFIED on this site. HIP recipes exist
+in every `build.sh` and exit with a clear message here (no ROCm): **untested**.
+Strong/weak entries above are runs that completed at 1/2/4 GPUs with the stated
+decks; where the deck is too small for a B200 (Nyx, GEOS, CP2K/DFT-FE size
+sweeps) they are completeness records, not scaling results; numerical
+acceptance of the strong/weak runs beyond the smoke criteria is not claimed.
+
+## Hard requirements (summary of the Level 3 policy)
+
+1. Full application workflow (mesher/solver/IO stages included where upstream
+   has them); no hotspot-only or single-kernel runs.
+2. `HPCPERF_GPUS=N|all` selects the GPU count; requested == launched. A rank
+   count the application's decomposition cannot support is an error -- never a
+   silent change of N, never silent GPU sharing, never a fallback to 1 GPU,
+   never a failure reported as PASS.
+3. Default policy is one MPI rank per GPU; if upstream officially recommends
+   another model (threads per GPU, MPI+OpenMP, several GPUs per rank) the
+   application follows upstream and its README says so. All five first-batch
+   applications document one rank per GPU.
+4. Every application defines smoke / strong / weak inputs (global size,
+   per-rank size, memory estimate, process topology, expected runtime,
+   validation quantity), the rank->GPU mapping and multi-node requirements.
+5. 40/80-GPU shapes are `DRY-RUN / UNVALIDATED` until a real allocation
+   exists; multi-node is BLOCKED/UNVERIFIED on this site; HIP is `untested`
+   without an AMD GPU.
+6. Toolchain follows the application's officially supported versions, not
+   Level 1's pins; compatibility modifications are classified (A none,
+   B build-system-only, C environment, D source-level compatibility) -- E
+   algorithm/performance modifications are forbidden in bring-up.
+7. The validated Level 2 dependency tree (`.deps/install`) is never modified.
+
+## Correctness policy as applied
+
+Exit code is never sufficient. Each `validate.sh` uses the application's own
+mechanism and prints a single `... validation (N GPU, ...): PASS|FAIL` line:
+LAMMPS thermo vs the shipped reference log (bit-identical here); SPARTA
+statistical stats vs the shipped reference log with justified tolerances
+(particle count exact, temperature 2 %, collision attempts 15 %); WarpX
+upstream's analytic Langmuir-wave regression test (5e-2) and charge
+conservation (1e-11) read from the plotfile, plus exact particle conservation;
+SPECFEM3D reference seismograms through upstream's comparison script
+(correlation, misfit, time shift); nekRS upstream's `--cimode` CI checks on the
+analytic Ethier solution. For these five, no tolerance was loosened to obtain a
+PASS and no precision or physics setting was changed. Second batch: CP2K
+regtest tolerances + MD energy consistency, QMCPACK `check_scalars.py`, DFT-FE
+upstream GPU reference, GEOS geos-ats metrics/restart baseline, Nyx official
+`fcompare` tolerances -- with one recorded exception: the Nyx heat/cool
+tolerance 5e-5 (upstream's value for that deck) was adopted after a first run at
+the adiabatic 2e-10 had FAILED, and the `I_R` field of that deck is not accepted
+by any tolerance (PENDING), see `nyx/README.md`.
+
+## Dependency isolation
+
+Every application owns a private tree -- no shared Level 3 install root:
 
 ```
-level3/<application_name>/
-├── README.md
-├── source or integration files
-├── build files
-├── configs/
-├── jobs/
-└── run.sh
+.deps/level3/<app>/{src,build,install,logs}     patched source copy (where needed), deps, install, logs
+_upstream/level3/<Name>                        shallow upstream checkout at the selected tag (read-only)
+build/level3/<app>/<cuda|hip>                  application build tree (+ run/ directories of run.sh)
 ```
+
+Installs carry `.hpcperf-l3-fingerprint` (schema `l3-1`: application, upstream
+commit, dependency versions, compiler, Fortran compiler, CUDA/ROCm, GPU arch,
+MPI, CMake/configure options, GPU-aware-MPI setting, patch list, site profile,
+Spack lock hash, container image hash, build time). A recorded fingerprint
+that differs from the requested configuration fails fast
+(`level3/tools/l3_common.sh`).
+
+Spack, when chosen, uses one environment per application and backend
+(`level3/envs/<app>/{cuda,rocm}/spack.yaml` + `spack.lock`); containers, when
+chosen, commit the `.def`, build script, image SHA256 and README -- never the
+`.sif`. Neither is used by the first batch (see BUILD_STRATEGY.md for why).
+
+## Runtime
+
+Launches go through the common launcher (`HPCPERF_GPUS`, `HPCPERF_NODES`,
+`HPCPERF_GPUS_PER_NODE`, `HPCPERF_CPUS_PER_RANK`, `HPCPERF_SCALE_MODE`,
+`HPCPERF_SITE_PROFILE`, `HPCPERF_DRY_RUN=1`) with the per-rank GPU wrapper
+(each rank sees one GPU; expected vs observed GPU audited). Level 3 refers to
+it through `HPCPERF_RUNTIME_DIR` (default `level2/tools`); the plan to move
+the shared tools to `tools/runtime/` without breaking Level 2 is in
+[../tools/runtime/README.md](../tools/runtime/README.md).
+
+Site/transport observations recorded in the READMEs (single node, `pml ob1 /
+btl self,sm,smcuda`): GPU-aware MPI makes WarpX's 4-GPU step 3x slower
+(0.081 vs 0.026 s/step) but LAMMPS 2.5x faster (2.38 vs 5.87 s); SPARTA is
+indifferent. Defaults stay upstream's; this is a performance topic for a later
+round, not a bring-up change. Open MPI's one-sided layer still selects
+`osc ucx` on this node and aborts inside `uct_ib` with 4 ranks (nekRS uses
+`MPI_Win_lock`); nekRS' `run.sh` sets `OMPI_MCA_osc=^ucx`, which is proposed
+for the gmu-hopper site profile in the runtime commonization PR.
+
+## Per-application layout
+
+```
+level3/<app>/
+├── README.md        provenance, version/commit, license, LOC, build strategy, changes (A-D), execution model,
+│                    inputs (smoke/strong/weak), validation, 1/2/4-GPU results, dry-runs, limitations
+├── fetch.sh         shallow clone at the recorded tag/commit (no source trees committed)
+├── build.sh         native build into .deps/level3/<app>, fingerprinted; HIP branch present, untested
+├── run.sh           HPCPERF_GPUS + HPCPERF_SCALE_MODE aware, launched via the common launcher
+├── validate.sh      upstream correctness mechanism, PASS/FAIL line, exit code
+└── patches/         compatibility patches (classified, documented; SPECFEM3D, nekRS)
+```
+
+Inputs are upstream's own decks referenced from the read-only checkout;
+derived decks (size, steps, topology, diagnostics) are written into the build
+tree at run time and documented per application, so no upstream input file is
+modified and nothing large is committed.
diff --git a/level3/SECOND_BATCH_STATUS.md b/level3/SECOND_BATCH_STATUS.md
new file mode 100644
index 0000000..be36bbf
--- /dev/null
+++ b/level3/SECOND_BATCH_STATUS.md
@@ -0,0 +1,297 @@
+# Level 3 second batch -- status (Nyx, CP2K, QMCPACK, DFT-FE, GEOS)
+
+Branch `level3/second-batch-bringup` (worktree `HPC-Performance-AI-b2`, from the
+first-batch checkpoint `366b72f`). Node dgx003 (4x B200, CUDA 13.2.78, conda
+Open MPI 5.0.10, system GCC 14.2.1 / conda GCC 13.3.0), Slurm job 9552083.
+Everything below was produced on that allocation between 2026-09-05 and
+2026-09-08. Last update: **2026-09-08 -- CP2K, QMCPACK, DFT-FE, GEOS VALIDATED_PASS at
+1/2/4 GPUs (results of 2026-09-05/06, commits listed at the end); Nyx MiniSB and
+LyA-adiabatic VALIDATED_PASS at 1/2/4 GPUs (re-run 2026-09-07 at the joint HEAD), Nyx
+LyA heat/cool STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING (not a full pass, see
+`nyx/README.md`); open items per application (QMCPACK walker-memory anomaly, GEOS
+flow/well unit tests, Nyx I_R, size of the official GEOS/Nyx decks).** The branch is
+being submitted as a Draft PR; see "Joint-HEAD regression" below for what was and was
+not re-run.
+
+States: `PLANNED` (not started) / `NOT_RUN` (deliberately not run, reason given) /
+`BUILD_PASS` / `COMPLETED` (ran to completion, timings or plans recorded, no
+correctness claim) / `VALIDATED_PASS` (validate.sh PASS) / `FAILED` / `BLOCKED` /
+`UNVERIFIED`.
+
+## Status table
+
+| Application | Selected version / SHA | Build strategy | Compiler / Toolkit | Dependency probe | CUDA build | 1-GPU smoke | 2-GPU correctness | 4-GPU correctness | Strong | Weak / size sweep | 40/80 dry-run | Multi-node | HIP | Source changes | Blocker |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| Nyx | 26.09 `e06eabc1` + external AMReX 26.09 `a52ca733` (+ SUNDIALS 7.2.1 for heat/cool) | NATIVE, per-profile AMReX/SUNDIALS | conda GCC 13.3.0 + nvcc 13.2.78, sm_100 | COMPLETED (AMReX archs = sm_100 verified; SUNDIALS CUDA examples 5/6, see README) | BUILD_PASS (2 profiles + CPU reference profiles) | VALIDATED_PASS (MiniSB, LyA-adiabatic); LyA heat/cool **STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING** (exit 3, not a pass) | VALIDATED_PASS (adiabatic decks); heat/cool PENDING | VALIDATED_PASS (adiabatic decks); heat/cool PENDING | COMPLETED (LyA 64^3 adiabatic; synthetic 256^3; heat/cool LyA 64^3: 2.85/2.62/2.65 s on 1/2/4 GPUs -- too small to scale, fixed-cost dominated; the heat/cool timings are completeness records of a PENDING case) | COMPLETED (synthetic 64^3/rank, labelled synthetic) | COMPLETED (8/40/80 planned or refused as designed) | BLOCKED (site) | NOT_RUN (no ROCm) | none (class A) | large ICs not public |
+| CP2K | v2026.2 `67b5da87`; DBCSR 2.10.0; toolchain OpenBLAS 0.3.33 | NATIVE + upstream toolchain (B200 back-port patch) | system GCC 14.2.1 + nvcc 13.2.78, sm_100 | COMPLETED (DBCSR ctest 19/19 on 4 GPUs) | BUILD_PASS (attempt 3: toolchain OpenBLAS via rpath order; attempts 1-2 resolved BLAS to the conda pthreads OpenBLAS -- archived under `install/ATTEMPT-*`) | VALIDATED_PASS (attempt-3 binary: regtests within upstream tol, H2O-64 MD 10/10 SCF converged) | VALIDATED_PASS (max FORCE_EVAL diff 1.1e-11 Ha vs 1 GPU) | VALIDATED_PASS (8.6e-12 Ha) | COMPLETED (H2O-128, 10 MD steps: 119.8 / 90.8 / 60.0 s on 1/2/4 GPUs = 1.32x / 2.0x; 8 OpenMP threads per rank) | COMPLETED (size sweep, 32 waters per GPU: H2O-32 on 1 GPU 17.3 s, H2O-64 on 2 GPUs 32.9 s, H2O-128 on 4 GPUs 62.8 s -- the GPW cost per molecule grows with system size, so this is a size sweep, not an iso-efficiency series) | COMPLETED (8/40/80 strong planned; weak 40/80 REFUSED: no upstream H2O-1280/2560 deck) | BLOCKED (site) | NOT_RUN | toolchain patch (class B) + node adaptations (C) | none |
+| QMCPACK | v4.4.0 `2601d62e`; LLVM 23.1.0, HDF5 1.14.5, Boost 1.90 | NATIVE + private LLVM offload toolchain | clang 23.1.0 (host) + nvcc 13.2.78, `QMC_GPU=openmp;cuda` sm_100 | COMPLETED (offload probe PASS 1/2/4 ranks; unit ctests 64/64; deterministic diamond ctests 526/526) | BUILD_PASS (982 s) | VALIDATED_PASS (DMC E = -21.8492 +- 0.0152 vs ref -21.844975 +- 0.02, -0.32 sigma) | VALIDATED_PASS (-21.8566 +- 0.0126; -0.58 sigma vs ref, 0.37 sigma vs 1-GPU) | VALIDATED_PASS (-21.8313 +- 0.0105; +0.68 sigma vs ref, 0.97 sigma vs 1-GPU) | FAILED at 4096 walkers (cuSOLVER INTERNAL_ERROR: device memory ~320 MB/walker exhausted, see README) -> COMPLETED with the verbatim 256-walker deck over 1/2/4 GPUs: 534 / 332 / 211 s (1.61x, 2.53x; the validation runs of the same deck: 316/254 s); DMC energies -21.8365 +- 0.0088 / -21.8511 +- 0.0100 Ha, upstream check_scalars pass | FAILED at 1024 walkers/GPU (same cause) -> COMPLETED with 256 walkers/GPU: 534 / 632 / 656 s on 1/2/4 GPUs (84 % / 81 % weak efficiency; 512 and 1024 walkers in total), DMC -21.8466 +- 0.0077 / -21.8433 +- 0.0034 Ha (error bars shrink as 1/sqrt(walkers) as they should), check_scalars pass | COMPLETED (8/40/80 planned, HYPOTHETICAL) | BLOCKED (site) | NOT_RUN | none (class A); HDF5 zlib discovery flag (C) | NiO datasets external |
+| DFT-FE | 1.2.0 `7147faa5`; deal.II **9.6.2** (9.7.1 attempt failed: API removals), ELPA 2026.02.001 (sm_100 kernels), p4est 2.8.7 | NATIVE (install_DFTFE recipe transcribed) | system GCC 14.2.1 + nvcc 13.2.78, sm_100 | COMPLETED (ELPA GPU probe PASS: ELPA's analytic 1-stage and 2-stage GPU eigensolver tests on 1/2/4 GPUs, max eigenvalue error <= 7.3e-15 (tol 5e-14), eigenvector error <= 1.0e-11 (tol 6e-10), GPU timers present, audit 0 mismatch; two earlier probe versions mis-parsed -- recorded) | BUILD_PASS (deal.II 9.6.2 rebuild + 2-line `std::isnan` patch) | VALIDATED_PASS (al_md 32-atom BOMD, 4 steps, vs upstream's GPU reference: e0, MD energies, temperatures, forces identical at printed precision, d = 0.0) | VALIDATED_PASS (identical to the reference and to the 1-GPU run) | VALIDATED_PASS (identical) | COMPLETED (LLZO 192 atoms / 720 states: 295 / 163 / 96 s on 1/2/4 GPUs = 1.81x / 3.07x; E = -3579.26588980 Ha identical on 1/2/4 GPUs) | COMPLETED (derived Al supercells, 32 atoms per GPU: 32 / 64 / 128 atoms on 1/2/4 GPUs: 213 / 284 / 378 s -- KS-DFT cost grows superlinearly with atoms, SYNTHETIC series labelled as such) | COMPLETED (8/40/80 planned, HYPOTHETICAL) | BLOCKED (site) | NOT_RUN | 2-line `std::isnan` patch (D); p4est-setup.sh + ELPA configure adaptations (C) | ELPA CPU cross-check programs not built with the GPU configuration (SKIPPED, recorded) |
+| GEOS | develop `b7a0f13305` (2026-09-04) + thirdPartyLibs `9b55672` (TPL 361-1070) | NATIVE (thirdPartyLibs superbuild + host-config) | system GCC 14.2.1 + nvcc 13.2.78, sm_100; hypre device (`ENABLE_HYPRE_DEVICE=CUDA`) | COMPLETED with findings (TPL superbuild: libHYPRE.a / libRAJA.a embed sm_100 only; GEOS/LvArray ctest on GPU 0: **254 / 261 passed** -- failed: `testMath` (device `asinhf` vs host within 1 float ulp), `testErrorHandling` (an abort-on-purpose test under `mpirun -np 1`/prterun), and **5 fluid-flow/well physics tests** -- `testCompMultiphaseFlow` (analytical vs numerical phase-mobility derivatives, error norm ~1.0), `testCompMultiphaseFlowHybrid` (flux Jacobian, 0.008-0.15), `testThermalEstimator{Prod,Inj}Well`, `testReservoirThermalSinglePhaseMSWells_RateInj` (ExternalError in the well solvers) -- cause not determined; these modules are UNVERIFIED on this stack, see section) | BUILD_PASS (1718 s at -j32; attempts: conda 32-bit metis.h shadowing the TPL one -> host-config include order; BLT CUDA runtime smoke test vs CUDA 13 -> back-port of LLNL/blt 38b46203) | VALIDATED_PASS (beam 80x8x4, hypre GMRES+AMG on device: geos-ats curve metric 1.376e-4 <= 2e-4 = upstream's own baseline value; shipped direct-solver deck vs the public restart baseline: 1191 arrays/attributes, 0 disagree, worst rel 3.6e-12; history vs baseline history rel 5e-12) | VALIDATED_PASS (curve metric 1.376e-4; vs 1 GPU rel L-inf 6.2e-9, metric 6.6e-10) | VALIDATED_PASS (curve metric 1.376e-4; vs 1 GPU rel L-inf 1.1e-8, metric 8.4e-10) | COMPLETED (beamBending_benchmark 160x16x8 = 20 480 C3D8, 10 quasi-static steps: GEOS run time 6.7 / 10.9 / 13.3 s on 1/2/4 GPUs -- the official beam decks are far too small for a B200; fixed hypre-setup/I/O cost dominates and more ranks cost more) | COMPLETED (refinement weak, 80x8x4 elements per GPU: 80x8x4 / 160x8x4 / 160x16x4 -> 9.5 / 14.2 / 8.7 s; same caveat) | COMPLETED (8/40/80 strong + weak planned, HYPOTHETICAL; partitions 2x2x2 / 5x4x2 / 5x4x4; 3 ranks refused: cannot partition 80x8x4) | BLOCKED (site) | NOT_RUN | 3 thirdPartyLibs build-system patches (B) + 1 BLT smoke-test back-port (D); host-config include order, ENABLE_HYPREDRV=OFF (C); no GEOS/LvArray/hypre source change | beam decks too small for GPU scaling (bigger official meshes = `HPCPERF_GEOS_WEAK_NX`); 5 compositional-flow / well unit tests FAIL on this GPU build (not the validated workflow; unresolved) |
+
+## Per application
+
+### Nyx -- `level3/nyx/` (commits 769482f, 1f7e14f, c7b90f9; validator rework f231217, fc4d2a1 and the 2026-09-08 layout/verdict commit)
+
+Official decks (MiniSB `inputs.32 nyx.ppm_type=0`, LyA `inputs.rt.garuda`, LyA
+heat/cool `inputs.rt`, 10 steps as in upstream's nightly GPU suite) on 1/2/4 GPUs;
+criteria: completeness/finiteness, the strict `nyx_fcompare_check.py` wrapper around
+`fcompare -n 0 --rel_tol` (Header/box-layout structure, raw finiteness, one parsed row
+per variable, zero-reference rule, parser/tool consistency) at MiniSB's official
+tolerance 2e-10 (applied to LyA-adiabatic as well, a project choice stricter than
+upstream's 5e-9) and 5e-5 for heat/cool (upstream's plt00354 tolerance, adopted on
+2026-09-06 after a first heat/cool run at 2e-10 FAILED), DM particle identity tracking
+across rank counts (`nyx_particle_compare.py` -- AMReX's own tool cannot compare
+different rank counts), CPU-backend cross-reference (1e-8 pre-fixed), baryon mass
+conservation 1e-9. **Heat/cool**: the SDC reaction-integral field `I_R` -- consumed by
+the next step's hydro predictor -- varies O(1) relative between any two runs of the
+10-step deck while every state field agrees to 1e-13; no criterion derived from the
+state tolerance accepts it and its mechanism is not identified, so the case verdict is
+`STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING` (validate.sh exit 3), not a full pass, and
+it is excluded from the correctness-pass count and from any performance summary.
+Strong/weak/dry-run numbers and the full derivation are in `level3/nyx/README.md`
+(heat/cool LyA 64^3 strong: 2.85 / 2.62 / 2.65 s on 1/2/4 GPUs -- too small to scale, a
+completeness record of a PENDING case; dry-runs 8 planned, 40 IMBALANCED, 80 refused as
+designed). **Remaining**: the I_R question (CPU-vs-CPU and GPU-vs-GPU repeats at the
+original configuration, per-step attribution, an independent tolerance comparison) --
+not attempted in this round.
+
+### CP2K -- `level3/cp2k/` (commit b04db93)
+
+Toolchain: `install_cp2k_toolchain.sh --gpu-ver=B200` with the back-port of
+upstream's B200 commit `378b2fab` (DBCSR gets `GPU_ARCH_NUMBER_B200 100`,
+`parameters_H100.json` reused as `parameters_B200.json`); DBCSR 2.10.0 test build
+and its ctest suite (19/19, 4 ranks x 4 threads on 4 GPUs) before CP2K;
+`CMAKE_CUDA_ARCHITECTURES=100`. Validation at 1/2/4 GPUs x 8 threads: adapted
+subset of upstream's regression tests (Ar, H2O-geoopt, pyridine, H2-big-1,
+H2-big-5) all within upstream tolerances (max |diff| 6.0e-14); H2O-64 GPW MD 10
+steps: all 10 MD-step SCF cycles converged, GRID GPU tasks / DBCSR ACC device /
+pw_gpu timers observed, per-step FORCE_EVAL energies agree with the 1-GPU run to
+<= 8.6e-12 Ha (pre-fixed 1e-8; upstream's CPU 1e-10 also met). Documented
+peculiarity: the upstream deck's initial ATOMIC-guess SCF does not converge
+within MAX_SCF=50 and says so (`IGNORE_CONVERGENCE_FAILURE`); reported, not
+hidden, negative-tested. GPU stages confirmed: DBCSR sparse multiply (ACC), GRID
+collocate/integrate, PW FFT (`pw_gpu_*`); ScaLAPACK diagonalisation/Cholesky
+remain CPU (as configured upstream without ELPA/cuSOLVERMp). Strong scaling
+(H2O-128, 10 MD steps, toolchain OpenBLAS binary): 119.8 / 90.8 / 60.0 s on 1/2/4 GPUs
+(1.32x, 2.0x -- the 64/128-water GPW MD is communication/latency heavy at 8 threads
+per rank); size sweep H2O-32/64/128 on 1/2/4 GPUs: 17.3 / 32.9 / 62.8 s. All runs
+record `blas_resolved=.../openblas-0.3.33/lib/libopenblas.so.0`.
+
+### QMCPACK -- `level3/qmcpack/`
+
+Toolchain: LLVM 23.1.0 built from source on local scratch (clang/lld + openmp/offload
+runtimes with the `nvptx64-nvidia-cuda` GPU runtimes target; the official binary
+release has no device runtime), offload probe (`toolchain/probe_offload.sh`,
+`OMP_TARGET_OFFLOAD=MANDATORY`, target region not on the initial device, numerics
+vs host, MPI 2/4 ranks through the launcher) PASS before any QMCPACK build. HDF5
+1.14.5 parallel, Boost 1.90 headers, OpenBLAS 0.3.30 private. QMCPACK 4.4.0
+`QMC_GPU="openmp;cuda" QMC_GPU_ARCHS=sm_100 QMC_MPI=ON BUILD_UNIT_TESTS=ON`, cuobjdump
+archs sm_100. Validation design (`validate.sh`): upstream unit ctests (64/64 PASS)
+and the deterministic diamond-case ctests (526/526 PASS), then
+`tests/solids/diamondC_2x1x1_pp/qmc_short_vmcbatch_dmcbatch.in.xml` verbatim on N
+GPUs with upstream's own `check_scalars.py` criterion (DMC total energy
+-21.844975 +- 0.02 Ha, 3 sigma, 2 equilibration blocks) plus completeness, device
+code-path evidence ("Running OpenMP offload code path", device memory allocated
+via the offload runtime) and, for N > 1, statistical consistency with the 1-GPU
+run (3 sqrt(sigma_1^2 + sigma_N^2)). Results (256 walkers, 8 crowds/rank):
+1 GPU -21.8492 +- 0.0152 Ha (534 s), 2 GPUs -21.8566 +- 0.0126 (316 s, 128 walkers/rank),
+4 GPUs -21.8313 +- 0.0105 (254 s, 64 walkers/rank); all within upstream's 3-sigma window
+and mutually consistent (<= 0.97 sigma); launcher audit N verified / 0 mismatch at every
+N. VALIDATED_PASS 1/2/4. Strong = fixed `total_walkers`, weak = `walkers_per_rank`:
+the first series (4096 total / 1024 per rank) FAILED in cuSOLVER on every rank --
+device memory exhausted at ~320 MB per walker although QMCPACK's allocators report
+~27 MiB (open issue, documented in the README, not worked around; run.sh now refuses
+> 300 walkers/GPU unless forced). Re-defined series with the verbatim population:
+strong (256 walkers total) 534 / 332 / 211 s on 1/2/4 GPUs (1.61x, 2.53x -- 64
+walkers per GPU at 4 GPUs is far below what a B200 needs), weak (256 walkers per GPU)
+534 / 632 / 656 s (84 % / 81 %); every run's DMC energy passes upstream's
+`check_scalars.py` window and the launcher audit (`qmc_summary.txt` in each run
+dir). NiO performance decks not used (data only via anl.box.com).
+
+### DFT-FE -- `level3/dftfe/`
+
+Dependency chain: OpenBLAS, ScaLAPACK 2.2.2, libxc 7.0.0, spglib, ALGLIB, p4est 2.8.7
+(three adaptations of dftfe's Cray-oriented `p4est-setup.sh`: MPI wrappers,
+`LIBS=-lm`, 2.8.7 header location), Kokkos 4.6.00 Serial, **deal.II 9.6.2** (attempt 1
+with 9.7.1 -- the version the current install_DFTFE recipe pairs with dftfe
+*develop* -- fails: 9.7 removed three APIs release 1.2.0 uses; 9.6.2 keeps them
+deprecated), ELPA 2026.02.001 with sm_100 kernels (`-march=native` for its AVX-512
+probe, ScaLAPACK/OpenBLAS paths in `LDFLAGS` for its cublas check), DFT-FE real with
+a 2-line `std::isnan` patch (unqualified `isnan` in a template, GCC 14). Validation
+(`validate.sh`, `dftfe_check.py`): upstream GPU regression deck `Input_MD_0.prm`
+(32-atom Al BOMD, 4 steps, USE GPU) on 1/2/4 GPUs vs upstream's own GPU reference
+`accuracyBenchmarks/output_MD_0` under pre-fixed tolerances (1e-5 Ha ground state,
+2e-5 Ha per MD step, 0.1 K, 2e-5 Ha/Bohr forces): **every quantity identical at the
+printed precision** (d = 0.0) at 1, 2 and 4 GPUs, and 2/4-GPU runs identical to the
+1-GPU run; launcher audit N verified / 0 mismatch. Criterion [0], the independent
+ELPA GPU-kernel probe (`elpa_probe.sh`: ELPA's own `validate_*_gpu_analytic` programs,
+1-stage and 2-stage solvers, na=2000 nev=1000, 1/2/4 GPUs): PASS on all six runs,
+max eigenvalue error 2.0e-15..7.3e-15 (ELPA's limit 5e-14), max eigenvector error
+1.5e-12..1.0e-11 (limit 6e-10), exit 0, ELPA's GPU timers in the output, audit 0
+mismatch. Two probe versions before it were wrong and are recorded (wrapper-script
+names; then the residual/orthogonality lines of ELPA's *random-matrix* programs, which
+the analytic programs do not print -- the empty grep aborted the script under
+`pipefail`, so the validations of 05:01-05:03 UTC FAILED on [0] only and were re-run at
+05:04-05:12 UTC with the corrected probe: VALIDATED_PASS 1/2/4). Strong (LLZO, 192
+atoms, 720 states, ELPA): 295 / 163 / 96 s on 1/2/4 GPUs (1.81x / 3.07x), ground-state
+energy -3579.26588980 Ha identical at every N. Weak (derived Al supercells, 32 atoms per
+GPU; synthetic, same construction as upstream's dftfe-benchmarks Mo series): 32 / 64 /
+128 atoms in 213 / 284 / 378 s -- the KS-DFT cost per atom grows with system size
+(more states x more DOFs), so this is a size sweep at constant atoms/GPU, not an
+iso-efficiency series. Dry-runs 8/40/80 planned (HYPOTHETICAL). ELPA's CPU test
+programs are not built with the GPU configuration, so the CPU cross-check is SKIPPED
+(recorded in the probe file).
+
+### GEOS -- `level3/geos/`
+
+TPL superbuild COMPLETED after three build-system fixes to thirdPartyLibs
+(superlu_dist hash typo; RAJA vectorization layer not compilable by nvcc 13.2 +
+GCC 14/x86-64-v3 -> `RAJA_ENABLE_VECTORIZATION=OFF` as upstream does for ROCm;
+hdf5 step generator) and node adaptations (`config-build.py` deletes existing
+build trees; hypredrive step lacks Umpire's include path -> `ENABLE_HYPREDRV=OFF`,
+a GEOS-documented option, the beam workflow does not use it). GEOS build: the conda
+MPI include dir (added by GEOS as `-isystem` ahead of the TPLs) carries an unrelated
+32-bit `metis.h` that broke the 64-bit ParMETIS static assertion -> TPL
+METIS/ParMETIS include dirs first (`-I`) in the host-config; BLT 0.6.2's CUDA
+runtime smoke test uses `cudaDeviceProp::memoryClockRate` (removed in CUDA 13) ->
+back-port of upstream BLT commit `38b46203` (`patches/geos-blt-0001`), tests kept
+at upstream's default ON. "CUDA 13 blocked" upstream is a Spack/uberenv `raja ^cuda@13:`
+conflict only; the native superbuild has no such check. GEOS built in 1718 s at
+-j32 (BUILD_PASS; `geosx`, `libHYPRE.a`, `libRAJA.a` embed sm_100 only).
+
+Validation (`validate.sh`, criteria = geos-ats's own checks for `beamBending`,
+re-implemented with the metrics of geosPythonPackages `curve_check.py` /
+`restart_check.py` / `permute_array.py` and verified against upstream's published
+baseline before the GEOS runs): the derived `beamBending_benchmark` deck at the smoke
+mesh (80x8x4, GMRES + hypre AMG **on the device**) on 1/2/4 GPUs -- geos-ats curve
+metric `||u - u_analytic||_2 / N` = 1.3761e-4 at every N (tolerance 2e-4; upstream's own
+baseline scores the same 1.3761e-4, i.e. the metric measures the mesh's discretisation
+error, which the partitioning does not change); 2/4-GPU histories vs the 1-GPU history:
+rel L-inf 6.2e-9 / 1.1e-8 (gate 1e-4, the 1e-5 information threshold met), geos-ats
+metric 6.6e-10 / 8.4e-10; the shipped serial-direct-solver smoke deck on 1 rank vs the
+public integrated-test baseline `beamBending_smoke_01` (tarball
+`pr3994-17525-4ae3593`, sha256 `de62cec7...`): restart file at cycle 10, 1191
+arrays/datasets/attributes under upstream's atol 1e-3 / rtol 1e-7 and default
+exclusions, **0 disagree, worst relative difference 3.6e-12** (`maxForce`); its
+history vs the baseline history rel 5.1e-12. Launcher audit N verified / 0 mismatch at
+every N. VALIDATED_PASS 1/2/4. Two checker corrections were needed before this result
+and are recorded in the README: (1) the per-time relative L-inf reading of the 2e-4
+tolerance would have failed upstream's own baseline (9.1e-4) -- replaced by the geos-ats
+metric before the first run; (2) the first restart comparison read the LvArray
+`__values__` datasets raw: a device build stores 2-D node/element fields in another
+memory permutation than the CPU build that produced the baseline (12 of 93 LvArrays), so
+the raw datasets differ while the arrays are equal -- fixed with geos-ats's permutation
+handling and re-validated (run ids in the run manifests). Three `LinearSolverParameters`
+values (`krylovMaxRestart` 100 vs 200, `amgCoarseningType` PMIS vs HMIS,
+`amgSmootherType` l1jacobi vs l1sgs) are GEOS's compile-time GPU-build defaults
+(`LinearSolverParameters.hpp`) and are reported, not gating (unused by the direct
+solver). Scaling: strong (`beamBending_benchmark.xml` verbatim, 160x16x8) 6.7 / 10.9 /
+13.3 s GEOS run time on 1/2/4 GPUs; refinement-weak (80x8x4 per GPU) 9.5 / 14.2 / 8.7 s
+-- the official beam decks are far too small for a B200 (hypre setup and silo/HDF5
+output dominate), so these are completeness records, not scaling results; larger
+meshes are one variable away (`HPCPERF_GEOS_WEAK_NX`) but were not run to keep the
+"official small workflow" definition. Dry-runs 8/40/80 strong + weak planned
+(partitions 2x2x2, 5x4x2, 5x4x4); 3 ranks refused (cannot partition 80x8x4).
+
+Unit tests (upstream default `ENABLE_TESTS=ON`, `ctest -j4` on GPU 0, 351 s, log
+`.deps/level3/geos/<profile>/logs/geos-ctest.log`): **254 of 261 passed**. The seven
+failures, characterised from the log: (1) `testMath` -- LvArray's
+`TestComplexMath/8.asinh` for `float` on the device: CUDA's `asinhf(5)` differs from the
+host value by at least one float ulp while the test allows `epsilon` (CUDA documents up
+to 3 ulp for `asinhf`); a libm-precision test, no GEOS code path; (2) `testErrorHandling`
+-- `testYamlFileAssertOutput` deliberately triggers `abort()`; under `mpirun -np 1`
+prterun reports the abort and the test exits non-zero (test-harness/launcher
+interaction, the preceding sub-tests pass); (3)-(7) **five fluid-flow and well physics
+tests fail on this GPU build**: `testCompMultiphaseFlow`
+(`derivativeNumericalCheck_phaseMobility`: analytical vs finite-difference derivatives
+disagree grossly, e.g. 653.7 vs -3.67, error norm ~1.0 against a 0.05 limit),
+`testCompMultiphaseFlowHybrid` (`jacobianNumericalCheck_flux`, error norms 0.008-0.15
+against 0.005), `testThermalEstimatorProdWell`, `testThermalEstimatorInjWell`,
+`testReservoirThermalSinglePhaseMSWells_RateInj` (GEOS `ExternalError` raised in the
+well solvers after warnings in `WellControls.cpp:1229` / `SinglePhaseWell.cpp:205`).
+These are not precision-level differences and were not investigated further (candidates:
+nvcc 13.2 + GCC 14 host code generation, RAJA 2026.07.0 on sm_100, or tests that upstream
+does not run on GPU); **the compositional multiphase flow and well modules of this GEOS
+build must be treated as UNVERIFIED**. The validated beam workflow (solid mechanics, hypre
+AMG on the device) is unaffected: its restart state agrees with upstream's public baseline
+to 3.6e-12.
+
+## Cross-cutting
+
+- Launcher: `level2/tools/hpcperf_mpi_launch.sh` unchanged (`HPCPERF_RUNTIME_DIR`
+  semantics kept); one rank per GPU, per-rank `CUDA_VISIBLE_DEVICES` wrapper,
+  nvidia-smi audit (verified / unverified / mismatch) in every run log; 8/40/80 GPUs
+  only as HYPOTHETICAL dry-runs (`HPCPERF_DRY_RUN=1 HPCPERF_NODES=N/4`), multi-node
+  BLOCKED/UNVERIFIED on this site.
+- Shared infra changes (committed separately): `l3_paths_profile`, `l3_version_mm`,
+  `l3_clean_conda_build_env`, real-grep guard, cuobjdump-based backend check for
+  statically linked cudart (9327c26, 0b9daaf); validator negative tests
+  `level3/tools/tests/test_l3_validators.sh` (352cbb3); 2026-09-07/08:
+  `HPCPERF_L3_RUN_SUBDIR` (regression run trees that never overwrite `run/`),
+  `l3_clean_env.sh`/`l3_clean_env_exec` (allow-listed `env -i` with a credential
+  deny-rule, used by CP2K's toolchain installer), `l3_run_recorded` + `l3_verdict.py`
+  (queue steps record their exit code and continue; classes PASS / PENDING /
+  UNSUPPORTED_LAYOUT / FAIL / MISSING, exit 3 and 4 never counted as PASS), all with
+  CPU tests in `level3/tools/tests/run_all.sh`.
+
+## Joint-HEAD regression (2026-09-07, code state `fc4d2a1`)
+
+After the shared-helper change (every `run.sh`/`validate.sh` builds its run directory
+under `$L3_RUN_SUBDIR`) and the Nyx validator rework, the following was re-run on
+dgx003 at the joint HEAD with `HPCPERF_L3_RUN_SUBDIR=run.regress-fc4d2a1` (new run
+ids, historical `run/` trees untouched; logs `build/level3/regress-firstbatch-fc4d2a1/`
+and `build/level3/nyx/regress-fc4d2a1/`, summary `SUMMARY_TABLE.md` produced by
+`l3_verdict.py`):
+
+| Scope | validate.sh calls | Result |
+|---|---|---|
+| First batch: LAMMPS, SPARTA, WarpX, SPECFEM3D smoke at 1/2/4 GPUs | 12 | 12 PASS |
+| nekRS ethier: hypregpu cimode 2 and 3, cpucoarse cimode 2, each at 1/2/4 GPUs | 9 | 9 PASS |
+| Nyx MiniSB + LyA-adiabatic at 1/2/4 GPUs | 3 | 3 PASS |
+| Nyx LyA heat/cool at 1/2/4 GPUs | 3 | 3 STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING (exit 3) |
+| **Total** | **27** | **24 PASS, 3 PENDING, 0 FAIL** -- not "all PASS" |
+
+Launcher audits: 33 audit lines, 0 mismatch; four logs contain unverified ranks
+(SPARTA np1 0/0/1, SPARTA np2 0/0/2, WarpX np1 second run 0/0/1, WarpX np2 second run
+1/0/1 -- runs shorter than 0.3 s that the nvidia-smi sampling missed): a GPU-binding
+evidence gap for those runs, not a correctness signal; their correctness comes from the
+validator output, the binding evidence from the np4 and the other runs.
+**Not re-run on the GPU in this round**: CP2K, QMCPACK, DFT-FE, GEOS -- their change is
+the one-line run-directory variable; their validation status above is the historical
+result of 2026-09-05/06 (commits listed below), and their checkers are covered offline
+by `test_l3_validators.sh`. A GPU regression of these four under the new directory
+logic is a listed follow-up. CPU test suites at the same HEAD: `level3/tools/tests/run_all.sh`
+(four groups) and `level2/tools/tests/run_all.sh` (topology self-test, deps markers,
+env dep profiles, launcher dry-run, run.sh guards; test scripts last changed in 7f79d9c,
+2026-09-04) all pass -- this says nothing about the open items listed per application.
+- Private Toolkit/compiler exceptions: none for the CUDA Toolkit (13.2.78
+  everywhere); private compilers: LLVM 23.1.0 for QMCPACK (required by upstream's
+  GPU path), system GCC 14.2.1 (CP2K, DFT-FE, GEOS: one compiler for C/C++/Fortran)
+  vs conda GCC 13.3.0 (Nyx, first batch). No `/usr/local/cuda` link or driver touched.
+
+## Raw material
+
+- Builds/logs/fingerprints: `.deps/level3/<app>/<profile>/{logs,install/BUILD_INFO.txt,install/.hpcperf-l3-fingerprint}`
+  (LLVM/CP2K toolchain sources on `/tmp/hpcperf-l3-b2-scratch/`, logs copied under `.deps`).
+- Runs: `build/level3/<app>/<profile>/run/<case>.<mode>.np<N>[.t<T>]/` with
+  `run_manifest.txt` (run_id, exit code, binary/deck/input hashes, fingerprint
+  hash, GPU evidence), full stdout/stderr, validator summaries
+  (`validate.*.stdout`, `md_summary.txt`, `qmc_summary.txt`, `check_vs_*.txt`);
+  dry-runs under `.dryrun/`.
+- Patches: `level3/<app>/patches/*.patch` (header: source, rationale, conditions, impact, verification).
+
+## Commits on `level3/second-batch-bringup`
+
+366b72f first-batch checkpoint; tools 9327c26 (per-profile paths, static-cudart
+backend check), 0b9daaf (conda build variables cleared, real grep), 32e25e9 (foreign
+include/library search paths cleared), 352cbb3 (validator negative tests, 17/17);
+Nyx 769482f, 1f7e14f, c7b90f9; CP2K b04db93, 7408614 (toolchain OpenBLAS relink +
+BLAS guard); QMCPACK 166657e, 1e5d00d (population guard, 256-walker series); DFT-FE
+62f48e5, e91119c (ELPA probe fixed, validation 1/2/4 PASS); GEOS e354870 (build +
+beam validation), fbaa962 (unit-test probe findings); documentation commit (this
+file, `APPLICATION_AUDIT.md`, `BUILD_STRATEGY.md`); 2026-09-07: 6711980 (clean-env
+wrapper, `HPCPERF_L3_RUN_SUBDIR`), f231217 (Nyx strict comparator, PENDING verdict,
+tolerance provenance), fc4d2a1 (Nyx run.sh run-dir variable), 74c1d4d (CLAUDE.md);
+2026-09-08: box-layout classification, verdict classes, credential deny-rule, this
+status. The branch (first batch + second batch + shared helpers) is submitted as a
+Draft PR against `main` on 2026-09-08; no merge.
diff --git a/level3/cp2k/README.md b/level3/cp2k/README.md
new file mode 100644
index 0000000..860a4ae
--- /dev/null
+++ b/level3/cp2k/README.md
@@ -0,0 +1,124 @@
+# CP2K -- Level 3 second batch
+
+Quickstep GPW density-functional theory (SCF/OT) and Born-Oppenheimer MD,
+psmp binary (MPI + OpenMP + CUDA), GPU-accelerated DBCSR (sparse matrix
+multiply, JIT libsmm_acc kernels), DBM, GRID (collocate/integrate) and PW (FFT).
+
+## Provenance / versions
+
+| Item | Value |
+|---|---|
+| CP2K | tag `v2026.2`, `67b5da876dd6a76b8b021d5a04d1c81ba79a4c50` (2026-07-15), GPL-2.0-or-later; CMake-only build |
+| Dependencies | upstream's own bootstrap `tools/toolchain/install_cp2k_toolchain.sh` (from the same tag): OpenBLAS 0.3.33 (SAPPHIRERAPIDS detected), ScaLAPACK 2.2.3, FFTW 3.3.11, libint 2.13.1 (lmax 5), libxc 7.0.0, LIBXSMM 2.0.0 / LIBXS 1.0.0, spglib 2.7.0, **DBCSR 2.10.0** (sha256 `3d897220...`), Eigen 5.0.1; ELPA/COSMA/SIRIUS/tblite/libvori/HDF5/PLUMED/... **off** (minimal GPW stack) |
+| Compilers | **system GCC 14.2.1** for C/C++/Fortran (the conda GCC 13.3 has no gfortran; one GCC for all three languages avoids the mixed-LTO/PIE issues met by SPECFEM/nekRS); nvcc 13.2.78 with g++ 14.2.1 host (probe OK on B200) |
+| MPI | conda Open MPI 5.0.10 (site-validated transport), wrappers redirected via `OMPI_CC/CXX/FC` (probe: C and `mpi_f08` programs build and run with 2 ranks) |
+| GPU target | sm_100 (`-DCMAKE_CUDA_ARCHITECTURES=100`, accepted by v2026.2's CMake); toolchain `--gpu-ver=B200` |
+| Build strategy | NATIVE + upstream toolchain (Spack rejected: recipes cap `cuda_arch` at 90 and the local Spack is 2025-05) |
+| Profile | `cuda132-gcc142-ompi5010` |
+
+### B200 support: upstream backport, not an unsourced sed
+
+v2026.2 knows H100/GB10 but not B200; upstream master added B200 in commit
+`378b2fab2f20224e5468d3279fa7f051403977c1` ("Add NVIDIA B200 GPU support
+(#5788)", 2026-08-20). `patches/0001-toolchain-b200-backport-cp2k-378b2fab.patch`
+back-ports that commit's two toolchain hunks (`--gpu-ver=B200 -> ARCH_NUM 100`;
+`install_dbcsr.sh`: add `B200 -> GPU_ARCH_NUMBER 100` to DBCSR 2.10.0's
+CMakeLists and copy `parameters_H100.json -> parameters_B200.json`) to the
+v2026.2 toolchain (class B). The CMakeLists.txt hunk of that commit is not needed
+because CP2K itself is configured with `CMAKE_CUDA_ARCHITECTURES=100`. So:
+**DBCSR device code is native sm_100; the libsmm_acc kernel *parameters* are the
+H100 set reused, not B200-tuned** (upstream's own approach for GB10 and B200).
+
+### Node adaptations (class C, recorded)
+
+- DBCSR 2.10.0's `cmake/GetGitRevisionDescription.cmake` aborts when its source
+  lives inside a git *worktree* (it treats the absolute gitdir in the `.git` file
+  as relative): the toolchain's private copy therefore lives on local scratch
+  (`/tmp/hpcperf-l3-b2-scratch/cp2k-toolchain/<profile>`, symlinked from
+  `.deps/level3/cp2k/<profile>/src/toolchain`); the install prefix, `setup`,
+  `toolchain.conf` and all logs stay under the profile tree.
+- DBCSR's own ctest suite launches `mpiexec -n 4` itself; under this 1-task-slot
+  Slurm allocation PRRTE's slot accounting is relaxed for those launches
+  (`PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe`), the same bookkeeping
+  relaxation the common launcher applies (4 ranks on 4 GPUs, no GPU sharing).
+
+### BLAS/LAPACK actually linked: three attempts (recorded under `install/ATTEMPT-*`)
+
+1. First build: CP2K's CMake found the toolchain OpenBLAS (`-L.../openblas-0.3.33/lib
+   -lopenblas`), but the conda `LDFLAGS` leaked into the link (`-Wl,--disable-new-dtags`
+   + the MPI wrapper's `-Wl,-rpath <conda lib>`), so `libopenblas.so.0` resolved at
+   run time to the **conda pthreads OpenBLAS** ("OpenBLAS Warning : Detect OpenMP
+   Loop" in every run). All validations of that build passed (numerics are
+   BLAS-implementation independent within the tolerances), but the configuration
+   was not the recorded one and the OpenMP x pthreads oversubscription made its
+   timings meaningless -- found 2026-09-06 while preparing the strong-scaling runs.
+2. `l3_clean_conda_build_env` + `CMAKE_INSTALL_RPATH` with the toolchain dirs: the
+   wrapper's rpath still came first in the link line -> same resolution; `run.sh`'s
+   new guard (`ldd libcp2k.so` must resolve BLAS under the toolchain) refused every
+   run of this build.
+3. `CP2K_BLAS_VENDOR=CUSTOM` with the toolchain `libopenblas.a` and the toolchain
+   rpath in the linker flags: CMake still records a dynamic `libopenblas.so.0`
+   dependency, but the RPATH now lists the toolchain directories first and `ldd`
+   resolves it to `toolchain/openblas-0.3.33/lib/libopenblas.so.0` (OpenBLAS 0.3.33,
+   `USE_OPENMP=1`, the toolchain's own build). This is the validated configuration;
+   `run_manifest.txt` records `blas_resolved=`.
+
+## DBCSR verified before CP2K (build.sh stage B)
+
+DBCSR 2.10.0 test build (`USE_ACCEL=cuda WITH_GPU=B200`, MPI + OpenMP,
+`BUILD_TESTING=ON`, libdbcsr.a device code `sm_100`), its official ctest suite
+with 4 MPI ranks x 4 OpenMP threads on the 4 B200s: **19/19 passed** (10
+dbcsr_perf inputs, dbcsr_unittest1-4, tensor/tas unit tests, csr conversions,
+dbcsr_test, dbcsr_tensor_test). Log: `.deps/level3/cp2k/<profile>/logs/dbcsr-ctest.log`.
+
+## Cases and validation
+
+`run.sh`: `h2o` (benchmarks/QS/H2O-<S>.inp, upstream's GPW-DFT MD benchmark, 10 NVE
+steps; smoke S=64, strong S=128 fixed over rank counts, weak = **size sweep**
+S=32*N -- not a strict weak-scaling series, see run.sh header) and `regtest`
+(a single upstream regression input). One MPI rank per GPU, OpenMP threads per
+rank (`HPCPERF_CPUS_PER_RANK`, default 8), input used verbatim, `CP2K_DATA_DIR`
+= the checkout's data/.
+
+`validate.sh`: [1] an adapted subset of upstream's regression tests
+(`tests/QS/regtest-gpw-1`: Ar, H2O-geoopt, pyridine; `regtest-dm-ls-scf-1`:
+H2-big-1, H2-big-5) run on N GPUs and compared with the upstream reference
+values and tolerances from `TEST_FILES.toml` through the same matcher
+definitions (`tests/matchers.py`: `E_total` = last "Total energy:" col 3, `M011`
+= last "ENERGY| Total FORCE_EVAL" col 9); [2] H2O-64 MD checked by
+`cp2k_md_summary.py --check`: 10 steps reached, **every MD-step SCF cycle
+converged**, finite energies, GPU evidence from CP2K's own output (cp2kflags
+`offload_cuda dbcsr_acc`, `DBCSR| ACC: Number of devices/node >= 1`, GRID task
+statistics with tasks executed on the GPU, `pw_gpu_*` timers when they reach the
+timing report); for N > 1 the per-step `ENERGY| Total FORCE_EVAL` energies of MD
+steps 1..10 must agree with the 1-GPU run within **1e-8 Ha** (pre-fixed;
+upstream's `check-release-comparison.py` demands 1e-10 across CPU MPIxOMP
+layouts -- printed as well).
+
+**The initial SCF of upstream's H2O-64 deck does not converge -- by design.**
+`benchmarks/QS/H2O-*.inp` start from `SCF_GUESS ATOMIC` with the default
+`MAX_SCF 50`, no outer SCF, and declare `IGNORE_CONVERGENCE_FAILURE`; the first
+cycle stops after 50 OT/DIIS iterations ("Leaving inner SCF loop after reaching
+50 steps", gradient 4e-5) and MD starts from that state. The checker reports this
+explicitly (`initial_scf_converged=False initial_scf_iterations=50
+deck_ignore_convergence_failure=True`) and would FAIL if the deck did not declare
+`IGNORE_CONVERGENCE_FAILURE` or if any of the 10 MD-step SCF cycles were not
+converged (negative-tested in `level3/tools/tests/test_l3_validators.sh`).
+The deck is used verbatim; loosening/repairing it was not attempted.
+
+### Results (2026-09-06, profile `cuda132-gcc142-ompi5010`, 8 OpenMP threads per rank)
+
+| GPUs | regtests (5) vs upstream refs | H2O-64 MD | FORCE_EVAL energies vs 1 GPU (steps 1..10) | launcher audit |
+|---|---|---|---|---|
+| 1 | all within tolerance (max \|diff\| 6.0e-14, tol 8e-14..3e-13) | 10/10 MD SCFs converged, GRID GPU tasks 7.5e7, DBCSR ACC 1 device | -- | 1 verified |
+| 2 | all within tolerance | 10/10, GRID GPU 3.7e7, `pw_gpu_c1dr3d_3d_ps/pw_gpu_r3dc1d_3d_ps` in timing report | max 5.7e-12 Ha (1e-10 also met) | 2 verified, 0 mismatch |
+| 4 | all within tolerance | 10/10, GRID GPU 1.9e7, pw_gpu timers present | max 8.6e-12 Ha (1e-10 also met) | 4 verified, 0 mismatch |
+
+VALIDATED_PASS at 1/2/4 GPUs -- first with the attempt-1 binary (conda BLAS, see above) and
+again with the attempt-3 binary (toolchain OpenBLAS; 2026-09-06 04:09-04:12 UTC: regtests within
+upstream tolerances, H2O-64 10/10 MD-step SCFs converged, FORCE_EVAL energies vs 1 GPU: max 1.1e-11
+Ha on 2 GPUs, 8.6e-12 Ha on 4 GPUs, `blas_resolved=` recorded in every manifest). Runs:
+`build/level3/cp2k/cuda132-gcc142-ompi5010/run/` (`validate.*.stdout`, per-run `cp2k.out`,
+`md_summary.txt`, `run_manifest.txt`). Strong (H2O-128) / size-sweep timings:
+`SECOND_BATCH_STATUS.md` (H2O-128 on 1 GPU: 119.8 s with the toolchain OpenBLAS vs 204.9 s
+with the conda pthreads OpenBLAS of attempt 1 -- the BLAS mix-up was also a 1.7x slowdown).
diff --git a/level3/cp2k/build.sh b/level3/cp2k/build.sh
new file mode 100755
index 0000000..941c11b
--- /dev/null
+++ b/level3/cp2k/build.sh
@@ -0,0 +1,186 @@
+#!/usr/bin/env bash
+# Build CP2K v2026.2 (psmp: MPI + OpenMP + CUDA) with upstream's own dependency
+# bootstrap (tools/toolchain/install_cp2k_toolchain.sh, "NATIVE+toolchain") and
+# upstream's CMake, in a private profile tree. Staged:
+#   [A] toolchain: OpenBLAS, ScaLAPACK, FFTW3, libint (lmax 5), libxc, LIBXSMM/LIBXS,
+#       spglib, DBCSR 2.10.0 (CPU + CUDA sm_100)   -> .deps/level3/cp2k/<profile>/install/toolchain
+#   [B] DBCSR official tests (its own ctest suite, BUILD_TESTING=ON, CUDA sm_100, MPI+OpenMP)
+#       -> build/dbcsr-test; results in logs/dbcsr-ctest.log  (gate: all tests must pass)
+#   [C] CP2K CMake configure/build/install                    -> install/cp2k/bin/cp2k.psmp
+#
+#   ./build.sh [CUDA]                 (HIP: not attempted here -- no ROCm on this node)
+#   HPCPERF_BUILD_JOBS=N              (default 32; also NPROCS_OVERWRITE for the toolchain)
+#   HPCPERF_CP2K_PROFILE=<name>       (override the derived profile name)
+#   HPCPERF_CP2K_SKIP_DBCSR_TEST=1    (debug only: skip stage B; the fingerprint then records dbcsr_tests=SKIPPED)
+#
+# Toolchain (profile cuda132-gcc142-ompi5010):
+#   compilers   system GCC 14.2.1 (gcc/g++/gfortran) for everything incl. the nvcc host
+#               compiler (the conda GCC 13.3 has no gfortran; a single GCC for C/C++/
+#               Fortran avoids the mixed-GCC LTO/PIE issues met by SPECFEM/nekRS)
+#   MPI         conda Open MPI 5.0.10 (the site-validated launcher/transport); wrappers
+#               redirected to the system GCC via OMPI_CC/OMPI_CXX/OMPI_FC (probe: C and
+#               mpi_f08 Fortran programs build and run with 2 ranks)
+#   CUDA        13.2.78, sm_100 (--gpu-ver=B200 -> ARCH_NUM 100 via the backported upstream
+#               patch in patches/; CP2K itself gets -DCMAKE_CUDA_ARCHITECTURES=100, which
+#               v2026.2 accepts natively)
+#   options     --with-elpa=no --with-cosma=no --with-sirius=no --with-tblite=no --with-libvori=no
+#               (minimal GPW-DFT stack: the selected cases use OT, not diagonalisation);
+#               everything the toolchain would download beyond that is left off
+# Modification classes: B (toolchain scripts: patches/0001-*, upstream backport) + C
+# (OMPI_* compiler redirection, NPROCS_OVERWRITE). No CP2K or DBCSR source changed by us;
+# the DBCSR CMakeLists/parameters edit is upstream's own sed (see the patch header).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+l3_isolate_build_env
+l3_clean_conda_build_env   # conda's LDFLAGS carry -Wl,--disable-new-dtags/-rpath <conda lib>: they made cp2k.psmp/libcp2k.so
+                           # resolve libopenblas.so.0 to the conda OpenBLAS (pthreads build) instead of the toolchain's (found 2026-09-06)
+
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+[ "$BACKEND" = CUDA ] || { echo "build.sh: only CUDA is implemented for CP2K here (HIP: no ROCm on this node, UNTESTED)" >&2; exit 2; }
+SRC="$R/_upstream/level3/cp2k"
+[ -f "$SRC/CMakeLists.txt" ] && [ -f "$SRC/tools/toolchain/install_cp2k_toolchain.sh" ] || { echo "build.sh: run $HERE/fetch.sh first" >&2; exit 1; }
+SHA="$(git -C "$SRC" rev-parse HEAD)"
+# system GCC 14 toolchain for C/C++/Fortran; conda Open MPI wrappers redirected to it
+export CC=/usr/bin/gcc CXX=/usr/bin/g++ FC=/usr/bin/gfortran F90=/usr/bin/gfortran F77=/usr/bin/gfortran
+export OMPI_CC=/usr/bin/gcc OMPI_CXX=/usr/bin/g++ OMPI_FC=/usr/bin/gfortran CUDAHOSTCXX=/usr/bin/g++
+for c in gcc g++ gfortran; do [ "$(command -v $c)" = "/usr/bin/$c" ] || { echo "build.sh: '$c' resolves to $(command -v $c), expected /usr/bin/$c" >&2; exit 1; }; done
+for c in mpicc mpic++ mpifort mpiexec cmake python3 nvcc; do command -v $c >/dev/null || { echo "build.sh: $c not in PATH" >&2; exit 1; }; done
+GCC_MM="$(l3_version_mm "$(/usr/bin/gcc -dumpfullversion)")"
+OMPI_V="$(mpirun --version 2>/dev/null | head -1 | /usr/bin/grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)"
+ARCH="${HPCPERF_CUDA_ARCH:-$(l3_gpu_arch)}"; [ "$ARCH" = 100 ] || { echo "build.sh: this profile is defined for sm_100 (B200); detected sm_$ARCH -- set HPCPERF_CUDA_ARCH deliberately if you mean it" >&2; exit 1; }
+GPUVER=B200
+PROFILE="${HPCPERF_CP2K_PROFILE:-cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-ompi$(echo "$OMPI_V" | tr -d .)}"
+l3_paths_profile cp2k "$PROFILE"
+JOBS="${HPCPERF_BUILD_JOBS:-32}"; export NPROCS_OVERWRITE="$JOBS"
+# The toolchain's private source/build copy lives on the node's LOCAL disk, outside any git
+# work tree: DBCSR 2.10.0's cmake/GetGitRevisionDescription.cmake walks up from its source
+# directory, finds this repository's `.git` *file* (git worktree) and mis-resolves the
+# absolute gitdir it points to as a relative path -> "file failed to open for reading" and
+# a configure abort. Everything the later stages need (install prefix, setup/toolchain.conf,
+# logs) stays under the profile tree; $L3_SRC/toolchain is a symlink to the scratch copy.
+TC_SCRATCH="${HPCPERF_CP2K_TOOLCHAIN_SCRATCH:-/tmp/hpcperf-l3-b2-scratch/cp2k-toolchain/$PROFILE}"
+mkdir -p "$(dirname "$TC_SCRATCH")"; [ -L "$L3_SRC/toolchain" ] || { rm -rf "$L3_SRC/toolchain"; ln -sfn "$TC_SCRATCH" "$L3_SRC/toolchain"; }
+TC_SRC="$TC_SCRATCH"; TC_INSTALL="$L3_INSTALL/toolchain"; CP2K_PREFIX="$L3_INSTALL/cp2k"
+PATCHES=("$HERE/patches/0001-toolchain-b200-backport-cp2k-378b2fab.patch")
+TC_OPTS=(--install-dir="$TC_INSTALL" --mpi-mode=openmpi --math-mode=openblas --with-gcc=system --with-openmpi=system --with-cmake=system
+         --enable-cuda=yes --gpu-ver=$GPUVER --libint-lmax=5
+         --with-openblas=install --with-scalapack=install --with-fftw=install --with-libint=install --with-libxc=install
+         --with-libxsmm=install --with-libxs=install --with-dbcsr=install --with-spglib=install
+         --with-elpa=no --with-cosma=no --with-sirius=no --with-tblite=no --with-libvori=no --with-hdf5=no --with-plumed=no
+         --with-libtorch=no --with-gsl=no --with-dftd4=no --with-spla=no --with-spfft=no --with-gauxc=no --with-libsmeagol=no
+         --with-deepmd=no --with-ace=no --with-greenx=no --with-trexio=no --with-libfci=no --with-mcl=no --with-libgint=no --with-cusolvermp=no)
+# Installed binaries carry an RPATH to the toolchain library directories (and libcp2k.so) so that the BLAS/LAPACK,
+# ScaLAPACK, FFTW, libxc, ... actually used at run time are the toolchain's, independent of LD_LIBRARY_PATH ordering;
+# run.sh verifies the resolution with ldd before every run.
+TC_RPATH="$( { ls -d "$TC_INSTALL"/*/lib "$TC_INSTALL"/*/lib64 2>/dev/null || true; } | paste -sd';')"   # (ls exits 2 when no lib64 exists: keep set -e/pipefail quiet)
+# BLAS/LAPACK: the toolchain's static OpenBLAS (its own convention, MATH_LIBS="-l:libopenblas.a") through CP2K's CUSTOM
+# vendor -- a dynamic -lopenblas resolved at run time to whichever libopenblas.so.0 the loader met first (the conda MPI
+# wrapper puts its rpath before ours), which was the conda pthreads OpenBLAS in attempts 1 and 2.
+TC_OPENBLAS_A="$(ls "$TC_INSTALL"/openblas-*/lib/libopenblas.a 2>/dev/null | head -1)"
+[ -f "$TC_OPENBLAS_A" ] || { echo "build.sh: toolchain libopenblas.a not found under $TC_INSTALL" >&2; exit 1; }
+TC_RPATH_COLON="${TC_RPATH//;/:}"
+CP2K_CMAKE=(-DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON "-DCMAKE_INSTALL_PREFIX=$CP2K_PREFIX" "-DCP2K_DATA_DIR=$SRC/data"
+            "-DCMAKE_INSTALL_RPATH=$CP2K_PREFIX/lib;$TC_RPATH" -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON
+            "-DCMAKE_EXE_LINKER_FLAGS=-Wl,-rpath,$CP2K_PREFIX/lib:$TC_RPATH_COLON" "-DCMAKE_SHARED_LINKER_FLAGS=-Wl,-rpath,$CP2K_PREFIX/lib:$TC_RPATH_COLON"
+            -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ -DCMAKE_Fortran_COMPILER=/usr/bin/gfortran
+            -DCP2K_USE_MPI=ON -DCP2K_USE_MPI_F08=ON -DCP2K_USE_FFTW3=ON -DCP2K_USE_LIBXC=ON -DCP2K_USE_LIBINT2=ON
+            -DCP2K_USE_LIBXS=ON -DCP2K_USE_LIBXSMM=ON -DCP2K_USE_SPGLIB=ON
+            -DCP2K_USE_ELPA=OFF -DCP2K_USE_COSMA=OFF -DCP2K_USE_SIRIUS=OFF -DCP2K_USE_TBLITE=OFF -DCP2K_USE_VORI=OFF -DCP2K_USE_DFTD4=OFF
+            -DCP2K_USE_HDF5=OFF -DCP2K_USE_PLUMED=OFF -DCP2K_USE_LIBTORCH=OFF -DCP2K_USE_GAUXC=OFF -DCP2K_USE_GREENX=OFF -DCP2K_USE_TREXIO=OFF
+            -DCP2K_USE_ACE=OFF -DCP2K_USE_DEEPMD=OFF -DCP2K_USE_LIBFCI=OFF -DCP2K_USE_MIMIC=OFF -DCP2K_USE_LIBSMEAGOL=OFF -DCP2K_USE_SPLA=OFF
+            -DCP2K_BLAS_VENDOR=CUSTOM "-DCP2K_BLAS_LINK_LIBRARIES=$TC_OPENBLAS_A" "-DCP2K_LAPACK_LINK_LIBRARIES=$TC_OPENBLAS_A" -DCP2K_SCALAPACK_VENDOR=GENERIC
+            -DCP2K_USE_ACCEL=CUDA "-DCMAKE_CUDA_ARCHITECTURES=$ARCH" -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++)
+CMAKE_OPTS="blas=CUSTOM:toolchain-libopenblas.a(static) install_rpath=toolchain-first ${CP2K_CMAKE[*]} | toolchain: ${TC_OPTS[*]}"
+DEPS="toolchain(install_cp2k_toolchain.sh v2026.2 + backport 378b2fab) dbcsr=2.10.0(sha256 3d897220fbb4498215331efad6905eb7744881b4cf04eb5c5fb4db7c48a56ef9; B200 entry=arch 100, libsmm_acc parameters=H100 reused) openblas/scalapack/fftw3/libint(lmax5)/libxc/libxsmm/libxs/spglib=toolchain pins gcc=$(/usr/bin/gcc -dumpfullversion) openmpi=$OMPI_V profile=$PROFILE dbcsr_tests=$( [ -n "${HPCPERF_CP2K_SKIP_DBCSR_TEST:-}" ] && echo SKIPPED || echo required)"
+FP="$(l3_fingerprint_text cp2k "$SHA" cuda "$DEPS" "$CMAKE_OPTS" "not-used(DBCSR/DBM communicate through host buffers)" "${PATCHES[@]}")"
+l3_fingerprint_check "$L3_INSTALL" "$FP" || exit 1
+
+echo "# CP2K $BACKEND profile=$PROFILE: cp2k $SHA (v2026.2), gcc $(/usr/bin/gcc -dumpfullversion), gfortran $(/usr/bin/gfortran -dumpfullversion), $(mpirun --version | head -1), CUDA $(l3_cuda_version) sm_$ARCH, -j$JOBS"
+echo "# resources: toolchain ~1.5-3 h (libint lmax 5 dominates), DBCSR tests ~10 min, CP2K ~40-90 min; disk ~10-15 GB under $L3_DEPS"
+t0=$(date +%s)
+
+# [A] toolchain (private, patched copy of tools/toolchain; the checkout is never written to)
+if [ ! -f "$TC_INSTALL/.hpcperf-stage-done" ]; then
+    if [ ! -f "$TC_SRC/.hpcperf-patched" ]; then
+        rm -rf "$TC_SRC"; mkdir -p "$TC_SRC"; cp -r "$SRC/tools/toolchain/." "$TC_SRC/"
+        for p in "${PATCHES[@]}"; do (cd "$TC_SRC" && patch -p1 < "$p") > "$L3_LOGS/patch-$(basename "$p").log" 2>&1 || { cat "$L3_LOGS/patch-$(basename "$p").log"; echo "build.sh: patch $p failed" >&2; exit 1; }; done
+        sha256sum "${PATCHES[@]}" > "$TC_SRC/.hpcperf-patched"
+    fi
+    echo "# [A] toolchain: ${TC_OPTS[*]}"
+    # the toolchain installer writes `declare -x` of its whole environment into <install>/toolchain.env:
+    # run it under the allow-listed environment so that no login-shell secret can end up in that file
+    ( cd "$TC_SRC" && unset CMAKE_GENERATOR && l3_clean_env_exec ./install_cp2k_toolchain.sh "${TC_OPTS[@]}" ) > "$L3_LOGS/toolchain.log" 2>&1 \
+        || { tail -60 "$L3_LOGS/toolchain.log"; echo "build.sh: toolchain failed (log: $L3_LOGS/toolchain.log)" >&2; exit 1; }
+    [ -f "$TC_INSTALL/setup" ] && [ -f "$TC_INSTALL/toolchain.conf" ] || { echo "build.sh: toolchain produced no setup/toolchain.conf under $TC_INSTALL" >&2; exit 1; }
+    /usr/bin/grep -q 'GPU_ARCH_NUMBER_B200 100' "$TC_SRC/build/dbcsr-2.10.0/CMakeLists.txt" || { echo "build.sh: DBCSR CMakeLists did not receive the B200 (arch 100) entry" >&2; exit 1; }
+    touch "$TC_INSTALL/.hpcperf-stage-done"
+fi
+t1=$(date +%s)
+set +u; # shellcheck disable=SC1091
+source "$TC_INSTALL/setup"; set -u
+# the toolchain's setup may re-point compilers; keep the system GCC and the conda MPI wrappers
+export CC=/usr/bin/gcc CXX=/usr/bin/g++ FC=/usr/bin/gfortran OMPI_CC=/usr/bin/gcc OMPI_CXX=/usr/bin/g++ OMPI_FC=/usr/bin/gfortran
+
+# [B] DBCSR official tests on the GPU (from the toolchain's patched DBCSR source)
+DB_SRC="$TC_SRC/build/dbcsr-2.10.0"; DB_TEST="$L3_BUILD_DEPS/dbcsr-test"
+if [ -z "${HPCPERF_CP2K_SKIP_DBCSR_TEST:-}" ] && [ ! -f "$DB_TEST/.hpcperf-stage-done" ]; then
+    [ -f "$DB_SRC/CMakeLists.txt" ] || { echo "build.sh: DBCSR source $DB_SRC missing" >&2; exit 1; }
+    echo "# [B] DBCSR 2.10.0 tests: USE_ACCEL=cuda WITH_GPU=$GPUVER (arch $ARCH) USE_MPI=ON USE_OPENMP=ON BUILD_TESTING=ON"
+    mkdir -p "$DB_TEST"
+    cmake -S "$DB_SRC" -B "$DB_TEST" -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_TESTING=ON -DWITH_EXAMPLES=OFF \
+        -DUSE_MPI=ON -DUSE_MPI_F08=ON -DUSE_OPENMP=ON -DUSE_LIBXS=ON -DUSE_LIBXSMM=ON -DUSE_ACCEL=cuda "-DWITH_GPU=$GPUVER" \
+        -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++ -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ -DCMAKE_Fortran_COMPILER=/usr/bin/gfortran \
+        -DTEST_MPI_RANKS="${HPCPERF_CP2K_DBCSR_TEST_RANKS:-4}" -DTEST_OMP_THREADS=4 > "$L3_LOGS/dbcsr-test-configure.log" 2>&1 \
+        || { tail -30 "$L3_LOGS/dbcsr-test-configure.log"; echo "build.sh: DBCSR test configure failed" >&2; exit 1; }
+    cmake --build "$DB_TEST" -j "$JOBS" > "$L3_LOGS/dbcsr-test-build.log" 2>&1 || { tail -30 "$L3_LOGS/dbcsr-test-build.log"; echo "build.sh: DBCSR test build failed" >&2; exit 1; }
+    for l in "$DB_TEST"/src/libdbcsr*.so "$DB_TEST"/src/libdbcsr*.a; do [ -f "$l" ] || continue; echo "$(basename "$l"): $(cuobjdump --list-elf "$l" 2>/dev/null | /usr/bin/grep -o 'sm_[0-9]*' | sort -u | paste -sd,)"; done > "$L3_LOGS/dbcsr-test-archs.txt" || true
+    # ctest with the launcher's single-node transport. DBCSR's tests call `mpiexec -n 4` directly
+    # (no launcher wrapper): each rank binds device rank%ndev -> 4 ranks on 4 distinct GPUs
+    # (ranks <= GPUs). The Slurm allocation exposes 1 task slot, so PRRTE's slot accounting is
+    # relaxed for these launches exactly as the common launcher does (--map-by ...:OVERSUBSCRIBE
+    # bookkeeping; no GPU sharing).
+    ( cd "$DB_TEST" && OMPI_MCA_pml=ob1 OMPI_MCA_btl=self,sm,smcuda OMP_NUM_THREADS=4 \
+        PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe OMPI_MCA_rmaps_base_oversubscribe=true \
+        ctest --output-on-failure --timeout 1800 -j 1 ) > "$L3_LOGS/dbcsr-ctest.log" 2>&1 || true
+    if /usr/bin/grep -qE '100% tests passed, 0 tests failed' "$L3_LOGS/dbcsr-ctest.log"; then
+        touch "$DB_TEST/.hpcperf-stage-done"; /usr/bin/grep -E 'tests passed|Total Test time' "$L3_LOGS/dbcsr-ctest.log"
+    else
+        /usr/bin/grep -E 'tests passed|Failed|\*\*\*|Not Run|Timeout' "$L3_LOGS/dbcsr-ctest.log" | head -20
+        echo "build.sh: DBCSR official tests did not all pass on the GPU -- CP2K is not built on top of an unverified DBCSR (log: $L3_LOGS/dbcsr-ctest.log)" >&2; exit 1
+    fi
+fi
+t2=$(date +%s)
+
+# [C] CP2K
+echo "# [C] CP2K cmake: ${CP2K_CMAKE[*]}"
+mkdir -p "$L3_BUILD"
+cmake -S "$SRC" -B "$L3_BUILD" -G Ninja "${CP2K_CMAKE[@]}" > "$L3_LOGS/cp2k-configure.log" 2>&1 \
+    || { tail -60 "$L3_LOGS/cp2k-configure.log"; echo "build.sh: CP2K configure failed (log: $L3_LOGS/cp2k-configure.log)" >&2; exit 1; }
+cmake --build "$L3_BUILD" -j "$JOBS" > "$L3_LOGS/cp2k-build.log" 2>&1 \
+    || { tail -60 "$L3_LOGS/cp2k-build.log"; echo "build.sh: CP2K build failed (log: $L3_LOGS/cp2k-build.log)" >&2; exit 1; }
+cmake --install "$L3_BUILD" > "$L3_LOGS/cp2k-install.log" 2>&1 || { echo "build.sh: CP2K install failed" >&2; exit 1; }
+t3=$(date +%s)
+EXE="$CP2K_PREFIX/bin/cp2k.psmp"
+[ -x "$EXE" ] || { echo "build.sh: $EXE not produced" >&2; exit 1; }
+# device code lives in libcp2k.so (the psmp executable itself carries none); a `[ -f ] &&` chain
+# must not propagate a failing last test into the substitution under set -e
+archs="$(for l in "$EXE" "$CP2K_PREFIX"/lib/libcp2k*.so* "$CP2K_PREFIX"/lib64/libcp2k*.so*; do if [ -f "$l" ]; then cuobjdump --list-elf "$l" 2>/dev/null | /usr/bin/grep -o 'sm_[0-9]*' || true; fi; done | sort -u | paste -sd,)"
+[ "$archs" = "sm_$ARCH" ] || { echo "build.sh: CP2K device code embeds '$archs', expected sm_$ARCH -- refusing" >&2; exit 1; }
+l3_fingerprint_write "$L3_INSTALL" "$FP"
+{
+    echo "profile=$PROFILE cp2k=$SHA utc=$(date -u +%FT%TZ) jobs=$JOBS"
+    echo "seconds: toolchain=$((t1 - t0)) dbcsr_tests=$((t2 - t1)) cp2k=$((t3 - t2)) total=$(( $(date +%s) - t0 ))"
+    echo "cp2k.psmp sha256=$(l3_sha_file "$EXE") device_archs=$archs"
+    echo "cp2k version line: $(LD_LIBRARY_PATH="$CP2K_PREFIX/lib:$CP2K_PREFIX/lib64:${LD_LIBRARY_PATH:-}" "$EXE" --version 2>/dev/null | head -3 | paste -sd'|' || true)"
+    echo "runtime note: cp2k.psmp has no rpath to libcp2k.so -- run.sh sources $TC_INSTALL/setup and prepends $CP2K_PREFIX/lib to LD_LIBRARY_PATH"
+    echo "toolchain.conf:"; sed 's/^/  /' "$TC_INSTALL/toolchain.conf"
+    [ -f "$L3_LOGS/dbcsr-test-archs.txt" ] && { echo "dbcsr test libs:"; sed 's/^/  /' "$L3_LOGS/dbcsr-test-archs.txt"; }
+    /usr/bin/grep -E 'tests passed' "$L3_LOGS/dbcsr-ctest.log" 2>/dev/null | sed 's/^/dbcsr ctest: /'
+} > "$L3_INSTALL/BUILD_INFO.txt"
+echo "# built: toolchain $((t1 - t0)) s, DBCSR tests $((t2 - t1)) s, CP2K $((t3 - t2)) s -> $EXE ($archs)"
+exit 0
diff --git a/level3/cp2k/cp2k_md_summary.py b/level3/cp2k/cp2k_md_summary.py
new file mode 100755
index 0000000..31efd03
--- /dev/null
+++ b/level3/cp2k/cp2k_md_summary.py
@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+"""Summarise (and with --check, validate) a CP2K Quickstep MD run for validate.sh.
+
+usage: cp2k_md_summary.py <cp2k.out> <input.inp> [--check]
+
+Prints key=value lines (md_summary.txt).  With --check the run must satisfy:
+  * "PROGRAM ENDED AT" present and MD step 10 reached;
+  * exactly one SCF cycle before MD_INI (the initial SCF from SCF_GUESS) and one per MD
+    step afterwards; EVERY MD-step SCF cycle converged.  The initial cycle may end with
+    "SCF run NOT converged" ONLY if the deck itself declares IGNORE_CONVERGENCE_FAILURE
+    (upstream's benchmarks/QS/H2O-*.inp do: the ATOMIC-guess start does not converge
+    within the default MAX_SCF=50 OT iterations by design) -- its iteration count and
+    status are reported, never hidden;
+  * all FORCE_EVAL / MD energies finite;
+  * GPU evidence from CP2K's own output: cp2kflags contain offload_cuda and dbcsr_acc,
+    DBCSR reports >= 1 accelerator device per node, and the GRID task statistics show
+    collocate/integrate tasks executed on the GPU.  PW (FFT) GPU use is reported from the
+    pw_gpu_* timers when they reach the timing report, otherwise marked unverified.
+The energies compared across rank counts are the per-cycle "ENERGY| Total FORCE_EVAL"
+values (18 decimals), efe_step0 = initial SCF, efe_step1..10 = MD steps 1..10.
+"""
+import os
+import re
+import sys
+
+sys.path.insert(0, os.environ["L3_TOOLS"])
+from l3_check import ValidationError, require_finite  # noqa: E402
+
+out, inp = sys.argv[1], sys.argv[2]
+check = "--check" in sys.argv[3:]
+txt = open(out, errors="replace").read()
+deck = open(inp, errors="replace").read()
+ignore_conv = re.search(r"^\s*IGNORE_CONVERGENCE_FAILURE\b", deck, re.M) is not None
+header = txt[: txt.find(" GLOBAL|")] if " GLOBAL|" in txt else txt[:20000]
+
+md_ini = txt.find("MD_INI| MD initialization")
+events = [(m.start(), m.group(1) is None) for m in re.finditer(r"SCF run (NOT )?converged", txt)]
+init = [c for p, c in events if md_ini < 0 or p < md_ini]
+md = [c for p, c in events if md_ini >= 0 and p > md_ini]
+m_iter = re.search(r"Leaving inner SCF loop after reaching\s+(\d+) steps", txt)
+m_conv = re.search(r"SCF run converged in\s+(\d+) steps", txt)
+init_iters = int(m_iter.group(1)) if (m_iter and init and not init[0]) else (int(m_conv.group(1)) if m_conv else -1)
+steps = [int(m) for m in re.findall(r"MD\| Step number\s+(\d+)", txt)]
+efe = [float(x) for x in re.findall(r"ENERGY\| Total FORCE_EVAL \( QS \) energy \[hartree\]\s+(\S+)", txt)]
+epot = [float(x) for x in re.findall(r"MD\| Potential energy \[hartree\]\s+(\S+)", txt)]
+cons = [float(x) for x in re.findall(r"MD\| Conserved quantity \[hartree\]\s+(\S+)", txt)]
+m = re.search(r"DBCSR\| ACC: Number of devices/node\s+(\d+)", txt)
+ndev = int(m.group(1)) if m else -1
+flags_ok = ("offload_cuda" in header) and ("dbcsr_acc" in header)
+grid = {"GPU": 0, "CPU": 0}
+for backend, cnt in re.findall(r"^\s*\d+\s+(?:collocate|integrate) (?:ortho|general)\s+(GPU|CPU)\s+(\d+)", txt, re.M):
+    grid[backend] += int(cnt)
+pw_gpu = sorted(set(re.findall(r"^\s*(pw_gpu_\w+)", txt, re.M)))
+
+try:
+    if check:
+        if "PROGRAM ENDED AT" not in txt:
+            raise ValidationError("no 'PROGRAM ENDED AT' -- run incomplete")
+        if not steps or steps[-1] != 10:
+            raise ValidationError(f"MD reached step {steps[-1] if steps else None}, expected 10")
+        if md_ini < 0:
+            raise ValidationError("no MD_INI block")
+        if len(init) != 1:
+            raise ValidationError(f"expected exactly one SCF cycle before MD_INI, found {len(init)}")
+        if not init[0] and not ignore_conv:
+            raise ValidationError("initial SCF did not converge and the deck does not declare IGNORE_CONVERGENCE_FAILURE")
+        if len(md) != 10:
+            raise ValidationError(f"expected 10 MD-step SCF cycles, found {len(md)}")
+        if not all(md):
+            raise ValidationError(f"{sum(1 for c in md if not c)} of 10 MD-step SCF cycles did not converge")
+        if len(efe) != 11:
+            raise ValidationError(f"expected 11 FORCE_EVAL energies (initial + 10 steps), found {len(efe)}")
+        if len(epot) < 10 or len(cons) < 10:
+            raise ValidationError(f"expected 10 MD energy records, got {len(epot)}/{len(cons)}")
+        for i, e in enumerate(efe):
+            require_finite(f"FORCE_EVAL energy #{i}", e)
+        for e in epot + cons:
+            require_finite("MD energy", e)
+        if ndev < 1:
+            raise ValidationError(f"DBCSR reports {ndev} accelerator devices -- GPU backend not active")
+        if not flags_ok:
+            raise ValidationError("cp2kflags lack offload_cuda/dbcsr_acc -- not a CUDA-offload binary")
+        if grid["GPU"] <= 0:
+            raise ValidationError("GRID statistics show no collocate/integrate tasks executed on the GPU")
+    print(f"steps={steps[-1] if steps else -1} initial_scf_converged={init[0] if init else None} initial_scf_iterations={init_iters} "
+          f"deck_ignore_convergence_failure={ignore_conv} md_scf_converged={sum(1 for c in md if c)}/{len(md)}")
+    print(f"dbcsr_acc_devices_per_node={ndev} cp2kflags_offload_cuda_dbcsr_acc={flags_ok} grid_gpu_tasks={grid['GPU']} grid_cpu_tasks={grid['CPU']} "
+          f"pw_gpu_timers={','.join(pw_gpu) if pw_gpu else 'absent-from-timing-report(unverified)'}")
+    print(" ".join(f"efe_step{i}={e!r}" for i, e in enumerate(efe)))
+    if epot and cons:
+        print("epot_step1=%.12f epot_step10=%.12f cons_step1=%.12f cons_step10=%.12f cons_drift=%.3e"
+              % (epot[0], epot[-1], cons[0], cons[-1], cons[-1] - cons[0]))
+except ValidationError as ex:
+    print(f"  VALIDATION ERROR: {ex}")
+    sys.exit(1)
diff --git a/level3/cp2k/fetch.sh b/level3/cp2k/fetch.sh
new file mode 100755
index 0000000..04e2c57
--- /dev/null
+++ b/level3/cp2k/fetch.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+# Fetch CP2K at the pinned release (read-only checkout under _upstream/level3/cp2k).
+#
+#   CP2K v2026.2   67b5da876dd6a76b8b021d5a04d1c81ba79a4c50  (tag v2026.2, 2026-07-15; CMake-only build,
+#                  ships tools/toolchain/install_cp2k_toolchain.sh with DBCSR 2.10.0 pinned)
+#
+# DBCSR 2.10.0 (sha256 3d897220fbb4498215331efad6905eb7744881b4cf04eb5c5fb4db7c48a56ef9) and the
+# other dependencies are downloaded by the toolchain script itself (build.sh) into
+# the private toolchain copy -- nothing is fetched into the repository.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+DEST="$R/_upstream/level3/cp2k"
+TAG=v2026.2; SHA=67b5da876dd6a76b8b021d5a04d1c81ba79a4c50
+if [ ! -d "$DEST/.git" ]; then
+    mkdir -p "$(dirname "$DEST")"
+    git clone -q --branch "$TAG" --depth 1 https://github.com/cp2k/cp2k.git "$DEST"
+fi
+got="$(git -C "$DEST" rev-parse HEAD)"
+[ "$got" = "$SHA" ] || { echo "fetch.sh: $DEST is at $got, expected $SHA ($TAG)" >&2; exit 1; }
+echo "# CP2K $TAG $got -> $DEST (data/ $(du -sh "$DEST/data" | cut -f1), benchmarks/ $(du -sh "$DEST/benchmarks" | cut -f1))"
diff --git a/level3/cp2k/patches/0001-toolchain-b200-backport-cp2k-378b2fab.patch b/level3/cp2k/patches/0001-toolchain-b200-backport-cp2k-378b2fab.patch
new file mode 100644
index 0000000..7fbeb9e
--- /dev/null
+++ b/level3/cp2k/patches/0001-toolchain-b200-backport-cp2k-378b2fab.patch
@@ -0,0 +1,92 @@
+Backport of upstream CP2K commit 378b2fab2f20224e5468d3279fa7f051403977c1
+("Add NVIDIA B200 GPU support (#5788)", 2026-08-20, master) to the v2026.2
+toolchain scripts, restricted to the two files this build uses. Class B
+(build-system only). The CP2K CMakeLists.txt hunk of that commit
+(CP2K_SUPPORTED_CUDA_ARCHITECTURES += B200, CP2K_GPU_ARCH_NUMBER_B200 100) is NOT
+needed: v2026.2 accepts -DCMAKE_CUDA_ARCHITECTURES=100 directly (build.sh uses
+that instead of the legacy -DCP2K_WITH_GPU=B200 selector). The DBCSR hunk keeps
+upstream's behaviour: DBCSR 2.10.0 predates B200, so the toolchain adds the B200
+name with GPU_ARCH_NUMBER 100 to DBCSR's CMakeLists.txt (native sm_100 device code)
+and copies parameters_H100.json to parameters_B200.json -- i.e. the libsmm_acc
+kernel PARAMETERS are the H100 ones REUSED, not tuned for B200 (correctness on
+B200 is what the DBCSR tests in build.sh verify; performance tuning is out of scope).
+Apply inside a private copy of tools/toolchain with: patch -p1
+--- a/install_cp2k_toolchain.sh
++++ b/install_cp2k_toolchain.sh
+@@ -140,7 +140,7 @@ OPTIONS:
+                           Default = native
+   --gpu-ver               Select the target GPU architecture for compiling.
+                           Available options are: K20X, K40, K80, P100, V100,
+-                          A100, H100, GB10, A40, Mi50, Mi100, Mi250, and no.
++                          A100, H100, B200, GB10, A40, Mi50, Mi100, Mi250, and no.
+                           This option determines the value of nvcc -arch flag.
+                           Default = no
+   --libint-lmax           Maximum supported angular momentum by libint if the
+@@ -720,13 +720,13 @@ Otherwise use option no."
+     --gpu-ver=*)
+       user_input="${1#*=}"
+       case "${user_input}" in
+-        K20X | K40 | K80 | P100 | V100 | A100 | H100 | GB10 | A40 | Mi50 | Mi100 | Mi250 | no)
++        K20X | K40 | K80 | P100 | V100 | A100 | H100 | B200 | GB10 | A40 | Mi50 | Mi100 | Mi250 | no)
+           export GPUVER="${user_input}"
+           ;;
+         *)
+           echo "ERROR: Invalid value for --gpu-ver found."
+           echo "Currently only one of the following options is supported:
+-            K20X, K40, K80, P100, V100, A100, H100, GB10, A40, Mi50, Mi100, Mi250.
++            K20X, K40, K80, P100, V100, A100, H100, B200, GB10, A40, Mi50, Mi100, Mi250.
+ Otherwise use option no."
+           exit 1
+           ;;
+@@ -1104,7 +1104,7 @@ if [ "${ENABLE_GAUXC_CUTLASS}" = "__TRUE__" ]; then
+     report_error ${LINENO} "--enable-gauxc-cutlass requires --enable-cuda=yes."
+   fi
+   case "${GPUVER}" in
+-    A100 | A40 | H100 | GB10) ;;
++    A100 | A40 | H100 | B200 | GB10) ;;
+     *)
+       report_error ${LINENO} "--enable-gauxc-cutlass requires CUDA compute capability >= 8.0."
+       ;;
+@@ -1287,6 +1287,9 @@ case ${GPUVER} in
+   H100)
+     export ARCH_NUM="90"
+     ;;
++  B200)
++    export ARCH_NUM="100"
++    ;;
+   GB10)
+     export ARCH_NUM="121"
+     ;;
+@@ -1305,7 +1308,7 @@ case ${GPUVER} in
+   *)
+     echo "ERROR: Invalid value for --gpu-ver found."
+     echo "Currently only one of the following options is supported:
+-      K20X, K40, K80, P100, V100, A100, H100, A40, Mi50, Mi100, Mi250.
++      K20X, K40, K80, P100, V100, A100, H100, B200, GB10, A40, Mi50, Mi100, Mi250.
+ Otherwise use option no."
+     exit 1
+     ;;
+--- a/scripts/stage9/install_dbcsr.sh
++++ b/scripts/stage9/install_dbcsr.sh
+@@ -32,14 +32,15 @@ case "${with_dbcsr}" in
+       [ -d dbcsr-${dbcsr_ver} ] && rm -rf dbcsr-${dbcsr_ver}
+       tar -xzf dbcsr-${dbcsr_ver}.tar.gz
+       cd dbcsr-${dbcsr_ver}
+-      # DBCSR may predate GB10. Build native sm_121 code while
++      # DBCSR 2.10 predates GB10 and B200. Build native device code while
+       # reusing the closest available libsmm_acc parameters.
+-      if [ "${ENABLE_CUDA}" == "__TRUE__" ] && [ "${GPUVER}" == "GB10" ]; then
+-        if ! grep -q "GB10" CMakeLists.txt; then
+-          sed -i "s/    H100)/    H100\\n    GB10)/" CMakeLists.txt
+-          sed -i "/  set(GPU_ARCH_NUMBER_H100 90)/a\\  set(GPU_ARCH_NUMBER_GB10 121)" CMakeLists.txt
++      if [ "${ENABLE_CUDA}" == "__TRUE__" ] &&
++        { [ "${GPUVER}" == "GB10" ] || [ "${GPUVER}" == "B200" ]; }; then
++        if ! grep -q "${GPUVER}" CMakeLists.txt; then
++          sed -i "s/    H100)/    H100\\n    ${GPUVER})/" CMakeLists.txt
++          sed -i "/  set(GPU_ARCH_NUMBER_H100 90)/a\\  set(GPU_ARCH_NUMBER_${GPUVER} ${ARCH_NUM})" CMakeLists.txt
+           cp src/acc/libsmm_acc/parameters/parameters_H100.json \
+-            src/acc/libsmm_acc/parameters/parameters_GB10.json
++            "src/acc/libsmm_acc/parameters/parameters_${GPUVER}.json"
+         fi
+       fi
+       mkdir build-cpu
diff --git a/level3/cp2k/run.sh b/level3/cp2k/run.sh
new file mode 100755
index 0000000..0856570
--- /dev/null
+++ b/level3/cp2k/run.sh
@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+# Run CP2K (cp2k.psmp: MPI + OpenMP + CUDA) on N GPUs, one MPI rank per GPU with
+# OpenMP threads per rank -- the hybrid execution model upstream ships as
+# "psmp" and uses for its GPU regression tests.
+#
+#   ./run.sh [CUDA]
+#
+# Cases (HPCPERF_CP2K_CASE), upstream inputs from the read-only checkout:
+#   h2o (default)   benchmarks/QS/H2O-<S>.inp -- upstream's Quickstep GPW-DFT benchmark:
+#                   Born-Oppenheimer NVE MD of liquid water (TZV2P, 280 Ry, LDA/Pade, OT/DIIS),
+#                   10 MD steps, S water molecules (32/64/128/256/...):
+#                     smoke : S = HPCPERF_CP2K_SYSTEM (default 64)      -- correctness case
+#                     strong: S = HPCPERF_CP2K_SYSTEM (default 128), fixed across rank counts
+#                     weak  : S = 32*N (N ranks) -- a SIZE SWEEP, not a strict weak-scaling series:
+#                             the GPW/OT cost per electron is not constant with system size, so
+#                             "equal work per GPU" is not well defined here (see README)
+#   regtest         a single upstream regression-test input (HPCPERF_CP2K_REGTEST=<dir/file
+#                   under tests/>, e.g. QS/regtest-gpw-1/Ar.inp); validate.sh uses this with the
+#                   reference values/tolerances from the directory's TEST_FILES.toml
+#
+# Controls:
+#   HPCPERF_GPUS=N|all             MPI ranks (= GPUs; default 1)
+#   HPCPERF_CPUS_PER_RANK=T        OpenMP threads per rank (default 8; ranks*T <= CPUs/node enforced by the launcher)
+#   HPCPERF_SCALE_MODE             smoke|strong|weak (default smoke)
+#   HPCPERF_CP2K_SYSTEM            water molecules for h2o smoke/strong
+#   HPCPERF_CP2K_PROFILE           build profile (default cuda<tk>-gcc<v>-ompi<v>)
+# The input file is used VERBATIM (copied next to the output); CP2K_DATA_DIR points at
+# the checkout's data/ (basis sets, potentials). GPU binding: the launcher's per-rank
+# wrapper narrows CUDA_VISIBLE_DEVICES to one GPU; CP2K binds device (rank mod
+# #visible) = 0. Output <run_dir>/cp2k.out (+ stdout.log with the launcher audit) and a
+# run_manifest.txt with exit code and hashes.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"; [ $# -gt 0 ] && shift
+[ "$BACKEND" = CUDA ] || { echo "run.sh: only CUDA is built for CP2K here" >&2; exit 2; }
+GCC_MM="$(l3_version_mm "$(/usr/bin/gcc -dumpfullversion)")"; OMPI_V="$(mpirun --version | head -1 | /usr/bin/grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)"
+PROFILE="${HPCPERF_CP2K_PROFILE:-cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-ompi$(echo "$OMPI_V" | tr -d .)}"
+l3_paths_profile cp2k "$PROFILE"
+SRC="$R/_upstream/level3/cp2k"
+EXE="$L3_INSTALL/cp2k/bin/cp2k.psmp"
+[ -x "$EXE" ] || { echo "run.sh: $EXE not found -- run ./build.sh first (profile $PROFILE)" >&2; exit 1; }
+# runtime libraries: the toolchain's `setup` (OpenBLAS/ScaLAPACK/FFTW/libxc/libint/DBCSR/... lib dirs)
+# plus libcp2k.so itself (cp2k.psmp is installed without an rpath to it)
+[ -f "$L3_INSTALL/toolchain/setup" ] || { echo "run.sh: toolchain setup file missing under $L3_INSTALL/toolchain" >&2; exit 1; }
+set +u; # shellcheck disable=SC1091
+source "$L3_INSTALL/toolchain/setup"; set -u
+export LD_LIBRARY_PATH="$L3_INSTALL/cp2k/lib:$L3_INSTALL/cp2k/lib64:${LD_LIBRARY_PATH:-}"
+# the CUDA device code (GRID/DBM/PW kernels) is in libcp2k.so; DBCSR's libsmm_acc kernels are JIT-compiled (NVRTC)
+LIBCP2K=""; for f in "$L3_INSTALL/cp2k/lib/libcp2k.so" "$L3_INSTALL/cp2k/lib64/libcp2k.so"; do if [ -f "$f" ]; then LIBCP2K="$f"; break; fi; done
+[ -n "$LIBCP2K" ] || { echo "run.sh: libcp2k.so not found under $L3_INSTALL/cp2k" >&2; exit 1; }
+l3_binary_backend_check "$LIBCP2K" cuda || exit 1
+# the BLAS/LAPACK actually loaded must be the toolchain's OpenBLAS (not another libopenblas.so.0 on the loader path):
+# refuse to run otherwise, and record the resolved library in the manifest
+BLAS_RESOLVED="$(ldd "$LIBCP2K" 2>/dev/null | awk '/libopenblas|libblas|liblapack|libmkl/{print $3}' | head -1)"
+if [ -z "$BLAS_RESOLVED" ]; then
+    # no dynamic BLAS: it must be the toolchain's libopenblas.a linked statically into libcp2k.so (CP2K_BLAS_VENDOR=CUSTOM)
+    nm -D "$LIBCP2K" 2>/dev/null | /usr/bin/grep -qE ' T (dgemm_|openblas_get_config)$' || { echo "run.sh: libcp2k.so has neither a dynamic BLAS nor statically linked OpenBLAS symbols -- refusing" >&2; exit 1; }
+    BLAS_RESOLVED="static:$(/usr/bin/grep -oE 'blas=[^ ]+' "$L3_INSTALL/BUILD_INFO.txt" 2>/dev/null | head -1)"
+else
+    case "$BLAS_RESOLVED" in
+        "$L3_INSTALL"/toolchain/openblas-*/lib/*) : ;;
+        *) echo "run.sh: libcp2k.so resolves BLAS to '$BLAS_RESOLVED', not to the toolchain's OpenBLAS under $L3_INSTALL/toolchain -- refusing (rebuild with ./build.sh)" >&2; exit 1 ;;
+    esac
+fi
+CASE="${HPCPERF_CP2K_CASE:-h2o}"
+MODE="$(l3_scale_mode cp2k)" || exit 2
+N_RANKS="$(hpcperf_ranks cp2k yes)" || exit 2
+THREADS="${HPCPERF_CPUS_PER_RANK:-8}"
+case "$CASE" in
+    h2o)
+        case "$MODE" in
+            smoke)  S="${HPCPERF_CP2K_SYSTEM:-64}" ;;
+            strong) S="${HPCPERF_CP2K_SYSTEM:-128}" ;;
+            weak)   S=$((32 * N_RANKS)) ;;
+        esac
+        INP="$SRC/benchmarks/QS/H2O-$S.inp"; LABEL="h2o$S"
+        [ -f "$INP" ] || { echo "run.sh: no upstream input for $S water molecules ($INP); available: $(ls "$SRC/benchmarks/QS" | /usr/bin/grep -oE 'H2O-[0-9]+\.inp' | tr '\n' ' ')" >&2; exit 2; } ;;
+    regtest)
+        [ "$MODE" = smoke ] || { echo "run.sh: regtest inputs are smoke-only" >&2; exit 2; }
+        RT="${HPCPERF_CP2K_REGTEST:-}"; [ -n "$RT" ] || { echo "run.sh: set HPCPERF_CP2K_REGTEST=<dir/file under tests/>" >&2; exit 2; }
+        INP="$SRC/tests/$RT"; [ -f "$INP" ] || { echo "run.sh: $INP not found" >&2; exit 2; }
+        LABEL="regtest.$(echo "$RT" | tr '/' '_' | sed 's/\.inp$//')" ;;
+    *) echo "run.sh: HPCPERF_CP2K_CASE must be h2o or regtest" >&2; exit 2 ;;
+esac
+RUN_DIR="$(l3_rundir "$L3_BUILD/$L3_RUN_SUBDIR/$LABEL.$MODE.np$N_RANKS.t$THREADS")" || exit 2
+cp "$INP" "$RUN_DIR/input.inp"
+# regtest inputs may reference sibling files (basis sets, restart files) by relative name
+if [ "$CASE" = regtest ]; then for f in "$(dirname "$INP")"/*; do [ -f "$f" ] && [ "$(basename "$f")" != "$(basename "$INP")" ] && ln -sfn "$f" "$RUN_DIR/"; done; fi
+export CP2K_DATA_DIR="$SRC/data" OMP_NUM_THREADS="$THREADS" OMP_PROC_BIND=close OMP_PLACES=cores
+echo "# CP2K $BACKEND profile=$PROFILE case=$CASE ($LABEL) mode=$MODE ranks=$N_RANKS threads/rank=$THREADS input=$(realpath --relative-to="$SRC" "$INP") run_dir=$RUN_DIR"
+cd "$RUN_DIR"
+RUN_ID="$(l3_run_id)"
+set +e; set -o pipefail
+"$L3_LAUNCHER" --gpus "$N_RANKS" --cpus-per-rank "$THREADS" --bind wrapper -- "$EXE" -i input.inp -o cp2k.out 2>&1 | tee "$RUN_DIR/stdout.log"
+rc=$?
+set +o pipefail; set -e
+if [ -z "${HPCPERF_DRY_RUN:-}" ]; then
+    l3_manifest "$RUN_DIR" "run_id=$RUN_ID" "app=cp2k" "backend=$BACKEND" "profile=$PROFILE" "case=$CASE" "label=$LABEL" "mode=$MODE" \
+        "ranks=$N_RANKS" "threads_per_rank=$THREADS" "exit_code=$rc" "binary=$EXE" "binary_sha256=$(l3_sha_file "$EXE")" \
+        "input=$INP" "input_sha256=$(l3_sha_file "$INP")" "cp2k_data_dir=$CP2K_DATA_DIR" "blas_resolved=$BLAS_RESOLVED" \
+        "fingerprint_sha256=$(l3_sha_file "$L3_INSTALL/.hpcperf-l3-fingerprint")" "output=$RUN_DIR/cp2k.out" "utc=$(date -u +%FT%TZ)"
+    # GPU evidence lines from CP2K's own banner (DBCSR/GRID/DBM/PW backends) for the manifest
+    /usr/bin/grep -aE 'DBCSR\| ACC|GRID\| .*(GPU|backend)|DBM\| .*(GPU|backend)|PW_GPU|ACC\| |offload' "$RUN_DIR/cp2k.out" 2>/dev/null | head -12 | sed 's/^/gpu_evidence: /' >> "$RUN_DIR/run_manifest.txt" || true
+fi
+exit "$rc"
diff --git a/level3/cp2k/validate.sh b/level3/cp2k/validate.sh
new file mode 100755
index 0000000..e68f531
--- /dev/null
+++ b/level3/cp2k/validate.sh
@@ -0,0 +1,131 @@
+#!/usr/bin/env bash
+# Correctness check for CP2K (cp2k.psmp, CUDA) on N GPUs.
+#
+#   ./validate.sh [CUDA]        HPCPERF_GPUS=N (default 1), HPCPERF_CPUS_PER_RANK=T (default 8)
+#
+# [1] Upstream regression tests (adapted subset of tests/do_regtest.py): selected inputs
+#     are run through run.sh on N ranks/GPUs and the quantity named by the directory's
+#     TEST_FILES.toml matcher (E_total = last "Total energy:" column 3; M011 = last
+#     "ENERGY| Total FORCE_EVAL" column 9 -- tests/matchers.py) must match the upstream
+#     reference within the upstream tolerance. References/tolerances are upstream's; the
+#     GPU CI runs the same tests. Selection (HPCPERF_CP2K_REGTESTS): GPW/OT ground states and
+#     linear-scaling DBCSR-heavy cases.
+# [2] Science case benchmarks/QS/H2O-64.inp (GPW-DFT NVE MD, 10 steps) on N GPUs, checked
+#     by cp2k_md_summary.py --check: completeness (exit 0, PROGRAM ENDED, 10 MD steps), every
+#     MD-step SCF cycle converged (the initial ATOMIC-guess SCF of upstream's deck does not
+#     converge within MAX_SCF=50 -- the deck declares IGNORE_CONVERGENCE_FAILURE for exactly
+#     that; it is reported, and would FAIL without that declaration), finite energies, GPU
+#     evidence from CP2K's own output (cp2kflags offload_cuda+dbcsr_acc, DBCSR ACC devices
+#     >= 1, GRID tasks executed on GPU; PW GPU timers reported or unverified), and --
+#     pre-fixed -- the per-step "ENERGY| Total FORCE_EVAL" energies of MD steps 1..10 must
+#     agree with the 1-GPU run of the same binary/input within 1e-8 Ha (upstream's
+#     check-release-comparison.py demands 1e-10 across MPI x OMP layouts for single-point
+#     energies of these benchmarks on CPU; the OT convergence (EPS_SCF 1e-5) and GPU
+#     reduction order motivate the looser 1e-8 chosen here BEFORE the runs; the 1e-10
+#     result is printed too). NVE conserved-quantity drift is reported.
+# Every exit code is captured (timeout/nonzero -> FAIL); nothing is read from old runs.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+N="${HPCPERF_GPUS:-1}"; T="${HPCPERF_CPUS_PER_RANK:-8}"
+TIMEOUT="${HPCPERF_VALIDATE_TIMEOUT:-3600}"
+E_TOL=1e-8       # pre-fixed cross-rank-count tolerance on MD potential energies (Ha)
+REGTESTS="${HPCPERF_CP2K_REGTESTS:-QS/regtest-gpw-1/Ar.inp QS/regtest-gpw-1/H2O-geoopt.inp QS/regtest-gpw-1/pyridine.inp QS/regtest-dm-ls-scf-1/H2-big-1.inp QS/regtest-dm-ls-scf-1/H2-big-5.inp}"
+GCC_MM="$(l3_version_mm "$(/usr/bin/gcc -dumpfullversion)")"; OMPI_V="$(mpirun --version | head -1 | /usr/bin/grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)"
+PROFILE="${HPCPERF_CP2K_PROFILE:-cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-ompi$(echo "$OMPI_V" | tr -d .)}"
+SRC="$R/_upstream/level3/cp2k"; RUNS="$R/build/level3/cp2k/$PROFILE/$L3_RUN_SUBDIR"
+export HPCPERF_GPUS="$N" HPCPERF_CPUS_PER_RANK="$T" HPCPERF_SCALE_MODE=smoke
+mkdir -p "$RUNS"; ok=1
+fail() { echo "validate.sh: FAIL -- $*"; ok=0; }
+manifest_val() { /usr/bin/grep -m1 "^$2=" "$1/run_manifest.txt" 2>/dev/null | cut -d= -f2- || true; }
+
+echo "validate.sh: [1] CP2K $BACKEND upstream regression tests (adapted subset) on $N GPU(s) x $T threads [profile $PROFILE]"
+for rt in $REGTESTS; do
+    dir="$(dirname "$rt")"; file="$(basename "$rt")"; toml="$SRC/tests/$dir/TEST_FILES.toml"
+    [ -f "$toml" ] || { fail "$rt: $toml missing"; continue; }
+    spec="$(python3 - "$toml" "$file" <<'PY'
+import sys, re, tomllib
+d = tomllib.load(open(sys.argv[1], "rb")); e = d.get(sys.argv[2])
+if not e: sys.exit(1)
+for m in e: print(m["matcher"], repr(m["tol"]), repr(m["ref"]))
+PY
+)" || { fail "$rt: not listed in $toml"; continue; }
+    label="regtest.$(echo "$rt" | tr '/' '_' | sed 's/\.inp$//')"; D="$RUNS/$label.smoke.np$N.t$T"
+    rc=0; HPCPERF_CP2K_CASE=regtest HPCPERF_CP2K_REGTEST="$rt" timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" > "$RUNS/validate.$label.np$N.stdout" 2>&1 || rc=$?
+    if [ "$rc" -eq 124 ]; then fail "$rt timed out"; continue; fi
+    [ "$rc" -eq 0 ] || { fail "$rt: run.sh exited $rc (see $RUNS/validate.$label.np$N.stdout)"; continue; }
+    [ -f "$D/cp2k.out" ] || { fail "$rt: no cp2k.out"; continue; }
+    while read -r matcher tol ref; do
+        python3 - "$D/cp2k.out" "$matcher" "$tol" "$ref" "$rt" <<'PY' || ok=0
+import sys, re, os
+sys.path.insert(0, os.environ["L3_TOOLS"])
+from l3_check import require_finite, ValidationError
+out, matcher, tol, ref, name = sys.argv[1], sys.argv[2], float(sys.argv[3]), float(sys.argv[4]), sys.argv[5]
+patterns = {"E_total": ("Total energy:", 3), "M011": ("ENERGY| Total FORCE_EVAL", 9), "M002": ("MD| Potential energy", 5), "M007": ("OPT| Total energy [hartree]", 5)}
+if matcher not in patterns:
+    print(f"  VALIDATION ERROR: matcher {matcher} not implemented in this adapted subset"); sys.exit(1)
+pat, col = patterns[matcher]
+val = None
+for line in reversed(open(out, errors="replace").read().split("\n")):
+    if pat in line:
+        val = float(line.split()[col - 1]); break
+try:
+    if val is None: raise ValidationError(f"{name}: pattern {pat!r} not found in output")
+    require_finite(f"{name} {matcher}", val)
+    err = abs(val - ref)
+    print(f"    {name}: {matcher} = {val:.14f} ref {ref:.14f} |diff| = {err:.2e} tol {tol:.0e} {'ok' if err <= tol else 'BAD'}")
+    sys.exit(0 if err <= tol else 1)
+except ValidationError as ex:
+    print(f"  VALIDATION ERROR: {ex}"); sys.exit(1)
+PY
+    done <<< "$spec"
+    /usr/bin/grep -aq 'PROGRAM ENDED AT' "$D/cp2k.out" || fail "$rt: output lacks 'PROGRAM ENDED AT' (incomplete run)"
+done
+
+echo "validate.sh: [2] CP2K $BACKEND H2O-64 GPW-DFT NVE MD, 10 steps, on $N GPU(s) x $T threads"
+D="$RUNS/h2o64.smoke.np$N.t$T"
+rc=0; HPCPERF_CP2K_CASE=h2o HPCPERF_CP2K_SYSTEM=64 timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" > "$RUNS/validate.h2o64.np$N.stdout" 2>&1 || rc=$?
+/usr/bin/grep -aE '^# CP2K|hpcperf-launch: audit summary' "$RUNS/validate.h2o64.np$N.stdout" || true
+if [ "$rc" -eq 124 ]; then fail "h2o64 timed out after ${TIMEOUT}s"; elif [ "$rc" -ne 0 ]; then fail "h2o64 run.sh exited $rc"; fi
+if [ -f "$D/cp2k.out" ]; then
+    /usr/bin/grep -aE 'DBCSR\| ACC: (Number of devices|GPU backend)' "$D/cp2k.out" | head -2 | sed 's/^/    gpu: /' || true
+    python3 "$HERE/cp2k_md_summary.py" "$D/cp2k.out" "$D/input.inp" --check > "$D/md_summary.txt" || { fail "h2o64 np$N: MD completeness/convergence/finiteness/GPU-evidence check failed"; }
+    sed 's/^/    /' "$D/md_summary.txt"
+    if [ "$N" -gt 1 ]; then
+        REF="$RUNS/h2o64.smoke.np1.t$T"
+        if [ ! -f "$REF/md_summary.txt" ] || [ "$(manifest_val "$REF" binary_sha256)" != "$(manifest_val "$D" binary_sha256)" ]; then
+            echo "    (1-GPU reference missing/stale -- running H2O-64 on 1 GPU now)"
+            rc=0; HPCPERF_GPUS=1 HPCPERF_CP2K_CASE=h2o HPCPERF_CP2K_SYSTEM=64 timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" > "$RUNS/validate.h2o64.np1.stdout" 2>&1 || rc=$?
+            [ "$rc" -eq 0 ] || fail "h2o64 1-GPU reference run exited $rc"
+            if [ -f "$REF/cp2k.out" ]; then python3 "$HERE/cp2k_md_summary.py" "$REF/cp2k.out" "$REF/input.inp" --check > "$REF/md_summary.txt" || fail "h2o64 1-GPU reference run failed its own checks"; fi
+        fi
+        python3 - "$REF/md_summary.txt" "$D/md_summary.txt" "$E_TOL" <<'PY' || fail "h2o64 np$N: MD energies differ from the 1-GPU run beyond $E_TOL Ha"
+import sys, re
+def load(p):
+    d = {}
+    for l in open(p):
+        for k, v in re.findall(r"(\w+)=(-?[0-9.]+(?:e[-+]?\d+)?)", l): d[k] = float(v)
+    return d
+a, b, tol = load(sys.argv[1]), load(sys.argv[2]), float(sys.argv[3])
+keys = [f"efe_step{i}" for i in range(1, 11)]
+missing = [k for k in keys if k not in a or k not in b]
+if missing:
+    print(f"    missing energies in summaries: {missing}"); sys.exit(1)
+diffs = {k: abs(a[k] - b[k]) for k in keys}
+worst = max(diffs, key=diffs.get)
+for k in ("efe_step1", "efe_step10", worst) if worst not in ("efe_step1", "efe_step10") else ("efe_step1", "efe_step10"):
+    d = diffs[k]
+    print(f"    {k}: 1-GPU {a[k]:.12f}  N-GPU {b[k]:.12f}  |diff| = {d:.3e} Ha (tol {tol:.0e} {'ok' if d <= tol else 'BAD'}; upstream CPU cross-layout 1e-10: {'ok' if d <= 1e-10 else 'exceeded'})")
+print(f"    max |diff| over MD steps 1..10: {diffs[worst]:.3e} Ha ({worst}); initial-SCF energy |diff| = {abs(a.get('efe_step0', 0) - b.get('efe_step0', 0)):.3e} Ha (informational)")
+sys.exit(0 if all(d <= tol for d in diffs.values()) else 1)
+PY
+    fi
+fi
+
+if [ "$ok" -eq 1 ]; then echo "CP2K $BACKEND validation ($N GPU x $T threads; upstream regtests [$REGTESTS] within upstream tolerances, H2O-64 MD complete/all MD-step SCFs converged/finite, GPU backends active$( [ "$N" -gt 1 ] && echo ", MD-step FORCE_EVAL energies within $E_TOL Ha of 1-GPU" || true)): PASS"; exit 0; fi
+echo "CP2K $BACKEND validation ($N GPU): FAIL"; exit 1
diff --git a/level3/dftfe/README.md b/level3/dftfe/README.md
new file mode 100644
index 0000000..58ce2b2
--- /dev/null
+++ b/level3/dftfe/README.md
@@ -0,0 +1,122 @@
+# DFT-FE -- Level 3 second batch
+
+Real-space finite-element Kohn-Sham DFT (University of Michigan). GPU build
+(`WITH_GPU=ON GPU_LANG=cuda GPU_VENDOR=nvidia`): Chebyshev-filtered subspace
+iteration, dense linear algebra (cuBLAS), Poisson/Helmholtz solves and the
+ELPA eigensolver with NVIDIA GPU kernels.
+
+## Provenance / versions
+
+| Item | Value |
+|---|---|
+| DFT-FE | release 1.2.0, `7147faa51f7c9f3075fffaa5e48ba989bcd329c1` (branch `release1.2`), LGPL-2.1-or-later |
+| Recipe | upstream `install_DFTFE` (branch `frontierDevelop`: `dftfe2.sh` + `setupUser.sh`) transcribed to this node's toolchain in `build.sh`; every dependency tarball is pinned by sha256 in `fetch.sh` |
+| Dependencies (private, same profile) | OpenBLAS 0.3.30 (`TARGET=SAPPHIRERAPIDS`, single-threaded) -> ScaLAPACK 2.2.2 -> libxc 7.0.0 -> spglib `02159eef` -> ALGLIB 4.06.0 -> p4est 2.8.7 (dftfe's `p4est-setup.sh`, FAST+DEBUG) -> Kokkos 4.6.00 (Serial, deal.II only) -> deal.II **9.6.2** (see decision below) (MPI, p4est, 64-bit indices, LAPACK=OpenBLAS, bundled Boost, no TBB/taskflow) -> ELPA 2026.02.001 (`--enable-nvidia-gpu-kernels --with-NVIDIA-GPU-compute-capability=sm_100`) -> DFT-FE real (`CMAKE_CUDA_ARCHITECTURES=100 WITH_DCCL=OFF WITH_GPU_AWARE_MPI=OFF USE_64BIT_INT=ON`); libxml2 from the system |
+| Compilers | system GCC 14.2.1 (C/C++/Fortran) through the conda Open MPI 5.0.10 wrappers (`OMPI_CC/CXX/FC`), nvcc 13.2.78 with g++ 14.2.1 host |
+| GPU target | sm_100 for DFT-FE and for the ELPA kernels (ELPA's generic `--with-NVIDIA-GPU-compute-capability=sm_XX` path; sm_100 never used upstream) |
+| NCCL | not used (`WITH_DCCL=OFF`; DFT-FE's default all-reduce path) |
+| Build strategy | NATIVE (no Spack: the `dftfe` recipe is 0.6/2019 without GPU variants; no container) |
+| Profile | `cuda132-gcc142-ompi5010` (`HPCPERF_DFTFE_ELPA_GPU=OFF` would build `-elpacpu`, a named CPU-ELPA diagnostic profile, not used for the results) |
+
+### Node adaptations (class C, recorded in build.sh)
+
+- dftfe's `p4est-setup.sh` is written for Cray systems: it hardcodes
+  `CC=cc CXX=CC FC=ftn F77=ftn` (no MPI on this node's `cc`), relies on the Cray
+  wrappers' implicit `-lm`, and checks `<build>/src/p4est_config.h` while p4est
+  2.8.7 writes `<build>/config/p4est_config.h`. build.sh passes
+  `CC=mpicc CXX=mpicxx FC=mpifort F77=mpifort LIBS=-lm` through the script's
+  trailing configure arguments and seds the header path in its private copy.
+- The conda environment's `CFLAGS/LDFLAGS/AR/...` are cleared for the
+  system-GCC builds (`l3_clean_conda_build_env`); OpenBLAS 0.3.30 misdetects the
+  Xeon Platinum 8570 and is built with `TARGET=SAPPHIRERAPIDS`.
+- ELPA's configure expects the host SIMD flags in `CFLAGS` (it probes AVX-512
+  intrinsics with the given flags and aborts otherwise; EL10's GCC 14 defaults to
+  x86-64-v3 = AVX2): `-march=native` for C/C++/Fortran -- a node-specific
+  binary like OpenBLAS. Its later CUDA link checks (`cublasDgemm`) keep
+  `LIBS=-lscalapack` but replace `LDFLAGS` by the CUDA path, so the
+  ScaLAPACK/OpenBLAS `-L`/rpath entries are given in `LDFLAGS` as well.
+
+### The one DFT-FE source patch (class D, `patches/0001-std-isnan.patch`)
+
+Two lines in `src/atom/AtomicCenteredNonLocalOperator.cc` (785/791) call an
+unqualified `isnan(` inside a template; GCC 14 (libstdc++ 14) no longer finds
+it through `<cmath>`'s global fallback and stops with "there are no arguments to
+'isnan' that depend on a template parameter". The patch qualifies both calls as
+`std::isnan(` -- the same function, no numerical or algorithmic change (upstream
+develop has the same spelling). Applied by build.sh with `git apply` (idempotent,
+sha256 recorded in the fingerprint as `DFTFE_PATCH_SHA`).
+
+### deal.II version decision (a dependency choice, not a DFT-FE change)
+
+Release 1.2.0 pins deal.II **9.5.2** in its own `setupUser*.sh`; the current
+`install_DFTFE` recipe (frontierDevelop/master) builds deal.II **9.7.1** -- but
+together with the *develop* branch (`publicGithubDevelop`), not with the release.
+Attempt 1 here (1.2.0 + 9.7.1) failed to compile: 9.7 removed three APIs 1.2.0
+still uses (`Utilities::MPI::create_group`, `parallel::distributed::Triangulation::load(name, autopartition)`,
+`DataOutBase::VtkFlags::ZlibCompressionLevel`; upstream develop already carries the
+replacements). Rather than back-porting a growing set of develop changes into the
+release, the profile uses **deal.II 9.6.2**, the newest deal.II that still provides
+those (deprecated) APIs, so 1.2.0 builds unpatched; recorded as attempt 2. The
+9.7.1 build and its install were removed from the profile
+(`.deps/level3/dftfe/<profile>/ATTEMPT-1-dealii-9.7.1.txt`, logs `attempt1-*`).
+
+## ELPA GPU kernels verified independently (`elpa_probe.sh`)
+
+ELPA's own test programs (`validate_real_double_eigenvectors_{1stage,2stage_default_kernel}_gpu_analytic`,
+which call `e%set("nvidia-gpu", 1)`; the real binaries in `build/elpa/.libs/`, the
+top-level names being libtool relink wrappers) on 1, 2 and 4 GPUs (one rank per GPU
+through the common launcher, na=2000 nev=1000 nblk=32) and their CPU counterparts on
+1 rank when built (they are not, with the GPU configuration: SKIPPED, recorded).
+The programs diagonalise the analytic test matrix of `test/shared/test_analytic_template.F90`
+(known eigenpairs) and apply ELPA's own limits `max |lambda - lambda_exact| <= 5e-14`,
+`max |z - z_exact| <= 6e-10` (real double; `stop 1` on violation); the probe re-parses the
+printed "Maximum error in eigenvalues/eigenvectors", re-applies the limits, requires exit 0,
+ELPA's GPU timers (`trans_ev_real_double_gpu`, `gpublas_*`) in the output and a launcher
+audit with 0 mismatch; record `<install>/elpa/ELPA_GPU_PROBE.txt`, required PASS by
+`validate.sh`. Two false starts, both recorded in the script header: the first pass
+looked for ELPA's `*_default` wrapper-script names (MISSING), the second parsed the
+`%Error Residual/Orthogonality` lines of the *random-matrix* programs, which the analytic
+programs never print, and aborted under `pipefail`.
+
+## Cases and validation
+
+`run.sh`: `al_md` (upstream GPU regression deck `testsGPU/pseudopotential/real/Input_MD_0.prm`:
+32-atom fcc Al, order-3 FE, 85 states, BOMD 1400 K, 4 steps, `USE GPU = true`,
+`REPRODUCIBLE OUTPUT = true`) -- smoke/strong verbatim, weak = derived Al supercell
+series (32 atoms per GPU; SYNTHETIC, same construction as upstream's
+dftfe-benchmarks Mo series); `llzo` (`parameterFile_LLZO.prm`, 192 atoms, 720
+states, ELPA) as the heavier fixed-size strong case. One MPI rank per GPU, 1
+thread per rank (upstream's GPU job scripts), decks used verbatim except the
+documented weak derivation.
+
+`validate.sh`: [0] ELPA GPU probe PASS; [1] `al_md` on N GPUs vs upstream's own GPU
+reference `accuracyBenchmarks/output_MD_0` through `dftfe_check.py` (pre-fixed:
+ground-state energy 1e-5 Ha, per-step MD energies 2e-5 Ha, temperatures 0.1 K,
+forces 2e-5 Ha/Bohr; SCF converged, MD completed, finite), launcher audit "N
+verified, 0 mismatch"; N > 1 additionally vs the 1-GPU run under the same
+tolerances.
+
+### Results (2026-09-06, profile `cuda132-gcc142-ompi5010`, 1 rank per GPU)
+
+| Run | GPUs | atoms / KS DOFs | wall (s) | check |
+|---|---|---|---|---|
+| `al_md` smoke (= strong, verbatim deck) | 1 | 32 / 50 653 | 216 | vs upstream GPU reference: e0, 4 MD energies, temperatures, forces all **identical at printed precision** (max diff 0.0; SCF iterations 8/9/9/9 as upstream) |
+| `al_md` | 2 | 32 / 50 653 | 117 | identical to upstream reference and to the 1-GPU run |
+| `al_md` | 4 | 32 / 50 653 | 70 | identical to upstream reference and to the 1-GPU run |
+| weak x1 (2x2x2 cells) | 1 | 32 / 50 653 | 213 | complete (same deck as smoke) |
+| weak x2 (4x2x2 cells, derived) | 2 | 64 / 108 151 | 284 | complete, 4 MD steps |
+| weak x4 (4x4x2 cells, derived) | 4 | 128 / 230 917 | 378 | complete, 4 MD steps |
+| `llzo` strong (192 atoms, 720 states, ELPA, verbatim deck) | 1 / 2 / 4 | 192 / 97 336 | 295 / 163 / 96 (1.81x, 3.07x) | ground state, 19 SCF iterations, E = -3579.26588980 Ha **identical on 1, 2 and 4 GPUs** |
+| ELPA GPU probe, 1-stage (`validate_real_double_eigenvectors_1stage_gpu_analytic`, na=2000 nev=1000) | 1 / 2 / 4 | -- | -- | PASS: max eigenvalue error 2.7e-15 / 2.4e-15 / 2.0e-15 (limit 5e-14), eigenvector error 1.5e-12 / 1.7e-12 / 1.5e-12 (limit 6e-10), exit 0, GPU timers, audit 0 mismatch |
+| ELPA GPU probe, 2-stage (`..._2stage_default_kernel_gpu_analytic`) | 1 / 2 / 4 | -- | -- | PASS: 5.8e-15 / 7.3e-15 / 6.9e-15; 1.0e-11 / 1.0e-11 / 8.1e-12; CPU counterparts SKIPPED (not built) |
+
+**VALIDATED_PASS at 1, 2 and 4 GPUs** (validate.sh 2026-09-06 05:04-05:12 UTC: [0] probe
+PASS, [1] al_md within the pre-fixed tolerances of upstream's reference and of the
+1-GPU run, launcher audit N verified / 0 mismatch; the 05:01-05:03 UTC pass FAILED on
+[0] only, with the aborted probe -- kept in `validate.*.stdout` history).
+Strong scaling: 32-atom deck 1.85x / 3.1x on 2 / 4 GPUs (small problem, 50 k DOFs);
+192-atom LLZO 1.81x / 3.07x. Dry-runs: 8/40/80 GPUs planned (HYPOTHETICAL) for the
+strong decks; the weak series is defined for 1/2/4/8 GPUs only (40/80 refused by run.sh). The weak series is synthetic (32 atoms per GPU; the electronic-structure
+cost grows faster than linearly with the cell, so constant wall time is not
+expected -- reported, not "ideal"). Probe record: `<install>/elpa/ELPA_GPU_PROBE.txt`,
+per-program logs `build/level3/dftfe/<profile>/elpa_probe/`.
diff --git a/level3/dftfe/build.sh b/level3/dftfe/build.sh
new file mode 100755
index 0000000..dd1308b
--- /dev/null
+++ b/level3/dftfe/build.sh
@@ -0,0 +1,206 @@
+#!/usr/bin/env bash
+# Build DFT-FE 1.2.0 (real-valued executable; complex optional) with its full
+# dependency stack, following upstream's install_DFTFE recipe (frontierDevelop
+# dftfe2.sh / setupUser.sh) adapted to this node: system GCC 14.2.1 for
+# C/C++/Fortran, conda Open MPI 5.0.10 (wrappers redirected to that GCC), CUDA
+# 13.2 sm_100. Everything lives in one profile tree; each stage has a done-marker
+# so the script can be re-run (and re-parallelised) without redoing finished work.
+#
+#   ./build.sh                          HPCPERF_BUILD_JOBS=N (default 16)
+#   HPCPERF_DFTFE_COMPLEX=1             also build the complex-valued executable (k-points)
+#   HPCPERF_DFTFE_ELPA_GPU=OFF          diagnostic variant: ELPA without NVIDIA kernels
+#                                       (profile suffix -elpacpu; NEVER a silent substitute --
+#                                       the default profile builds the GPU ELPA)
+#
+# Stages  (install/<pkg>, build/<pkg>, logs/<pkg>-*.log):
+#   openblas 0.3.30 (single-threaded) -> scalapack 2.2.2 -> libxc 7.0.0 -> spglib -> alglib 4.06.0
+#   -> p4est 2.8.7 (dftfe's p4est-setup.sh: FAST+DEBUG) -> kokkos 4.6.00 (Serial, for deal.II)
+#   -> deal.II 9.6.2 (MPI, p4est, 64-bit indices, LAPACK=openblas, bundled boost, no TBB/taskflow)
+#   -> ELPA 2026.02.001 (--enable-nvidia-gpu-kernels --with-NVIDIA-GPU-compute-capability=sm_100,
+#      test programs built for elpa_probe.sh) -> DFT-FE (WITH_GPU=ON GPU_LANG=cuda GPU_VENDOR=nvidia,
+#      CMAKE_CUDA_ARCHITECTURES=100, WITH_DCCL=OFF (no NCCL), WITH_GPU_AWARE_MPI=OFF, USE_64BIT_INT=ON)
+# Modification classes: A (build options) + C (compiler/MPI wrapper environment). No source patched.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+l3_isolate_build_env
+l3_clean_conda_build_env   # system GCC 14 build: drop conda's CFLAGS/LDFLAGS/AR/...
+export CC=/usr/bin/gcc CXX=/usr/bin/g++ FC=/usr/bin/gfortran F77=/usr/bin/gfortran F90=/usr/bin/gfortran
+export OMPI_CC=/usr/bin/gcc OMPI_CXX=/usr/bin/g++ OMPI_FC=/usr/bin/gfortran CUDAHOSTCXX=/usr/bin/g++
+unset CMAKE_GENERATOR   # autotools/ExternalProject-style stages below drive make/ninja explicitly
+for c in mpicc mpicxx mpifort mpirun cmake ninja nvcc python3; do command -v $c >/dev/null || { echo "build.sh: $c not in PATH" >&2; exit 1; }; done
+SRC="$R/_upstream/level3/dftfe"; DL="$R/.deps/level3/dftfe/downloads"
+# deal.II 9.6.2: the newest deal.II whose API release 1.2.0 still compiles against (9.7 removed Utilities::MPI::create_group,
+# Triangulation::load(name, autopartition) and VtkFlags::ZlibCompressionLevel, all used by 1.2.0; 9.5.2 is the release's own pin;
+# 9.7.1 is what the current install_DFTFE recipe pairs with the *develop* branch) -- see README.
+DEALII_VER="${HPCPERF_DFTFE_DEALII:-9.6.2}"
+[ -f "$SRC/CMakeLists.txt" ] && [ -f "$DL/dealii-$DEALII_VER.tar.gz" ] || { echo "build.sh: run $HERE/fetch.sh first" >&2; exit 1; }
+SHA="$(git -C "$SRC" rev-parse HEAD)"
+# Source compatibility patches (headers in patches/*.patch): applied to the checkout idempotently; hashes in the fingerprint.
+DFTFE_PATCH_SHA=""
+for P in "$HERE"/patches/000*.patch; do
+    [ -f "$P" ] || continue
+    if git -C "$SRC" apply --check --reverse "$P" >/dev/null 2>&1; then :
+    elif git -C "$SRC" apply --check "$P" >/dev/null 2>&1; then git -C "$SRC" apply "$P"; echo "# applied $(basename "$P") to the dftfe checkout"
+    else echo "build.sh: $P is neither applied nor applicable to dftfe $SHA" >&2; exit 1; fi
+    DFTFE_PATCH_SHA="$DFTFE_PATCH_SHA${DFTFE_PATCH_SHA:+,}$(basename "$P" | cut -d- -f1)=$(sha256sum "$P" | cut -c1-12)"
+done
+ARCH="${HPCPERF_CUDA_ARCH:-$(l3_gpu_arch)}"; [ -n "$ARCH" ] || { echo "build.sh: no GPU arch detected" >&2; exit 1; }
+ELPA_GPU="$(echo "${HPCPERF_DFTFE_ELPA_GPU:-ON}" | tr '[:lower:]' '[:upper:]')"
+GCC_MM="$(l3_version_mm "$(/usr/bin/gcc -dumpfullversion)")"; OMPI_V="$(mpirun --version | head -1 | /usr/bin/grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)"
+PROFILE="${HPCPERF_DFTFE_PROFILE:-cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-ompi$(echo "$OMPI_V" | tr -d .)$( [ "$ELPA_GPU" = OFF ] && echo -elpacpu || true)}"
+l3_paths_profile dftfe "$PROFILE"
+JOBS="${HPCPERF_BUILD_JOBS:-16}"; INST="$L3_INSTALL"; BLD="$L3_BUILD_DEPS"; SRCD="$L3_SRC"
+CUDA_ROOT="${CUDA_HOME:-/usr/local/cuda}"
+DEPS_DESC="dftfe_patches[$DFTFE_PATCH_SHA] openblas=0.3.30 scalapack=2.2.2 libxc=7.0.0 spglib=02159eef alglib=4.06.0 p4est=2.8.7 kokkos=4.6.00(serial) dealii=$DEALII_VER elpa=2026.02.001(nvidia-gpu-kernels=$ELPA_GPU sm_$ARCH) gcc=$(/usr/bin/gcc -dumpfullversion) openmpi=$OMPI_V downloads_sha256=$(sha256sum "$DL/SHA256SUMS" | cut -c1-16) profile=$PROFILE"
+DFTFE_OPTS="WITH_GPU=ON GPU_LANG=cuda GPU_VENDOR=nvidia CMAKE_CUDA_ARCHITECTURES=$ARCH WITH_DCCL=OFF WITH_GPU_AWARE_MPI=OFF WITH_COMPLEX=OFF(+ON if HPCPERF_DFTFE_COMPLEX) USE_64BIT_INT=ON HIGHERQUAD_PSP=OFF WITH_TESTING=OFF BUILD_SHARED_LIBS=ON CMAKE_BUILD_TYPE=Release"
+FP="$(l3_fingerprint_text dftfe "$SHA" cuda "$DEPS_DESC" "$DFTFE_OPTS" "WITH_GPU_AWARE_MPI=OFF")"
+l3_fingerprint_check "$INST" "$FP" || exit 1
+echo "# DFT-FE profile=$PROFILE: dftfe $SHA (1.2.0), gcc $(/usr/bin/gcc -dumpfullversion), $(mpirun --version | head -1), CUDA $(l3_cuda_version) sm_$ARCH, -j$JOBS; expected 2.5-4 h (deal.II dominates), ~10 GB under $L3_DEPS"
+T0=$(date +%s)
+done_marker() { [ -f "$INST/$1/.hpcperf-stage-done" ]; }
+finish() { touch "$INST/$1/.hpcperf-stage-done"; echo "# stage $1 done ($(( $(date +%s) - T0 )) s since start)"; }
+untar() { # untar <tarball> <dest-parent> -> prints extracted dir
+    local tb=$1 dst=$2 top; mkdir -p "$dst"; top="$(tar -tzf "$tb" | head -1 | cut -d/ -f1)"
+    [ -d "$dst/$top" ] || tar -xzf "$tb" -C "$dst"; echo "$dst/$top"
+}
+run() { # run <log> <cmd...>
+    local log=$1; shift; "$@" > "$L3_LOGS/$log" 2>&1 || { tail -40 "$L3_LOGS/$log"; echo "build.sh: FAILED: $* (log $L3_LOGS/$log)" >&2; exit 1; }
+}
+
+# 1. OpenBLAS (BLAS+LAPACK for ScaLAPACK/deal.II/ELPA/DFT-FE), single-threaded, gfortran 14
+if ! done_marker openblas; then
+    d="$(untar "$DL/OpenBLAS-0.3.30.tar.gz" "$SRCD")"
+    run openblas-make.log make -C "$d" -j "$JOBS" USE_THREAD=0 USE_OPENMP=0 DYNAMIC_ARCH=0 NO_AFFINITY=1 TARGET="${HPCPERF_OPENBLAS_TARGET:-SAPPHIRERAPIDS}" CC=/usr/bin/gcc FC=/usr/bin/gfortran
+    run openblas-install.log make -C "$d" PREFIX="$INST/openblas" install
+    finish openblas
+fi
+BLAS="$INST/openblas/lib/libopenblas.so"
+# 2. ScaLAPACK 2.2.2 (reference), on OpenBLAS, MPI via the wrappers
+if ! done_marker scalapack; then
+    d="$(untar "$DL/v2.2.2.tar.gz" "$SRCD")"; mkdir -p "$BLD/scalapack"
+    run scalapack-configure.log cmake -S "$d" -B "$BLD/scalapack" -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DBUILD_STATIC_LIBS=OFF -DBUILD_TESTING=OFF \
+        -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_Fortran_COMPILER=/usr/bin/gfortran -DMPI_C_COMPILER="$(command -v mpicc)" -DMPI_Fortran_COMPILER="$(command -v mpifort)" \
+        -DCMAKE_C_FLAGS="-fPIC -Wno-error=implicit-function-declaration" -DCMAKE_Fortran_FLAGS="-fPIC -fallow-argument-mismatch" \
+        -DUSE_OPTIMIZED_LAPACK_BLAS=ON -DBLAS_LIBRARIES="$BLAS" -DLAPACK_LIBRARIES="$BLAS" -DCMAKE_INSTALL_PREFIX="$INST/scalapack"
+    run scalapack-build.log cmake --build "$BLD/scalapack" -j "$JOBS"; run scalapack-install.log cmake --install "$BLD/scalapack"
+    finish scalapack
+fi
+# 3. libxc 7.0.0
+if ! done_marker libxc; then
+    d="$(untar "$DL/libxc-7.0.0.tar.gz" "$SRCD")"; mkdir -p "$BLD/libxc"
+    run libxc-configure.log cmake -S "$d" -B "$BLD/libxc" -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
+        -DCMAKE_C_FLAGS="-O2 -fPIC" -DCMAKE_CXX_FLAGS="-O2 -fPIC" -DBUILD_SHARED_LIBS=ON -DBUILD_TESTING=OFF -DENABLE_FORTRAN=OFF -DCMAKE_INSTALL_PREFIX="$INST/libxc"
+    run libxc-build.log cmake --build "$BLD/libxc" -j "$JOBS"; run libxc-install.log cmake --install "$BLD/libxc"
+    finish libxc
+fi
+# 4. spglib (pinned commit)
+if ! done_marker spglib; then
+    mkdir -p "$BLD/spglib"
+    run spglib-configure.log cmake -S "$DL/spglib" -B "$BLD/spglib" -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
+        -DSPGLIB_WITH_Fortran=OFF -DSPGLIB_WITH_Python=OFF -DSPGLIB_WITH_TESTS=OFF -DCMAKE_INSTALL_PREFIX="$INST/spglib"
+    run spglib-build.log cmake --build "$BLD/spglib" -j "$JOBS"; run spglib-install.log cmake --install "$BLD/spglib"
+    finish spglib
+fi
+# 5. ALGLIB 4.06.0 (C++ sources -> shared library, as upstream's recipe)
+if ! done_marker alglib; then
+    d="$SRCD/alglib-cpp"; [ -d "$d" ] || { mkdir -p "$SRCD"; tar -xzf "$DL/alglib-4.06.0.cpp.gpl.tgz" -C "$SRCD"; }
+    mkdir -p "$INST/alglib"
+    ( cd "$d/src" && /usr/bin/g++ -o "$INST/alglib/libAlglib.so" -shared -fPIC -O2 ./*.cpp && cp ./*.h "$INST/alglib/" ) > "$L3_LOGS/alglib.log" 2>&1 || { tail -20 "$L3_LOGS/alglib.log"; echo "build.sh: ALGLIB failed" >&2; exit 1; }
+    finish alglib
+fi
+# 6. p4est 2.8.7 with dftfe's p4est-setup.sh (FAST + DEBUG trees, MPI, zlib required)
+if ! done_marker p4est; then
+    [ -f /usr/include/zlib.h ] || { echo "build.sh: /usr/include/zlib.h missing (zlib-devel) -- p4est/deal.II need zlib" >&2; exit 1; }
+    mkdir -p "$SRCD/p4est"; cp "$DL/p4est-2.8.7.tar.gz" "$DL/p4est-setup.sh" "$SRCD/p4est/"; chmod +x "$SRCD/p4est/p4est-setup.sh"
+    # p4est 2.8.7 writes its configured header to <build>/config/p4est_config.h; dftfe's p4est-setup.sh (written for
+    # older p4est layouts) looks for <build>/src/p4est_config.h in its zlib check -> point the check at the 2.8.7 location
+    sed -i 's|/src/p4est_config.h"|/config/p4est_config.h"|g' "$SRCD/p4est/p4est-setup.sh"
+    /usr/bin/grep -q 'config/p4est_config.h' "$SRCD/p4est/p4est-setup.sh" || { echo "build.sh: p4est-setup.sh zlib-check path adaptation failed" >&2; exit 1; }
+    # dftfe's p4est-setup.sh hardcodes the Cray wrappers (configure ... CC=cc CXX=CC FC=ftn F77=ftn "$@"); the
+    # trailing "$@" lets us override them with this node's Open MPI wrappers (last assignment wins in configure)
+    ( cd "$SRCD/p4est" && CFLAGS="-fPIC -O2" ./p4est-setup.sh p4est-2.8.7.tar.gz "$INST/p4est" CC=mpicc CXX=mpicxx FC=mpifort F77=mpifort LIBS=-lm )   # LIBS=-lm: the Cray wrappers link libm implicitly, gcc/mpicc do not (libsc examples: undefined sin/sqrt) > "$L3_LOGS/p4est.log" 2>&1 || { tail -30 "$L3_LOGS/p4est.log"; echo "build.sh: p4est failed" >&2; exit 1; }
+    [ -f "$INST/p4est/FAST/lib/libp4est.so" ] || [ -f "$INST/p4est/FAST/lib/libp4est.a" ] || { echo "build.sh: p4est FAST install missing" >&2; exit 1; }
+    finish p4est
+fi
+# 7. Kokkos 4.6.00 (Serial; deal.II's Kokkos dependency -- CPU only, as upstream)
+if ! done_marker kokkos; then
+    d="$(untar "$DL/4.6.00.tar.gz" "$SRCD")"; mkdir -p "$BLD/kokkos"
+    run kokkos-configure.log cmake -S "$d" -B "$BLD/kokkos" -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=/usr/bin/g++ -DCMAKE_CXX_STANDARD=17 \
+        -DCMAKE_CXX_FLAGS="-O2 -fPIC" -DKokkos_ENABLE_SERIAL=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX="$INST/kokkos"
+    run kokkos-build.log cmake --build "$BLD/kokkos" -j "$JOBS"; run kokkos-install.log cmake --install "$BLD/kokkos"
+    finish kokkos
+fi
+# 8. deal.II (version: DEALII_VER above)
+if ! done_marker dealii; then
+    d="$(untar "$DL/dealii-$DEALII_VER.tar.gz" "$SRCD")"; mkdir -p "$BLD/dealii"
+    run dealii-configure.log cmake -S "$d" -B "$BLD/dealii" -G Ninja -DCMAKE_BUILD_TYPE=Release \
+        -DCMAKE_C_COMPILER="$(command -v mpicc)" -DCMAKE_CXX_COMPILER="$(command -v mpicxx)" -DCMAKE_Fortran_COMPILER="$(command -v mpifort)" \
+        -DCMAKE_CXX_STANDARD=17 -DCMAKE_CXX_FLAGS="-std=c++17" -DDEAL_II_CXX_FLAGS_RELEASE=-O2 -DDEAL_II_ALLOW_PLATFORM_INTROSPECTION=OFF \
+        -DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_64BIT_INDICES=ON -DDEAL_II_WITH_COMPLEX_VALUES=ON -DDEAL_II_FORCE_BUNDLED_BOOST=ON \
+        -DDEAL_II_WITH_TASKFLOW=OFF -DDEAL_II_WITH_TBB=OFF -DDEAL_II_COMPONENT_EXAMPLES=OFF \
+        -DP4EST_DIR="$INST/p4est" -DKOKKOS_DIR="$INST/kokkos" -DDEAL_II_WITH_KOKKOS=ON \
+        -DDEAL_II_WITH_LAPACK=ON -DLAPACK_DIR="$INST/openblas" -DLAPACK_FOUND=true -DLAPACK_LIBRARIES="$BLAS" \
+        -DCMAKE_INSTALL_PREFIX="$INST/dealii"
+    run dealii-build.log cmake --build "$BLD/dealii" -j "$JOBS"; run dealii-install.log cmake --install "$BLD/dealii"
+    finish dealii
+fi
+# 9. ELPA 2026.02.001 (autotools). NVIDIA GPU kernels for sm_100 (generic --generate-code path of
+#    ELPA's configure). Test programs are built (make check TESTS=) for elpa_probe.sh; not run here.
+#    ELPA expects the host SIMD flags in CFLAGS (its configure probes AVX512 intrinsics with the given
+#    CFLAGS and aborts otherwise; EL10's GCC defaults to x86-64-v3 = AVX2 only): -march=native on this
+#    Xeon 8570 enables the AVX-512 CPU kernels -- a node-specific binary, like OpenBLAS's TARGET.
+#    LDFLAGS carries the ScaLAPACK/OpenBLAS library paths as well: ELPA's later CUDA checks (cublas) link
+#    with LIBS=-lscalapack but replace LDFLAGS by the CUDA path, so without them "cannot find -lscalapack".
+if ! done_marker elpa; then
+    d="$(untar "$DL/elpa-2026.02.001.tar.gz" "$SRCD")"; mkdir -p "$BLD/elpa"
+    GPUOPTS=(--disable-nvidia-gpu-kernels)
+    [ "$ELPA_GPU" = ON ] && GPUOPTS=(--enable-nvidia-gpu-kernels "--with-NVIDIA-GPU-compute-capability=sm_$ARCH" "--with-cuda-path=$CUDA_ROOT")
+    ( cd "$BLD/elpa" && "$d/configure" --prefix="$INST/elpa" CC="$(command -v mpicc)" CXX="$(command -v mpicxx)" FC="$(command -v mpifort)" \
+        CFLAGS="-O2 -fPIC -march=native" CXXFLAGS="-O2 -fPIC -std=c++17 -march=native" FCFLAGS="-O2 -fPIC -march=native" \
+        SCALAPACK_LDFLAGS="-L$INST/scalapack/lib -lscalapack -L$INST/openblas/lib -lopenblas -Wl,-rpath,$INST/scalapack/lib -Wl,-rpath,$INST/openblas/lib" \
+        SCALAPACK_FCFLAGS="-I$INST/scalapack/include" \
+        LDFLAGS="-L$INST/scalapack/lib -L$INST/openblas/lib -Wl,-rpath,$INST/scalapack/lib -Wl,-rpath,$INST/openblas/lib" \
+        --enable-shared --disable-static --enable-c-tests=no --enable-cpp-tests=no --enable-option-checking=fatal --disable-openmp \
+        "${GPUOPTS[@]}" ) > "$L3_LOGS/elpa-configure.log" 2>&1 || { tail -40 "$L3_LOGS/elpa-configure.log"; echo "build.sh: ELPA configure failed" >&2; exit 1; }
+    run elpa-build.log make -C "$BLD/elpa" -j "$JOBS"
+    run elpa-tests-build.log make -C "$BLD/elpa" -j "$JOBS" check TESTS=
+    run elpa-install.log make -C "$BLD/elpa" install
+    ls "$BLD/elpa"/validate_* > "$INST/elpa/TEST_PROGRAMS.txt" 2>/dev/null || true
+    finish elpa
+fi
+# 10. DFT-FE (real; complex on request)
+build_dftfe() { # build_dftfe <real|complex>
+    local kind=$1 cplx=OFF; [ "$kind" = complex ] && cplx=ON
+    mkdir -p "$L3_BUILD/$kind"
+    run "dftfe-$kind-configure.log" cmake -S "$SRC" -B "$L3_BUILD/$kind" -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17 \
+        -DCMAKE_CXX_COMPILER="$(command -v mpicxx)" -DCMAKE_CXX_FLAGS="-fPIC" -DCMAKE_CXX_FLAGS_RELEASE="-O2" \
+        -DDEAL_II_DIR="$INST/dealii" -DALGLIB_DIR="$INST/alglib" -DLIBXC_DIR="$INST/libxc" -DSPGLIB_DIR="$INST/spglib" \
+        -DXML_LIB_DIR=/usr/lib64 -DXML_INCLUDE_DIR=/usr/include/libxml2 -DCMAKE_PREFIX_PATH="$INST/elpa" \
+        -DWITH_MDI=OFF -DWITH_TORCH=OFF -DWITH_CUSTOMIZED_DEALII=OFF -DWITH_DCCL=OFF \
+        -DWITH_COMPLEX="$cplx" -DWITH_GPU=ON -DGPU_LANG=cuda -DGPU_VENDOR=nvidia -DWITH_GPU_AWARE_MPI=OFF \
+        "-DCMAKE_CUDA_ARCHITECTURES=$ARCH" -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++ \
+        -DWITH_TESTING=OFF -DHIGHERQUAD_PSP=OFF -DBUILD_SHARED_LIBS=ON -DUSE_64BIT_INT=ON
+    run "dftfe-$kind-build.log" cmake --build "$L3_BUILD/$kind" -j "$JOBS"
+    mkdir -p "$INST/bin"; cp -f "$L3_BUILD/$kind/dftfe" "$INST/bin/dftfe_$kind"
+    # (cuobjdump exits nonzero on a host-only binary: keep the substitutions from aborting the script under pipefail)
+    local archs; archs="$( { cuobjdump --list-elf "$INST/bin/dftfe_$kind" 2>/dev/null || true; } | { /usr/bin/grep -o 'sm_[0-9]*' || true; } | sort -u | paste -sd,)"
+    [ -n "$archs" ] || archs="$(for l in "$L3_BUILD/$kind"/libdftfe*.so*; do if [ -f "$l" ]; then cuobjdump --list-elf "$l" 2>/dev/null | /usr/bin/grep -o 'sm_[0-9]*' || true; fi; done | sort -u | paste -sd,)"
+    [ "$archs" = "sm_$ARCH" ] || { echo "build.sh: dftfe_$kind device code '$archs' != sm_$ARCH" >&2; exit 1; }
+    echo "dftfe_$kind sha256=$(l3_sha_file "$INST/bin/dftfe_$kind") device_archs=$archs" >> "$INST/BUILD_INFO.txt"
+}
+: > "$INST/BUILD_INFO.txt"
+build_dftfe real
+[ -n "${HPCPERF_DFTFE_COMPLEX:-}" ] && build_dftfe complex
+l3_fingerprint_write "$INST" "$FP"
+{
+    echo "profile=$PROFILE dftfe=$SHA utc=$(date -u +%FT%TZ) jobs=$JOBS total_seconds=$(( $(date +%s) - T0 ))"
+    echo "deps: $DEPS_DESC"; echo "dftfe options: $DFTFE_OPTS"
+    echo "elpa nvidia kernels: $ELPA_GPU ($(/usr/bin/grep -c 'nvidia' "$L3_LOGS/elpa-configure.log" 2>/dev/null || true) configure lines mention nvidia)"
+} >> "$INST/BUILD_INFO.txt"
+cat "$INST/BUILD_INFO.txt"
+echo "# DFT-FE built in $(( $(date +%s) - T0 )) s -> $INST/bin"
diff --git a/level3/dftfe/dftfe_check.py b/level3/dftfe/dftfe_check.py
new file mode 100755
index 0000000..2995bd2
--- /dev/null
+++ b/level3/dftfe/dftfe_check.py
@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+"""Compare a DFT-FE run (REPRODUCIBLE OUTPUT = true) with a reference output for validate.sh.
+
+usage: dftfe_check.py <dftfe.out> <reference.output> [--check] [--label NAME]
+                      [--tol-e0 1e-5] [--tol-emd 2e-5] [--tol-temp 0.1] [--tol-force 2e-5]
+
+Quantities (all parsed from DFT-FE's own output, both files):
+  e0        first ground-state "Total energy:" (Ha, printed with 8 decimals)
+  emd[k]    "Total Energy in Ha at timeIndex" of MD step k (5 decimals)        [MD decks]
+  temp[k]   "Temperature from velocities" of MD step k (K, 2 decimals)          [MD decks]
+  forces    "Absolute values of ion forces" per atom (Ha/Bohr, 6 decimals)      [when printed]
+  steps     number of MD steps; MD decks must end with "MD run completed successfully"
+Pre-fixed tolerances (absolute): |d e0| <= 1e-5 Ha, |d emd| <= 2e-5 Ha, |d temp| <= 0.1 K,
+max |d force| <= 2e-5 Ha/Bohr -- chosen BEFORE the runs from the deck's own convergence settings
+(SCF TOLERANCE 1e-5 on the density residual, energies quadratically converged) and the print
+resolution of the reference (5 decimals for the MD energies). Upstream's own GPU check is a plain
+`diff` against accuracyBenchmarks/, i.e. an even stricter expectation.
+With --check any violation, NaN/Inf, missing quantity, incomplete MD or step-count mismatch exits 1.
+"""
+import argparse
+import os
+import re
+import sys
+
+sys.path.insert(0, os.environ["L3_TOOLS"])
+from l3_check import ValidationError, require_finite  # noqa: E402
+
+ap = argparse.ArgumentParser()
+ap.add_argument("out"); ap.add_argument("ref"); ap.add_argument("--check", action="store_true"); ap.add_argument("--label", default="")
+ap.add_argument("--tol-e0", type=float, default=1e-5); ap.add_argument("--tol-emd", type=float, default=2e-5)
+ap.add_argument("--tol-temp", type=float, default=0.1); ap.add_argument("--tol-force", type=float, default=2e-5)
+a = ap.parse_args()
+
+
+def parse(path, name):
+    txt = open(path, errors="replace").read()
+    d = {"complete": "MD run completed successfully" in txt, "is_md": "MD STEP" in txt or "Molecular Dynamics" in txt}
+    e0 = re.findall(r"Total energy:\s+(\S+)", txt)
+    d["e0"] = require_finite(f"{name} e0", e0[0]) if e0 else None
+    d["scf_iters"] = [int(x) for x in re.findall(r"converged to the specified tolerance after:\s+(\d+) iterations", txt)]
+    d["emd"] = [require_finite(f"{name} emd", x) for x in re.findall(r"Total Energy in Ha at timeIndex\s+(\S+)", txt)]
+    d["temp"] = [require_finite(f"{name} temp", x) for x in re.findall(r"Temperature from velocities:\s+(\S+)", txt)]
+    d["steps"] = [int(x) for x in re.findall(r"MD STEP\s+(\d+)", txt)]
+    forces = []
+    for block in re.findall(r"Absolute values of ion forces[^\n]*\n(.*?)\n-{20,}", txt, re.S):
+        forces.append([tuple(require_finite(f"{name} force", v) for v in m.split(",")) for m in re.findall(r"AtomId\s+\d+:\s+([-0-9.eE,]+)", block)])
+    d["forces"] = forces
+    d["not_converged"] = "not converged" in txt.lower() or "NOT converged" in txt
+    return d
+
+
+try:
+    o, r = parse(a.out, "run"), parse(a.ref, "reference")
+    if a.check:
+        if o["e0"] is None:
+            raise ValidationError("run has no 'Total energy:' line (no completed SCF)")
+        if o["not_converged"]:
+            raise ValidationError("run reports a non-converged solve")
+        if r["is_md"]:
+            if not o["complete"]:
+                raise ValidationError("MD run did not end with 'MD run completed successfully'")
+            if len(o["emd"]) != len(r["emd"]) or len(o["steps"]) != len(r["steps"]):
+                raise ValidationError(f"MD step count {len(o['steps'])}/{len(o['emd'])} energies vs reference {len(r['steps'])}/{len(r['emd'])}")
+    de0 = abs(o["e0"] - r["e0"]) if (o["e0"] is not None and r["e0"] is not None) else None
+    demd = [abs(x - y) for x, y in zip(o["emd"], r["emd"])]
+    dtemp = [abs(x - y) for x, y in zip(o["temp"], r["temp"])]
+    dforce = None
+    if o["forces"] and r["forces"] and len(o["forces"][0]) == len(r["forces"][0]):
+        dforce = max(abs(x - y) for fo, fr in zip(o["forces"], r["forces"]) for po, pr in zip(fo, fr) for x, y in zip(po, pr))
+    tag = f"[{a.label}] " if a.label else ""
+    print(f"{tag}e0_run={o['e0']!r} e0_ref={r['e0']!r} d_e0={de0!r} tol_e0={a.tol_e0} scf_iters_run={o['scf_iters']} scf_iters_ref={r['scf_iters']}")
+    if r["is_md"] or o["is_md"]:
+        print(f"{tag}md_steps={len(o['steps'])} complete={o['complete']} emd_run={o['emd']} emd_ref={r['emd']} max_d_emd={max(demd) if demd else None!r} tol_emd={a.tol_emd}")
+        print(f"{tag}temp_run={o['temp']} temp_ref={r['temp']} max_d_temp={max(dtemp) if dtemp else None!r} tol_temp={a.tol_temp}")
+    print(f"{tag}force_blocks_run={len(o['forces'])} force_blocks_ref={len(r['forces'])} max_d_force={dforce!r} tol_force={a.tol_force}")
+    if a.check:
+        bad = []
+        if de0 is None or de0 > a.tol_e0: bad.append(f"e0 |diff| {de0} > {a.tol_e0}")
+        if any(x > a.tol_emd for x in demd): bad.append(f"MD energy |diff| {max(demd)} > {a.tol_emd}")
+        if any(x > a.tol_temp for x in dtemp): bad.append(f"temperature |diff| {max(dtemp)} > {a.tol_temp}")
+        if dforce is not None and dforce > a.tol_force: bad.append(f"force |diff| {dforce} > {a.tol_force}")
+        if bad:
+            raise ValidationError("; ".join(bad))
+        print(f"{tag}RESULT within tolerances")
+except ValidationError as ex:
+    print(f"  VALIDATION ERROR: {ex}")
+    sys.exit(1)
diff --git a/level3/dftfe/elpa_probe.sh b/level3/dftfe/elpa_probe.sh
new file mode 100755
index 0000000..671520b
--- /dev/null
+++ b/level3/dftfe/elpa_probe.sh
@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# Independent check of the ELPA NVIDIA-GPU kernels built for DFT-FE (before any DFT-FE run):
+# ELPA's own test programs (test/Fortran/test.F90 compiled as validate_*_gpu_analytic programs,
+# which call e%set("nvidia-gpu", 1)) diagonalise the analytic test matrix of
+# test/shared/test_analytic_template.F90 (known eigenpairs) and verify themselves
+#   max |lambda_i - lambda_i^exact|   <= 5e-14   (ELPA's tol_eigenvalues, real double)
+#   max |z_i - z_i^exact|             <= 6e-10   (ELPA's tol_eigenvectors, real double)
+# (nonzero exit -- `stop 1` -- on violation); this script re-parses the printed
+# "Maximum error in eigenvalues/eigenvectors" values and applies the same limits, runs the
+# 1-stage and 2-stage GPU solvers on 1, 2 and 4 GPUs (one MPI rank per GPU through the common
+# launcher, ELPA's process grid np_rows x np_cols) and the CPU versions of the same programs on
+# 1 rank as a cross-check when they exist. A GPU run whose errors are not within the limits,
+# that does not exit 0, or whose launcher audit shows a GPU mismatch -> FAIL; a run whose output
+# lacks the values -> FAIL (never an abort of this script).
+#
+#   ./elpa_probe.sh            HPCPERF_ELPA_NA=2000 HPCPERF_ELPA_NEV=1000 HPCPERF_ELPA_NBLK=32
+# Record: <install>/elpa/ELPA_GPU_PROBE.txt (PASS/FAIL + values); validate.sh requires PASS.
+# History: the first version looked for the "%Error Residual/Orthogonality" lines of ELPA's
+# *random-matrix* validate programs, which the analytic programs do not print (they print the
+# eigenvalue/eigenvector errors above) -- under `set -e -o pipefail` the empty grep aborted the
+# script after the first (successful) GPU run; fixed 2026-09-06, no run was ever mis-judged.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+GCC_MM="$(l3_version_mm "$(/usr/bin/gcc -dumpfullversion)")"; OMPI_V="$(mpirun --version | head -1 | /usr/bin/grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)"
+PROFILE="${HPCPERF_DFTFE_PROFILE:-cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-ompi$(echo "$OMPI_V" | tr -d .)}"
+l3_paths_profile dftfe "$PROFILE"
+EB="$L3_BUILD_DEPS/elpa"; INST="$L3_INSTALL"
+[ -f "$INST/elpa/.hpcperf-stage-done" ] || { echo "elpa_probe.sh: ELPA stage not built for profile $PROFILE" >&2; exit 1; }
+NA="${HPCPERF_ELPA_NA:-2000}"; NEV="${HPCPERF_ELPA_NEV:-1000}"; NBLK="${HPCPERF_ELPA_NBLK:-32}"
+TOL_EV=5e-14; TOL_Z=6e-10     # test_analytic_template.F90: tol_eigenvalues / tol_eigenvectors (real double)
+export LD_LIBRARY_PATH="$EB/.libs:$INST/elpa/lib:$INST/scalapack/lib:$INST/openblas/lib:${LD_LIBRARY_PATH:-}"
+OUT="$L3_BUILD/elpa_probe"; mkdir -p "$OUT"
+REC="$INST/elpa/ELPA_GPU_PROBE.txt"; : > "$REC.tmp"
+ok=1
+note() { echo "$*" | tee -a "$REC.tmp"; }
+run_case() { # run_case <program> <ranks> <gpus:yes|no>
+    # the top-level names in the build tree are libtool wrapper scripts (they would try to relink through mpicc);
+    # the real, already linked programs live in .libs/
+    local prog=$1 n=$2 gpu=$3 exe="$EB/.libs/$prog" log="$OUT/$prog.np$n.log" rc=0 res orth audit
+    if [ ! -x "$exe" ]; then
+        if [ "$gpu" = yes ]; then note "MISSING $prog (GPU test program not built)"; ok=0; else note "SKIPPED $prog (CPU variant not built with the GPU-enabled configuration; cross-check unavailable)"; fi
+        return
+    fi
+    if [ "$gpu" = yes ]; then
+        "$L3_LAUNCHER" --gpus "$n" --cpus-per-rank 4 --bind wrapper -- "$exe" "$NA" "$NEV" "$NBLK" > "$log" 2>&1 || rc=$?
+        audit="$( { /usr/bin/grep -a 'audit summary' "$log" || true; } | tail -1 | sed 's/.*audit summary: //')"
+    else
+        # CPU cross-check: one rank through the same launcher (no GPU binding; the CPU program ignores the device)
+        "$L3_LAUNCHER" --gpus 1 --bind none -- "$exe" "$NA" "$NEV" "$NBLK" > "$log" 2>&1 || rc=$?
+        audit="cpu"
+    fi
+    # the analytic test programs print these two lines (test_analytic_template.F90); missing -> FAIL below
+    ev="$( { /usr/bin/grep -a 'Maximum error in eigenvalues' "$log" || true; } | tail -1 | awk -F: '{print $2}' | tr -d ' ')"
+    zv="$( { /usr/bin/grep -a 'Maximum error in eigenvectors' "$log" || true; } | tail -1 | awk -F: '{print $2}' | tr -d ' ')"
+    gpu_evidence="$( { /usr/bin/grep -ac '_gpu\|gpublas_\|gpu_copy' "$log" || true; } )"
+    local verdict
+    verdict="$(python3 - "$rc" "$ev" "$zv" "$TOL_EV" "$TOL_Z" "$gpu" "$audit" "$gpu_evidence" <<'PY'
+import sys, math
+rc, ev, zv, tol_ev, tol_z, gpu, audit, gpu_evidence = sys.argv[1:9]
+try:
+    e, z = float(ev), float(zv)
+    fin = math.isfinite(e) and math.isfinite(z)
+except ValueError:
+    fin = False; e = z = float("nan")
+good = rc == "0" and fin and 0 <= e <= float(tol_ev) and 0 <= z <= float(tol_z)
+if gpu == "yes":
+    # launcher audit "N verified, 0 mismatch, ..." and ELPA's own GPU timers (trans_ev_*_gpu, gpublas_*) in the output
+    good = good and "mismatch" in audit and audit.split(",")[1].strip().startswith("0") and int(gpu_evidence or 0) > 0
+print("PASS" if good else "FAIL")
+PY
+)"
+    [ "$verdict" = PASS ] || ok=0
+    note "$verdict $prog ranks=$n gpu=$gpu na=$NA nev=$NEV nblk=$NBLK exit=$rc max_err_eigenvalues=${ev:-NA} (tol $TOL_EV) max_err_eigenvectors=${zv:-NA} (tol $TOL_Z) gpu_timer_lines=$gpu_evidence audit=[${audit:-none}] log=$log"
+}
+note "# ELPA 2026.02.001 GPU kernel probe, profile $PROFILE, $(date -u +%FT%TZ); limits: max eigenvalue error <= $TOL_EV, max eigenvector error <= $TOL_Z (ELPA's own real-double analytic-test tolerances), exit 0, GPU timers present, launcher audit 0 mismatch"
+for prog in validate_real_double_eigenvectors_1stage_gpu_analytic validate_real_double_eigenvectors_2stage_default_kernel_gpu_analytic; do   # (the *_default names are ELPA's .sh wrappers; these are the programs)
+    for n in 1 2 4; do run_case "$prog" "$n" yes; done
+done
+for prog in validate_real_double_eigenvectors_1stage_analytic validate_real_double_eigenvectors_2stage_default_kernel_analytic; do
+    run_case "$prog" 1 no
+done
+if [ "$ok" -eq 1 ]; then note "RESULT PASS"; else note "RESULT FAIL"; fi
+mv -f "$REC.tmp" "$REC"; echo "# record: $REC"; [ "$ok" -eq 1 ]
diff --git a/level3/dftfe/fetch.sh b/level3/dftfe/fetch.sh
new file mode 100755
index 0000000..2f3bd14
--- /dev/null
+++ b/level3/dftfe/fetch.sh
@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+# Fetch DFT-FE 1.2.0 and the pinned sources of its dependency stack (read-only).
+#
+#   DFT-FE 1.2.0   7147faa51f7c9f3075fffaa5e48ba989bcd329c1 (github.com/dftfeDevelopers/dftfe, 2025-08-17)
+#   install_DFTFE  reference recipe: branch frontierDevelop b27f89b (dftfe2.sh: deal.II 9.7.1, p4est 2.8.7,
+#                  Kokkos 4.6.00, ELPA 2026.02.001, libxc 7.0.0, ScaLAPACK 2.2.2, ALGLIB 4.06.0, spglib 02159eef)
+#   Tarballs (downloaded once into .deps/level3/dftfe/downloads, sha256 recorded in SHA256SUMS):
+#     OpenBLAS 0.3.30, ScaLAPACK 2.2.2, libxc 7.0.0, ALGLIB 4.06.0 (GPL), p4est 2.8.7 (+ dftfe's p4est-setup.sh),
+#     Kokkos 4.6.00, deal.II 9.7.1, ELPA 2026.02.001; spglib at commit 02159eef6e7349535049a43fe2272bb634c77945 (git).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+UP="$R/_upstream/level3"; DL="$R/.deps/level3/dftfe/downloads"
+SHA=7147faa51f7c9f3075fffaa5e48ba989bcd329c1
+mkdir -p "$UP" "$DL"
+if [ ! -d "$UP/dftfe/.git" ]; then
+    git clone -q https://github.com/dftfeDevelopers/dftfe.git "$UP/dftfe"
+    git -C "$UP/dftfe" checkout -q "$SHA"
+fi
+got="$(git -C "$UP/dftfe" rev-parse HEAD)"; [ "$got" = "$SHA" ] || { echo "fetch.sh: dftfe at $got, expected $SHA" >&2; exit 1; }
+declare -A URLS=(
+  [OpenBLAS-0.3.30.tar.gz]=https://github.com/OpenMathLib/OpenBLAS/releases/download/v0.3.30/OpenBLAS-0.3.30.tar.gz
+  [v2.2.2.tar.gz]=https://github.com/Reference-ScaLAPACK/scalapack/archive/refs/tags/v2.2.2.tar.gz
+  [libxc-7.0.0.tar.gz]=https://gitlab.com/libxc/libxc/-/archive/7.0.0/libxc-7.0.0.tar.gz
+  [alglib-4.06.0.cpp.gpl.tgz]=https://www.alglib.net/translator/re/alglib-4.06.0.cpp.gpl.tgz
+  [p4est-2.8.7.tar.gz]=https://p4est.github.io/release/p4est-2.8.7.tar.gz
+  [p4est-setup.sh]=https://raw.githubusercontent.com/dftfeDevelopers/dftfe/manual/p4est-setup.sh
+  [4.6.00.tar.gz]=https://github.com/kokkos/kokkos/archive/refs/tags/4.6.00.tar.gz
+  [dealii-9.7.1.tar.gz]=https://github.com/dealii/dealii/releases/download/v9.7.1/dealii-9.7.1.tar.gz
+  [dealii-9.6.2.tar.gz]=https://github.com/dealii/dealii/releases/download/v9.6.2/dealii-9.6.2.tar.gz
+  [elpa-2026.02.001.tar.gz]=https://elpa.mpcdf.mpg.de/software/tarball-archive/Releases/2026.02.001/elpa-2026.02.001.tar.gz
+)
+# expected sha256 (recorded at first download, 2026-09-05)
+declare -A SHAS=(
+  [OpenBLAS-0.3.30.tar.gz]=27342cff518646afb4c2b976d809102e368957974c250a25ccc965e53063c95d
+  [v2.2.2.tar.gz]=a2f0c9180a210bf7ffe126c9cb81099cf337da1a7120ddb4cbe4894eb7b7d022
+  [libxc-7.0.0.tar.gz]=8d4e343041c9cd869833822f57744872076ae709a613c118d70605539fb13a77
+  [alglib-4.06.0.cpp.gpl.tgz]=27fa4b0b8160cbb2ace4f69d1a8ef0990cf894c87dff3f46dc86ec8ac2d30c68
+  [p4est-2.8.7.tar.gz]=0a1e912f3529999ca6d62fee335d51f24b5650b586e95a03ef39ebf73936d7f4
+  [p4est-setup.sh]=86269c2ef751b33d06916299a3d15fa80f4c8c9de40eceedfb8b7f45bd90fb8b
+  [4.6.00.tar.gz]=348b2d860046fc3ddef5ca3a128317be1a6f3fa35196f268338a180fcae52264
+  [dealii-9.7.1.tar.gz]=0f2096ef83db54fdcebe9f3d148fa713f63f1c3f567941b53bcb4a1a8ea7de43
+  [dealii-9.6.2.tar.gz]=1051e332de3822488e91c2b0460681052a3c4c5ac261cdd7a6af784869a25523
+  [elpa-2026.02.001.tar.gz]=a379f27f4dbd27b2ee45017afec656d064301e97150c874649bdfd64957b75ed
+)
+for f in "${!URLS[@]}"; do
+    [ -s "$DL/$f" ] || curl -sSL -o "$DL/$f" "${URLS[$f]}"
+    got_sha="$(sha256sum "$DL/$f" | cut -d' ' -f1)"
+    [ "$got_sha" = "${SHAS[$f]}" ] || { echo "fetch.sh: $f sha256 $got_sha != expected ${SHAS[$f]}" >&2; exit 1; }
+done
+if [ ! -d "$DL/spglib/.git" ]; then git clone -q https://github.com/spglib/spglib.git "$DL/spglib"; git -C "$DL/spglib" checkout -q 02159eef6e7349535049a43fe2272bb634c77945; fi
+[ "$(git -C "$DL/spglib" rev-parse HEAD)" = 02159eef6e7349535049a43fe2272bb634c77945 ] || { echo "fetch.sh: spglib not at the pinned commit" >&2; exit 1; }
+( cd "$DL" && for f in "${!SHAS[@]}"; do echo "${SHAS[$f]}  $f"; done | sort -k2 > SHA256SUMS )
+echo "# dftfe $got; downloads under $DL (all sha256 verified):"; sed 's/^/#   /' "$DL/SHA256SUMS"; echo "#   spglib $(git -C "$DL/spglib" rev-parse HEAD)"
diff --git a/level3/dftfe/patches/0001-std-isnan.patch b/level3/dftfe/patches/0001-std-isnan.patch
new file mode 100644
index 0000000..0fdf719
--- /dev/null
+++ b/level3/dftfe/patches/0001-std-isnan.patch
@@ -0,0 +1,33 @@
+DFT-FE 1.2.0 (7147faa51f7c9f3075fffaa5e48ba989bcd329c1) -- qualify two isnan() calls as std::isnan
+
+Class: source compatibility fix (2 lines; no numerics: the calls only detect NaN
+in a diagnostic branch of AtomicCenteredNonLocalOperator::compute...).
+Problem: src/atom/AtomicCenteredNonLocalOperator.cc lines 785/791 call the
+unqualified `isnan(...)` inside a template; C++ provides std::isnan (<cmath>), and
+the unqualified spelling only compiles when some header drags in the C macro.
+With GCC 14.2.1 and deal.II 9.6.2's headers this fails ("there are no arguments to
+'isnan' that depend on a template parameter ... 'isnan' was not declared in this
+scope"). Upstream's publicGithubDevelop branch no longer contains these calls.
+Fix: `std::isnan`. Conditions: any compiler; applied for this profile. Impact:
+none on results. Verified: DFT-FE real builds; results validated in
+level3/dftfe/validate.sh. Applied by level3/dftfe/build.sh (git apply,
+idempotent); hash recorded in the profile fingerprint.
+
+--- a/src/atom/AtomicCenteredNonLocalOperator.cc
++++ b/src/atom/AtomicCenteredNonLocalOperator.cc
+@@ -782,13 +782,13 @@
+                         projectorMatricesReal[flattenedIndex];
+                       const double tempImag =
+                         projectorMatricesImag[flattenedIndex];
+-                      if (isnan(tempReal))
++                      if (std::isnan(tempReal))
+                         std::cout
+                           << "Real->Processor number and indices has nan: "
+                           << d_this_mpi_process << " " << iElemComp << " "
+                           << kPoint << " "
+                           << " " << beta << " " << iNode << std::endl;
+-                      if (isnan(tempImag))
++                      if (std::isnan(tempImag))
+                         std::cout
+                           << "Imag->Processor number and indices has nan: "
+                           << d_this_mpi_process << " " << iElemComp << " "
diff --git a/level3/dftfe/run.sh b/level3/dftfe/run.sh
new file mode 100755
index 0000000..4e10e17
--- /dev/null
+++ b/level3/dftfe/run.sh
@@ -0,0 +1,103 @@
+#!/usr/bin/env bash
+# Run DFT-FE (dftfe_real, CUDA) on N GPUs, one MPI rank per GPU, 1 thread per rank (as upstream's GPU
+# job scripts: OMP_NUM_THREADS=DFTFE_NUM_THREADS=DEAL_II_NUM_THREADS=1).
+#
+#   ./run.sh [CUDA]
+#
+# Cases (HPCPERF_DFTFE_CASE), inputs from the read-only checkout testsGPU/pseudopotential/real/:
+#   al_md (default)  Input_MD_0.prm -- upstream's GPU regression case: 32-atom fcc Al supercell
+#       (15.28 bohr cube, ONCV PBE, order-3 FE, 85 Kohn-Sham states, ANDERSON_WITH_KERKER mixing,
+#       SCF TOLERANCE 1e-5), Born-Oppenheimer NVE MD at 1400 K, 4 steps of 1 fs, USE GPU = true,
+#       REPRODUCIBLE OUTPUT = true; upstream reference accuracyBenchmarks/output_MD_0.
+#         smoke / strong : the deck verbatim (fixed 32-atom problem; strong scaling of a SMALL problem)
+#         weak           : derived Al supercell series, 32 atoms per GPU: N=1 -> 2x2x2 cells (32 atoms,
+#                          verbatim), N=2 -> 4x2x2 (64), N=4 -> 4x4x2 (128); coordinates replicated,
+#                          domain vectors scaled, NATOMS and NUMBER OF KOHN-SHAM WAVEFUNCTIONS (85 per
+#                          32 atoms) scaled; everything else verbatim (diff recorded). SYNTHETIC series
+#                          (same construction as upstream's dftfe-benchmarks Mo supercells).
+#   llzo             parameterFile_LLZO.prm -- 192-atom Li7La3Zr2O12 ground state (720 states, USE ELPA,
+#                    GPU); upstream reference accuracyBenchmarks/outputLLZO. smoke/strong verbatim only
+#                    (a heavier fixed-size strong-scaling case).
+# Controls: HPCPERF_GPUS=N|all, HPCPERF_CPUS_PER_RANK (CPU cores bound per rank, default 4; threads
+#           stay 1), HPCPERF_SCALE_MODE=smoke|strong|weak, HPCPERF_DFTFE_PROFILE, HPCPERF_DRY_RUN=1
+# Output: <run_dir>/dftfe.out (stdout+stderr incl. launcher audit), run_manifest.txt.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"; [ $# -gt 0 ] && shift
+[ "$BACKEND" = CUDA ] || { echo "run.sh: only CUDA is built for DFT-FE here" >&2; exit 2; }
+GCC_MM="$(l3_version_mm "$(/usr/bin/gcc -dumpfullversion)")"; OMPI_V="$(mpirun --version | head -1 | /usr/bin/grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)"
+PROFILE="${HPCPERF_DFTFE_PROFILE:-cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-ompi$(echo "$OMPI_V" | tr -d .)}"
+l3_paths_profile dftfe "$PROFILE"
+SRC="$R/_upstream/level3/dftfe"; T="$SRC/testsGPU/pseudopotential/real"; INST="$L3_INSTALL"
+EXE="$INST/bin/dftfe_real"
+[ -x "$EXE" ] || { echo "run.sh: $EXE not found -- run ./build.sh first (profile $PROFILE)" >&2; exit 1; }
+export LD_LIBRARY_PATH="$INST/dealii/lib:$INST/dealii/lib64:$INST/elpa/lib:$INST/scalapack/lib:$INST/openblas/lib:$INST/libxc/lib:$INST/libxc/lib64:$INST/spglib/lib:$INST/spglib/lib64:$INST/kokkos/lib:$INST/kokkos/lib64:$INST/p4est/FAST/lib:$INST/alglib:$L3_BUILD/real:${LD_LIBRARY_PATH:-}"
+l3_binary_backend_check "$EXE" cuda || exit 1
+CASE="${HPCPERF_DFTFE_CASE:-al_md}"
+MODE="$(l3_scale_mode dftfe)" || exit 2
+N_RANKS="$(hpcperf_ranks dftfe yes)" || exit 2
+CPUS="${HPCPERF_CPUS_PER_RANK:-4}"
+case "$CASE" in
+    al_md) PRM="$T/Input_MD_0.prm"; FILES="aluminumMD_coordinates.inp aluminumMD_domainBoundingVectors.inp aluminumMD_pseudo.inp Al.upf Mass.inp"; REF="$T/accuracyBenchmarks/output_MD_0" ;;
+    llzo)  PRM="$T/parameterFile_LLZO.prm"; FILES="coordinates_LLZO.inp domainVectors_LLZO.inp pseudo_LLZO.inp Li.upf La.upf O.upf Mass.inp"; REF="$T/accuracyBenchmarks/outputLLZO"
+           [ "$MODE" != weak ] || { echo "run.sh: llzo has no weak series" >&2; exit 2; }
+           for f in $(awk '{print $2}' "$T/pseudo_LLZO.inp"); do FILES="$FILES $f"; done ;;
+    *) echo "run.sh: HPCPERF_DFTFE_CASE must be al_md or llzo" >&2; exit 2 ;;
+esac
+[ -f "$PRM" ] || { echo "run.sh: $PRM missing (run fetch.sh)" >&2; exit 1; }
+LABEL="$CASE"; DERIV="verbatim"
+[ "$MODE" = weak ] && { LABEL="$CASE.x$N_RANKS"; DERIV="al-supercell x$N_RANKS (32 atoms/GPU)"; }
+RUN_DIR="$(l3_rundir "$L3_BUILD/$L3_RUN_SUBDIR/$LABEL.$MODE.np$N_RANKS")" || exit 2
+for f in $(echo "$FILES" | tr ' ' '\n' | sort -u); do [ -f "$T/$f" ] || { echo "run.sh: input $T/$f missing" >&2; exit 1; }; ln -sfn "$T/$f" "$RUN_DIR/$f"; done
+if [ "$MODE" = weak ] && [ "$N_RANKS" -gt 1 ]; then
+    case "$N_RANKS" in 2) REP="2 1 1" ;; 4) REP="2 2 1" ;; 8) REP="2 2 2" ;; *) echo "run.sh: weak series defined for 1/2/4/8 GPUs" >&2; exit 2 ;; esac
+    python3 - "$T" "$RUN_DIR" "$PRM" $REP <<'PY'
+import math, re, sys
+T, out, prm = sys.argv[1], sys.argv[2], sys.argv[3]; rx, ry, rz = (int(v) for v in sys.argv[4:7])
+coords = [l.split() for l in open(f"{T}/aluminumMD_coordinates.inp") if l.strip()]
+dom = [[float(v) for v in l.split()] for l in open(f"{T}/aluminumMD_domainBoundingVectors.inp") if l.strip()]
+new = []
+for i in range(rx):
+    for j in range(ry):
+        for k in range(rz):
+            for c in coords:
+                x, y, z = (float(c[2]) + i) / rx, (float(c[3]) + j) / ry, (float(c[4]) + k) / rz
+                new.append(f"{c[0]} {c[1]} {x:.12f} {y:.12f} {z:.12f}\n")
+open(f"{out}/al_coordinates.inp", "w").writelines(new)
+scale = [rx, ry, rz]
+open(f"{out}/al_domainBoundingVectors.inp", "w").writelines(" ".join(f"{v * scale[i]:.10f}" for v in dom[i]) + "\n" for i in range(3))
+n = len(new); nwf = math.ceil(85 * n / 32)
+txt = open(prm).read()
+txt, c1 = re.subn(r"(set NATOMS\s*=\s*)\d+", rf"\g<1>{n}", txt)
+txt, c2 = re.subn(r"(set NUMBER OF KOHN-SHAM WAVEFUNCTIONS\s*=\s*)\d+", rf"\g<1>{nwf}", txt)
+txt, c3 = re.subn(r"aluminumMD_coordinates\.inp", "al_coordinates.inp", txt)
+txt, c4 = re.subn(r"aluminumMD_domainBoundingVectors\.inp", "al_domainBoundingVectors.inp", txt)
+if not (c1 == c2 == c3 == c4 == 1): sys.exit(f"run.sh: deck derivation touched unexpected counts {c1, c2, c3, c4}")
+open(f"{out}/parameters.prm", "w").write(txt)
+print(f"# derived Al supercell: {rx}x{ry}x{rz} replication -> {n} atoms, {nwf} Kohn-Sham states")
+PY
+    diff -u "$PRM" "$RUN_DIR/parameters.prm" > "$RUN_DIR/deck.diff" || true
+else
+    cp "$PRM" "$RUN_DIR/parameters.prm"
+fi
+export OMP_NUM_THREADS=1 DFTFE_NUM_THREADS=1 DEAL_II_NUM_THREADS=1
+echo "# DFT-FE $BACKEND profile=$PROFILE case=$CASE mode=$MODE ($DERIV) ranks=$N_RANKS deck=$(realpath --relative-to="$SRC" "$PRM") run_dir=$RUN_DIR"
+cd "$RUN_DIR"
+RUN_ID="$(l3_run_id)"
+set +e; set -o pipefail
+"$L3_LAUNCHER" --gpus "$N_RANKS" --cpus-per-rank "$CPUS" --bind wrapper -- "$EXE" parameters.prm 2>&1 | tee "$RUN_DIR/dftfe.out"
+rc=$?
+set +o pipefail; set -e
+if [ -z "${HPCPERF_DRY_RUN:-}" ]; then
+    l3_manifest "$RUN_DIR" "run_id=$RUN_ID" "app=dftfe" "backend=$BACKEND" "profile=$PROFILE" "case=$CASE" "label=$LABEL" "mode=$MODE" "deck_derivation=$DERIV" \
+        "ranks=$N_RANKS" "threads_per_rank=1" "exit_code=$rc" "binary=$EXE" "binary_sha256=$(l3_sha_file "$EXE")" \
+        "upstream_prm=$PRM" "upstream_prm_sha256=$(l3_sha_file "$PRM")" "deck_sha256=$(l3_sha_file "$RUN_DIR/parameters.prm")" "reference_output=$REF" "reference_sha256=$(l3_sha_file "$REF")" \
+        "fingerprint_sha256=$(l3_sha_file "$INST/.hpcperf-l3-fingerprint")" "elpa_gpu_probe=$(/usr/bin/grep -m1 '^RESULT' "$INST/elpa/ELPA_GPU_PROBE.txt" 2>/dev/null || echo NOT_RUN)" "utc=$(date -u +%FT%TZ)"
+    /usr/bin/grep -aE 'Device|GPU|device' "$RUN_DIR/dftfe.out" 2>/dev/null | head -6 | sed 's/^/gpu_evidence: /' >> "$RUN_DIR/run_manifest.txt" || true
+fi
+exit "$rc"
diff --git a/level3/dftfe/validate.sh b/level3/dftfe/validate.sh
new file mode 100755
index 0000000..cb9658f
--- /dev/null
+++ b/level3/dftfe/validate.sh
@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# Correctness check for DFT-FE (dftfe_real, CUDA) on N GPUs.
+#
+#   ./validate.sh [CUDA]       HPCPERF_GPUS=N (default 1)
+#
+# [0] ELPA GPU kernels verified independently: <install>/elpa/ELPA_GPU_PROBE.txt must say RESULT PASS
+#     (elpa_probe.sh: ELPA's own analytic-matrix test programs with the sm_100 GPU kernels on 1/2/4 GPUs,
+#     ELPA's eigenvalue/eigenvector error limits, exit 0, GPU timers present, launcher audit 0 mismatch).
+# [1] al_md (upstream GPU regression deck Input_MD_0.prm, verbatim) on N GPUs, dftfe_check.py --check
+#     against upstream's own GPU reference accuracyBenchmarks/output_MD_0: SCF converged, MD completed
+#     (4 steps), ground-state energy within 1e-5 Ha, per-step MD total energies within 2e-5 Ha,
+#     temperatures within 0.1 K, ion forces within 2e-5 Ha/Bohr (pre-fixed, see dftfe_check.py), every
+#     value finite; GPU evidence: the deck's USE GPU = true is honoured by a CUDA binary (backend check
+#     in run.sh) and the launcher audit must report every rank verified on its own GPU with 0 mismatch.
+#     For N > 1 the same quantities are ALSO compared with the 1-GPU run of the same binary/deck under
+#     the same tolerances (rank-count independence).
+# Exit codes are captured (timeout/nonzero -> FAIL); nothing is read from old runs except the 1-GPU
+# reference, which is re-run when its binary hash differs.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+N="${HPCPERF_GPUS:-1}"
+TIMEOUT="${HPCPERF_VALIDATE_TIMEOUT:-3600}"
+GCC_MM="$(l3_version_mm "$(/usr/bin/gcc -dumpfullversion)")"; OMPI_V="$(mpirun --version | head -1 | /usr/bin/grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)"
+PROFILE="${HPCPERF_DFTFE_PROFILE:-cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-ompi$(echo "$OMPI_V" | tr -d .)}"
+l3_paths_profile dftfe "$PROFILE"
+SRC="$R/_upstream/level3/dftfe"; RUNS="$L3_BUILD/$L3_RUN_SUBDIR"; INST="$L3_INSTALL"
+REF="$SRC/testsGPU/pseudopotential/real/accuracyBenchmarks/output_MD_0"
+export HPCPERF_GPUS="$N" HPCPERF_SCALE_MODE=smoke HPCPERF_DFTFE_CASE=al_md
+mkdir -p "$RUNS"; ok=1
+fail() { echo "validate.sh: FAIL -- $*"; ok=0; }
+manifest_val() { /usr/bin/grep -m1 "^$2=" "$1/run_manifest.txt" 2>/dev/null | cut -d= -f2- || true; }
+
+echo "validate.sh: [0] ELPA GPU kernel probe record [profile $PROFILE]"
+if [ -f "$INST/elpa/ELPA_GPU_PROBE.txt" ]; then
+    /usr/bin/grep -E '^(PASS|FAIL|MISSING|RESULT)' "$INST/elpa/ELPA_GPU_PROBE.txt" | sed 's/^/    /' | cut -c1-200
+    /usr/bin/grep -q '^RESULT PASS' "$INST/elpa/ELPA_GPU_PROBE.txt" || fail "ELPA GPU probe did not PASS"
+else
+    fail "ELPA GPU probe record missing -- run ./elpa_probe.sh first"
+fi
+
+echo "validate.sh: [1] DFT-FE $BACKEND al_md (Input_MD_0.prm: 32-atom Al BOMD, 4 steps) on $N GPU(s) vs upstream GPU reference"
+D="$RUNS/al_md.smoke.np$N"
+rc=0; timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" > "$RUNS/validate.al_md.np$N.stdout" 2>&1 || rc=$?
+/usr/bin/grep -aE '^# DFT-FE|hpcperf-launch: audit summary' "$RUNS/validate.al_md.np$N.stdout" || true
+if [ "$rc" -eq 124 ]; then fail "al_md timed out after ${TIMEOUT}s"; elif [ "$rc" -ne 0 ]; then fail "al_md run.sh exited $rc (see $RUNS/validate.al_md.np$N.stdout)"; fi
+audit="$(/usr/bin/grep -a 'audit summary' "$RUNS/validate.al_md.np$N.stdout" | tail -1)"
+echo "$audit" | /usr/bin/grep -qE "audit summary: $N verified, 0 mismatch, 0 unverified" || fail "al_md np$N: launcher GPU audit is not '$N verified, 0 mismatch, 0 unverified' (${audit:-no audit line})"
+if [ -f "$D/dftfe.out" ]; then
+    python3 "$HERE/dftfe_check.py" "$D/dftfe.out" "$REF" --check --label "np$N vs upstream" > "$D/check_vs_upstream.txt" || fail "al_md np$N: mismatch against upstream reference / incomplete / non-finite"
+    sed 's/^/    /' "$D/check_vs_upstream.txt"
+    if [ "$N" -gt 1 ]; then
+        R1="$RUNS/al_md.smoke.np1"
+        if [ ! -f "$R1/dftfe.out" ] || [ "$(manifest_val "$R1" binary_sha256)" != "$(manifest_val "$D" binary_sha256)" ] || [ "$(manifest_val "$R1" exit_code)" != 0 ]; then
+            echo "    (1-GPU reference missing/stale -- running al_md on 1 GPU now)"
+            rc=0; HPCPERF_GPUS=1 timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" > "$RUNS/validate.al_md.np1.stdout" 2>&1 || rc=$?
+            [ "$rc" -eq 0 ] || fail "al_md 1-GPU reference run exited $rc"
+        fi
+        python3 "$HERE/dftfe_check.py" "$D/dftfe.out" "$R1/dftfe.out" --check --label "np$N vs np1" > "$D/check_vs_np1.txt" || fail "al_md np$N: mismatch against the 1-GPU run"
+        sed 's/^/    /' "$D/check_vs_np1.txt"
+    fi
+fi
+
+if [ "$ok" -eq 1 ]; then echo "DFT-FE $BACKEND validation ($N GPU; ELPA GPU probe PASS, al_md complete/finite within pre-fixed tolerances of upstream's GPU reference$( [ "$N" -gt 1 ] && echo " and of the 1-GPU run" || true), every rank verified on its own GPU): PASS"; exit 0; fi
+echo "DFT-FE $BACKEND validation ($N GPU): FAIL"; exit 1
diff --git a/level3/geos/README.md b/level3/geos/README.md
new file mode 100644
index 0000000..b7094e8
--- /dev/null
+++ b/level3/geos/README.md
@@ -0,0 +1,168 @@
+# GEOS -- Level 3 second batch
+
+Multiphysics reservoir/geomechanics simulator (LLNL); here the solid-mechanics
+`beamBending` workflow (linear-elastic cantilever, analytic solution shipped
+upstream) with the hypre linear solver on the GPU (RAJA/CHAI/Umpire device
+kernels via LvArray).
+
+## Provenance / versions
+
+| Item | Value |
+|---|---|
+| GEOS | `develop` snapshot `b7a0f13305277c3d825ee93f34946ac4e8c94fee` (2026-09-04, fixed SHA; the last tag 1.2.0 is from 2024-10 and pins RAJA 2024.07/hypre 2.31 with CUDA <= 12.5 heritage -- see `APPLICATION_AUDIT.md` for the stable-vs-develop decision) with submodules (LvArray, BLT `9ff77344` = 0.6.2-era, PVTPackage, hdf5_interface, ...), LGPL-2.1 |
+| thirdPartyLibs | `9b55672f6f8a73d02fd632396eb0410e58c9b120` = the `GEOS_TPL_TAG 361-1070` that this GEOS commit requires (the repo has no git tags); RAJA/CHAI/Umpire/camp **2026.07.0**, hypre `f1374fb6` (`--with-cuda --enable-cusparse --enable-cusolver --enable-unified-memory --with-umpire --with-gpu-arch=100`), HDF5 1.12.1, conduit 0.9.5, silo 4.11.1, VTK 9.7.0, pugixml, fmt, SuiteSparse 5.10.1, ParMETIS/METIS, SuperLU_DIST `0f6efc3`, Scotch; Trilinos/PETSc/Caliper/MathPresso **off** |
+| Compilers | system GCC 14.2.1 (C/C++/Fortran), nvcc 13.2.78 with g++ 14.2.1 host (`-allow-unsupported-compiler` not needed), conda Open MPI 5.0.10 (`OMPI_CC/CXX/FC` -> system GCC) |
+| GPU target | `CMAKE_CUDA_ARCHITECTURES 100` / `CUDA_ARCH sm_100` in the host-config; hypre device backend (`ENABLE_HYPRE_DEVICE=CUDA`), `ENABLE_HYPRE_GPU_AWARE_MPI=OFF` (GEOS' default pinned-host-buffer path) |
+| BLAS/LAPACK | private OpenBLAS 0.3.30 (`TARGET=SAPPHIRERAPIDS`, single-threaded) |
+| Build strategy | NATIVE: upstream `thirdPartyLibs` superbuild (`scripts/config-build.py -hc <host-config>`) + GEOS with the same host-config; no Spack (`geosx` recipe is uberenv-only), no container |
+| Host-config | `host-configs/gmu-hopper-gcc14-ompi5010-cuda132-sm100.cmake` (paths injected as `-D HPCPERF_*` by build.sh) |
+| Profile | `cuda132-gcc142-ompi5010` |
+
+## Build-system patches to thirdPartyLibs (class B, `patches/`)
+
+- `0001-tpl-superlu_dist-url-hash-typo.patch`: upstream's `SUPERLU_URL_HASH` is a
+  65-character string (also on master, checked 2026-09-05), so the superbuild's
+  download step can never verify it; the tarball's real sha256 is that string
+  minus the trailing character. No code impact.
+- `0002-tpl-raja-vectorization-overridable.patch`: upstream forces
+  `RAJA_ENABLE_VECTORIZATION=ON` for every non-HIP build (OFF for ROCm because the
+  layer "does not compile with the ROCm toolchain"). With nvcc 13.2 + EL10's GCC
+  14 (default `-march=x86-64-v3` -> `__AVX2__` in every nvcc host pass) RAJA's
+  AVX2 tensor-register headers fail to compile in every CUDA translation unit.
+  The patch only makes the option overridable; build.sh passes
+  `-D RAJA_ENABLE_VECTORIZATION=OFF` (upstream's own ROCm choice). GEOS/LvArray do
+  not use the experimental `RAJA::expt` tensor layer.
+- `0003-tpl-hdf5-step-generator-and-build-command.patch`: the hdf5 step is the
+  only CMake-based TPL step without `CMAKE_GENERATOR ${TPL_GENERATOR}` and with
+  a hardcoded `make`; under a Ninja-generated superbuild (`config-build.py -n`)
+  its sub-build inherits Ninja and `make` finds no Makefile. Same generator and
+  build/install commands as every other step; no effect on the built HDF5.
+- Not built (disabled, no patch): **hypredrive** (optional hypre driver library;
+  GEOS' `ENABLE_HYPREDRV` defaults to OFF and the beam workflow does not use it).
+  Upstream's TPL step compiles it against the Umpire-enabled hypre without
+  Umpire's include path (`umpire/config.hpp: No such file`) -- disabled with
+  `-D ENABLE_HYPREDRV=OFF` rather than patched.
+- Node adaptations (no patch): `config-build.py` deletes an existing build tree
+  before configuring, so build.sh configures once and re-asserts cache values
+  (`RAJA_ENABLE_VECTORIZATION`, `ENABLE_HYPREDRV`, `NUM_PROC`) with plain `cmake`
+  on re-runs; stale ExternalProject configure stamps left by an interrupted run
+  are removed when the sub-build has no Makefile.
+
+## GEOS itself: one BLT back-port (class D) and one include-order adaptation (class C)
+
+- `patches/geos-blt-0001-cuda13-memoryClockRate.patch` (applied to the BLT
+  submodule `src/cmake/blt`, `9ff77344`): BLT's CUDA runtime smoke test
+  (`cmake/thirdparty/BLTCudaRuntimeSmokeTest.cpp` — the tiny program built with
+  `ENABLE_TESTS=ON`, upstream's default) reads `cudaDeviceProp::memoryClockRate`,
+  which CUDA 13 removed from the struct. The patch is the back-port of upstream
+  LLNL/blt commit `38b46203` ("use cudaDeviceGetAttribute for memoryClockRate on
+  CUDA >= 13"): `#if CUDART_VERSION >= 13000 cudaDeviceGetAttribute(&rate,
+  cudaDevAttrMemoryClockRate, i) #else prop.memoryClockRate #endif`. Only the
+  smoke-test program changes; no GEOS code, no kernel. The alternative --
+  switching the tests off -- cascaded through `ENABLE_BENCHMARKS`, the GEOS
+  `GEOS_ENABLE_TESTS`/LvArray `ENABLE_EXAMPLES` and HPCReact test switches and
+  was abandoned so that the tree stays at upstream's defaults (the 100+ GEOS/LvArray
+  unit-test executables are built but **not run** here -- see the status table).
+- The conda Open MPI include directory (`$CONDA_PREFIX/include`, added by GEOS as
+  `-isystem` from the MPI imported target) carries an unrelated 32-bit `metis.h`
+  (2013) that shadowed the TPL's 64-bit one and broke ParMETIS' `static_assert(
+  sizeof(idx_t) == 8)` in `VTKMeshGenerator`; the host-config puts the TPL
+  METIS/ParMETIS include dirs first with `-I` in `CMAKE_CXX_FLAGS`/`CMAKE_CUDA_FLAGS`
+  (`-I` is searched before every `-isystem`). No source change.
+- Build time (`BUILD_INFO.txt`): GEOS 1718 s at `-j32` (the heaviest translation
+  unit, `CompositionalMultiphaseBase.cpp`, takes ~26 min alone in cicc/ptxas);
+  `cuobjdump` on `geosx`, `libHYPRE.a`, `libRAJA.a`: sm_100 only.
+
+## Cases, launch, validation
+
+`run.sh` (see its header): `beam` = `inputFiles/solidMechanics/beamBending_*.xml`
+family; smoke `80x8x4` elements with the hypre AMG (GPU) or direct solver, strong
+`160x16x8` (upstream's `beamBending_benchmark.xml` size, fixed over rank counts),
+weak = mesh refined with the rank count (elements per GPU constant). Partitions
+through GEOS' `-x -y -z` (product = ranks; the launcher gives one GPU per rank;
+GEOS itself has no device-selection logic, so the per-rank `CUDA_VISIBLE_DEVICES`
+wrapper is what binds it), output prefix `-o`.
+
+`validate.sh` re-implements geos-ats's checks for this test (`beamBending.ats`:
+`CurveCheckParameters(tolerance=[0.0002], script beamBending_curve.py)` and
+`RestartcheckParameters(atol=1e-3, rtol=1e-7)`) in `geos_beam_check.py`, with the
+metrics of geosPythonPackages `curve_check.py` / `restart_check.py`, and was
+checked against upstream's published baseline **before** any GEOS run here:
+[1] completion (exit 0, `displacement_history.hdf5` with the 10 output times,
+finite, launcher audit N verified / 0 mismatch); [2] the geos-ats "script" curve
+check, `||u - u_analytic||_2 / N <= 0.0002` over the whole (10 x 81 x 3) trace vs
+`beamBending_curve.py` (Euler-Bernoulli, upstream's 1.043 factor) -- upstream's
+own 80x8x4 baseline scores 1.376e-4, i.e. this tolerance sits just above the
+mesh's discretisation error, whose per-time relative L-inf is 9.1e-4 (printed as
+information; my first reading of "2e-4 relative per time" would have failed
+upstream's own baseline and was replaced before the runs); [3] rank-count
+independence of the AMG runs: N-GPU vs 1-GPU history (trace nodes matched by
+reference position), relative L-inf <= 1e-4 (100x krylovTol; whether 1e-5 is met
+is printed -- upstream's direct-solver 1- and 8-rank baselines agree to 1.5e-12)
+and the geos-ats baseline metric <= 0.0002; [4] the shipped serial-direct-solver
+smoke deck on 1 rank vs the public baseline `beamBending_smoke_01`
+(`baseline_integratedTests-pr3994-17525-4ae3593.tar.gz`, the `.integrated_tests.yaml`
+baseline of this commit): restart file at cycle 10, every dataset and attribute
+with upstream's tolerances and default exclusions (`commandLine`, `schema`,
+`globalToLocalMap`, `timeHistoryOutput*/restart`), plus its history vs the
+baseline history (metric <= 0.0002, rel L-inf <= 1e-4).
+
+The first restart comparison read the LvArray `__values__` datasets raw and reported
+23 disagreements: a device build stores 2-D node/element fields (displacements,
+stresses, `nodeList`, ...) in a different memory permutation than the host build that
+produced the baseline (`__permutation__` differs for 12 of the 93 LvArrays), so the raw
+datasets differ while the logical arrays are equal. geos-ats handles this with
+`permute_array.py`; `geos_beam_check.py` now does the same (unit-checked against
+upstream's own permutation test cases) and the validation was re-run (run ids in the
+manifests; the first pass is kept in `validate.*.stdout` history only as the FAIL it
+was). Three `LinearSolverParameters` values differ by construction --
+`krylovMaxRestart` 100 vs 200, `amgCoarseningType` PMIS vs HMIS, `amgSmootherType`
+l1jacobi vs l1sgs are GEOS's compile-time defaults for `GEOS_USE_HYPRE_DEVICE` builds
+(`src/coreComponents/linearAlgebra/utilities/LinearSolverParameters.hpp`) -- they are
+reported by the checker and not gating (the direct solver of the compared deck does not
+use them; nothing else may differ).
+
+### Results (2026-09-06, profile `cuda132-gcc142-ompi5010`, one rank per GPU)
+
+| Run | GPUs | Mesh (C3D8) / partitions | Solver | geos-ats curve metric (tol 2e-4) | vs 1 GPU: rel L-inf / metric | GEOS run time | Audit |
+|---|---|---|---|---|---|---|---|
+| validate | 1 | 80x8x4 / 1x1x1 | GMRES + hypre AMG (device) | 1.3761e-4 | -- | 7.1 s | 1 verified |
+| validate | 2 | 80x8x4 / 2x1x1 | same | 1.3761e-4 | 6.2e-9 / 6.6e-10 | -- | 2 verified, 0 mismatch |
+| validate | 4 | 80x8x4 / 2x2x1 | same | 1.3761e-4 | 1.1e-8 / 8.4e-10 | -- | 4 verified, 0 mismatch |
+| validate, shipped smoke deck | 1 | 80x8x4 / 1x1x1 | serial direct | 1.3761e-4; restart vs upstream baseline: 1191 arrays/attributes, 0 disagree, worst rel 3.6e-12; history vs baseline rel 5.1e-12 | -- | 6.5 s | 1 verified |
+| strong (`beamBending_benchmark.xml` verbatim) | 1 / 2 / 4 | 160x16x8 / 1x1x1, 2x1x1, 2x2x1 | GMRES + AMG (device) | -- | -- | 6.7 / 10.9 / 13.3 s | all verified |
+| weak (80x8x4 per GPU) | 1 / 2 / 4 | 80x8x4, 160x8x4, 160x16x4 | GMRES + AMG (device) | -- | -- | 9.5 / 14.2 / 8.7 s | all verified |
+
+VALIDATED_PASS at 1/2/4 GPUs. Upstream's own baseline scores the same 1.3761e-4 on the
+curve metric (the mesh's discretisation error; the tolerance 2e-4 sits just above it),
+and upstream's 1- and 8-rank baselines agree to 1.5e-12 -- the same order as our
+direct-solver run vs their baseline. The scaling numbers are completeness records
+only: the official beam decks are far too small for a B200 (20 480 elements at most;
+hypre setup, silo/HDF5 output and MPI latency dominate, more ranks cost more). Larger
+meshes are one variable away (`HPCPERF_GEOS_WEAK_NX`, or an upstream
+`performanceBenchmarks` case) but were not run so that the workflow stays the
+official small one. Dry-runs 8/40/80 (strong + weak): planned as HYPOTHETICAL
+(partitions 2x2x2 / 5x4x2 / 5x4x4; weak meshes up to 400x32x16); 3 ranks are refused
+(80x8x4 cannot be partitioned). Raw material: `build/level3/geos/<profile>/run/`
+(`stdout.log`, `displacement_history.hdf5`, restart files, `analytic_check.txt`,
+`cross_rank_check.txt`, `restart_baseline_check.txt`, `history_baseline_check.txt`,
+`run_manifest.txt`), `.deps/level3/geos/<profile>/install/BUILD_INFO.txt`.
+
+### Unit tests (dependency probe, not a validation criterion of the beam workflow)
+
+`ctest -j4` in `build/level3/geos/<profile>` on GPU 0 (upstream default `ENABLE_TESTS=ON`,
+`PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe` for the tests' own `mpirun -np 1`
+launches): **254 / 261 passed** in 351 s, log
+`.deps/level3/geos/<profile>/logs/geos-ctest.log`. Failed: `testMath` (LvArray
+`TestComplexMath/8.asinh`, `float` on the device: CUDA's `asinhf(5)` differs from the host
+value by >= 1 float ulp, the test allows `epsilon`; CUDA documents up to 3 ulp), `testErrorHandling`
+(`testYamlFileAssertOutput` aborts on purpose; prterun turns the abort into a non-zero test
+exit), and five fluid-flow/well physics tests -- `testCompMultiphaseFlow`
+(phase-mobility derivative check, analytical vs finite-difference error norm ~1.0),
+`testCompMultiphaseFlowHybrid` (flux Jacobian, error norms 0.008-0.15),
+`testThermalEstimatorProdWell`, `testThermalEstimatorInjWell`,
+`testReservoirThermalSinglePhaseMSWells_RateInj` (`ExternalError` in the well solvers).
+The last five are gross, not precision-level, and were not investigated: **the
+compositional multiphase flow and well modules of this build are UNVERIFIED**; the
+validated solid-mechanics workflow is not affected (its restart state matches upstream's
+baseline to 3.6e-12).
diff --git a/level3/geos/build.sh b/level3/geos/build.sh
new file mode 100755
index 0000000..61c06c3
--- /dev/null
+++ b/level3/geos/build.sh
@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+# Build GEOS (develop snapshot b7a0f133) with upstream's thirdPartyLibs superbuild
+# (TPL 9b55672 = tag 361-1070) and a node host-config, in a private profile tree.
+#
+#   ./build.sh [tpl|geos|all]      (default all)   HPCPERF_BUILD_JOBS=N (default 32)
+#
+# Stages:
+#   [0] OpenBLAS 0.3.30 (single-threaded; BLAS/LAPACK for hypre/SuiteSparse/SuperLU_DIST)
+#   [1] thirdPartyLibs superbuild: RAJA/CHAI/Umpire/camp 2026.07.0 (CUDA sm_100), hypre f1374fb6
+#       (--with-cuda --enable-cusparse --enable-cusolver --enable-unified-memory --with-umpire,
+#       gpu-arch 100, mixedint), hypredrive, conduit 0.9.5, HDF5 1.12.1, silo 4.11.1, VTK,
+#       pugixml, ParMETIS/METIS, SuperLU_DIST, SuiteSparse 5.10.1, Scotch, fmt
+#       (Trilinos/PETSc/Caliper/MathPresso/Doxygen/Uncrustify OFF)      -> install/tpl
+#   [2] GEOS: scripts/config-build.py with the same host-config + GEOS_TPL_DIR -> install/geos
+# Toolchain: system GCC 14.2.1 (C/C++; CUDA host), conda Open MPI 5.0.10 (wrappers redirected
+# via OMPI_CC/OMPI_CXX), CUDA 13.2, sm_100. Profile cuda132-gcc142-ompi5010.
+# Modification class: A/B (host-config + build options; no source patched unless a
+# patches/ file exists and is listed below -- none at the time of writing).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+l3_isolate_build_env
+l3_clean_conda_build_env   # system GCC 14 build: drop conda's CFLAGS/LDFLAGS/AR/...
+export CC=/usr/bin/gcc CXX=/usr/bin/g++ FC=/usr/bin/gfortran OMPI_CC=/usr/bin/gcc OMPI_CXX=/usr/bin/g++ OMPI_FC=/usr/bin/gfortran CUDAHOSTCXX=/usr/bin/g++
+unset CMAKE_GENERATOR
+STAGE="${1:-all}"
+GEOS_SRC="$R/_upstream/level3/GEOS"; TPL_SRC="$R/_upstream/level3/thirdPartyLibs"
+[ -f "$GEOS_SRC/src/CMakeLists.txt" ] && [ -f "$TPL_SRC/CMakeLists.txt" ] || { echo "build.sh: run $HERE/fetch.sh first" >&2; exit 1; }
+GEOS_SHA="$(git -C "$GEOS_SRC" rev-parse HEAD)"; TPL_SHA="$(git -C "$TPL_SRC" rev-parse HEAD)"
+# TPL build-system patches (headers in patches/*.patch): 0001 fixes upstream's 65-character superlu_dist URL hash,
+# 0002 makes RAJA_ENABLE_VECTORIZATION overridable (nvcc 13.2 + GCC 14/x86-64-v3 cannot compile RAJA's AVX2 tensor layer),
+# 0003 gives the hdf5 step the superbuild's sub-project generator and build/install commands like every other step.
+# Applied to the TPL checkout idempotently; their hashes are recorded in the fingerprint.
+# GEOS-side patch: BLT submodule smoke test vs CUDA 13 (back-port of LLNL/blt 38b46203), same idempotent scheme
+BLT_SRC="$GEOS_SRC/src/cmake/blt"
+for BLT_PATCH in "$HERE"/patches/geos-blt-*.patch; do
+    [ -f "$BLT_PATCH" ] || continue
+    if git -C "$BLT_SRC" apply --check --reverse "$BLT_PATCH" >/dev/null 2>&1; then :
+    elif git -C "$BLT_SRC" apply --check "$BLT_PATCH" >/dev/null 2>&1; then git -C "$BLT_SRC" apply "$BLT_PATCH"; echo "# applied $(basename "$BLT_PATCH") to the GEOS BLT submodule"
+    else echo "build.sh: $BLT_PATCH is neither applied nor applicable to BLT $(git -C "$BLT_SRC" rev-parse --short HEAD)" >&2; exit 1; fi
+    GEOS_PATCH_SHA="${GEOS_PATCH_SHA:-}${GEOS_PATCH_SHA:+,}$(basename "$BLT_PATCH" .patch)=$(sha256sum "$BLT_PATCH" | cut -c1-12)"
+done
+TPL_PATCH_SHA=""
+for TPL_PATCH in "$HERE"/patches/000*.patch; do
+    if git -C "$TPL_SRC" apply --check --reverse "$TPL_PATCH" >/dev/null 2>&1; then :
+    elif git -C "$TPL_SRC" apply --check "$TPL_PATCH" >/dev/null 2>&1; then git -C "$TPL_SRC" apply "$TPL_PATCH"; echo "# applied $(basename "$TPL_PATCH") to the thirdPartyLibs checkout"
+    else echo "build.sh: $TPL_PATCH is neither applied nor applicable to thirdPartyLibs $TPL_SHA" >&2; exit 1; fi
+    TPL_PATCH_SHA="$TPL_PATCH_SHA${TPL_PATCH_SHA:+,}$(basename "$TPL_PATCH" | cut -d- -f1)=$(sha256sum "$TPL_PATCH" | cut -c1-12)"
+done
+ARCH="${HPCPERF_CUDA_ARCH:-$(l3_gpu_arch)}"; [ "$ARCH" = 100 ] || { echo "build.sh: host-config is written for sm_100 (got sm_$ARCH)" >&2; exit 1; }
+GCC_MM="$(l3_version_mm "$(/usr/bin/gcc -dumpfullversion)")"; OMPI_V="$(mpirun --version | head -1 | /usr/bin/grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)"
+PROFILE="${HPCPERF_GEOS_PROFILE:-cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-ompi$(echo "$OMPI_V" | tr -d .)}"
+l3_paths_profile geos "$PROFILE"
+JOBS="${HPCPERF_BUILD_JOBS:-32}"
+HC="$HERE/host-configs/gmu-hopper-gcc14-ompi5010-cuda132-sm100.cmake"
+MPI_BIN="$(dirname "$(command -v mpicc)")"; CUDA_ROOT="${CUDA_HOME:-/usr/local/cuda}"
+OPENBLAS="$L3_INSTALL/openblas/lib/libopenblas.so"
+HCDEFS=(-D "HPCPERF_MPI_BIN=$MPI_BIN" -D "HPCPERF_OPENBLAS_LIB=$OPENBLAS" -D "HPCPERF_CUDA_ROOT=$CUDA_ROOT")
+DEPS="geos_patches[${GEOS_PATCH_SHA:-none}] tpl=361-1070($TPL_SHA)+patches[$TPL_PATCH_SHA] raja/chai/umpire=2026.07.0 hypre=f1374fb6(cuda,unified-memory,umpire,gpu-arch=$ARCH) openblas=0.3.30 gcc=$(/usr/bin/gcc -dumpfullversion) openmpi=$OMPI_V profile=$PROFILE"
+OPTS="host-config=$(basename "$HC") RAJA_ENABLE_VECTORIZATION=OFF ENABLE_HYPREDRV=OFF(host-config+TPL) tpl_metis_include_first ENABLE_TESTS=ON(upstream default) ENABLE_CUDA=ON CUDA_ARCH=sm_$ARCH GEOS_LA_INTERFACE=Hypre ENABLE_HYPRE_DEVICE=CUDA ENABLE_TRILINOS=OFF ENABLE_PETSC=OFF ENABLE_OPENMP=OFF ENABLE_CALIPER=OFF ENABLE_MATHPRESSO=OFF ENABLE_VTK=ON ENABLE_SUPERLU_DIST=ON ENABLE_SUITESPARSE=ON ENABLE_SCOTCH=ON CMAKE_BUILD_TYPE=Release"
+FP="$(l3_fingerprint_text geos "$GEOS_SHA" cuda "$DEPS" "$OPTS" "ENABLE_HYPRE_GPU_AWARE_MPI=OFF (GEOS pinned host buffers)")"
+l3_fingerprint_check "$L3_INSTALL" "$FP" || exit 1
+echo "# GEOS profile=$PROFILE: GEOS $GEOS_SHA, TPL $TPL_SHA, gcc $(/usr/bin/gcc -dumpfullversion), $(mpirun --version | head -1), CUDA $(l3_cuda_version) sm_$ARCH, -j$JOBS"
+echo "# resources: TPL superbuild ~2-3 h (VTK, hypre-CUDA, RAJA suite; downloads ~500 MB), GEOS ~1.5-2.5 h; ~25 GB under $L3_DEPS"
+T0=$(date +%s)
+run() { local log=$1; shift; "$@" > "$L3_LOGS/$log" 2>&1 || { tail -50 "$L3_LOGS/$log"; echo "build.sh: FAILED: $* (log $L3_LOGS/$log)" >&2; exit 1; }; }
+
+# [0] OpenBLAS
+if [ ! -f "$L3_INSTALL/openblas/.hpcperf-stage-done" ]; then
+    TB="$R/.deps/level3/dftfe/downloads/OpenBLAS-0.3.30.tar.gz"
+    [ -f "$TB" ] || { mkdir -p "$L3_DEPS/downloads"; TB="$L3_DEPS/downloads/OpenBLAS-0.3.30.tar.gz"; curl -sSL -o "$TB" https://github.com/OpenMathLib/OpenBLAS/releases/download/v0.3.30/OpenBLAS-0.3.30.tar.gz; }
+    echo "$(sha256sum "$TB" | cut -d' ' -f1) OpenBLAS-0.3.30.tar.gz" > "$L3_LOGS/openblas.sha256"
+    d="$L3_SRC/OpenBLAS-0.3.30"; [ -d "$d" ] || tar -xzf "$TB" -C "$L3_SRC"
+    run openblas-make.log make -C "$d" -j "$JOBS" USE_THREAD=0 USE_OPENMP=0 DYNAMIC_ARCH=0 NO_AFFINITY=1 TARGET="${HPCPERF_OPENBLAS_TARGET:-SAPPHIRERAPIDS}" CC=/usr/bin/gcc FC=/usr/bin/gfortran
+    run openblas-install.log make -C "$d" PREFIX="$L3_INSTALL/openblas" install
+    touch "$L3_INSTALL/openblas/.hpcperf-stage-done"
+fi
+T1=$(date +%s)
+# [1] thirdPartyLibs superbuild
+if [ "$STAGE" != geos ] && [ ! -f "$L3_INSTALL/tpl/.hpcperf-stage-done" ]; then
+    # config-build.py DELETES an existing build directory before configuring; configure only once and let
+    # a re-run continue the ninja build (CMakeLists changes -- e.g. the TPL patch -- trigger cmake's own re-run)
+    # vectorization: see patches/0002 (upstream's own choice for ROCm); NUM_PROC = make -j inside the TPL steps;
+    # hypredrive (optional hypre driver library, GEOS' ENABLE_HYPREDRV defaults to OFF and is not used here) is left out:
+    # upstream's TPL step compiles it against a hypre built with Umpire but without Umpire's include path (fails on
+    # `umpire/config.hpp: No such file`) -- not needed for the beam workflow, so disabled rather than patched.
+    TPLDEFS=(-D RAJA_ENABLE_VECTORIZATION:BOOL=OFF -D ENABLE_HYPREDRV:BOOL=OFF -D "NUM_PROC=$JOBS")
+    if [ ! -f "$L3_BUILD_DEPS/tpl/build.ninja" ]; then
+        ( cd "$TPL_SRC" && python3 scripts/config-build.py -hc "$HC" -bt Release -bp "$L3_BUILD_DEPS/tpl" -ip "$L3_INSTALL/tpl" -n "${HCDEFS[@]}" "${TPLDEFS[@]}" ) > "$L3_LOGS/tpl-configure.log" 2>&1 \
+            || { tail -40 "$L3_LOGS/tpl-configure.log"; echo "build.sh: TPL configure failed" >&2; exit 1; }
+    else  # existing tree: re-assert the cache values without config-build.py (which would delete the tree)
+        run tpl-reconfigure.log cmake "${TPLDEFS[@]}" "$L3_BUILD_DEPS/tpl"
+    fi
+    run tpl-build.log ninja -C "$L3_BUILD_DEPS/tpl" -j "$JOBS"
+    for d in raja chai hypre hdf5 conduit vtk pugixml suitesparse superlu_dist parmetis scotch; do [ -d "$L3_INSTALL/tpl/$d" ] || echo "build.sh: WARNING TPL install lacks $d" >&2; done
+    [ -d "$L3_INSTALL/tpl/hypre" ] && [ -d "$L3_INSTALL/tpl/raja" ] || { echo "build.sh: TPL install incomplete (hypre/raja missing)" >&2; exit 1; }
+    for l in "$L3_INSTALL"/tpl/hypre/lib/libHYPRE*.a "$L3_INSTALL"/tpl/raja/lib/libRAJA*.a; do [ -f "$l" ] && echo "$(basename "$l"): $(cuobjdump --list-elf "$l" 2>/dev/null | /usr/bin/grep -o 'sm_[0-9]*' | sort -u | paste -sd,)"; done > "$L3_LOGS/tpl-device-archs.txt" || true
+    touch "$L3_INSTALL/tpl/.hpcperf-stage-done"
+fi
+T2=$(date +%s)
+[ "$STAGE" = tpl ] && { echo "# TPL stage done ($((T2 - T0)) s)"; exit 0; }
+# [2] GEOS
+mkdir -p "$L3_BUILD"
+if [ ! -f "$L3_BUILD/build.ninja" ]; then   # same reason as above: config-build.py deletes an existing build tree
+    ( cd "$GEOS_SRC" && python3 scripts/config-build.py -hc "$HC" -bt Release -bp "$L3_BUILD" -ip "$L3_INSTALL/geos" -n "${HCDEFS[@]}" -D "HPCPERF_GEOS_TPL_DIR=$L3_INSTALL/tpl" -D "GEOS_TPL_DIR=$L3_INSTALL/tpl" ) > "$L3_LOGS/geos-configure.log" 2>&1 \
+        || { tail -60 "$L3_LOGS/geos-configure.log"; echo "build.sh: GEOS configure failed" >&2; exit 1; }
+else  # existing tree: re-read the host-config (cache FORCE values) without config-build.py
+    run geos-reconfigure.log cmake "${HCDEFS[@]}" -D "HPCPERF_GEOS_TPL_DIR=$L3_INSTALL/tpl" -D "GEOS_TPL_DIR=$L3_INSTALL/tpl" -C "$HC" "$L3_BUILD"   # -D before -C: the host-config reads HPCPERF_*
+fi
+run geos-build.log ninja -C "$L3_BUILD" -j "$JOBS"
+run geos-install.log ninja -C "$L3_BUILD" install
+T3=$(date +%s)
+EXE="$L3_INSTALL/geos/bin/geosx"; [ -x "$EXE" ] || EXE="$(find "$L3_INSTALL/geos/bin" -maxdepth 1 -type f -name 'geos*' | head -1)"
+[ -n "$EXE" ] && [ -x "$EXE" ] || { echo "build.sh: GEOS executable not found under $L3_INSTALL/geos/bin" >&2; exit 1; }
+archs="$(cuobjdump --list-elf "$EXE" 2>/dev/null | /usr/bin/grep -o 'sm_[0-9]*' | sort -u | paste -sd,)"
+[ -n "$archs" ] || archs="$(for l in "$L3_INSTALL"/geos/lib/*.so "$L3_INSTALL"/geos/lib64/*.so; do [ -f "$l" ] && cuobjdump --list-elf "$l" 2>/dev/null | /usr/bin/grep -o 'sm_[0-9]*'; done | sort -u | paste -sd,)"
+[ "$archs" = "sm_$ARCH" ] || { echo "build.sh: GEOS device code '$archs' != sm_$ARCH" >&2; exit 1; }
+l3_fingerprint_write "$L3_INSTALL" "$FP"
+{
+    echo "profile=$PROFILE geos=$GEOS_SHA tpl=$TPL_SHA utc=$(date -u +%FT%TZ) jobs=$JOBS"
+    echo "seconds: openblas=$((T1 - T0)) tpl=$((T2 - T1)) geos=$((T3 - T2)) total=$((T3 - T0))"
+    echo "geos exe=$EXE sha256=$(l3_sha_file "$EXE") device_archs=$archs"
+    [ -f "$L3_LOGS/tpl-device-archs.txt" ] && sed 's/^/tpl: /' "$L3_LOGS/tpl-device-archs.txt"
+    "$EXE" --help 2>&1 | head -3 | sed 's/^/help: /'
+} > "$L3_INSTALL/BUILD_INFO.txt"; cat "$L3_INSTALL/BUILD_INFO.txt"
+echo "# GEOS built: openblas $((T1 - T0)) s, TPL $((T2 - T1)) s, GEOS $((T3 - T2)) s"
diff --git a/level3/geos/fetch.sh b/level3/geos/fetch.sh
new file mode 100755
index 0000000..208d384
--- /dev/null
+++ b/level3/geos/fetch.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Fetch GEOS and its thirdPartyLibs superbuild at fixed commits (read-only checkouts).
+#
+#   GEOS develop     b7a0f13305277c3d825ee93f34946ac4e8c94fee (2026-09-04, "fix: SinglePhaseWell Thermal (#4126)")
+#                    submodules: BLT, LvArray, HPCReact, hdf5_interface (as pinned by that commit)
+#   thirdPartyLibs   9b55672f6f8a73d02fd632396eb0410e58c9b120 (2026-08-26, "Update TPLs (#361)" = the
+#                    GEOS_TPL_TAG 361-1070 GEOS develop's .devcontainer/CI images are built from)
+#
+# Why a fixed develop snapshot and not release 1.2.0 (2024-10): 1.2.0 pins TPL tag 284-535
+# (RAJA/CHAI/Umpire 2024.07, hypre 2.31, CUDA <= 12.5 heritage) and its CI never went
+# beyond CUDA 12.5; the 2026 TPL set (RAJA/CHAI/Umpire 2026.07.0 -- the versions Level 2
+# already builds with CUDA 13.2 on this node -- hypre f1374fb6, VTK 9.7) is the one with
+# a plausible CUDA 13.2 / sm_100 path. The snapshot is pinned by SHA; nothing floats.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+UP="$R/_upstream/level3"
+GEOS_SHA=b7a0f13305277c3d825ee93f34946ac4e8c94fee
+TPL_SHA=9b55672f6f8a73d02fd632396eb0410e58c9b120
+mkdir -p "$UP"
+if [ ! -d "$UP/GEOS/.git" ]; then
+    git clone -q --branch develop --depth 1 https://github.com/GEOS-DEV/GEOS.git "$UP/GEOS"
+    git -C "$UP/GEOS" fetch -q --depth 1 origin "$GEOS_SHA"; git -C "$UP/GEOS" checkout -q "$GEOS_SHA"
+    git -C "$UP/GEOS" submodule update -q --init --depth 1
+fi
+g="$(git -C "$UP/GEOS" rev-parse HEAD)"; [ "$g" = "$GEOS_SHA" ] || { echo "fetch.sh: GEOS at $g, expected $GEOS_SHA" >&2; exit 1; }
+if [ ! -d "$UP/thirdPartyLibs/.git" ]; then
+    git clone -q https://github.com/GEOS-DEV/thirdPartyLibs.git "$UP/thirdPartyLibs"
+    git -C "$UP/thirdPartyLibs" checkout -q "$TPL_SHA"; git -C "$UP/thirdPartyLibs" submodule update -q --init --depth 1
+fi
+t="$(git -C "$UP/thirdPartyLibs" rev-parse HEAD)"; [ "$t" = "$TPL_SHA" ] || { echo "fetch.sh: thirdPartyLibs at $t, expected $TPL_SHA" >&2; exit 1; }
+echo "# GEOS $g (develop snapshot); thirdPartyLibs $t (TPL tag 361-1070)"
+git -C "$UP/GEOS" submodule status | sed 's/^/#   GEOS submodule /'
+git -C "$UP/thirdPartyLibs" submodule status | sed 's/^/#   TPL submodule /'
diff --git a/level3/geos/geos_beam_check.py b/level3/geos/geos_beam_check.py
new file mode 100755
index 0000000..ba9752a
--- /dev/null
+++ b/level3/geos/geos_beam_check.py
@@ -0,0 +1,238 @@
+#!/usr/bin/env python3
+"""GEOS beam-bending checks on the TimeHistory output (displacement_history.hdf5).
+
+  geos_beam_check.py analytic <run_dir> --tol T --curve beamBending_curve.py
+        geos-ats "script" curve check: the displacement trace vs upstream's analytic reference
+        (inputFiles/solidMechanics/beamBending_curve.py, imported and called exactly as geos-ats does);
+        gate = geos-ats check_diff() metric ||u - u_script||_2 / N <= T over the whole (ntime, nnodes, 3)
+        array (T = 0.0002 in beamBending.ats). The per-output-time relative L-inf of uy is printed as
+        information (it is the mesh's discretisation error, ~9e-4 for 80x8x4 in upstream's own baseline).
+  geos_beam_check.py compare <ref_dir> <run_dir> [--tol REL [--info-tol REL2]] [--l2n-tol T]
+        two histories (trace nodes matched by reference position when partitions order them
+        differently): --tol gates on the relative L-inf max|u_ref-u|/max|u_ref| (cross-rank-count
+        consistency), --l2n-tol gates on the geos-ats "baseline" curve metric ||u - u_ref||_2 / N.
+  geos_beam_check.py restart <run_dir> <baseline_dir> --rtol R --atol A
+        GEOS restart HDF5 files vs upstream's integrated-test baseline (same rank count), geos-ats
+        restart_check.py rules (see RESTART_EXCLUDE / restart()).
+
+Every value is required finite. Exit 0 on agreement, 1 on disagreement, 2 on structural problems
+(missing file/dataset, NaN, shape). Run with the profile's venv python (h5py) -- see validate.sh.
+"""
+import argparse, glob, importlib.util, math, os, sys
+import numpy as np
+import h5py
+
+
+def die(msg, code=2):
+    print(f"  VALIDATION ERROR: {msg}"); sys.exit(code)
+
+
+def load_history(run_dir):
+    f = os.path.join(run_dir, "displacement_history.hdf5")
+    if not os.path.isfile(f): die(f"{f} missing (TimeHistory output not produced)")
+    with h5py.File(f, "r") as h:
+        keys = list(h.keys())
+        need = ["totalDisplacement trace", "totalDisplacement Time", "totalDisplacement ReferencePosition trace"]
+        for k in need:
+            if k not in h: die(f"dataset {k!r} missing in {f} (have {keys})")
+        d = {k: np.array(h[k]) for k in need}
+    for k, v in d.items():
+        if not np.isfinite(v).all(): die(f"non-finite values in {k}")
+    return d
+
+
+def geos_ats_diff(target, baseline):
+    """geos-ats curve_check.py check_diff(): ||t-b||_2 / N over the whole (ntime, nnodes, 3) array [m]."""
+    dx = target - baseline
+    return math.sqrt(float(np.sum(dx * dx))) / dx.size
+
+
+def analytic(run_dir, tol, curve_py):
+    d = load_history(run_dir)
+    spec = importlib.util.spec_from_file_location("beam_curve", curve_py)
+    mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)
+    ref = mod.curve(**d)                          # shape (ntime, nnodes, 3), y-component filled, x/z zero
+    disp = d["totalDisplacement trace"]
+    if ref.shape != disp.shape: die(f"reference shape {ref.shape} != data shape {disp.shape}")
+    t = np.squeeze(d["totalDisplacement Time"][:, 0])
+    worst = 0.0
+    for i, tb in enumerate(t):
+        num = np.abs(disp[i, :, 1] - ref[i, :, 1]).max(); den = np.abs(ref[i, :, 1]).max()
+        rel = num / den if den > 0 else num
+        worst = max(worst, rel)
+        print(f"    t={tb:5.1f}: max|uy_geos - uy_analytic| = {num:.4e}, max|uy_analytic| = {den:.4e}, rel = {rel:.3e}")
+    diff = geos_ats_diff(disp, ref)
+    print(f"    (info) per-time relative L-inf of uy vs the Euler-Bernoulli curve: worst {worst:.3e} over {len(t)} output times (discretisation error of the mesh)")
+    print(f"    analytic beam check (geos-ats curve_check metric ||u - u_script||_2 / N over {disp.size} values): {diff:.4e} (tol {tol:.0e}) {'ok' if diff <= tol else 'BAD'}")
+    return 0 if diff <= tol else 1
+
+
+def aligned_traces(ref_dir, run_dir):
+    a, b = load_history(ref_dir), load_history(run_dir)
+    ua, ub = a["totalDisplacement trace"], b["totalDisplacement trace"]
+    if ua.shape != ub.shape: die(f"totalDisplacement trace: shape {ua.shape} vs {ub.shape}")
+    ta, tb = np.squeeze(a["totalDisplacement Time"][:, 0]), np.squeeze(b["totalDisplacement Time"][:, 0])
+    if not np.allclose(ta, tb, rtol=0, atol=1e-12): die(f"output times differ: {ta} vs {tb}")
+    pa = a["totalDisplacement ReferencePosition trace"][0]; pb = b["totalDisplacement ReferencePosition trace"][0]
+    if pa.shape != pb.shape: die(f"trace node count differs: {pa.shape} vs {pb.shape}")
+    if not np.array_equal(pa, pb):
+        # node ordering of the trace set differs across partitions: match nodes by reference position
+        ia = np.lexsort(pa.T[::-1]); ib = np.lexsort(pb.T[::-1])
+        if not np.allclose(pa[ia], pb[ib], rtol=0, atol=1e-12): die("trace node positions differ between the runs")
+        ua, ub = ua[:, ia, :], ub[:, ib, :]
+        print(f"    (trace nodes re-ordered by reference position: {len(ia)} nodes)")
+    return ua, ub
+
+
+def compare(ref_dir, run_dir, tol, info_tol, l2n_tol):
+    ua, ub = aligned_traces(ref_dir, run_dir)
+    num = np.abs(ua - ub).max(); den = np.abs(ua).max(); rel = num / den if den > 0 else num
+    diff = geos_ats_diff(ub, ua)
+    ok = True
+    if tol is not None:
+        ok &= rel <= tol
+        print(f"    cross-run displacement history: max|u_ref - u| = {num:.4e}, max|u_ref| = {den:.4e}, rel L-inf = {rel:.3e} (tol {tol:.0e}) {'ok' if rel <= tol else 'BAD'}"
+              + (f"; {'also within' if rel <= info_tol else 'not within'} {info_tol:.0e} (info)" if info_tol is not None else ""))
+    if l2n_tol is not None:
+        ok &= diff <= l2n_tol
+        print(f"    geos-ats baseline curve metric ||u - u_ref||_2 / N = {diff:.4e} (tol {l2n_tol:.0e}) {'ok' if diff <= l2n_tol else 'BAD'}")
+    elif tol is not None:
+        print(f"    (info) geos-ats curve metric ||u - u_ref||_2 / N = {diff:.4e}")
+    return 0 if ok else 1
+
+
+# Same rules as geos-ats restart_check.py (geosPythonPackages, geos-ats/src/geos/ats/helpers/
+# restart_check.py + permute_array.py): default exclusions EXCLUDE_DEFAULT; groups walked
+# recursively with their attributes; an LvArray group (__dimensions__/__permutation__/__values__)
+# is compared as the logical array -- values reshaped through the stored permutation on each side
+# (a device build stores 2-D node/element fields in a different memory layout than a host build,
+# so the raw __values__ and __permutation__ datasets differ while the arrays are equal); float
+# entries agree iff |x-b| <= atol OR |x-b| <= rtol*|b| (the "(1+max|b|)" scaling is commented out
+# upstream), integer/boolean and string data must be identical; a child present on one side only
+# is an error.
+import re
+RESTART_EXCLUDE = [re.compile(p) for p in (".*/commandLine", ".*/schema$", ".*/globalToLocalMap", ".*/timeHistoryOutput.*/restart")]
+# Compile-time defaults of GEOS's LinearSolverParameters that differ between device and host builds
+# (src/coreComponents/linearAlgebra/utilities/LinearSolverParameters.hpp under GEOS_USE_HYPRE_DEVICE:
+# krylov.maxRestart 100 vs 200, amg.coarseningType PMIS vs HMIS, amg.smootherType l1jacobi vs l1sgs).
+# The public baseline was produced by a host build, so a GPU build legitimately stores other values
+# for these three; they are reported but not gating (the direct-solver deck compared here does not
+# use them, and every other dataset must still agree).
+DEVICE_DEFAULTS = re.compile(r".*/LinearSolverParameters/(krylovMaxRestart|amgCoarseningType|amgSmootherType)(/__values__)?$")
+LVARRAY_KEYS = ("__dimensions__", "__permutation__", "__values__")
+INT_KINDS = set("?bBiumMV"); FLOAT_KINDS = set("fc"); STR_KINDS = set("SaU")
+
+
+def permute_array(data, shape, perm):
+    """geos-ats permute_array.permuteArray: stored (memory-order) values -> logical array of `shape`."""
+    shape, perm = np.asarray(shape), np.asarray(perm)
+    if shape.ndim != 1 or perm.ndim != 1 or shape.size != perm.size: return None, "shape/permutation are not 1-D of equal length"
+    if np.prod(shape) != data.size: return None, f"shape {shape} (size {np.prod(shape)}) vs stored size {data.size}"
+    if np.any(np.sort(perm) != np.arange(shape.size)): return None, f"invalid permutation {perm}"
+    data = data.reshape(shape[perm])
+    rev = np.empty_like(perm); rev[perm] = np.arange(perm.size)
+    data = np.transpose(data, rev)
+    if tuple(data.shape) != tuple(shape): return None, "reshape failed"
+    return data, None
+
+
+def restart(run_dir, baseline_dir, rtol, atol):
+    def find_root(d):
+        roots = sorted(glob.glob(os.path.join(d, "**", "*restart_*.root"), recursive=True))
+        if not roots: die(f"no restart .root file under {d}")
+        return roots[-1]
+    ra, rb = find_root(run_dir), find_root(baseline_dir)
+    print(f"    restart files: {os.path.basename(ra)} vs baseline {os.path.basename(rb)}")
+    da, db = ra[:-5], rb[:-5]          # <name>.root -> <name>/ directory with rank_*.hdf5
+    fa, fb = sorted(glob.glob(os.path.join(da, "rank_*.hdf5"))), sorted(glob.glob(os.path.join(db, "rank_*.hdf5")))
+    if len(fa) != len(fb) or not fa: die(f"rank file count differs or empty: {len(fa)} vs {len(fb)} (baseline is for a fixed rank count)")
+    stats = {"ncmp": 0, "nfail": 0, "nexcl": 0, "nlv": 0, "nperm": 0, "ndev": 0, "worst": 0.0, "worst_name": ""}
+
+    def fail(msg): stats["nfail"] += 1; print(f"    {msg}")
+
+    def cmp(path, A, B):
+        A, B = np.asarray(A), np.asarray(B)
+        if A.shape == (): A = A.reshape(1)
+        if B.shape == (): B = B.reshape(1)
+        stats["ncmp"] += 1
+        ka, kb = A.dtype.kind, B.dtype.kind
+        if A.size == 0 and B.size == 0: return
+        if A.shape != B.shape: fail(f"{path}: shape {A.shape} vs {B.shape}"); return
+        exact = (ka in INT_KINDS and kb in INT_KINDS) or (ka in STR_KINDS and kb in STR_KINDS) or ka == "O" or kb == "O"
+        if exact:
+            if not np.array_equal(A, B):
+                what = f"{path}: exact ({A.dtype}) content differs" + (f": {A.flat[0]!r} (run) vs {B.flat[0]!r} (baseline)" if A.size == 1 else "")
+                if DEVICE_DEFAULTS.match(path): stats["ndev"] += 1; print(f"    [device-build default, not gating] {what}")
+                else: fail(what)
+            return
+        if ka in FLOAT_KINDS | INT_KINDS and kb in FLOAT_KINDS | INT_KINDS:
+            A, B = A.astype(float), B.astype(float)
+            if not (np.isfinite(A).all() and np.isfinite(B).all()): fail(f"{path}: non-finite values"); return
+            diff = np.abs(A - B); bad = (diff > atol) & (diff > rtol * np.abs(B))
+            if bad.any(): fail(f"{path}: {bad.sum()} of {bad.size} values outside atol {atol:g} / rtol {rtol:g} (max diff {diff.max():.3e})")
+            if np.abs(B).max() > atol:       # relative figure only meaningful for non-vanishing fields
+                m = diff.max() / np.abs(B).max()
+                if m > stats["worst"]: stats["worst"], stats["worst_name"] = m, path
+            return
+        fail(f"{path}: unrecognised type combination {A.dtype} / {B.dtype}")
+
+    def cmp_attrs(path, oa, ob):
+        for k in sorted(set(oa.attrs) | set(ob.attrs)):
+            if k not in oa.attrs or k not in ob.attrs: fail(f"{path}.attrs[{k}]: present on one side only"); continue
+            cmp(f"{path}.attrs[{k}]", oa.attrs[k], ob.attrs[k])
+
+    def walk(ga, gb):
+        path = ga.name
+        cmp_attrs(path, ga, gb)
+        children = set(ga.keys()) | set(gb.keys())
+        if all(k in ga and k in gb for k in LVARRAY_KEYS):
+            for k in LVARRAY_KEYS: children.discard(k)
+            stats["nlv"] += 1
+            dims_a, dims_b = np.asarray(ga["__dimensions__"][()]), np.asarray(gb["__dimensions__"][()])
+            perm_a, perm_b = np.asarray(ga["__permutation__"][()]), np.asarray(gb["__permutation__"][()])
+            if dims_a.shape != dims_b.shape or np.any(dims_a != dims_b): fail(f"{path}: LvArray dimensions differ: {dims_a} vs {dims_b}")
+            else:
+                if not np.array_equal(perm_a, perm_b): stats["nperm"] += 1
+                va, ea = permute_array(np.asarray(ga["__values__"][()]), dims_a, perm_a)
+                vb, eb = permute_array(np.asarray(gb["__values__"][()]), dims_b, perm_b)
+                if va is None: fail(f"{path}: cannot permute the run's LvArray: {ea}")
+                elif vb is None: fail(f"{path}: cannot permute the baseline's LvArray: {eb}")
+                else: cmp(path, va, vb)
+        for name in sorted(children):
+            p = path.rstrip("/") + "/" + name
+            if any(r.match(p) for r in RESTART_EXCLUDE): stats["nexcl"] += 1; continue
+            if name not in gb: fail(f"{p}: in the run but not in the baseline"); continue
+            if name not in ga: fail(f"{p}: in the baseline but not in the run"); continue
+            a, b = ga[name], gb[name]
+            if isinstance(a, h5py.Group) != isinstance(b, h5py.Group): fail(f"{p}: group on one side, dataset on the other"); continue
+            if isinstance(a, h5py.Group): walk(a, b)
+            else: cmp(p, a[()], b[()]); cmp_attrs(p, a, b)
+
+    for x, y in zip(fa, fb):
+        with h5py.File(x, "r") as ha, h5py.File(y, "r") as hb:
+            walk(ha, hb)
+    print(f"    restart vs baseline: {stats['ncmp']} arrays/datasets/attributes compared over {len(fa)} rank file(s) "
+          f"({stats['nlv']} LvArrays, {stats['nperm']} of them stored with a different memory permutation; {stats['nexcl']} children excluded by the "
+          f"geos-ats default patterns; {stats['ndev']} device-build solver defaults reported, not gating), {stats['nfail']} disagree; "
+          f"worst relative diff {stats['worst']:.3e} ({stats['worst_name']})")
+    return 0 if stats["nfail"] == 0 else 1
+
+
+def main():
+    ap = argparse.ArgumentParser(); sub = ap.add_subparsers(dest="cmd", required=True)
+    p = sub.add_parser("analytic"); p.add_argument("run_dir"); p.add_argument("--tol", type=float, required=True); p.add_argument("--curve", required=True)
+    p = sub.add_parser("compare"); p.add_argument("ref_dir"); p.add_argument("run_dir")
+    p.add_argument("--tol", type=float, help="gate: relative L-inf of the displacement history vs ref")
+    p.add_argument("--info-tol", type=float, help="also report whether this tighter relative L-inf is met (not gating)")
+    p.add_argument("--l2n-tol", type=float, help="gate: geos-ats curve metric ||u - u_ref||_2 / N (baseline curve check)")
+    p = sub.add_parser("restart"); p.add_argument("run_dir"); p.add_argument("baseline_dir"); p.add_argument("--rtol", type=float, required=True); p.add_argument("--atol", type=float, required=True)
+    a = ap.parse_args()
+    if a.cmd == "analytic": return analytic(a.run_dir, a.tol, a.curve)
+    if a.cmd == "compare":
+        if a.tol is None and a.l2n_tol is None: die("compare: give --tol and/or --l2n-tol")
+        return compare(a.ref_dir, a.run_dir, a.tol, a.info_tol, a.l2n_tol)
+    return restart(a.run_dir, a.baseline_dir, a.rtol, a.atol)
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/level3/geos/host-configs/gmu-hopper-gcc14-ompi5010-cuda132-sm100.cmake b/level3/geos/host-configs/gmu-hopper-gcc14-ompi5010-cuda132-sm100.cmake
new file mode 100644
index 0000000..bfe14e1
--- /dev/null
+++ b/level3/geos/host-configs/gmu-hopper-gcc14-ompi5010-cuda132-sm100.cmake
@@ -0,0 +1,80 @@
+# GEOS / thirdPartyLibs host-config for GMU Hopper dgx003 (4x B200, RHEL 10):
+# system GCC 14.2.1, conda Open MPI 5.0.10 (wrappers redirected to that GCC via
+# OMPI_CC/OMPI_CXX in build.sh), CUDA 13.2 sm_100, hypre on the device, no Trilinos,
+# no PETSc, no OpenMP (as upstream's CUDA CI rows: ENABLE_HYPRE_DEVICE=CUDA,
+# ENABLE_TRILINOS=OFF), no Caliper/MathPresso/Doxygen (not needed by the cases run).
+# Modelled on thirdPartyLibs docker/Stanford/sherlock-gcc10-cuda12-sm70.cmake and
+# GEOS host-configs/environment.cmake (which hard-codes CUDA_ARCH sm_86 for CI).
+# Paths under HPCPERF_L3_* are injected by build.sh through -D on the command line.
+set(CONFIG_NAME "gmu-hopper-gcc14-ompi5010-cuda132-sm100" CACHE PATH "" FORCE)
+
+set(CMAKE_C_COMPILER "/usr/bin/gcc" CACHE PATH "" FORCE)
+set(CMAKE_CXX_COMPILER "/usr/bin/g++" CACHE PATH "" FORCE)
+set(CMAKE_Fortran_COMPILER "/usr/bin/gfortran" CACHE PATH "" FORCE)
+set(ENABLE_FORTRAN OFF CACHE BOOL "" FORCE)
+
+set(ENABLE_MPI ON CACHE BOOL "" FORCE)
+set(MPI_C_COMPILER "${HPCPERF_MPI_BIN}/mpicc" CACHE PATH "" FORCE)
+set(MPI_CXX_COMPILER "${HPCPERF_MPI_BIN}/mpicxx" CACHE PATH "" FORCE)
+set(MPI_Fortran_COMPILER "${HPCPERF_MPI_BIN}/mpifort" CACHE PATH "" FORCE)
+set(MPIEXEC_EXECUTABLE "${HPCPERF_MPI_BIN}/mpirun" CACHE PATH "" FORCE)
+set(MPIEXEC_NUMPROC_FLAG "-np" CACHE STRING "" FORCE)
+set(ENABLE_WRAP_ALL_TESTS_WITH_MPIEXEC ON CACHE BOOL "")
+set(ENABLE_GTEST_DEATH_TESTS ON CACHE BOOL "" FORCE)
+
+set(ENABLE_OPENMP OFF CACHE BOOL "" FORCE)
+
+# BLAS/LAPACK: private OpenBLAS built by build.sh (single-threaded)
+set(BLAS_LIBRARIES "${HPCPERF_OPENBLAS_LIB}" CACHE PATH "" FORCE)
+set(LAPACK_LIBRARIES "${HPCPERF_OPENBLAS_LIB}" CACHE PATH "" FORCE)
+
+# CUDA
+set(ENABLE_CUDA ON CACHE BOOL "" FORCE)
+set(CUDA_TOOLKIT_ROOT_DIR "${HPCPERF_CUDA_ROOT}" CACHE PATH "" FORCE)
+set(CMAKE_CUDA_COMPILER "${HPCPERF_CUDA_ROOT}/bin/nvcc" CACHE STRING "" FORCE)
+set(CMAKE_CUDA_HOST_COMPILER "/usr/bin/g++" CACHE STRING "" FORCE)
+set(CMAKE_CUDA_ARCHITECTURES "100" CACHE STRING "" FORCE)
+set(CUDA_ARCH "sm_100" CACHE STRING "" FORCE)
+set(CMAKE_CUDA_FLAGS "-restrict -arch ${CUDA_ARCH} --expt-extended-lambda --expt-relaxed-constexpr -Werror cross-execution-space-call,reorder,deprecated-declarations" CACHE STRING "" FORCE)
+set(CMAKE_CUDA_FLAGS_RELEASE "-O3 -DNDEBUG -Xcompiler -DNDEBUG -Xcompiler -O3" CACHE STRING "" FORCE)
+set(CMAKE_CUDA_FLAGS_RELWITHDEBINFO "-g -lineinfo ${CMAKE_CUDA_FLAGS_RELEASE}" CACHE STRING "" FORCE)
+set(CMAKE_CUDA_FLAGS_DEBUG "-g -G -O0 -Xcompiler -O0" CACHE STRING "" FORCE)
+
+# Linear algebra interface: hypre on the GPU; Trilinos/PETSc off (upstream CUDA CI configuration)
+set(GEOS_LA_INTERFACE "Hypre" CACHE STRING "" FORCE)
+set(ENABLE_HYPRE ON CACHE BOOL "" FORCE)
+set(ENABLE_HYPRE_DEVICE "CUDA" CACHE STRING "" FORCE)
+set(ENABLE_TRILINOS OFF CACHE BOOL "" FORCE)
+# hypredrive (hypre driver library) is GEOS develop's default companion of hypre, but the TPL superbuild's hypredrive
+# step does not compile against the Umpire-enabled hypre here (missing Umpire include path) and the beam workflow
+# does not use it; GEOS documents "-DENABLE_HYPREDRV=OFF" for exactly this (SetupGeosxThirdParty.cmake).
+set(ENABLE_HYPREDRV OFF CACHE BOOL "" FORCE)
+set(ENABLE_PETSC OFF CACHE BOOL "" FORCE)
+set(ENABLE_SUPERLU_DIST ON CACHE BOOL "" FORCE)
+set(ENABLE_SUITESPARSE ON CACHE BOOL "" FORCE)
+set(ENABLE_SCOTCH ON CACHE BOOL "" FORCE)
+set(ENABLE_VTK ON CACHE BOOL "" FORCE)
+
+# Tests stay at upstream's default (ENABLE_TESTS ON, gtest/gbenchmark built); BLT 0.6.2's CUDA runtime smoke test needs
+# the back-port level3/geos/patches/geos-blt-0001-cuda13-memoryClockRate.patch (CUDA 13 removed cudaDeviceProp::memoryClockRate).
+set(ENABLE_CALIPER OFF CACHE BOOL "" FORCE)
+set(ENABLE_MATHPRESSO OFF CACHE BOOL "" FORCE)
+set(ENABLE_DOXYGEN OFF CACHE BOOL "" FORCE)
+set(ENABLE_UNCRUSTIFY OFF CACHE BOOL "" FORCE)
+set(ENABLE_XML_UPDATES OFF CACHE BOOL "" FORCE)
+set(ENABLE_PYGEOSX OFF CACHE BOOL "" FORCE)
+
+# GEOS build: point at the private TPL install when provided
+if(DEFINED HPCPERF_GEOS_TPL_DIR)
+  set(GEOS_TPL_DIR "${HPCPERF_GEOS_TPL_DIR}" CACHE PATH "" FORCE)
+  # GEOS adds the MPI include directory (= the conda environment's include/) as -isystem AHEAD of the TPL include
+  # directories, and that directory carries an unrelated 32-bit metis.h (a 2013 conda package): ParMETISInterface.cpp
+  # then fails its "ParMETIS must be built with 64-bit indices" static assertion although the TPL METIS/ParMETIS are
+  # 64-bit. -I directories are searched before -isystem ones, so the TPL METIS/ParMETIS headers are put first here
+  # (host and CUDA compile lines); no source change.
+  set(CMAKE_CXX_FLAGS "-I${HPCPERF_GEOS_TPL_DIR}/parmetis/include -I${HPCPERF_GEOS_TPL_DIR}/metis/include" CACHE STRING "" FORCE)
+  set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -I${HPCPERF_GEOS_TPL_DIR}/parmetis/include -I${HPCPERF_GEOS_TPL_DIR}/metis/include" CACHE STRING "" FORCE)
+  if(EXISTS "${CMAKE_CURRENT_LIST_DIR}/../../../_upstream/level3/GEOS/host-configs/tpls.cmake")
+    include("${CMAKE_CURRENT_LIST_DIR}/../../../_upstream/level3/GEOS/host-configs/tpls.cmake")
+  endif()
+endif()
diff --git a/level3/geos/patches/0001-tpl-superlu_dist-url-hash-typo.patch b/level3/geos/patches/0001-tpl-superlu_dist-url-hash-typo.patch
new file mode 100644
index 0000000..292326f
--- /dev/null
+++ b/level3/geos/patches/0001-tpl-superlu_dist-url-hash-typo.patch
@@ -0,0 +1,32 @@
+GEOS thirdPartyLibs 9b55672f6f8a73d02fd632396eb0410e58c9b120 (TPL tag 361-1070) -- superlu_dist URL hash typo
+
+Class: build-system compatibility fix (no source/algorithm change).
+Problem: CMakeLists.txt sets SUPERLU_URL_HASH to a 65-character string
+("...a2edf58a"); a SHA-256 has 64 hex characters, so ExternalProject's
+URL_HASH check can never succeed and the superlu_dist download step fails
+("does not match expected value ... Each download failed!").  The tarball
+https://github.com/xiaoyeli/superlu_dist/archive/0f6efc3.tar.gz actually
+downloaded on 2026-09-05 has sha256
+49cd47da22fe72a37011e8dfaf94bd63ad8d61cc8144eac7a571bcbd0a2edf58, i.e. the
+upstream string minus its trailing extra "a".  The same 65-character string is
+still present on upstream master (checked 2026-09-05); upstream CI evidently
+builds superlu_dist from a pre-populated mirror, not through this hash.
+Fix: drop the trailing character.  Impact: none on the built code; the hash now
+verifies the very tarball upstream intended.  Verified: TPL superbuild proceeds
+past the superlu_dist download.
+Applied by level3/geos/build.sh (git apply, idempotent); recorded in the
+profile fingerprint.
+
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index ab7fe24..9a6ecac 100644
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -851,7 +851,7 @@ list(APPEND build_list metis )
+ if( ENABLE_MPI AND ENABLE_SUPERLU_DIST )
+   set(SUPERLU_DIR "${CMAKE_INSTALL_PREFIX}/superlu_dist")
+   set(SUPERLU_URL "https://github.com/xiaoyeli/superlu_dist/archive/0f6efc3.tar.gz")
+-  set(SUPERLU_URL_HASH "49cd47da22fe72a37011e8dfaf94bd63ad8d61cc8144eac7a571bcbd0a2edf58a")
++  set(SUPERLU_URL_HASH "49cd47da22fe72a37011e8dfaf94bd63ad8d61cc8144eac7a571bcbd0a2edf58")
+ 
+   if ( CMAKE_CXX_COMPILER_ID STREQUAL "Clang" AND CMAKE_CXX_COMPILER_VERSION VERSION_GREATER 16.0 )
+     set(SUPERLU_C_FLAGS "-fPIC ${C_FLAGS_NO_WARNINGS}")
diff --git a/level3/geos/patches/0002-tpl-raja-vectorization-overridable.patch b/level3/geos/patches/0002-tpl-raja-vectorization-overridable.patch
new file mode 100644
index 0000000..48e80c3
--- /dev/null
+++ b/level3/geos/patches/0002-tpl-raja-vectorization-overridable.patch
@@ -0,0 +1,40 @@
+GEOS thirdPartyLibs 9b55672f6f8a73d02fd632396eb0410e58c9b120 (TPL tag 361-1070) -- make RAJA_ENABLE_VECTORIZATION overridable
+
+Class: build-system compatibility fix (no source/algorithm change in RAJA or GEOS).
+Problem: CMakeLists.txt unconditionally forces RAJA_ENABLE_VECTORIZATION=ON for
+every non-HIP build (upstream switches it OFF for ROCm because "RAJA's
+vectorization layer does not compile with the ROCm toolchain"). On this node the
+same layer does not compile with nvcc 13.2.78 + system GCC 14.2.1 as host
+compiler: EL10's GCC defaults to -march=x86-64-v3, so __AVX2__ is defined in
+every nvcc host pass and RAJA/policy/tensor/arch/avx2/avx2_int32.hpp is pulled
+in; nvcc's front end rejects GCC 14's __m256i intrinsics there
+('"__m256i" ... is a vector type', broadcast/self_type errors) in every CUDA
+translation unit that includes RAJA/RAJA.hpp (RAJA's own exercises fail first).
+Fix: only default the option when the caller has not set it, so build.sh can pass
+-D RAJA_ENABLE_VECTORIZATION=OFF (exactly the switch upstream uses for ROCm).
+Impact: RAJA's experimental RAJA::expt tensor/vector-register layer is not
+built; GEOS and LvArray do not use it (no RAJA::expt tensor symbols in
+GEOS/src or LvArray/src -- checked 2026-09-06), RAJA::forall/kernel/CUDA
+policies are unaffected. Verified: TPL superbuild proceeds past RAJA with the
+option OFF. Applied by level3/geos/build.sh (git apply, idempotent); the
+option and the patch hash are recorded in the profile fingerprint.
+
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -397,10 +397,14 @@
+ message(STATUS "Building RAJA found at ${RAJA_URL}")
+ 
+ # RAJA's vectorization layer does not compile with the ROCm toolchain.
+-if( ENABLE_HIP )
++# (hpcperf: honour a value given on the command line, e.g. -D RAJA_ENABLE_VECTORIZATION=OFF for
++#  toolchains whose nvcc cannot compile the layer's AVX2 register code -- see level3/geos/patches/0002)
++if( NOT DEFINED RAJA_ENABLE_VECTORIZATION )
++  if( ENABLE_HIP )
+     set( RAJA_ENABLE_VECTORIZATION OFF )
+-else()
++  else()
+     set( RAJA_ENABLE_VECTORIZATION ON )
++  endif()
+ endif()
+ 
+ ExternalProject_Add( raja
diff --git a/level3/geos/patches/0003-tpl-hdf5-step-generator-and-build-command.patch b/level3/geos/patches/0003-tpl-hdf5-step-generator-and-build-command.patch
new file mode 100644
index 0000000..122ff67
--- /dev/null
+++ b/level3/geos/patches/0003-tpl-hdf5-step-generator-and-build-command.patch
@@ -0,0 +1,30 @@
+GEOS thirdPartyLibs 9b55672f6f8a73d02fd632396eb0410e58c9b120 (TPL tag 361-1070) -- hdf5 step: use the superbuild's generator and build/install commands
+
+Class: build-system compatibility fix (no source/algorithm change).
+Problem: every CMake-based ExternalProject step passes
+`CMAKE_GENERATOR ${TPL_GENERATOR}` and uses ${TPL_BUILD_COMMAND}/${TPL_INSTALL_COMMAND}
+(make / ninja, selected by ENABLE_NINJA, CMakeLists.txt lines 86-96) -- except
+hdf5, which passes no generator and hardcodes `make -j ${NUM_PROC}` / `make install`.
+When the superbuild itself is generated with Ninja (`scripts/config-build.py -n`,
+upstream's documented option) the hdf5 sub-build inherits the Ninja generator
+from the parent and its hardcoded make step fails with "make: *** No targets
+specified and no makefile found". Fix: give the hdf5 step the same generator and
+build/install commands as all other steps. Impact: none on the built HDF5 1.12.1
+(same configure options; only the generator/driver of its own build). Verified:
+TPL superbuild proceeds past hdf5 under a Ninja-generated superbuild. Applied by
+level3/geos/build.sh (git apply, idempotent); recorded in the profile fingerprint.
+
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -259,8 +259,9 @@
+                      URL_HASH SHA256=${HDF5_URL_HASH}
+                      PREFIX ${PROJECT_BINARY_DIR}/hdf5
+                      INSTALL_DIR ${HDF5_DIR}
+-                     BUILD_COMMAND make -j ${NUM_PROC}
+-                     INSTALL_COMMAND make install
++                     BUILD_COMMAND ${TPL_BUILD_COMMAND}
++                     INSTALL_COMMAND ${TPL_INSTALL_COMMAND}
++                     CMAKE_GENERATOR ${TPL_GENERATOR}
+                      CMAKE_ARGS -D CMAKE_C_COMPILER:STRING=${HDF5_C_COMPILER}
+                                 -D CMAKE_CXX_COMPILER:STRING=${HDF5_CXX_COMPILER}
+                                 -D CMAKE_C_FLAGS:STRING=${C_FLAGS_NO_WARNINGS}
diff --git a/level3/geos/patches/geos-blt-0001-cuda13-memoryClockRate.patch b/level3/geos/patches/geos-blt-0001-cuda13-memoryClockRate.patch
new file mode 100644
index 0000000..58e7277
--- /dev/null
+++ b/level3/geos/patches/geos-blt-0001-cuda13-memoryClockRate.patch
@@ -0,0 +1,44 @@
+GEOS submodule src/cmake/blt (BLT 0.6.2, 9ff77344f0b2a6ee345e452bddd6bfd46cbbfa35) -- CUDA 13: memoryClockRate
+
+Class: build-system/test-fixture compatibility fix, back-port of upstream BLT
+commit 38b46203cf42 ("Set memory clock rate based on the cuda runtime version",
+LLNL/blt develop, 2025-09-07); applied to the smoke test only.
+Problem: CUDA 13 removed cudaDeviceProp::memoryClockRate; BLT 0.6.2's CUDA
+runtime smoke test (built whenever ENABLE_TESTS is on, GEOS' default) fails to
+compile with nvcc 13.2 ("'struct cudaDeviceProp' has no member named
+'memoryClockRate'"), which stops the GEOS build although nothing in GEOS itself
+is affected. Fix: query cudaDevAttrMemoryClockRate through
+cudaDeviceGetAttribute when CUDART_VERSION >= 13000, exactly as upstream BLT
+does. Impact: none on GEOS (a smoke-test executable). Verified: the target
+compiles; GEOS builds with upstream's default ENABLE_TESTS=ON. Applied by
+level3/geos/build.sh (git apply in the submodule, idempotent); hash recorded in
+the profile fingerprint.
+
+--- a/tests/smoke/blt_cuda_runtime_smoke.cpp
++++ b/tests/smoke/blt_cuda_runtime_smoke.cpp
+@@ -27,15 +27,23 @@
+   for (int i = 0; i < nDevices; i++)
+   {
+     cudaDeviceProp prop;
++    int memoryClockRate;
+     cudaGetDeviceProperties(&prop, i);
++
++#if CUDART_VERSION >= 13000
++    cudaDeviceGetAttribute(&memoryClockRate, cudaDevAttrMemoryClockRate, i);
++#else
++    memoryClockRate = prop.memoryClockRate;
++#endif
++
+     printf("Device Number: %d\n", i);
+     printf("  Device name: %s\n", prop.name);
+     printf("  Memory Clock Rate (KHz): %d\n",
+-           prop.memoryClockRate);
++           memoryClockRate);
+     printf("  Memory Bus Width (bits): %d\n",
+            prop.memoryBusWidth);
+     printf("  Peak Memory Bandwidth (GB/s): %f\n\n",
+-           2.0*prop.memoryClockRate*(prop.memoryBusWidth/8)/1.0e6);
++           2.0*memoryClockRate*(prop.memoryBusWidth/8)/1.0e6);
+   }
+ 
+   return 0;
diff --git a/level3/geos/run.sh b/level3/geos/run.sh
new file mode 100755
index 0000000..a75b0a1
--- /dev/null
+++ b/level3/geos/run.sh
@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+# Run GEOS (develop snapshot, CUDA sm_100, hypre on device) on N GPUs, one MPI
+# rank per GPU (upstream's model: --ntasks-per-gpu=1 / jsrun -g 1; GEOS itself
+# does not select a device -- the launcher's per-rank wrapper exposes one GPU).
+#
+#   ./run.sh [CUDA] [extra geos args...]
+#
+# Cases (HPCPERF_GEOS_CASE), upstream inputs under inputFiles/solidMechanics/:
+#   beam (default)   Beam bending (quasi-static linear elasticity, SolidMechanicsLagrangianFEM,
+#                    traction ramp over 10 steps, analytic Euler-Bernoulli reference shipped as
+#                    beamBending_curve.py). Decks:
+#     smoke  : beamBending_smoke.xml mesh (80x8x4 C3D8) with the benchmark deck's iterative solver
+#              (GMRES + AMG on hypre, krylovTol 1e-6 -- the GPU linear-solver path); the shipped
+#              smoke deck uses a serial direct solver (directParallel=0), kept for the 1-rank
+#              official-baseline comparison in validate.sh (HPCPERF_GEOS_SOLVER=direct)
+#     strong : beamBending_benchmark.xml as shipped (160x16x8, GMRES+AMG) -- fixed mesh over ranks
+#     weak   : the beam refined so that elements/rank stays ~constant (nx,ny,nz scaled with the
+#              partition grid; same physical problem, finer mesh) -- labelled "refinement weak"
+#   Partitions come from the launcher's topology helper with the divisibility constraint
+#   (nx % PX == 0 ...); GEOS receives -x PX -y PY -z PZ so requested == decomposed ranks.
+#
+# Controls: HPCPERF_GPUS, HPCPERF_SCALE_MODE, HPCPERF_GEOS_PROFILE, HPCPERF_GEOS_SOLVER=amg|direct
+# (smoke only), HPCPERF_GEOS_WEAK_NX (elements per rank along x at PX=1, default 80).
+# Output: <run_dir>/ (GEOS -o), displacement_history.hdf5 (TimeHistory), silo/, restart files
+# (only for HPCPERF_GEOS_SOLVER=direct runs); stdout.log; run_manifest.txt.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"; [ $# -gt 0 ] && shift
+[ "$BACKEND" = CUDA ] || { echo "run.sh: only CUDA is built for GEOS here" >&2; exit 2; }
+GCC_MM="$(l3_version_mm "$(/usr/bin/gcc -dumpfullversion)")"; OMPI_V="$(mpirun --version | head -1 | /usr/bin/grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)"
+PROFILE="${HPCPERF_GEOS_PROFILE:-cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-ompi$(echo "$OMPI_V" | tr -d .)}"
+l3_paths_profile geos "$PROFILE"
+SRC="$R/_upstream/level3/GEOS"
+EXE="$L3_INSTALL/geos/bin/geosx"; [ -x "$EXE" ] || EXE="$(find "$L3_INSTALL/geos/bin" -maxdepth 1 -type f -name 'geos*' 2>/dev/null | head -1)"
+[ -n "$EXE" ] && [ -x "$EXE" ] || { echo "run.sh: GEOS executable not found under $L3_INSTALL/geos/bin -- run ./build.sh" >&2; exit 1; }
+l3_binary_backend_check "$EXE" cuda || exit 1
+CASE="${HPCPERF_GEOS_CASE:-beam}"; [ "$CASE" = beam ] || { echo "run.sh: HPCPERF_GEOS_CASE must be beam" >&2; exit 2; }
+MODE="$(l3_scale_mode geos)" || exit 2
+N_RANKS="$(hpcperf_ranks geos yes)" || exit 2
+SOLVER="${HPCPERF_GEOS_SOLVER:-amg}"
+hpcperf_forbid_args geos -x -y -z -i -o --input --output --x-partitions --y-partitions --z-partitions -- "$@" || exit 2
+D="$SRC/inputFiles/solidMechanics"
+case "$MODE" in
+    smoke)  NX=80; NY=8; NZ=4; LABEL="beam80.$SOLVER" ;;
+    strong) NX=160; NY=16; NZ=8; LABEL="beam160.amg"; SOLVER=amg ;;
+    weak)   L="${HPCPERF_GEOS_WEAK_NX:-80}"; SOLVER=amg
+            TOPO="$(hpcperf_topology geos "$N_RANKS")" || exit 2; read -r PX PY PZ <<< "$TOPO"
+            NX=$((L * PX)); NY=$((L / 10 * PY)); NZ=$((L / 20 * PZ)); LABEL="beam.amg" ;;
+esac
+[ "$MODE" = smoke ] && case "$SOLVER" in amg|direct) ;; *) echo "run.sh: HPCPERF_GEOS_SOLVER must be amg or direct" >&2; exit 2;; esac
+if [ "$MODE" != weak ]; then
+    TOPO="$(hpcperf_topology geos "$N_RANKS" --divides "$NX,$NY,$NZ")" || { echo "run.sh: $N_RANKS ranks cannot partition the ${NX}x${NY}x${NZ} mesh (each factor must divide its extent)" >&2; exit 2; }
+    read -r PX PY PZ <<< "$TOPO"
+fi
+ELEMS=$((NX * NY * NZ))
+RUN_DIR="$(l3_rundir "$L3_BUILD/$L3_RUN_SUBDIR/$LABEL.$MODE.np$N_RANKS")" || exit 2
+# Derived deck (class A): upstream smoke/benchmark deck with the mesh resolution and, for
+# the smoke+amg variant, the benchmark's LinearSolverParameters; base file copied alongside
+# so the <Included> relative reference resolves. Physics, BCs, material, time stepping and
+# the analytic reference (beamBending_curve.py) are upstream's, untouched.
+cp "$D/beamBending_base.xml" "$RUN_DIR/"
+if [ "$MODE" = smoke ] && [ "$SOLVER" = direct ]; then
+    cp "$D/beamBending_smoke.xml" "$RUN_DIR/input.xml"; DECK="$D/beamBending_smoke.xml"
+else
+    DECK="$D/beamBending_benchmark.xml"
+    sed -e "s|nx=\"{ 160 }\"|nx=\"{ $NX }\"|; s|ny=\"{ 16 }\"|ny=\"{ $NY }\"|; s|nz=\"{ 8 }\"|nz=\"{ $NZ }\"|" "$DECK" > "$RUN_DIR/input.xml"
+    /usr/bin/grep -q "nx=\"{ $NX }\"" "$RUN_DIR/input.xml" || { echo "run.sh: mesh substitution failed" >&2; exit 1; }
+fi
+echo "# GEOS $BACKEND profile=$PROFILE case=$CASE mode=$MODE ranks=$N_RANKS mesh=${NX}x${NY}x${NZ} ($ELEMS elements) partitions=${PX}x${PY}x${PZ} solver=$SOLVER deck=$(realpath --relative-to="$SRC" "$DECK") run_dir=$RUN_DIR"
+cd "$RUN_DIR"
+RUN_ID="$(l3_run_id)"
+set +e; set -o pipefail
+"$L3_LAUNCHER" --gpus "$N_RANKS" --bind wrapper -- "$EXE" -i input.xml -x "$PX" -y "$PY" -z "$PZ" -o "$RUN_DIR" "$@" 2>&1 | tee "$RUN_DIR/stdout.log"
+rc=$?
+set +o pipefail; set -e
+if [ -z "${HPCPERF_DRY_RUN:-}" ]; then
+    l3_manifest "$RUN_DIR" "run_id=$RUN_ID" "app=geos" "backend=$BACKEND" "profile=$PROFILE" "case=$CASE" "mode=$MODE" "solver=$SOLVER" \
+        "ranks=$N_RANKS" "mesh=${NX}x${NY}x${NZ}" "elements=$ELEMS" "partitions=${PX}x${PY}x${PZ}" "exit_code=$rc" \
+        "binary=$EXE" "binary_sha256=$(l3_sha_file "$EXE")" "deck=$DECK" "deck_sha256=$(l3_sha_file "$DECK")" "input_sha256=$(l3_sha_file "$RUN_DIR/input.xml")" \
+        "fingerprint_sha256=$(l3_sha_file "$L3_INSTALL/.hpcperf-l3-fingerprint")" "stdout=$RUN_DIR/stdout.log" "utc=$(date -u +%FT%TZ)"
+fi
+exit "$rc"
diff --git a/level3/geos/validate.sh b/level3/geos/validate.sh
new file mode 100755
index 0000000..21675e0
--- /dev/null
+++ b/level3/geos/validate.sh
@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+# Correctness check for GEOS (beam bending, hypre AMG on the GPU) on N GPUs.
+#
+#   ./validate.sh [CUDA]        HPCPERF_GPUS=N (default 1)
+#
+# Criteria (fixed before the runs; [2]-[4] are geos-ats's own checks for this test, re-implemented
+# in geos_beam_check.py with the metrics of geosPythonPackages curve_check.py / restart_check.py and
+# verified against upstream's published baseline before any GEOS run here):
+#  [1] completeness: run.sh exit 0 (timeout -> FAIL), displacement_history.hdf5 written with
+#      the 10 output times, all values finite, launcher audit "N verified, 0 mismatch".
+#  [2] analytic reference (beamBending.ats CurveCheckParameters, script beamBending_curve.py =
+#      Euler-Bernoulli with upstream's 1.043 discretisation factor): geos-ats metric
+#      ||u - u_script||_2 / N <= HPCPERF_GEOS_ANALYTIC_TOL (default 0.0002, the .ats tolerance;
+#      upstream's own 80x8x4 baseline scores 1.376e-4 -- the mesh's discretisation error, which is
+#      why a per-time relative L-inf reading of the same number would fail even upstream's baseline;
+#      that reading is printed as information only). Every run, every N.
+#  [3] rank-count independence (AMG deck): N-GPU history vs the 1-GPU history of the same binary/deck,
+#      relative L-inf <= 1e-4 (gate; 100x the GMRES krylovTol 1e-6 -- the two runs converge to
+#      different iterates of the same tolerance through different AMG hierarchies -- and below the
+#      official discretisation-level tolerance) and geos-ats baseline metric ||u - u_1GPU||_2 / N
+#      <= 2e-4; whether the tighter 1e-5 is also met is printed (upstream's own direct-solver
+#      baselines for 1 and 8 ranks agree to 1.5e-12).
+#  [4] official baselines (1 rank only; the public integrated-test baseline beamBending_smoke_01
+#      is for 1 rank with the shipped smoke deck): (a) every dataset/attribute of the GEOS restart
+#      file at cycle 10 vs the baseline with atol 1e-3 / rtol 1e-7 (RestartcheckParameters) and
+#      the geos-ats default exclusions; (b) the history vs the baseline's history, geos-ats baseline
+#      curve metric <= 0.0002 and relative L-inf <= 1e-4. Requires the baseline tarball under
+#      .deps/level3/geos/downloads (recorded with its sha256).
+# The GPU is exercised through RAJA/CHAI kernels (assembly) and hypre's device AMG/GMRES for
+# the amg solver runs; the launcher audits one GPU per rank.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+N="${HPCPERF_GPUS:-1}"; TIMEOUT="${HPCPERF_VALIDATE_TIMEOUT:-1800}"
+ATOL=1e-3; RTOL=1e-7                  # restart baseline (beamBending.ats)
+CURVE_TOL="${HPCPERF_GEOS_ANALYTIC_TOL:-2e-4}"  # geos-ats ||.||_2/N metric (script and baseline curve checks)
+XRANK_TOL=1e-4; XRANK_INFO=1e-5                 # relative L-inf between histories
+GCC_MM="$(l3_version_mm "$(/usr/bin/gcc -dumpfullversion)")"; OMPI_V="$(mpirun --version | head -1 | /usr/bin/grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)"
+PROFILE="${HPCPERF_GEOS_PROFILE:-cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-ompi$(echo "$OMPI_V" | tr -d .)}"
+RUNS="$R/build/level3/geos/$PROFILE/$L3_RUN_SUBDIR"; SRC="$R/_upstream/level3/GEOS"
+PY="$R/.deps/level3/geos/$PROFILE/install/venv/bin/python"; [ -x "$PY" ] || { echo "validate.sh: FAIL -- venv python with h5py missing ($PY)"; exit 1; }
+CURVE="$SRC/inputFiles/solidMechanics/beamBending_curve.py"
+CHECK="$HERE/geos_beam_check.py"
+export HPCPERF_GPUS="$N" HPCPERF_SCALE_MODE=smoke
+mkdir -p "$RUNS"; ok=1
+fail() { echo "validate.sh: FAIL -- $*"; ok=0; }
+manifest_val() { /usr/bin/grep -m1 "^$2=" "$1/run_manifest.txt" 2>/dev/null | cut -d= -f2- || true; }
+run_beam() { # run_beam <solver> <n> <out>
+    local rc=0; HPCPERF_GEOS_SOLVER="$1" HPCPERF_GPUS="$2" timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" > "$3" 2>&1 || rc=$?; return $rc
+}
+
+echo "validate.sh: === GEOS $BACKEND beam bending (80x8x4), hypre GMRES+AMG on device, $N GPU(s) [profile $PROFILE] ==="
+D="$RUNS/beam80.amg.smoke.np$N"
+rc=0; run_beam amg "$N" "$RUNS/validate.beam80.amg.np$N.stdout" || rc=$?
+/usr/bin/grep -aE '^# GEOS|hpcperf-launch: audit summary|Error|error' "$RUNS/validate.beam80.amg.np$N.stdout" | head -6 || true
+if [ "$rc" -eq 124 ]; then fail "beam80.amg np$N timed out"; elif [ "$rc" -ne 0 ]; then fail "beam80.amg np$N run.sh exited $rc"; else
+    [ -f "$D/displacement_history.hdf5" ] || fail "beam80.amg np$N: displacement_history.hdf5 missing"
+    if [ -f "$D/displacement_history.hdf5" ]; then
+        "$PY" "$CHECK" analytic "$D" --tol "$CURVE_TOL" --curve "$CURVE" | tee "$D/analytic_check.txt" | tail -3 || fail "beam80.amg np$N: analytic beam check failed"
+        if [ "$N" -gt 1 ]; then
+            REF="$RUNS/beam80.amg.smoke.np1"
+            if [ ! -f "$REF/displacement_history.hdf5" ] || [ "$(manifest_val "$REF" binary_sha256)" != "$(manifest_val "$D" binary_sha256)" ]; then
+                echo "    (1-GPU reference missing/stale -- running it now)"; rc=0; run_beam amg 1 "$RUNS/validate.beam80.amg.np1.stdout" || fail "1-GPU reference run exited $?"
+            fi
+            "$PY" "$CHECK" compare "$REF" "$D" --tol "$XRANK_TOL" --info-tol "$XRANK_INFO" --l2n-tol "$CURVE_TOL" | tee "$D/cross_rank_check.txt" | tail -3 || fail "beam80.amg np$N: history differs from the 1-GPU run (rel L-inf > $XRANK_TOL or ||.||_2/N > $CURVE_TOL)"
+        fi
+    fi
+fi
+
+if [ "$N" -eq 1 ]; then
+    echo "validate.sh: === GEOS $BACKEND shipped smoke deck (serial direct solver) on 1 rank: analytic check + official restart baseline ==="
+    DD="$RUNS/beam80.direct.smoke.np1"
+    rc=0; run_beam direct 1 "$RUNS/validate.beam80.direct.np1.stdout" || rc=$?
+    if [ "$rc" -ne 0 ]; then fail "beam80.direct np1 run.sh exited $rc"; else
+        "$PY" "$CHECK" analytic "$DD" --tol "$CURVE_TOL" --curve "$CURVE" | tee "$DD/analytic_check.txt" | tail -1 || fail "beam80.direct np1: analytic beam check failed"
+        BASE_TGZ="$(ls "$R"/.deps/level3/geos/downloads/baseline_integratedTests-*.tar.gz 2>/dev/null | head -1)"
+        if [ -n "$BASE_TGZ" ]; then
+            BD="$R/.deps/level3/geos/downloads/baseline_extracted/solidMechanics/beamBending_smoke_01"
+            [ -d "$BD" ] || { mkdir -p "$R/.deps/level3/geos/downloads/baseline_extracted"; tar -xzf "$BASE_TGZ" -C "$R/.deps/level3/geos/downloads/baseline_extracted" ./solidMechanics/beamBending_smoke_01 2>/dev/null || tar -xzf "$BASE_TGZ" -C "$R/.deps/level3/geos/downloads/baseline_extracted" solidMechanics/beamBending_smoke_01; }
+            [ -d "$BD" ] || fail "baseline beamBending_smoke_01 not found in $BASE_TGZ"
+            [ -d "$BD" ] && { "$PY" "$CHECK" restart "$DD" "$BD" --rtol "$RTOL" --atol "$ATOL" | tee "$DD/restart_baseline_check.txt" | tail -2 || fail "beam80.direct np1: restart file disagrees with the upstream baseline (atol $ATOL rtol $RTOL)"; }
+            [ -d "$BD" ] && { "$PY" "$CHECK" compare "$BD" "$DD" --tol "$XRANK_TOL" --info-tol "$XRANK_INFO" --l2n-tol "$CURVE_TOL" | tee "$DD/history_baseline_check.txt" | tail -3 || fail "beam80.direct np1: history disagrees with the upstream baseline history"; }
+            echo "    baseline: $(basename "$BASE_TGZ") sha256=$(l3_sha_file "$BASE_TGZ")"
+        else
+            fail "no integrated-test baseline tarball under .deps/level3/geos/downloads (official restart comparison NOT_RUN)"
+        fi
+    fi
+fi
+
+if [ "$ok" -eq 1 ]; then echo "GEOS $BACKEND validation ($N GPU; geos-ats analytic curve metric <= $CURVE_TOL$( [ "$N" -gt 1 ] && echo ", history within rel $XRANK_TOL / metric $CURVE_TOL of 1-GPU" || echo ", shipped smoke deck vs upstream restart baseline atol $ATOL rtol $RTOL + baseline history"), finite, complete): PASS"; exit 0; fi
+echo "GEOS $BACKEND validation ($N GPU): FAIL"; exit 1
diff --git a/level3/lammps/README.md b/level3/lammps/README.md
new file mode 100644
index 0000000..3785623
--- /dev/null
+++ b/level3/lammps/README.md
@@ -0,0 +1,126 @@
+# LAMMPS (Level 3)
+
+Full classical molecular dynamics (neighbor lists, short- and long-range
+forces, spatial decomposition, MPI halo exchange) -- run as the complete
+application through its own input scripts, KOKKOS package on the GPU.
+
+## Provenance
+
+- Official repository: https://github.com/lammps/lammps (docs
+  https://docs.lammps.org/, Kokkos: https://docs.lammps.org/Speed_kokkos.html)
+- Release policy: `stable_*` tags with `_updateN` bug-fix updates; `patch_*`
+  are feature releases (GitHub marks them pre-release).
+- Selected: **`stable_22Jul2025_update6`** (released 2026-09-03), commit
+  `9c5ab448c78a14fd534619622162ba418d6a1fb1`, fetched by `fetch.sh` into
+  `_upstream/level3/lammps` (shallow, read-only).
+- License: GPL-2.0 (`LICENSE`).
+- Application-owned LOC (cloc 2.06, code lines): `src/` **852,527** in
+  3,865 files = 743,761 outside `src/KOKKOS` + 108,766 in `src/KOKKOS`
+  (the GPU package). Bundled `lib/kokkos` (Kokkos 4.6.2) is counted
+  separately and not modified.
+
+## Build strategy: NATIVE (upstream CMake + bundled Kokkos 4.6.2)
+
+`build.sh CUDA` = LAMMPS' documented Kokkos/CUDA recipe:
+`nvcc_wrapper` (host compiler conda GCC 13.3.0) as CXX, `PKG_KOKKOS`,
+`Kokkos_ENABLE_CUDA`, `Kokkos_ARCH_BLACKWELL100` (sm_100; the bundled
+Kokkos 4.6.2 supports it), `Kokkos_ENABLE_OPENMP/SERIAL`, `FFT_KOKKOS=CUFFT`,
+`FFT=KISS` (host), `BUILD_MPI` (conda Open MPI 5.0.10, CUDA-aware), C++17,
+packages `MOLECULE KSPACE MANYBODY RIGID GRANULAR` (what `bench/` needs),
+`WITH_JPEG=no WITH_PNG=no` (no `jpeglib.h` on the node; image dumps unused).
+Build time on dgx003: **220 s** at `-j32` (708 targets). Warnings: 663
+lines, essentially all `nvcc_wrapper: multiple optimization flags` (conda
+`-O2` + Release `-O3`) plus two upstream unused-variable notes (`#550-D`,
+`#177-D`); no errors. Install prefix `.deps/level3/lammps/install`
+(fingerprinted: upstream commit, Kokkos 4.6.2, compiler, CUDA 13.2.78, MPI,
+CMake options, GPU-aware setting).
+
+Why not the others: upstream does not recommend Spack for GPU builds (the
+Spack `lammps` package exists but the local Spack checkout is 2025-05 and
+lacks `cuda_arch=100`); no Apptainer on the node and a container would not
+provide the host MPI/transport; site modules are broken on dgx003. Level 2's
+Kokkos 5.2.1 is **not** used: LAMMPS requires an external Kokkos
+`>= 4.6.02` and pins 4.6.2 internally -- the bundled one is the supported
+configuration.
+
+HIP: `build.sh HIP` carries the upstream `Kokkos_ENABLE_HIP` +
+`Kokkos_ARCH_AMD_GFX950` + `FFT_KOKKOS=HIPFFT` recipe and exits with a clear
+message here (no ROCm). **Untested.**
+
+## Changes from upstream
+
+Class **A -- no source modification.** `run.sh` writes a *derived* copy of
+`bench/in.lj` into the build tree with `run 100` -> `run ${steps}` and, in
+weak mode, a `processors ${px} ${py} ${pz}` line; the upstream file is
+untouched, and with the default 100 steps the derived deck is semantically
+identical.
+
+## Execution model
+
+One MPI rank per GPU (upstream `Speed_kokkos`), `-k on g 1 t 1 -sf kk
+-pk kokkos newton on neigh half gpu/aware on`. Ranks are launched by the
+common launcher with the per-rank GPU wrapper, so each rank sees exactly one
+GPU (`g 1`) and the launcher audits expected vs observed GPU. GPU-aware MPI
+(`gpu/aware on`, LAMMPS default) is used with the CUDA-aware conda Open MPI;
+`HPCPERF_LAMMPS_GPU_AWARE=off` selects host-staged communication. Any rank
+count is legal: LAMMPS factors the box into a processor grid itself (weak mode
+passes the grid explicitly). Threads per rank: 1 (`HPCPERF_CPUS_PER_RANK`
+sets `t`), as upstream recommends for GPU runs.
+
+## Inputs (`HPCPERF_SCALE_MODE`)
+
+| Mode | Global box (fcc cells) | Atoms | Per rank @4 GPU | Topology | Steps | Memory/GPU (est.) | Runtime on B200 | Validation quantity |
+|---|---|---|---|---|---|---|---|---|
+| smoke (default) | 20^3 (upstream `in.lj`) | 32,000 | 8,000 | LAMMPS auto | 100 | < 0.1 GB | 0.02-0.09 s | thermo vs upstream reference log |
+| strong | (20*S)^3, S=`HPCPERF_LAMMPS_STRONG` (8) = 160^3 | 16,384,000 | 4,096,000 | LAMMPS auto | 100 | ~2 GB | 1.1 s (1 GPU) / 2.3 s (4 GPU) | same thermo table |
+| weak | (20*L*P)^3-shaped, L=`HPCPERF_LAMMPS_LOCAL` (4): 80^3 cells/rank | 2,048,000 x N | 2,048,000 | `hpcperf_topology.py` grid = `processors` | 100 | ~0.3 GB | 1.1 s (4 GPU) | same thermo table |
+
+Memory estimate: ~120 B/atom for LJ with Kokkos neighbor lists (55
+neighbors/atom, half list) -- well below the 180 GB of a B200 at every size
+above; the strong default is deliberately a *correctness* size (100 steps run
+in seconds), not a performance deck. The 4-GPU strong run (2.29 s) being
+slower than 1 GPU (1.12 s) at 16.4M atoms/100 steps is the expected
+communication-dominated behaviour of a fixed small workload and is **not** a
+scaling result.
+
+## Validation (`validate.sh`, upstream mechanism)
+
+Thermo output (Temp, E_pair, TotEng, Press at steps 0 and 100) of the
+unmodified `bench/in.lj` is compared with the reference log LAMMPS ships,
+`bench/log.15Jul25.lj.fixed.g++.1` (CPU, 1 process; `velocity ... loop geom`
+makes the initial state machine- and rank-count-independent). Tolerances:
+1e-8 relative at step 0 (deterministic), 1e-5 at step 100 (reduction-order
+divergence); with N > 1 GPUs the N-rank run is also compared with this build's
+1-GPU run. Observed on dgx003 (2026-09-04): **all eight quantities identical
+to the reference to every printed digit (rel 0.00e+00) at 1, 2 and 4 GPUs**.
+
+## Results on dgx003 (4x B200, CUDA 13.2.78, Slurm job 9552083)
+
+| Run | Ranks x GPUs | rank->GPU | CPU binding | Topology | Problem | Loop time | Validation |
+|---|---|---|---|---|---|---|---|
+| smoke | 1 x 1 | wrapper (1 visible GPU/rank); audit 1/1 verified | runtime default, `t 1` | 1x1x1 | 32k atoms, 100 steps | 0.020 s | PASS |
+| smoke | 2 x 2 | wrapper; audit 1 verified / 1 unverified (0.07 s run, too short to sample) | runtime default | LAMMPS auto | 32k atoms | 0.070 s | PASS (vs ref and vs 1-GPU) |
+| smoke | 4 x 4 | wrapper; audit 4/4 verified | runtime default | LAMMPS auto | 32k atoms | 0.085 s | PASS (vs ref and vs 1-GPU) |
+| strong | 1 x 1 | wrapper | runtime default | 1x1x1 | 16.4M atoms | 1.118 s | run completes; thermo consistent |
+| strong | 4 x 4 | wrapper; 4/4 verified | runtime default | LAMMPS auto | 16.4M atoms | 2.295 s | run completes |
+| weak | 4 x 4 | wrapper; 4/4 verified | runtime default | 2x2x1 (`processors`) | 8.19M atoms (2.05M/rank) | 1.128 s | run completes |
+
+Dry-runs (`HPCPERF_DRY_RUN=1`, hypothetical allocations) -- **DRY-RUN /
+UNVALIDATED**, nothing executed:
+
+| GPUs | Nodes x GPUs/node | Mode | Global box | Per rank | Ranks/node | Launch |
+|---|---|---|---|---|---|---|
+| 8 | 1 x 8 | strong | 160^3 cells = 16.4M atoms | 2.05M | 8 | `mpirun -np 8 --host dgx003:8 --map-by ppr:8:node ...` (single node) |
+| 40 | 5 x 8 | weak | 400x320x160 = 81.9M atoms | 2.05M | 8 | `mpirun -np 40 --host <5 nodes>:8 --map-by ppr:8:node` -- multi-node BLOCKED on this site |
+| 80 | 10 x 8 | weak | 400x320x320 = 163.8M atoms | 2.05M | 8 | `mpirun -np 80 ...` -- multi-node BLOCKED on this site |
+
+## Limitations
+
+- Multi-node: BLOCKED/UNVERIFIED on this site (transport); 40/80-GPU shapes
+  are plans only.
+- HIP: recipe present, untested (no AMD GPU).
+- The benchmark family here is `bench/in.lj`; `in.eam`, `in.rhodo`
+  (pppm/kk + cuFFT), `in.chain`, `in.chute` build with this package set but
+  have no wrappers yet.
+- Weak-mode `processors` grid comes from the generic balanced factorization;
+  LAMMPS' own auto grid is used in smoke/strong.
diff --git a/level3/lammps/build.sh b/level3/lammps/build.sh
new file mode 100755
index 0000000..4ed409c
--- /dev/null
+++ b/level3/lammps/build.sh
@@ -0,0 +1,85 @@
+#!/usr/bin/env bash
+# Build LAMMPS (KOKKOS package, CUDA or HIP) with upstream's native CMake and
+# the Kokkos version LAMMPS bundles (lib/kokkos) -- the upstream-supported path.
+#
+#   ./build.sh [CUDA|HIP]        (default CUDA)
+#
+# Layout (Level 3 dependency isolation, nothing shared with Level 2):
+#   source    _upstream/level3/lammps        (fetch.sh; the app owns its Kokkos)
+#   build     build/level3/lammps/<cuda|hip>
+#   install   .deps/level3/lammps/install    (+ .hpcperf-l3-fingerprint)
+#   logs      .deps/level3/lammps/logs
+#
+# Toolchain: conda GCC 13.3.0 as nvcc_wrapper host compiler (LAMMPS documents
+# GCC >= 8 and C++17), system CUDA, conda Open MPI 5.0.10 (CUDA-aware).
+# Packages: KOKKOS + MOLECULE, KSPACE (pppm/kk needs FFT_KOKKOS=CUFFT/HIPFFT),
+# MANYBODY, RIGID, GRANULAR -- the set the bench/ inputs need.
+#
+# Modification class: A (no upstream source modified; build flags only).
+#
+# Environment overrides:
+#   HPCPERF_CUDA_ARCH   numeric compute capability (default: detected; 100 -> Kokkos_ARCH_BLACKWELL100)
+#   HPCPERF_HIP_ARCH    Kokkos AMD arch name for HIP (default AMD_GFX950); HIP is UNTESTED here
+#   HPCPERF_BUILD_JOBS  parallel jobs (default 32)
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+# conda activation scripts are not set -u safe
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+l3_isolate_build_env    # Level 3 builds must not see Level 2 .deps/install prefixes
+
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+SRC="$R/_upstream/level3/lammps"
+[ -f "$SRC/cmake/CMakeLists.txt" ] || { echo "build.sh: LAMMPS source missing -- run $HERE/fetch.sh first" >&2; exit 1; }
+SHA="$(git -C "$SRC" rev-parse HEAD)"
+KOKKOS_VER="$(sed -n 's/^set(Kokkos_VERSION_\(MAJOR\|MINOR\|PATCH\) \([0-9]*\))/\2/p' "$SRC/lib/kokkos/CMakeLists.txt" | paste -sd.)"
+l3_paths lammps
+BUILD_DIR="$R/build/level3/lammps/$MODEL"
+JOBS="${HPCPERF_BUILD_JOBS:-32}"
+PKGS=(-DPKG_KOKKOS=yes -DPKG_MOLECULE=yes -DPKG_KSPACE=yes -DPKG_MANYBODY=yes -DPKG_RIGID=yes -DPKG_GRANULAR=yes)
+
+case "$BACKEND" in
+    CUDA)
+        ARCH="${HPCPERF_CUDA_ARCH:-$(l3_gpu_arch)}"
+        case "$ARCH" in
+            100) KARCH=BLACKWELL100;; 120) KARCH=BLACKWELL120;; 90) KARCH=HOPPER90;; 80) KARCH=AMPERE80;;
+            *) echo "build.sh: no Kokkos arch mapping for compute capability '$ARCH' (set HPCPERF_CUDA_ARCH)" >&2; exit 2;;
+        esac
+        export NVCC_WRAPPER_DEFAULT_COMPILER="$CXX"
+        GPU_FLAGS=(-DCMAKE_CXX_COMPILER="$SRC/lib/kokkos/bin/nvcc_wrapper"
+                   -DKokkos_ENABLE_CUDA=yes "-DKokkos_ARCH_$KARCH=yes" -DFFT_KOKKOS=CUFFT)
+        ARCHNOTE="sm_$ARCH ($KARCH)" ;;
+    HIP)
+        command -v hipcc >/dev/null 2>&1 || { echo "build.sh: HIP requested but hipcc not found -- HIP build is UNTESTED on this machine (no ROCm)" >&2; exit 1; }
+        KARCH="${HPCPERF_HIP_ARCH:-AMD_GFX950}"
+        GPU_FLAGS=(-DCMAKE_CXX_COMPILER=hipcc -DKokkos_ENABLE_HIP=yes "-DKokkos_ARCH_$KARCH=yes" -DFFT_KOKKOS=HIPFFT)
+        ARCHNOTE="$KARCH" ;;
+    *) echo "usage: $0 [CUDA|HIP]" >&2; exit 2 ;;
+esac
+
+# Image output libs are disabled: the node has a libjpeg runtime but no headers
+# (jpeglib.h), and dump image is not part of any benchmark here.
+CMAKE_OPTS="BUILD_MPI=yes BUILD_OMP=yes CXX_STANDARD=17 Kokkos_ENABLE_${BACKEND}=yes Kokkos_ARCH_${KARCH} Kokkos_ENABLE_OPENMP=yes Kokkos_ENABLE_SERIAL=yes FFT=KISS FFT_KOKKOS=${GPU_FLAGS[-1]#-DFFT_KOKKOS=} WITH_JPEG=no WITH_PNG=no PKGS=KOKKOS,MOLECULE,KSPACE,MANYBODY,RIGID,GRANULAR"
+FP="$(l3_fingerprint_text lammps "$SHA" "$MODEL" "kokkos(bundled)=$KOKKOS_VER" "$CMAKE_OPTS" "runtime(-pk kokkos gpu/aware)")"
+l3_fingerprint_check "$L3_INSTALL" "$FP" || exit 1
+
+echo "# LAMMPS $BACKEND: upstream $SHA, bundled Kokkos $KOKKOS_VER, arch $ARCHNOTE, MPI $(mpirun --version 2>/dev/null | head -1)"
+mkdir -p "$BUILD_DIR"
+cmake -S "$SRC/cmake" -B "$BUILD_DIR" -G Ninja \
+    -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="$L3_INSTALL" \
+    -DCMAKE_CXX_STANDARD=17 -DBUILD_MPI=yes -DBUILD_OMP=yes -DLAMMPS_MACHINE="kokkos_$MODEL" \
+    -DKokkos_ENABLE_OPENMP=yes -DKokkos_ENABLE_SERIAL=yes -DFFT=KISS \
+    -DWITH_JPEG=no -DWITH_PNG=no -DWITH_GZIP=yes \
+    "${GPU_FLAGS[@]}" "${PKGS[@]}" > "$L3_LOGS/configure-$MODEL.log" 2>&1 \
+    || { tail -30 "$L3_LOGS/configure-$MODEL.log"; echo "build.sh: configure failed (log: $L3_LOGS/configure-$MODEL.log)" >&2; exit 1; }
+t0=$(date +%s)
+cmake --build "$BUILD_DIR" -j "$JOBS" > "$L3_LOGS/build-$MODEL.log" 2>&1 \
+    || { tail -30 "$L3_LOGS/build-$MODEL.log"; echo "build.sh: build failed (log: $L3_LOGS/build-$MODEL.log)" >&2; exit 1; }
+cmake --install "$BUILD_DIR" > "$L3_LOGS/install-$MODEL.log" 2>&1 || { echo "build.sh: install failed" >&2; exit 1; }
+l3_fingerprint_write "$L3_INSTALL" "$FP"
+echo "# built in $(( $(date +%s)-t0 )) s: $BUILD_DIR/lmp_kokkos_$MODEL (installed under $L3_INSTALL)"
+echo "# compiler warning lines: $(grep -c 'warning' "$L3_LOGS/build-$MODEL.log" || true)"
diff --git a/level3/lammps/fetch.sh b/level3/lammps/fetch.sh
new file mode 100755
index 0000000..cc803c4
--- /dev/null
+++ b/level3/lammps/fetch.sh
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Fetch the LAMMPS upstream source at the recorded stable release into the
+# read-only reference checkout _upstream/level3/lammps (gitignored). Nothing is
+# built here. Re-running is idempotent; a checkout at a different commit is an
+# error (delete it to re-fetch), never silently reused.
+#
+#   ./fetch.sh
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+
+# Official repository and the selected stable release (see README.md).
+UPSTREAM_URL="https://github.com/lammps/lammps.git"
+UPSTREAM_TAG="stable_22Jul2025_update6"
+UPSTREAM_SHA="9c5ab448c78a14fd534619622162ba418d6a1fb1"
+DEST="$R/_upstream/level3/lammps"
+
+if [ -d "$DEST/.git" ]; then
+    have="$(git -C "$DEST" rev-parse HEAD)"
+    if [ "$have" = "$UPSTREAM_SHA" ]; then
+        echo "fetch.sh: $DEST already at $UPSTREAM_TAG ($UPSTREAM_SHA)"; exit 0
+    fi
+    echo "fetch.sh: $DEST is at $have, not the recorded $UPSTREAM_SHA ($UPSTREAM_TAG); remove it to re-fetch" >&2
+    exit 1
+fi
+mkdir -p "$(dirname "$DEST")"
+echo "fetch.sh: cloning $UPSTREAM_URL @ $UPSTREAM_TAG (shallow)"
+git clone --quiet --depth 1 --branch "$UPSTREAM_TAG" "$UPSTREAM_URL" "$DEST"
+have="$(git -C "$DEST" rev-parse HEAD)"
+[ "$have" = "$UPSTREAM_SHA" ] || { echo "fetch.sh: tag $UPSTREAM_TAG resolved to $have, expected $UPSTREAM_SHA" >&2; exit 1; }
+echo "fetch.sh: ok -> $DEST ($UPSTREAM_SHA)"
diff --git a/level3/lammps/run.sh b/level3/lammps/run.sh
new file mode 100755
index 0000000..ee03f3b
--- /dev/null
+++ b/level3/lammps/run.sh
@@ -0,0 +1,97 @@
+#!/usr/bin/env bash
+# Run the LAMMPS Lennard-Jones benchmark (upstream bench/in.lj) on N GPUs.
+#
+#   ./run.sh [CUDA|HIP] [extra lmp args...]
+#
+# Execution model (upstream Speed_kokkos): one MPI rank per GPU, KOKKOS
+# package on the device, one host thread per rank. Ranks are launched through
+# the common launcher with the per-rank GPU wrapper, so every rank sees exactly
+# one GPU and LAMMPS is started with `-k on g 1`; the launcher audits the
+# rank->GPU mapping. GPU-aware MPI (device-buffer halo exchange,
+# `-pk kokkos gpu/aware on`) is the LAMMPS default and matches the CUDA-aware
+# Open MPI of this repository; HPCPERF_LAMMPS_GPU_AWARE=off disables it.
+#
+# Resource / size controls (common Level 3 parameters):
+#   HPCPERF_GPUS=N|all        ranks = GPUs (default 1)
+#   HPCPERF_SCALE_MODE        smoke | strong | weak   (default smoke)
+#     smoke  : upstream bench/in.lj as shipped: 20^3 fcc cells = 32,000 atoms,
+#              100 steps; bring-up / correctness (reference log in bench/)
+#     strong : ONE fixed global box, (20*S)^3 cells with S=HPCPERF_LAMMPS_STRONG
+#              (default 8: 160^3 cells = 16,384,000 atoms), decomposed by
+#              LAMMPS over the ranks (any N is legal; LAMMPS factors the grid)
+#     weak   : fixed work per rank: (20*L)^3 cells per rank, L=HPCPERF_LAMMPS_LOCAL
+#              (default 4: 80^3 cells = 2,048,000 atoms/rank); the box is
+#              20*L*PX x 20*L*PY x 20*L*PZ with PXxPYxPZ from hpcperf_topology.py
+#              and the same grid is passed to LAMMPS `processors`
+#   HPCPERF_LAMMPS_STEPS      MD steps (default 100, the bench convention)
+#   HPCPERF_LAMMPS_GPU_AWARE  on|off (default on)
+#
+# LAMMPS decomposes the box into a PxQxR processor grid automatically for any
+# rank count; in weak mode the grid is set explicitly to match the box shape.
+# Extra args are appended to the lmp command line; arguments that would change
+# the validated problem (-in, -var x/y/z, -k, -sf, -pk) are rejected.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"; [ $# -gt 0 ] && shift
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+EXE="$R/build/level3/lammps/$MODEL/lmp_kokkos_$MODEL"
+[ -x "$EXE" ] || { echo "run.sh: $EXE not found -- run ./build.sh $BACKEND first" >&2; exit 1; }
+SRC="$R/_upstream/level3/lammps"
+
+N_RANKS="$(hpcperf_ranks lammps yes)" || exit 2
+hpcperf_forbid_args lammps -in -i -var -v -k -kokkos -sf -suffix -pk -package -log -- "$@" || exit 2
+MODE="$(l3_scale_mode lammps)" || exit 2
+STEPS="${HPCPERF_LAMMPS_STEPS:-100}"
+GAM="${HPCPERF_LAMMPS_GPU_AWARE:-on}"
+
+PROCS=()
+case "$MODE" in
+    smoke)  X=1; Y=1; Z=1 ;;
+    strong) S="${HPCPERF_LAMMPS_STRONG:-8}"; X=$S; Y=$S; Z=$S ;;
+    weak)   L="${HPCPERF_LAMMPS_LOCAL:-4}"
+            TOPO="$(hpcperf_topology lammps "$N_RANKS")" || exit 2
+            read -r PX PY PZ <<< "$TOPO"
+            X=$((L * PX)); Y=$((L * PY)); Z=$((L * PZ))
+            PROCS=(-var px "$PX" -var py "$PY" -var pz "$PZ") ;;
+esac
+ATOMS=$(( 4 * 20 * X * 20 * Y * 20 * Z ))
+RUN_DIR="$R/build/level3/lammps/$MODEL/$L3_RUN_SUBDIR"
+# A dry-run must never touch real results: it writes its derived deck and would-be
+# log into a throwaway .dryrun/ subdir instead of the real run directory.
+[ -n "${HPCPERF_DRY_RUN:-}" ] && RUN_DIR="$RUN_DIR/.dryrun"
+mkdir -p "$RUN_DIR"
+LOG="$RUN_DIR/log.$MODE.np$N_RANKS.lammps"
+rm -f "$LOG"      # validate only against THIS run's output; never a stale log left by a failed run
+# Derived deck (upstream bench/in.lj untouched): `run 100` -> `run ${steps}`,
+# and in weak mode a `processors ${px} ${py} ${pz}` line before create_box so
+# the rank grid matches the box shape. With steps=100 and no processors line
+# the derived deck is semantically identical to upstream's.
+IN="$RUN_DIR/in.lj.$MODE"
+{
+    if [ "${#PROCS[@]}" -gt 0 ]; then
+        sed -e 's/^create_box.*/processors      ${px} ${py} ${pz}\n&/' -e 's/^run[[:space:]].*/run             ${steps}/' "$SRC/bench/in.lj"
+    else
+        sed -e 's/^run[[:space:]].*/run             ${steps}/' "$SRC/bench/in.lj"
+    fi
+} > "$IN"
+
+echo "# LAMMPS $BACKEND: mode=$MODE ranks=$N_RANKS box=$((20*X))x$((20*Y))x$((20*Z)) fcc cells = $ATOMS atoms ($((ATOMS / N_RANKS))/rank), $STEPS steps, gpu-aware=$GAM, log=$LOG"
+RUN_ID="$(l3_run_id)"
+rc=0
+"$L3_LAUNCHER" --gpus "$N_RANKS" --bind wrapper -- \
+    "$EXE" -k on g 1 t "${HPCPERF_CPUS_PER_RANK:-1}" -sf kk -pk kokkos newton on neigh half gpu/aware "$GAM" \
+    -in "$IN" -var x "$X" -var y "$Y" -var z "$Z" "${PROCS[@]}" -var steps "$STEPS" \
+    -log "$LOG" -echo none "$@" || rc=$?
+if [ -z "${HPCPERF_DRY_RUN:-}" ]; then
+    l3_manifest "$RUN_DIR" "run_id=$RUN_ID" "app=lammps" "backend=$BACKEND" "mode=$MODE" \
+        "ranks=$N_RANKS" "atoms=$ATOMS" "steps=$STEPS" "gpu_aware=$GAM" "exit_code=$rc" \
+        "binary=$EXE" "binary_sha256=$(l3_sha_file "$EXE")" "input=$IN" "input_sha256=$(l3_sha_file "$IN")" \
+        "log=$LOG" "utc=$(date -u +%FT%TZ)"
+fi
+exit "$rc"
diff --git a/level3/lammps/validate.sh b/level3/lammps/validate.sh
new file mode 100755
index 0000000..61c1bc1
--- /dev/null
+++ b/level3/lammps/validate.sh
@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+# Correctness check for the LAMMPS Kokkos build (upstream mechanism: compare
+# the thermodynamic output of bench/in.lj with the reference log LAMMPS ships,
+# bench/log.15Jul25.lj.fixed.g++.1, a CPU run of the same 32,000-atom, 100-step
+# problem; `velocity ... loop geom` makes the initial state machine- and
+# rank-count-independent).
+#
+#   ./validate.sh [CUDA|HIP]        HPCPERF_GPUS=N (default 1) selects the rank count
+#
+# What is compared (Step 0 and Step 100 rows: Temp, E_pair, TotEng, Press):
+#   Step 0  : relative tolerance 1e-8 -- deterministic initial state.
+#   Step 100: relative tolerance 1e-5 -- reduction-order divergence only.
+# With HPCPERF_GPUS>1 the N-rank result is also compared with this build's 1-GPU
+# result (rank-count independence). Adapted subset of upstream's regression
+# check: upstream's tools/regression-tests/run_tests.py compares every thermo
+# column of every logged step with per-quantity tolerances; here the four
+# state variables at the first and last step are checked (the quantities that
+# move if the physics or force summation is wrong). No tolerance is loosened.
+#
+# Reproducibility rules enforced here:
+#   * the run's real exit code is captured; a nonzero exit, a timeout, a missing
+#     log or a non-finite value is a FAIL (never a PASS on a stale log);
+#   * run.sh removes its target log before launching, so only THIS run's output
+#     is validated;
+#   * NaN/Inf in any compared quantity is rejected explicitly (l3_check).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+N="${HPCPERF_GPUS:-1}"
+REF="$R/_upstream/level3/lammps/bench/log.15Jul25.lj.fixed.g++.1"
+RUN_DIR="$R/build/level3/lammps/$MODEL/$L3_RUN_SUBDIR"
+TIMEOUT="${HPCPERF_VALIDATE_TIMEOUT:-900}"
+[ -f "$REF" ] || { echo "validate.sh: reference log $REF missing (run fetch.sh)" >&2; exit 1; }
+
+unset HPCPERF_SCALE_MODE
+export HPCPERF_GPUS="$N"
+echo "validate.sh: LAMMPS $BACKEND smoke (bench/in.lj, 32000 atoms, 100 steps) on $N GPU(s)"
+
+# run once, on N GPUs, capturing the real exit code (no pipe swallows it)
+run_once() { # <ngpu> <stdout-file>
+    local ng=$1 out=$2 rc=0
+    HPCPERF_GPUS="$ng" HPCPERF_SCALE_MODE=smoke timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" > "$out" 2>&1 || rc=$?
+    return $rc
+}
+VOUT="$RUN_DIR/validate.smoke.np$N.stdout"; mkdir -p "$RUN_DIR"
+rc=0; run_once "$N" "$VOUT" || rc=$?
+grep -aE '^#|hpcperf-launch: audit summary|Loop time|ERROR|abort' "$VOUT" || true
+if [ "$rc" -eq 124 ]; then echo "validate.sh: FAIL -- run timed out after ${TIMEOUT}s"; exit 1; fi
+[ "$rc" -eq 0 ] || { echo "validate.sh: FAIL -- run.sh exited $rc (see $VOUT)"; exit 1; }
+LOG="$RUN_DIR/log.smoke.np$N.lammps"
+[ -f "$LOG" ] || { echo "validate.sh: FAIL -- no log produced ($LOG)"; exit 1; }
+if [ "$N" -gt 1 ] && [ ! -f "$RUN_DIR/log.smoke.np1.lammps" ]; then
+    echo "validate.sh: producing the 1-GPU reference run for rank-count comparison"
+    r1=0; run_once 1 "$RUN_DIR/validate.smoke.np1.stdout" || r1=$?
+    [ "$r1" -eq 0 ] && [ -f "$RUN_DIR/log.smoke.np1.lammps" ] || { echo "validate.sh: FAIL -- 1-GPU reference run failed (rc=$r1)"; exit 1; }
+fi
+
+python3 - "$LOG" "$REF" "$N" "$RUN_DIR/log.smoke.np1.lammps" <<'PY'
+import re, sys, os
+sys.path.insert(0, os.environ["L3_TOOLS"])
+from l3_check import require_finite, ValidationError
+def thermo(path):
+    rows = {}
+    lines = open(path).read().splitlines()
+    for i, ln in enumerate(lines):
+        if ln.split()[:2] == ["Step", "Temp"]:
+            cols = ln.split()
+            for row in lines[i+1:]:
+                p = row.split()
+                if not p or not re.match(r'^\d+$', p[0]): break
+                rows[int(p[0])] = dict(zip(cols[1:], p[1:]))
+    return rows
+log, ref, n, log1 = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
+keys = ["Temp", "E_pair", "TotEng", "Press"]; tol = {0: 1e-8, 100: 1e-5}
+try:
+    got, want = thermo(log), thermo(ref)
+    for step in tol:
+        if step not in got:  raise ValidationError(f"run log has no Step {step} row (got {sorted(got)}); run did not complete")
+        if step not in want: raise ValidationError(f"reference log has no Step {step} row")
+        for k in keys:
+            if k not in got[step]:  raise ValidationError(f"field '{k}' missing at step {step} in run log")
+    ok = True
+    def cmp(a, b, label):
+        global ok
+        for step, t in tol.items():
+            for k in keys:
+                x = require_finite(f"{label} step{step} {k} (got)", a[step][k])
+                y = require_finite(f"{label} step{step} {k} (ref)", b[step][k])
+                rel = abs(x - y) / max(abs(y), 1e-30)
+                if rel > t: ok = False
+                print(f"  {label}: step {step:>3} {k:<7} got {x: .10g} ref {y: .10g} rel {rel:.2e} (tol {t:.0e}) {'ok ' if rel <= t else 'BAD'}")
+    print(f"[1] {n}-GPU run vs upstream CPU reference log (adapted subset: 4 state vars at step 0 and 100):")
+    cmp(got, want, "vs-ref")
+    if n > 1:
+        g1 = thermo(log1)
+        for step in tol:
+            if step not in g1: raise ValidationError(f"1-GPU log has no Step {step} row")
+        print(f"[2] {n}-GPU run vs this build's 1-GPU run (rank-count independence):")
+        cmp(got, g1, "vs-1gpu")
+except ValidationError as ex:
+    print(f"  VALIDATION ERROR: {ex}")
+    print(f"LAMMPS {sys.argv and ''}validation ({n} GPU): FAIL"); sys.exit(1)
+print(f"LAMMPS CUDA validation ({n} GPU, bench/in.lj vs log.15Jul25.lj.fixed.g++.1): {'PASS' if ok else 'FAIL'}")
+sys.exit(0 if ok else 1)
+PY
diff --git a/level3/nekrs/COMPATIBILITY.md b/level3/nekrs/COMPATIBILITY.md
new file mode 100644
index 0000000..2a5f46b
--- /dev/null
+++ b/level3/nekrs/COMPATIBILITY.md
@@ -0,0 +1,200 @@
+# nekRS -- CUDA / dependency compatibility (v26.0 on dgx003, B200, CUDA 13.2.78)
+
+Scope: the exact software *combination* used for the Level 3 nekRS bring-up, and
+what is and is not verified about it. This document separates four things the
+review asked never to be conflated:
+
+1. what upstream **documents**;
+2. what upstream **CI / site testing** actually exercises;
+3. the **local build** result for this exact combination;
+4. the **local run** result for this exact case.
+
+"Verified" below means *observed on this node*, never "endorsed by upstream".
+Where no upstream evidence was found, the entry is `NOT_FOUND` / `UNVERIFIED` --
+not "upstream forbids it".
+
+Upstream references consulted (read at the pinned commit / current docs):
+`github.com/Nek5000/nekRS` and `/releases`; the pinned
+`CMakeLists.txt` and `cmake/hypre.cmake` at
+`96b3cf9e5bacede16568826c04a21bc0fe50dc7d`; `nekrs.readthedocs.io/quickstart`;
+`Nek5000/nekRS_HPCsupport`; `hypre-space/hypre`; the CCCL 3.0 migration guide
+(`nvidia.github.io/cccl/.../3.0_migration_guide.html`); the Blackwell
+compatibility guide (`docs.nvidia.com/cuda/blackwell-compatibility-guide`).
+
+## The combination
+
+| Component | Value |
+|---|---|
+| nekRS | v26.0, commit `96b3cf9e5bacede16568826c04a21bc0fe50dc7d` |
+| vendored HYPRE | 2.32.0 (`3rd_party/hypre`, squashed subtree) |
+| vendored OCCA | 2.0.0-dev (`OCCA_VERSION_STR`) |
+| CUDA toolkit | 13.2.78 (`/usr/local/cuda` -> cuda-13.2); Thrust/CCCL 3.2 (`THRUST_VERSION 300200`) |
+| GPU / arch | B200, sm_100 |
+| host compiler | conda GCC 13.3.0 (C/C++), system gfortran 14.2.1 (Fortran) |
+| MPI | conda Open MPI 5.0.10 (CUDA-aware) |
+| default build option | `OCCA_ENABLE_CUDA=ON`, `ENABLE_HYPRE_GPU=ON` (variant `hypregpu`) |
+
+## `cmake/hypre.cmake` at this commit already has a CUDA >= 13 branch
+
+The pinned `cmake/hypre.cmake` contains:
+
+```
+if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL "13.0.0")
+    set(HYPRE_DEVICE_ARCH "HYPRE_CUDA_SM=80 90")
+elseif(CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.0.0")
+    set(HYPRE_DEVICE_ARCH "HYPRE_CUDA_SM=70 80")
+endif()
+```
+
+So nekRS v26.0 **is aware of CUDA 13** (a dedicated branch exists) -- it is wrong
+to say "nekRS does not support CUDA 13". Equally, the branch only sets the SASS
+list to `80 90` (no sm_100, no PTX) and the presence of a branch is **not**
+evidence that CUDA 13.2.78 + B200 + vendored HYPRE 2.32.0 passed any upstream CI:
+upstream CI (`.github/workflows/ci.yml`) builds the serial/CPU backend with
+MPICH + gfortran on ubuntu; the nekRS_HPCsupport machine files target
+Frontier/Perlmutter/Polaris/etc., none of them a B200/CUDA-13 combination
+(NOT_FOUND for this exact stack).
+
+## Four problem classes, separated
+
+### A. HYPRE GPU source vs CCCL/Thrust 3.2 API  (compile, GPU-only code)
+
+`ENABLE_HYPRE_GPU=ON` compiles HYPRE's device sources with nvcc against CUDA 13's
+Thrust 3.2. Three incompatibilities were hit and are addressed by the three
+patches (all confined to HYPRE's *device* code / device headers):
+
+| Symptom (build error) | Cause | Real API change? | Patch |
+|---|---|---|---|
+| `namespace "thrust" has no member "make_reverse_iterator"` / `"pair"` | `<thrust/iterator/reverse_iterator.h>` and `<thrust/pair.h>` are no longer pulled in transitively by other thrust headers in CCCL 3.x | **No** -- the names still exist (`thrust::make_reverse_iterator`/`reverse_iterator` are re-exported from `cuda::std` in `thrust/iterator/reverse_iterator.h`); they were merely not *visible* without an explicit include | 0003 (add the includes to `device_utils.h` and the pre-generated `_hypre_utilities.hpp`) |
+| `thrust::pair<...>` as a declared result type of `reduce_by_key` | same non-transitive-include issue at the point of use | **No** (visibility) -- resolved either by the include or by letting `auto` take the library's real return type | 0002 (`auto`) + 0003 (include) |
+| `thrust::not1` undefined | `thrust::not1` (C++17-deprecated) was **removed** in CCCL 3.x | **Yes -- genuinely removed**; documented replacement is `thrust::not_fn` (= `cuda::std::not_fn`) | 0003 (`not1` -> `not_fn`, 16 sites) |
+
+So only `not1` is a true API removal; the `pair`/`reverse_iterator` errors are
+missing explicit includes, not deleted APIs. This distinction is recorded per the
+review's instruction.
+
+Upstream fix status: **UNVERIFIED / NOT looked up in detail.** HYPRE's own master
+is understood to build under CUDA 13, and the CCCL migration guide documents the
+`not1 -> not_fn` change, but no specific HYPRE commit/PR or nekRS-vendored-HYPRE
+bump was located and pinned. The patches are therefore labelled **project-local
+compatibility patches**, not backports of a known upstream commit. Whether nekRS
+`next` already vendors a CUDA-13-ready HYPRE was not checked (route A, below).
+
+### B. HYPRE prebuilt device library arch list (sm_100 coverage)
+
+Patch 0001 changes `HYPRE_CUDA_SM=80 90` to `80 90 100` so the device library
+carries Blackwell SASS. Without it HYPRE's device kernels would have no sm_100
+code (and the branch emits no PTX to JIT from). This matters **only when GPU
+HYPRE is actually used** (see the coarse-solver finding below).
+
+### C. OCCA CUDA / JIT
+
+OCCA derives `-arch=sm_<device>` at run time and JIT-compiles the OKL kernels
+with the host nvcc; no patch needed. Verified locally: `active occa mode: CUDA`,
+kernels compiled, the main solver runs on the GPU (the launcher's per-rank GPU
+audit reports the ranks on distinct GPUs).
+
+### D. MPI one-sided (OSC) UCX runtime failure
+
+Independent of the above: nekRS uses `MPI_Win_lock`, and Open MPI's default
+`osc ucx` aborts in `uct_ib` at 4 ranks on this node. Worked around at run time
+with `OMPI_MCA_osc=^ucx` in `run.sh` (nekRS-local; not a global default). This is
+a transport/runtime issue, not a CUDA/HYPRE compatibility issue.
+
+Fixing any one of A/B/C/D does not fix the others; they are tracked separately.
+
+## The load-bearing finding: the benchmark case runs the HYPRE coarse solve on the CPU
+
+For the ethier case (the only nekRS case wired up), the six recorded logs
+(`smoke.np{1,2,4}.cimode2`, `strong.np{1,4}`, `weak.np4`) all show:
+
+```
+FLUID PRESSURE MULTIGRID COARSE SOLVER: BOOMERAMG
+FLUID PRESSURE MULTIGRID COARSE SOLVER LOCATION: CPU
+FLUID PRESSURE MULTIGRID COARSE SOLVER PRECISION: FP32
+```
+
+Cross-checked against the source, not just the logs:
+- `src/platform/par/parsePreconditioner.hpp` sets the default
+  `... MULTIGRID COARSE SOLVER LOCATION = CPU` and `... PRECISION = FP32`;
+- `src/core/elliptic/elliptic.cpp` sets `... COARSE SOLVER LOCATION = CPU`;
+- `examples/ethier/ethier.par` does **not** set a coarse-solver location, and
+  `--cimode 2` (`examples/ethier/ci.inc`) does **not** set one either -> the
+  default (CPU) applies;
+- only `--cimode 3` sets `FLUID PRESSURE PRECONDITIONER = MULTIGRID+SEMFEM` and
+  `... COARSE SOLVER LOCATION = DEVICE` (falling back to CPU only when the run is
+  serial / CPU-backend).
+
+Consequences (stated precisely):
+- The nekRS **main application does run on the GPU** (OCCA CUDA) -- this is NOT a
+  CPU-only application, and multi-GPU main-solve is real (per-rank GPU audit).
+- The HYPRE **BoomerAMG coarse solve for the ethier cimode-2 case runs on the
+  CPU**. The GPU HYPRE device library built by `ENABLE_HYPRE_GPU=ON` is compiled
+  but **not exercised** by this case.
+- Therefore the earlier "9/9 CI checks passed" for cimode 2 validated *the CUDA
+  main application + a CPU coarse solve*. It did **not** validate GPU HYPRE. A
+  case that selects `COARSE SOLVER LOCATION = DEVICE` (cimode 3) is required to
+  exercise GPU HYPRE on the target GPU; that verification is tracked in the
+  matrix below and must not be claimed from the cimode-2 result.
+
+("Library compiled/loaded" is not "kernel executed" -- the judgement here is from
+the solver-configuration lines the run prints and the source defaults, not from a
+source grep alone.)
+
+## Compatibility / verification matrix
+
+| variant | ENABLE_HYPRE_GPU | patches | cimode | runtime coarse | build (dgx003, CUDA 13.2.78, sm_100) | main app CUDA multi-GPU | GPU HYPRE coarse | official CI for this stack |
+|---|---|---|---|---|---|---|---|---|
+| hypregpu (default) | ON | 0001+0002+0003 | 2 | CPU | ok (~30 min, prior round) | VERIFIED 1/2/4 GPU, 9/9 | not exercised (CPU coarse) | NOT_FOUND |
+| hypregpu | ON | 0001+0002+0003 | 3 | DEVICE (GPU) | (same install) | VERIFIED 1/4 GPU, 9/9 | **VERIFIED** 1/4 GPU (9/9, coarse=DEVICE) | NOT_FOUND |
+| cpucoarse (candidate) | OFF | **none** | 2 | CPU | ok, **113 s, 0 patches** | VERIFIED 1/2/4 GPU, 9/9 | not built | n/a |
+| cpucoarse | OFF | none | 3 | (DEVICE requested) | (same install) | n/a | **explicitly rejected** -- nekRS aborts: `HYPRE+DEVICE not enabled! Recompile with -DENABLE_HYPRE_GPU=ON` (exit 1); NO silent CPU fallback | n/a |
+
+Results (dgx003, 2026-09-05), all through the common launcher (one rank per GPU,
+per-rank GPU wrapper, audit verified on distinct GPUs), analytic Ethier solution,
+EPS 0.3, stricter validator (real exit code, complete 9/9 check set, coarse
+location asserted, NaN/Inf rejected):
+
+- hypregpu cimode 2: PASS 1/2/4 GPU, coarse=CPU.
+- hypregpu cimode 3: PASS 1/4 GPU, coarse=DEVICE -> **the GPU HYPRE coarse solve
+  enabled by the three patches is verified correct on B200/CUDA 13.2**, not just
+  compiled.
+- cpucoarse cimode 2: PASS 1/2/4 GPU, coarse=CPU, main app on distinct GPUs ->
+  the entire current Ethier workload runs correctly with GPU HYPRE OFF and NO
+  patches (build 113 s vs the ~30 min patched GPU-HYPRE build).
+- cpucoarse cimode 3: nekRS's own `HYPRE+DEVICE not enabled!` check aborts the
+  run (exit 1); the validator FAILs (coarse != DEVICE). No silent downgrade.
+
+Finding: the three patches are needed ONLY to build/exercise GPU HYPRE. For the
+current CPU-coarse Ethier workload the `cpucoarse` variant is a smaller,
+patch-free, faster-to-build configuration that runs the same GPU main
+application. The `hypregpu` variant remains the one that can run (and is verified
+to run) the GPU coarse solve.
+
+## What is and is not claimed for the default (hypregpu) variant
+
+- CLAIMED (verified locally): CUDA main application, multi-GPU (1/2/4), on
+  distinct GPUs; ethier cimode-2 correctness (analytic solution, 9/9 checks) with
+  a **CPU** HYPRE coarse solve.
+- NOT CLAIMED: that GPU HYPRE coarse solve is correct (cimode 2 does not run it);
+  that the CUDA 13.2.78 + B200 + HYPRE 2.32.0 combination is upstream-CI-certified
+  (NOT_FOUND); that CPU-coarse is the best configuration at 40/80 GPUs
+  (UNVERIFIED -- CPU coarse scalability is a separate, unmeasured question).
+
+## Routes if GPU HYPRE coarse is actually required (audited, not all built)
+
+- **A -- upstream integration**: pin a newer nekRS release / a fixed
+  master/next commit / an upstream HYPRE bump that resolves CCCL-3 + sm_100.
+  STATUS: NOT_RUN (not audited to a specific SHA this round).
+- **B -- keep CUDA 13.2 + HYPRE 2.32.0 + the local patches** as a
+  project-maintained compatibility variant, and verify GPU coarse with cimode 3.
+  STATUS: the cimode-3 verification is attempted this round (matrix).
+- **C -- an earlier B200-capable CUDA (12.8/12.9) for nekRS only**: would still
+  need HYPRE_CUDA_SM to include sm_100 and re-verification; does not
+  automatically avoid the arch-target问题. Must not touch the system CUDA symlink
+  or the other four apps. STATUS: NOT_RUN.
+- **D -- replace the vendored HYPRE**: dependency-upgrade experiment; needs
+  host/device wrapper, single/mixedint, precision, HYPRE_Int/BigInt and link
+  checks. Not to be recorded as an LLM/source optimisation. STATUS: NOT_RUN.
+
+Routes A/C/D are UNVERIFIED this round by design (no extra large builds started).
diff --git a/level3/nekrs/README.md b/level3/nekrs/README.md
new file mode 100644
index 0000000..b2bffd5
--- /dev/null
+++ b/level3/nekrs/README.md
@@ -0,0 +1,188 @@
+# nekRS (Level 3)
+
+Spectral-element incompressible Navier-Stokes (pressure Poisson with p-multigrid
++ HYPRE coarse grid, velocity Helmholtz solves, subcycled advection, passive
+scalars, gather-scatter halo exchange) -- the complete solver driven by its own
+case files, with all GPU kernels JIT-compiled by OCCA at run time.
+
+## Provenance
+
+- Official repository: https://github.com/Nek5000/nekRS (`master` = latest
+  stable release); docs https://nekrs.readthedocs.io/
+- Release policy: tagged releases on `master` (v26.0 2026-01-27; previous
+  v23.0 2023-05); `next` is the preview branch.
+- Selected: **v26.0**, commit `96b3cf9e5bacede16568826c04a21bc0fe50dc7d`,
+  fetched by `fetch.sh` into `_upstream/level3/nekRS` (shallow, read-only).
+- License: BSD-3-Clause.
+- Application-owned LOC (cloc 2.06, code lines): `src/` **53,131** (C++
+  35,784; headers 12,322; C 3,782; Fortran 1,104). Vendored third-party
+  libraries in `3rd_party/` are counted separately (~2.27 M: LAPACK 829,795,
+  ADIOS2 618,827, HYPRE 445,492, CVODE 169,316, OCCA 89,888, Nek5000 87,143,
+  gslib 13,038, parRSB 6,450).
+
+## Build strategy: NATIVE (upstream CMake, vendored libraries)
+
+`build.sh CUDA` = upstream's CMake route (`build.sh` upstream is only a
+wrapper with interactive prompts): `CC=mpicc CXX=mpicxx FC=mpif90 cmake -G
+"Unix Makefiles" -DOCCA_ENABLE_CUDA=ON -DOCCA_ENABLE_HIP=OFF
+-DOCCA_ENABLE_DPCPP=OFF -DENABLE_HYPRE_GPU=ON -DENABLE_ADIOS=OFF
+-DENABLE_CVODE=OFF -DNEKRS_BUILD_FLOAT=OFF`, install = `NEKRS_HOME` =
+`.deps/level3/nekrs/install` (with `nekrs.conf` recording the JIT toolchain:
+`OCCA_CXX` = conda g++ 13.3.0, `OCCA_CUDA_COMPILER_FLAGS = -w -O3 -lineinfo
+--use_fast_math`, `NEKRS_GPU_MPI = 0`). Toolchain: conda GCC 13.3.0 through the
+conda Open MPI 5.0.10 wrappers for C/C++, system gfortran 14.2.1 through
+`mpif90` (`OMPI_FC`) for the Nek5000 interface and the vendored LAPACK, CUDA
+13.2.78 for HYPRE's device build and for the run-time JIT. OKL kernels are
+compiled for the device found at run time (sm_100 here); HYPRE's device
+kernels are compiled for sm_80/90/100 (patch below). Build time: the single
+full compile of all vendored libraries + nekRS took ~30 min at `-j32` (its
+exact wall time was not captured because the first complete pass failed at the
+install step; the final incremental install took 27 s); **0 compiler warning
+lines** in the final pass. Fingerprint: upstream commit, vendored library
+versions, compilers, CUDA 13.2.78, MPI, CMake options, patch list.
+
+Why not the others: the Spack `nekrs` recipe is stale (23.0, option names that
+no longer exist); all dependencies are vendored, so Spack would add nothing;
+no Apptainer on the node and no upstream image; JIT couples nekRS to the host
+compiler/nvcc anyway; site modules broken. HIP: `build.sh HIP` selects
+`OCCA_ENABLE_HIP` and exits with a clear message here (no ROCm). **Untested.**
+
+## Changes from upstream (all recorded in `patches/` and `build.sh`)
+
+| Class | Change | Size / reason |
+|---|---|---|
+| B | `0001-hypre-cuda-sm100.patch`: `cmake/hypre.cmake` `HYPRE_CUDA_SM=80 90` -> `80 90 100` for CUDA >= 13 | 1 line; upstream lists no Blackwell SASS (SASS only, no PTX) |
+| D | `0002-hypre-cuda13-thrust-pair.patch`: vendored HYPRE 2.32.0 declares `thrust::reduce_by_key` results as `thrust::pair<...>`, a name the Thrust 3.2 of CUDA 13 no longer exposes there -> `auto` | 2 lines |
+| D | `0003-hypre-cuda13-thrust3-compat.patch`: explicit `<thrust/iterator/reverse_iterator.h>` / `<thrust/pair.h>` includes (no longer transitive) in HYPRE's `device_utils.h` **and** in the pre-generated concatenated `_hypre_utilities.hpp` the sources include; `thrust::not1` (removed) -> its documented replacement `thrust::not_fn` (16 uses) | 36 lines, mechanical |
+| B | CMake generator pinned to upstream's Unix Makefiles (`build.sh`): the conda environment exports `CMAKE_GENERATOR=Ninja`, under which HYPRE's ExternalProject install rule (`$(MAKE) install`) is invalid | build.sh only |
+| C | `OMPI_FC=/usr/bin/gfortran` (no conda gfortran); `LDFLAGS += -fno-lto` (CMake's FortranCInterface probe compiles with `-flto -ffat-lto-objects`; GCC 13 bytecode vs the gfortran 14 link driver); `FFLAGS += -fPIC` (conda GCC links PIE); `unset AR` (HYPRE's configure takes `$AR` as the full archive command; conda exports the bare tool); run time: `unset CMAKE_GENERATOR` (UDF build), `OMPI_MCA_osc=^ucx` (see execution model), `ulimit -s unlimited` (as upstream's `nrsqsub_utils`: the Nek5000 side keeps lx1^3*lelt work arrays on the stack; the h-refined cases segfault in `useric` at the default 8 MB) | environment |
+
+No numerics changed. HYPRE 2.32.0 + CUDA 13 is not an upstream-validated
+combination; the patches only restore names the CCCL removed.
+
+## Execution model
+
+One MPI rank per GPU (upstream: "NekRS binds 1 GPU to 1 MPI rank"). nekRS'
+default is `--device-id LOCAL-RANK`; under the common launcher's per-rank
+wrapper each rank sees exactly one GPU, so `run.sh` passes `--device-id 0` and
+the launcher audits the mapping (4/4 verified). GPU-aware MPI is upstream's
+default OFF (`NEKRS_GPU_MPI=0`; RELEASE.md warns enabling it "may cause a
+performance regression"); `HPCPERF_NEKRS_GPU_MPI=1` turns it on. nekRS uses
+MPI one-sided operations (`MPI_Win_lock`): with the default Open MPI selection
+these went through `osc ucx` -> UCX/InfiniBand even on one node and aborted
+in `uct_ib` with 4 ranks ("'abort' is not implemented for protocol
+amo64/fetch"); `run.sh` sets `OMPI_MCA_osc=^ucx`, the one-sided counterpart of
+the site profile's `pml ob1 / btl self,sm,smcuda`, and 4 ranks then pass
+(candidate for the gmu-hopper site profile). Rank count is unconstrained
+(parRSB graph partitioning). The first run of a new kernel set JIT-compiles
+OKL kernels with nvcc into `build/level3/nekrs/cuda/cache` (minutes; shared by
+later runs), and the case's `.usr` file is compiled with `mpif90` at run time.
+
+## Inputs (`HPCPERF_SCALE_MODE`, case `examples/ethier`)
+
+| Mode | Elements | Order N | Grid points | Per rank @4 GPU | Steps | GPU memory (est.) | Validation quantity |
+|---|---|---|---|---|---|---|---|
+| smoke (default) | 32 (upstream `ethier.par`) | 9 | 32,000 | 8 elements | 100 (CI mode: 30) | < 0.1 GB | upstream `--cimode 2` CI checks (analytic solution) |
+| strong | 32 x H^3, H=`HPCPERF_NEKRS_HREFINE` (10): 32,000 (`ethierRefine.par`, `hrefine`) | 7 | 16.4 M | 8,000 | 100 | ~2 GB | run completes; L2 errors vs exact solution printed every step |
+| weak | 32 x H^3 with H = round(cbrt(250 N)) -> ~8,000 elements/rank (upstream's reference load, kershaw README "E/GPU=8000") | 7 | 4.1 M x N | 6.9k-8.5k (integer H) | 100 | ~2 GB | run completes |
+
+Memory estimate ~60 KB per element at N=7 (velocity, pressure, two scalars,
+multistep history, preconditioner). Derived `ethier.par` files change only
+`hrefine` and `numSteps` (class A).
+
+## Coarse-solver location: what runs on the GPU (read `COMPATIBILITY.md`)
+
+The ethier case's HYPRE BoomerAMG coarse solve runs where the cimode selects:
+
+- `--cimode 2` (and the default `.par`): `FLUID PRESSURE MULTIGRID COARSE SOLVER
+  LOCATION = CPU` -- the coarse solve is on the **host**. The nekRS main
+  application (advection, Helmholtz, pressure pMG smoother, gather-scatter) runs
+  on the **GPU** via OCCA/CUDA. So this is not a CPU-only application, but a
+  cimode-2 PASS does **not** exercise GPU HYPRE.
+- `--cimode 3`: `FLUID PRESSURE PRECONDITIONER = MULTIGRID+SEMFEM`, `COARSE
+  SOLVER LOCATION = DEVICE` -- the coarse solve runs on the **GPU** (this is the
+  mode that exercises the GPU HYPRE built by `ENABLE_HYPRE_GPU=ON` and the three
+  patches).
+
+Two build variants exist (isolated src/build/install/JIT-cache; select with
+`HPCPERF_NEKRS_HYPRE_GPU`/`HPCPERF_NEKRS_VARIANT`):
+
+| variant | `ENABLE_HYPRE_GPU` | patches | GPU HYPRE coarse (cimode 3) | CPU coarse (cimode 2) |
+|---|---|---|---|---|
+| `hypregpu` (default) | ON | 0001+0002+0003 | built + **verified** (cimode 3, 1/4 GPU, 9/9, coarse=DEVICE) | verified 1/2/4 |
+| `cpucoarse` (candidate) | OFF | none | not built; a DEVICE request is **explicitly rejected** by nekRS (`HYPRE+DEVICE not enabled!`, exit 1), no silent fallback | verified 1/2/4 (candidate, 0 patches, 113 s build) |
+
+Build/run isolation per variant: `hypregpu` keeps the legacy paths
+(`.deps/level3/nekrs/{src,install}`, `build/level3/nekrs/cuda`); other variants
+use `.deps/level3/nekrs/<variant>/{src,install}` and `build/level3/nekrs/<variant>.<backend>`
+with their own OCCA/nekRS JIT cache. The source-copy cache key is the upstream
+SHA plus the ordered patch-content hash, so the two variants never share a
+patched/unpatched tree.
+
+Which to make default is a decision for review: `cpucoarse` is minimal (no
+patches, fast build) and covers the current CPU-coarse workload; `hypregpu` is
+required if a case selects GPU (DEVICE) coarse. CPU-coarse scalability at 40/80
+GPUs is UNVERIFIED and is not claimed to be optimal at all scales.
+
+## Validation (`validate.sh`, upstream mechanism)
+
+`nekrs --setup ethier --cimode 2` is one of the modes upstream's CI runs on this
+case (`.github/workflows/ci.yml`): it fixes the solver settings (velocity
+solver +BLOCK, subcycling 1, tolerances 1e-12/1e-10, 30 steps) and at the last
+step checks the L2 errors of velocity, pressure and both scalars against the
+exact Ethier-Steinman solution (references in `examples/ethier/ci.inc`,
+relative tolerance EPS = 0.3) plus the iteration counts of the pressure,
+velocity and scalar solves. nekRS prints `CI test <...> passed|failed` per
+check and exits non-zero on failure; `validate.sh` uses that verdict unchanged
+(upstream runs it on CPUs with 2 ranks; here the CUDA backend on 1, 2 and 4
+ranks). The validator now (a) captures the run's real exit code (nonzero/timeout ->
+FAIL), (b) requires the COMPLETE set of CI checks (9 for cimode 2/3, not merely
+"some passed"), (c) asserts the coarse-solver LOCATION recorded in the log
+matches the cimode (CPU for 2, DEVICE for 3 -- no silent fallback), and (d)
+rejects NaN/Inf. `HPCPERF_NEKRS_CIMODE` selects the mode.
+
+Observed on dgx003 (2026-09-05), `hypregpu` variant:
+- `--cimode 2` (CPU coarse): **PASS at 1, 2, 4 GPUs, 9/9 checks, coarse=CPU**.
+- `--cimode 3` (DEVICE / GPU HYPRE coarse): **PASS at 1 and 4 GPUs, 9/9 checks,
+  coarse=DEVICE** -- this is the run that actually exercises the GPU HYPRE coarse
+  solve the three patches enable, verified against the analytic solution.
+Earlier CI L2 errors (cimode 2): velocity 2.78e-10, pressure 6.98e-10, scalars
+6.67e-12 / 7.49e-12 (CI references 2.77e-10 / 7.14e-10 / 7.49e-12 / 7.22e-12).
+
+## Results on dgx003 (4x B200, CUDA 13.2.78, Slurm job 9552083)
+
+| Run | Ranks x GPUs | rank->GPU | CPU binding | Topology | Problem | Time | Validation |
+|---|---|---|---|---|---|---|---|
+| smoke/CI | 1 x 1 | wrapper, `--device-id 0`; audit 1/1 verified | runtime default | parRSB | 32 el., N=9, 30 steps | 1.18 s for 30 steps (8.9 ms/step) | PASS 9/9 |
+| smoke/CI | 2 x 2 | wrapper; 2/2 verified | runtime default | parRSB | same | 1.66 s (24.7 ms/step) | PASS 9/9 |
+| smoke/CI | 4 x 4 | wrapper; 4/4 verified | runtime default | parRSB | same | 1.40 s | PASS 9/9 |
+| strong | 1 x 1 | wrapper; 1/1 verified | runtime default | parRSB | 32,000 el., N=7 (16.4 M points), 100 steps | 177.2 s (1.77 s/step) | completes; L2 err vs exact at step 100: u 2.88e-11, p 1.69e-10, s00 1.38e-10, s01 1.39e-10 |
+| strong | 4 x 4 | wrapper; 4/4 verified | runtime default | parRSB | 32,000 el. (8,000/rank) | 62.5 s (0.63 s/step) | completes; L2 err u 2.87e-11, p 1.71e-10, s00 1.38e-10, s01 1.39e-10 |
+| weak | 4 x 4 | wrapper; 4/4 verified | runtime default | parRSB | H=10 -> 32,000 el. (8,000/rank; coincides with strong at N=4) | 65.6 s (0.66 s/step) | completes; same L2 errors |
+
+The 32-element CI case is launch/communication bound (2 and 4 GPUs are slower
+per step than 1); it is a correctness case, not a scaling result. The
+h-refined runs keep upstream's CI-oriented solver tolerances
+(`residualTol` 1e-12 velocity/scalars, 1e-8 pressure), so their step times
+are not performance figures either; the 1 -> 4 GPU ratio (2.8x) is recorded
+as observed.
+
+Dry-runs (`HPCPERF_DRY_RUN=1`, hypothetical allocations) -- **DRY-RUN /
+UNVALIDATED**, nothing executed:
+
+| GPUs | Nodes x GPUs/node | Mode | hrefine | Elements | Per rank | Launch |
+|---|---|---|---|---|---|---|
+| 8 | 1 x 8 | strong | 10 | 32,000 | 4,000 | `mpirun -np 8 --host dgx003:8 --map-by ppr:8:node ...` (single node) |
+| 40 | 5 x 8 | weak | 22 | 340,736 | 8,518 | 5 nodes x 8 -- multi-node BLOCKED on this site |
+| 80 | 10 x 8 | weak | 27 | 629,856 | 7,873 | 10 nodes x 8 -- multi-node BLOCKED on this site |
+
+## Limitations
+
+- Multi-node: BLOCKED/UNVERIFIED on this site; 40/80-GPU shapes are plans
+  (multi-node also needs the JIT cache strategy `NEKRS_CACHE_LOCAL/BCAST`).
+- HIP: untested; no gfx950 statement upstream.
+- Vendored HYPRE 2.32.0 needed CUDA 13 compatibility patches (above); the
+  combination is not upstream-validated.
+- Only the ethier family is wrapped (CI case + h-refined strong/weak); turbPipe,
+  kershaw (needs `genbox`, not shipped), tgv, pb146 build with this install
+  but have no wrappers yet. ADIOS2 checkpointing and CVODE are not built.
diff --git a/level3/nekrs/build.sh b/level3/nekrs/build.sh
new file mode 100755
index 0000000..cece0a2
--- /dev/null
+++ b/level3/nekrs/build.sh
@@ -0,0 +1,148 @@
+#!/usr/bin/env bash
+# Build nekRS (OCCA/CUDA backend, HYPRE on GPU) with upstream's CMake, from a
+# private copy of the vendored source tree.
+#
+#   ./build.sh [CUDA|HIP]        (default CUDA)
+#
+# Layout (Level 3 isolation): read-only clone _upstream/level3/nekRS; patched
+# private source copy .deps/level3/nekrs/src (upstream's third-party libraries
+# -- OCCA, HYPRE 2.32, gslib, Nek5000, LAPACK -- are vendored in-tree and built
+# by nekRS' own superbuild, nothing is shared with other applications); build
+# build/level3/nekrs/<cuda|hip>; install .deps/level3/nekrs/install
+# (= NEKRS_HOME, with nekrs.conf recording the JIT toolchain); logs
+# .deps/level3/nekrs/logs.
+#
+# Toolchain (upstream: GNU >= 9.1, MPI-3.1 with Fortran bindings, CMake >= 3.21,
+# CUDA >= 12): CC/CXX/FC = conda Open MPI wrappers (mpicc/mpicxx/mpif90 around
+# conda GCC 13.3.0). The conda environment has no gfortran, so the Fortran
+# wrapper is pointed at the system gfortran 14.2.1 through OMPI_FC (class C,
+# environment only); the conda MPI Fortran modules (.mod format 15) load under
+# gfortran 14 and the mixed link (gfortran-14 objects + conda libgfortran) was
+# tested with a 2-rank MPI Fortran program before this recipe was written.
+#
+# Options vs upstream defaults (class A, documented CMake options):
+# OCCA_ENABLE_HIP/DPCPP=OFF (CUDA only; no ROCm/SYCL here), ENABLE_ADIOS=OFF
+# (ADIOS2 checkpoint backend not needed; native .fld output stays),
+# NEKRS_BUILD_FLOAT=OFF (skip the second, fp32 solver build), ENABLE_CVODE
+# off (default). Patch (class B, 1 line): cmake/hypre.cmake adds sm_100 to the
+# HYPRE device architectures for CUDA >= 13 (upstream lists 80 90 only, which
+# would leave HYPRE's device kernels without Blackwell code); OKL kernels are
+# JIT-compiled by OCCA for the device it finds at run time (sm_100 here).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+l3_isolate_build_env    # Level 3 builds must not see Level 2 .deps/install prefixes
+
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+UP="$R/_upstream/level3/nekRS"
+[ -f "$UP/CMakeLists.txt" ] || { echo "build.sh: $UP missing -- run $HERE/fetch.sh first" >&2; exit 1; }
+SHA="$(git -C "$UP" rev-parse HEAD)"
+l3_paths nekrs
+
+# --- variant selection (multi-variant build/install/cache isolation) ----------
+# HPCPERF_NEKRS_HYPRE_GPU=ON|OFF selects whether the vendored HYPRE is built with
+# its CUDA device backend (GPU coarse solve possible) or host-only (CPU coarse
+# only). This is INDEPENDENT of OCCA_ENABLE_CUDA: the main application is on the
+# GPU either way. The default variant 'hypregpu' (ENABLE_HYPRE_GPU=ON) keeps the
+# existing legacy paths so the already-validated install is untouched; any other
+# variant gets a fully separate src/build/install/logs and its own JIT cache.
+HYPRE_GPU="${HPCPERF_NEKRS_HYPRE_GPU:-ON}"
+case "$HYPRE_GPU" in ON|OFF) : ;; *) echo "build.sh: HPCPERF_NEKRS_HYPRE_GPU must be ON or OFF" >&2; exit 2 ;; esac
+VARIANT="${HPCPERF_NEKRS_VARIANT:-$([ "$HYPRE_GPU" = ON ] && echo hypregpu || echo cpucoarse)}"
+if [ "$VARIANT" = hypregpu ]; then
+    BUILD_DIR="$R/build/level3/nekrs/$MODEL"                 # legacy layout (unchanged)
+else
+    L3_DEPS="$L3_R/.deps/level3/nekrs/$VARIANT"
+    L3_SRC="$L3_DEPS/src"; L3_INSTALL="$L3_DEPS/install"; L3_LOGS="$L3_DEPS/logs"
+    mkdir -p "$L3_SRC" "$L3_INSTALL" "$L3_LOGS"
+    BUILD_DIR="$R/build/level3/nekrs/$VARIANT.$MODEL"
+fi
+JOBS="${HPCPERF_BUILD_JOBS:-32}"
+SYS_FC="${HPCPERF_SYSTEM_GFORTRAN:-/usr/bin/gfortran}"
+[ -x "$SYS_FC" ] || { echo "build.sh: no gfortran at $SYS_FC (set HPCPERF_SYSTEM_GFORTRAN); the conda env has none" >&2; exit 1; }
+export OMPI_FC="$SYS_FC"
+# Mixed GCC majors (conda GCC 13 for C/C++, system gfortran 14): CMake's FortranCInterface detection
+# compiles its probe objects with -flto=auto -ffat-lto-objects (GCC >= 12) and links them with the
+# Fortran driver, whose lto1 rejects GCC 13 bytecode ("LTO version 13.1 instead of 14.0"). Linking with
+# -fno-lto uses the fat objects' regular code instead (verified with a 2-language test program). nekRS
+# itself does not use LTO, so this changes nothing else. Class C (link flag).
+export LDFLAGS="${LDFLAGS:-} -fno-lto"
+# The conda GCC links position-independent executables by default while the system gfortran emits
+# non-PIE objects ("relocation R_X86_64_32S ... can not be used when making a PIE object"); every
+# Fortran object that ends up in nekrs (Nek5000 interface, vendored LAPACK) therefore needs -fPIC.
+# Class C (compile flag; no effect on numerics).
+export FFLAGS="${FFLAGS:-} -fPIC"
+# The conda environment exports AR=x86_64-conda-linux-gnu-ar; the vendored HYPRE's configure takes $AR
+# verbatim as the full archive command (its default is "ar -rcu"), so the bare tool name makes every
+# `ar libHYPRE_*.a ...` call fail with a usage error. Unset -> HYPRE's own default. Class C.
+unset AR
+# The three HYPRE patches are needed ONLY to compile HYPRE's CUDA device backend against CUDA 13:
+#   0001 (class B, 1 line): add sm_100 to HYPRE_CUDA_SM (device SASS list).
+#   0002 (class D, 2 lines): thrust::reduce_by_key result type -> auto (device_utils.c/csr_matop_device.c).
+#   0003 (class D, 36 lines): explicit <thrust/iterator/reverse_iterator.h> + <thrust/pair.h> includes and
+#         thrust::not1 -> thrust::not_fn, all in HYPRE's *device* sources / device_utils headers.
+# All three touch code that is compiled by nvcc ONLY when ENABLE_HYPRE_GPU=ON. With ENABLE_HYPRE_GPU=OFF
+# HYPRE is host-only and that code is not compiled, so NO patch is applied (the candidate tests whether
+# the unused GPU component -- and therefore the patches -- can be dropped entirely).
+if [ "$HYPRE_GPU" = ON ]; then
+    PATCHES=("$HERE/patches/0001-hypre-cuda-sm100.patch" "$HERE/patches/0002-hypre-cuda13-thrust-pair.patch" "$HERE/patches/0003-hypre-cuda13-thrust3-compat.patch")
+else
+    PATCHES=()
+fi
+
+case "$BACKEND" in
+    CUDA) command -v nvcc >/dev/null || { echo "build.sh: nvcc not on PATH" >&2; exit 1; }
+          OCCA_FLAGS=(-DOCCA_ENABLE_CUDA=ON -DOCCA_ENABLE_HIP=OFF -DOCCA_ENABLE_DPCPP=OFF -DOCCA_ENABLE_OPENCL=OFF); ARCHNOTE="sm_$(l3_gpu_arch) (JIT at run time)" ;;
+    HIP)  command -v hipcc >/dev/null 2>&1 || { echo "build.sh: HIP requested but hipcc not found -- HIP build is UNTESTED on this machine (no ROCm)" >&2; exit 1; }
+          OCCA_FLAGS=(-DOCCA_ENABLE_CUDA=OFF -DOCCA_ENABLE_HIP=ON -DOCCA_ENABLE_DPCPP=OFF -DOCCA_ENABLE_OPENCL=OFF); ARCHNOTE="gfx950 (JIT at run time)" ;;
+    *) echo "usage: $0 [CUDA|HIP]" >&2; exit 2 ;;
+esac
+CMAKE_OPTS="variant=$VARIANT ${OCCA_FLAGS[*]} ENABLE_HYPRE_GPU=$HYPRE_GPU ENABLE_ADIOS=OFF ENABLE_CVODE=OFF NEKRS_BUILD_FLOAT=OFF NEKRS_GPU_MPI=OFF(default; runtime NEKRS_GPU_MPI) CC=mpicc CXX=mpicxx FC=mpif90(OMPI_FC=$SYS_FC)"
+PATCHNAMES=(); for p in "${PATCHES[@]}"; do PATCHNAMES+=("$(basename "$p")"); done
+FP="$(l3_fingerprint_text nekrs "$SHA" "$MODEL" "vendored: occa=2.0.0-dev hypre=2.32.0 gslib nek5000 lapack (in-tree)" "$CMAKE_OPTS" "runtime(NEKRS_GPU_MPI, default 0)" "${PATCHNAMES[@]}")"
+l3_fingerprint_check "$L3_INSTALL" "$FP" || exit 1
+
+echo "# nekRS $BACKEND: variant=$VARIANT ENABLE_HYPRE_GPU=$HYPRE_GPU patches=${#PATCHES[@]} upstream $SHA (v26.0), arch $ARCHNOTE, install=$L3_INSTALL"
+# private source copy (upstream clone stays pristine). The copy is ~280 MB in many small files (slow on
+# this filesystem), so it is reused only when it already holds this upstream commit AND this exact patch
+# series. The cache key is the upstream SHA plus the ORDERED patch-CONTENT hash, so editing a patch (even
+# without renaming it) invalidates the copy. A missing patch is a hard error.
+for p in "${PATCHES[@]}"; do [ -f "$p" ] || { echo "build.sh: patch $p missing" >&2; exit 1; }; done
+PATCH_SERIES_HASH="$(for p in "${PATCHES[@]}"; do sha256sum "$p" | cut -d' ' -f1; done | sha256sum | cut -d' ' -f1)"
+STAMP="$SHA $PATCH_SERIES_HASH"
+if [ -f "$L3_SRC/.hpcperf-src-stamp" ] && [ "$(cat "$L3_SRC/.hpcperf-src-stamp")" = "$STAMP" ]; then
+    echo "# reusing patched source copy $L3_SRC ($STAMP)"
+else
+    rm -rf "$L3_SRC"; mkdir -p "$L3_SRC"
+    rsync -a --exclude .git "$UP/" "$L3_SRC/"    # examples/ and doc/ are installed by nekRS' CMake
+    for p in "${PATCHES[@]}"; do
+        (cd "$L3_SRC" && patch -p1 --forward --silent < "$p") || { echo "build.sh: patch $(basename "$p") failed to apply" >&2; exit 1; }
+        echo "# applied $(basename "$p")"
+    done
+    echo "$STAMP" > "$L3_SRC/.hpcperf-src-stamp"
+fi
+# fresh configure every time: CMake caches CMAKE_EXE_LINKER_FLAGS and the Fortran/C detection results
+# from the first configure of a build directory, so environment fixes would otherwise not take effect.
+# HPCPERF_L3_INCREMENTAL=1 keeps an existing build tree (only for re-running install/fingerprint after a
+# late failure with an unchanged toolchain).
+[ -n "${HPCPERF_L3_INCREMENTAL:-}" ] || rm -rf "$BUILD_DIR"; mkdir -p "$BUILD_DIR"
+# upstream's build.sh uses CMake's default generator (Unix Makefiles); the conda environment exports
+# CMAKE_GENERATOR=Ninja, under which nekRS' generated build file is invalid ("bad $-escape" in a
+# vendored-library rule), so the generator is pinned to upstream's
+CC=mpicc CXX=mpicxx FC=mpif90 cmake -S "$L3_SRC" -B "$BUILD_DIR" -G "Unix Makefiles" -Wfatal-errors \
+    -DCMAKE_INSTALL_PREFIX="$L3_INSTALL" \
+    "${OCCA_FLAGS[@]}" -DENABLE_HYPRE_GPU="$HYPRE_GPU" -DENABLE_ADIOS=OFF -DENABLE_CVODE=OFF -DNEKRS_BUILD_FLOAT=OFF \
+    > "$L3_LOGS/configure-$MODEL.log" 2>&1 \
+    || { tail -40 "$L3_LOGS/configure-$MODEL.log"; echo "build.sh: configure failed (log: $L3_LOGS/configure-$MODEL.log)" >&2; exit 1; }
+t0=$(date +%s)
+cmake --build "$BUILD_DIR" --target install -j "$JOBS" > "$L3_LOGS/build-$MODEL.log" 2>&1 \
+    || { tail -40 "$L3_LOGS/build-$MODEL.log"; echo "build.sh: build failed (log: $L3_LOGS/build-$MODEL.log)" >&2; exit 1; }
+l3_fingerprint_write "$L3_INSTALL" "$FP"
+echo "# built + installed in $(( $(date +%s)-t0 )) s: $L3_INSTALL/bin/nekrs (NEKRS_HOME=$L3_INSTALL)"
+echo "# compiler warning lines: $(grep -c 'warning' "$L3_LOGS/build-$MODEL.log" || true)"
+grep -E 'OCCA_CXX|OCCA_CUDA_COMPILER|NEKRS_FC|NEKRS_GPU_MPI|BACKEND' "$L3_INSTALL/nekrs.conf" 2>/dev/null | sed 's/^/# nekrs.conf: /'
diff --git a/level3/nekrs/fetch.sh b/level3/nekrs/fetch.sh
new file mode 100755
index 0000000..112061e
--- /dev/null
+++ b/level3/nekrs/fetch.sh
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+# Fetch nekRS at the latest release tag into _upstream/level3/nekRS
+# (gitignored, read-only). nekRS vendors all third-party libraries as squashed
+# subtrees (OCCA, HYPRE, gslib, Nek5000, ADIOS2, ...), so no submodules and no
+# configure-time downloads are involved. Idempotent; a checkout at another
+# commit is an error, never silently reused.
+#
+#   ./fetch.sh
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+UPSTREAM_URL="https://github.com/Nek5000/nekRS.git"
+UPSTREAM_TAG="v26.0"
+UPSTREAM_SHA="96b3cf9e5bacede16568826c04a21bc0fe50dc7d"
+DEST="$R/_upstream/level3/nekRS"
+
+if [ -d "$DEST/.git" ]; then
+    have="$(git -C "$DEST" rev-parse HEAD)"
+    if [ "$have" = "$UPSTREAM_SHA" ]; then echo "fetch.sh: $DEST already at $UPSTREAM_TAG ($UPSTREAM_SHA)"; exit 0; fi
+    echo "fetch.sh: $DEST is at $have, not the recorded $UPSTREAM_SHA ($UPSTREAM_TAG); remove it to re-fetch" >&2; exit 1
+fi
+mkdir -p "$(dirname "$DEST")"
+echo "fetch.sh: cloning $UPSTREAM_URL @ $UPSTREAM_TAG (shallow)"
+git clone --quiet --depth 1 --branch "$UPSTREAM_TAG" "$UPSTREAM_URL" "$DEST"
+have="$(git -C "$DEST" rev-parse HEAD)"
+[ "$have" = "$UPSTREAM_SHA" ] || { echo "fetch.sh: tag $UPSTREAM_TAG resolved to $have, expected $UPSTREAM_SHA" >&2; exit 1; }
+echo "fetch.sh: ok -> $DEST ($UPSTREAM_SHA)"
diff --git a/level3/nekrs/patches/0001-hypre-cuda-sm100.patch b/level3/nekrs/patches/0001-hypre-cuda-sm100.patch
new file mode 100644
index 0000000..a6e1bd8
--- /dev/null
+++ b/level3/nekrs/patches/0001-hypre-cuda-sm100.patch
@@ -0,0 +1,11 @@
+--- a/cmake/hypre.cmake
++++ b/cmake/hypre.cmake
+@@ -58,7 +58,7 @@
+   set(HYPRE_BACKEND "--with-cuda" "--with-cuda-home=${CUDAToolkit_LIBRARY_ROOT}")
+ 
+   if(CUDAToolkit_VERSION VERSION_GREATER_EQUAL "13.0.0")
+-    set(HYPRE_DEVICE_ARCH "HYPRE_CUDA_SM=80 90")
++    set(HYPRE_DEVICE_ARCH "HYPRE_CUDA_SM=80 90 100")
+     #disable for now as it might not play well with all MPI implementations
+     #set(HYPRE_CONFIGURE_FLAGS "--enable-device-malloc-async")
+   elseif(CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.0.0")
diff --git a/level3/nekrs/patches/0002-hypre-cuda13-thrust-pair.patch b/level3/nekrs/patches/0002-hypre-cuda13-thrust-pair.patch
new file mode 100644
index 0000000..6c9c7c9
--- /dev/null
+++ b/level3/nekrs/patches/0002-hypre-cuda13-thrust-pair.patch
@@ -0,0 +1,22 @@
+--- a/3rd_party/hypre/src/utilities/device_utils.c
++++ b/3rd_party/hypre/src/utilities/device_utils.c
+@@ -1491,7 +1491,7 @@
+ #else
+       HYPRE_THRUST_CALL(sort_by_key, map2, map2 + ny, y);
+ 
+-      thrust::pair<HYPRE_Int*, HYPRE_Real*> new_end = HYPRE_THRUST_CALL( reduce_by_key,
++      auto new_end = HYPRE_THRUST_CALL( reduce_by_key,
+                                                                          map2,
+                                                                          map2 + ny,
+                                                                          y,
+--- a/3rd_party/hypre/src/seq_mv/csr_matop_device.c
++++ b/3rd_party/hypre/src/seq_mv/csr_matop_device.c
+@@ -1034,7 +1034,7 @@
+    hypre_assert(new_end.first - reduced_col_indices == new_end.second - reduced_col_nnz);
+    num_reduced_col_indices = new_end.first - reduced_col_indices;
+ #else
+-   thrust::pair<HYPRE_Int*, HYPRE_Int*> new_end =
++   auto new_end =
+       HYPRE_THRUST_CALL(reduce_by_key, A_j_sorted, A_j_sorted + nnz_A,
+                         thrust::make_constant_iterator(1),
+                         reduced_col_indices,
diff --git a/level3/nekrs/patches/0003-hypre-cuda13-thrust3-compat.patch b/level3/nekrs/patches/0003-hypre-cuda13-thrust3-compat.patch
new file mode 100644
index 0000000..0fd3964
--- /dev/null
+++ b/level3/nekrs/patches/0003-hypre-cuda13-thrust3-compat.patch
@@ -0,0 +1,178 @@
+--- a/3rd_party/hypre/src/utilities/device_utils.h
++++ b/3rd_party/hypre/src/utilities/device_utils.h
+@@ -164,6 +164,8 @@
+ #include <thrust/iterator/constant_iterator.h>
+ #include <thrust/iterator/counting_iterator.h>
+ #include <thrust/iterator/zip_iterator.h>
++#include <thrust/iterator/reverse_iterator.h>
++#include <thrust/pair.h>
+ #include <thrust/transform.h>
+ #include <thrust/functional.h>
+ #include <thrust/gather.h>
+--- a/3rd_party/hypre/src/utilities/_hypre_utilities.hpp
++++ b/3rd_party/hypre/src/utilities/_hypre_utilities.hpp
+@@ -359,6 +359,8 @@
+ #include <thrust/iterator/constant_iterator.h>
+ #include <thrust/iterator/counting_iterator.h>
+ #include <thrust/iterator/zip_iterator.h>
++#include <thrust/iterator/reverse_iterator.h>
++#include <thrust/pair.h>
+ #include <thrust/transform.h>
+ #include <thrust/functional.h>
+ #include <thrust/gather.h>
+--- a/3rd_party/hypre/src/IJ_mv/IJMatrix_parcsr_device.c
++++ b/3rd_party/hypre/src/IJ_mv/IJMatrix_parcsr_device.c
+@@ -656,7 +656,7 @@
+                             is_on_proc,                                                                                                               /* stencil */
+                             thrust::make_zip_iterator(thrust::make_tuple(off_proc_i,      off_proc_j,      off_proc_data,
+                                                                          off_proc_sora)),       /* result */
+-                            thrust::not1(thrust::identity<char>()) );
++                            thrust::not_fn(thrust::identity<char>()) );
+ 
+          hypre_assert(thrust::get<0>(new_end1.get_iterator_tuple()) - off_proc_i == nelms_off);
+ 
+@@ -668,7 +668,7 @@
+                             thrust::make_zip_iterator(thrust::make_tuple(stack_i + nelms, stack_j + nelms, stack_data + nelms,
+                                                                          stack_sora + nelms)),  /* last */
+                             is_on_proc,                                                                                                               /* stencil */
+-                            thrust::not1(thrust::identity<char>()) );
++                            thrust::not_fn(thrust::identity<char>()) );
+ 
+          hypre_assert(thrust::get<0>(new_end2.get_iterator_tuple()) - stack_i == nelms_on);
+ #endif
+--- a/3rd_party/hypre/src/IJ_mv/IJVector_parcsr_device.c
++++ b/3rd_party/hypre/src/IJ_mv/IJVector_parcsr_device.c
+@@ -522,7 +522,7 @@
+                             is_on_proc,                                                         /* stencil */
+                             thrust::make_zip_iterator(thrust::make_tuple(off_proc_i,      off_proc_data,
+                                                                          off_proc_sora)),       /* result */
+-                            thrust::not1(thrust::identity<char>()) );
++                            thrust::not_fn(thrust::identity<char>()) );
+ 
+          hypre_assert(thrust::get<0>(new_end1.get_iterator_tuple()) - off_proc_i == nelms_off);
+ 
+@@ -534,7 +534,7 @@
+                             thrust::make_zip_iterator(thrust::make_tuple(stack_i + nelms, stack_data + nelms,
+                                                                          stack_sora + nelms)),  /* last */
+                             is_on_proc,                                                         /* stencil */
+-                            thrust::not1(thrust::identity<char>()) );
++                            thrust::not_fn(thrust::identity<char>()) );
+ 
+          hypre_assert(thrust::get<0>(new_end2.get_iterator_tuple()) - stack_i == nelms_on);
+ #endif
+--- a/3rd_party/hypre/src/parcsr_ls/ams.c
++++ b/3rd_party/hypre/src/parcsr_ls/ams.c
+@@ -697,7 +697,7 @@
+          1.0 );
+ #else
+          thrust::identity<HYPRE_Complex> identity;
+-         HYPRE_THRUST_CALL( replace_if, l1_norm, l1_norm + num_rows, thrust::not1(identity), 1.0 );
++         HYPRE_THRUST_CALL( replace_if, l1_norm, l1_norm + num_rows, thrust::not_fn(identity), 1.0 );
+ #endif
+       }
+       else
+@@ -768,7 +768,7 @@
+       HYPRE_THRUST_CALL( transform_if, l1_norm, l1_norm + num_rows, diag_tmp, l1_norm,
+                          thrust::negate<HYPRE_Real>(),
+                          is_negative<HYPRE_Real>() );
+-      //bool any_zero = HYPRE_THRUST_CALL( any_of, l1_norm, l1_norm + num_rows, thrust::not1(thrust::identity<HYPRE_Complex>()) );
++      //bool any_zero = HYPRE_THRUST_CALL( any_of, l1_norm, l1_norm + num_rows, thrust::not_fn(thrust::identity<HYPRE_Complex>()) );
+       bool any_zero = 0.0 == HYPRE_THRUST_CALL( reduce, l1_norm, l1_norm + num_rows, 1.0,
+                                                 thrust::minimum<HYPRE_Real>() );
+ #endif
+--- a/3rd_party/hypre/src/parcsr_ls/par_mod_multi_interp_device.c
++++ b/3rd_party/hypre/src/parcsr_ls/par_mod_multi_interp_device.c
+@@ -327,7 +327,7 @@
+                                               thrust::make_counting_iterator(n_fine),
+                                               CF_marker,
+                                               points_left,
+-                                              thrust::not1(equal<HYPRE_Int>(1)) );
++                                              thrust::not_fn(equal<HYPRE_Int>(1)) );
+    remaining = points_end - points_left;
+ 
+    /* Cpts; number of C pts */
+@@ -541,7 +541,7 @@
+                                       points_left_old + remaining,
+                                       diag_shifts,
+                                       points_left,
+-                                      thrust::not1(thrust::identity<HYPRE_Int>()) );
++                                      thrust::not_fn(thrust::identity<HYPRE_Int>()) );
+ #endif
+ 
+          hypre_assert(new_end - points_left == cnt_rem);
+@@ -1637,7 +1637,7 @@
+                       fine_to_coarse,
+                       fine_to_coarse + n_fine,
+                       pass_marker,
+-                      thrust::not1(equal<HYPRE_Int>(color)),
++                      thrust::not_fn(equal<HYPRE_Int>(color)),
+                       -1 );
+ #endif
+ }
+--- a/3rd_party/hypre/src/parcsr_mv/par_csr_triplemat_device.c
++++ b/3rd_party/hypre/src/parcsr_mv/par_csr_triplemat_device.c
+@@ -229,7 +229,7 @@
+                                                                 hypre_CSRMatrixData(Cbar))) + hypre_CSRMatrixNumNonzeros(Cbar),
+                    hypre_CSRMatrixJ(Cbar),
+                    thrust::make_zip_iterator(thrust::make_tuple(C_offd_ii, C_offd_j, C_offd_a)),
+-                   thrust::not1(pred) );
++                   thrust::not_fn(pred) );
+       hypre_assert( thrust::get<0>(new_end.get_iterator_tuple()) == C_offd_ii + nnz_C_offd );
+ #endif
+ 
+@@ -958,7 +958,7 @@
+                                                                                      Cext_bigj)) + Cext_nnz,
+                                         Cext_bigj,
+                                         thrust::make_zip_iterator(thrust::make_tuple(work, big_work)),
+-                                        thrust::not1(pred1) );
++                                        thrust::not_fn(pred1) );
+ 
+       HYPRE_Int Cext_offd_nnz = thrust::get<0>(off_end.get_iterator_tuple()) - work;
+ #endif
+@@ -1220,7 +1220,7 @@
+                                 thrust::make_zip_iterator(thrust::make_tuple(zmp_i, zmp_j, zmp_a)) + local_nnz_C,
+                                 zmp_j,
+                                 thrust::make_zip_iterator(thrust::make_tuple(C_offd_ii, C_offd_j, C_offd_a)),
+-                                thrust::not1(pred) );
++                                thrust::not_fn(pred) );
+    hypre_assert( thrust::get<0>(new_end.get_iterator_tuple()) == C_offd_ii + nnz_C_offd );
+ #endif
+    hypreDevice_CsrRowIndicesToPtrs_v2(hypre_CSRMatrixNumRows(C_offd), nnz_C_offd, C_offd_ii,
+--- a/3rd_party/hypre/src/seq_mv/csr_matop_device.c
++++ b/3rd_party/hypre/src/seq_mv/csr_matop_device.c
+@@ -681,7 +681,7 @@
+                         B_ext_bigj,                                                            /* stencil */
+                         thrust::make_zip_iterator(thrust::make_tuple(B_ext_offd_ii, B_ext_offd_bigj, B_ext_offd_data,
+                                                                      B_ext_offd_xata)),        /* result */
+-                        thrust::not1(pred1) );
++                        thrust::not_fn(pred1) );
+ 
+       hypre_assert( thrust::get<0>(new_end.get_iterator_tuple()) == B_ext_offd_ii + B_ext_offd_nnz );
+ #endif
+@@ -708,7 +708,7 @@
+                         B_ext_bigj,                                                            /* stencil */
+                         thrust::make_zip_iterator(thrust::make_tuple(B_ext_offd_ii, B_ext_offd_bigj,
+                                                                      B_ext_offd_data)),        /* result */
+-                        thrust::not1(pred1) );
++                        thrust::not_fn(pred1) );
+ 
+       hypre_assert( thrust::get<0>(new_end.get_iterator_tuple()) == B_ext_offd_ii + B_ext_offd_nnz );
+ #endif
+@@ -2145,7 +2145,7 @@
+       new_nnz = HYPRE_THRUST_CALL( count_if,
+                                    A_data,
+                                    A_data + nnz,
+-                                   thrust::not1(less_than<HYPRE_Complex>(tol)) );
++                                   thrust::not_fn(less_than<HYPRE_Complex>(tol)) );
+    }
+    else
+    {
+@@ -2199,7 +2199,7 @@
+                                         thrust::make_zip_iterator(thrust::make_tuple(A_ii, A_j, A_data)) + nnz,
+                                         A_data,
+                                         thrust::make_zip_iterator(thrust::make_tuple(new_ii, new_j, new_data)),
+-                                        thrust::not1(less_than<HYPRE_Complex>(tol)) );
++                                        thrust::not_fn(less_than<HYPRE_Complex>(tol)) );
+ 
+       hypre_assert( thrust::get<0>(new_end.get_iterator_tuple()) == new_ii + new_nnz );
+    }
diff --git a/level3/nekrs/run.sh b/level3/nekrs/run.sh
new file mode 100755
index 0000000..409d544
--- /dev/null
+++ b/level3/nekrs/run.sh
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+# Run nekRS' ethier case (examples/ethier: 3D incompressible Navier-Stokes with
+# two passive scalars on the Ethier-Steinman exact solution; the full solver
+# stack -- pressure Poisson with pMG+HYPRE coarse grid, velocity Helmholtz,
+# subcycled advection, gather-scatter halo exchange) on N GPUs.
+#
+#   ./run.sh [CUDA|HIP] [extra nekrs args, e.g. --cimode 2]
+#
+# Execution model (upstream: "NekRS binds 1 GPU to 1 MPI rank"): one MPI rank
+# per GPU. nekRS' own default is --device-id LOCAL-RANK; under the common
+# launcher's per-rank wrapper each rank sees exactly one GPU, so the run passes
+# --device-id 0 (the mapping is audited by the launcher). GPU-aware MPI is
+# upstream-default OFF (RELEASE.md: enabling it "may cause a performance
+# regression"); HPCPERF_NEKRS_GPU_MPI=1 turns it on (NEKRS_GPU_MPI env).
+# Rank count is unconstrained (graph partitioning), but must stay well below
+# the element count.
+#
+# Resource / size controls:
+#   HPCPERF_GPUS=N|all       ranks = GPUs (default 1)
+#   HPCPERF_SCALE_MODE       smoke | strong | weak  (default smoke)
+#     smoke  : upstream ethier.par as shipped: 32 elements, N=9, 100 steps
+#     strong : ethierRefine.par with hrefine=H (H=HPCPERF_NEKRS_HREFINE, default
+#              10): 32*H^3 = 32,000 elements, N=7 (16.4M grid points), fixed
+#     weak   : hrefine chosen per rank count so that elements/rank ~ 8000
+#              (upstream's reference load, kershaw README "E/GPU=8000"):
+#              H = round(cbrt(250 N)); integer H makes the per-rank count vary
+#              between ~6.9k and ~8.5k -- printed and recorded
+#   HPCPERF_NEKRS_STEPS      time steps (default: upstream 100)
+#   HPCPERF_NEKRS_GPU_MPI    0|1 (default 0 = upstream default)
+#
+# The case directory (re2/usr/udf/oudf/par) is copied into the build tree;
+# strong/weak write a derived ethier.par from upstream's ethierRefine.par with
+# only `hrefine` and `numSteps` changed (class A). The OCCA JIT cache is shared
+# per build (NEKRS_CACHE_DIR) -- the first run of a new kernel set compiles OKL
+# kernels with nvcc, which takes minutes and is not part of the solve time.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"; [ $# -gt 0 ] && shift
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+l3_paths nekrs
+# variant selection must match build.sh: the default 'hypregpu' uses the legacy layout; any other
+# variant (e.g. cpucoarse = ENABLE_HYPRE_GPU=OFF) has its own install and JIT cache.
+VARIANT="${HPCPERF_NEKRS_VARIANT:-$([ "${HPCPERF_NEKRS_HYPRE_GPU:-ON}" = ON ] && echo hypregpu || echo cpucoarse)}"
+if [ "$VARIANT" = hypregpu ]; then
+    BUILD_DIR="$R/build/level3/nekrs/$MODEL"
+else
+    L3_INSTALL="$L3_R/.deps/level3/nekrs/$VARIANT/install"
+    BUILD_DIR="$R/build/level3/nekrs/$VARIANT.$MODEL"
+fi
+export NEKRS_HOME="$L3_INSTALL"
+EXE="$NEKRS_HOME/bin/nekrs"
+[ -x "$EXE" ] || { echo "run.sh: $EXE not found for variant '$VARIANT' -- run HPCPERF_NEKRS_VARIANT=$VARIANT ./build.sh $BACKEND first" >&2; exit 1; }
+CASE_SRC="$R/_upstream/level3/nekRS/examples/ethier"
+[ -f "$CASE_SRC/ethier.re2" ] || { echo "run.sh: $CASE_SRC missing (run fetch.sh)" >&2; exit 1; }
+
+N_RANKS="$(hpcperf_ranks nekrs yes)" || exit 2
+hpcperf_forbid_args nekrs --setup --device-id --backend -- "$@" || exit 2
+MODE="$(l3_scale_mode nekrs)" || exit 2
+STEPS="${HPCPERF_NEKRS_STEPS:-100}"
+case "$MODE" in
+    smoke)  H=0; ORDER=9 ;;
+    strong) H="${HPCPERF_NEKRS_HREFINE:-10}"; ORDER=7 ;;
+    weak)   H="$(python3 -c "import math; print(max(1, round((250*$N_RANKS)**(1/3))))")"; ORDER=7 ;;
+esac
+if [ "$H" -gt 0 ]; then ELEMS=$((32 * H * H * H)); else ELEMS=32; fi
+POINTS=$((ELEMS * (ORDER + 1) * (ORDER + 1) * (ORDER + 1)))
+
+# l3_rundir: dry-run gets a throwaway dir instead of rm -rf'ing the real run directory.
+RUN_DIR="$(l3_rundir "$BUILD_DIR/$L3_RUN_SUBDIR/$MODE.np$N_RANKS")" || exit 2
+cp "$CASE_SRC"/* "$RUN_DIR"/     # complete upstream case directory (re2, usr, udf, CASEDATA include, ci.inc, par files)
+if [ "$MODE" = smoke ]; then
+    sed -e "s/^numSteps *=.*/numSteps = $STEPS/" "$CASE_SRC/ethier.par" > "$RUN_DIR/ethier.par"
+else
+    # derived from upstream ethierRefine.par: hrefine and numSteps only
+    sed -e "s/^hrefine *=.*/hrefine = $H/" -e "s/^numSteps *=.*/numSteps = $STEPS/" "$CASE_SRC/ethierRefine.par" > "$RUN_DIR/ethier.par"
+fi
+export NEKRS_CACHE_DIR="$BUILD_DIR/cache"; mkdir -p "$NEKRS_CACHE_DIR"
+export NEKRS_GPU_MPI="${HPCPERF_NEKRS_GPU_MPI:-0}"
+# nekRS uses MPI one-sided operations (MPI_Win_lock); Open MPI's default one-sided component on this
+# node is `osc ucx`, which goes through UCX/InfiniBand even on one node and aborts in uct_ib with 4 ranks
+# ("'abort' is not implemented for protocol amo64/fetch"). The site profile already keeps point-to-point
+# off UCX (pml ob1 / btl self,sm,smcuda); the same is done here for one-sided (class C, site transport).
+export OMPI_MCA_osc="${HPCPERF_NEKRS_OSC:-^ucx}"
+export OMPI_FC="${HPCPERF_SYSTEM_GFORTRAN:-/usr/bin/gfortran}"   # .usr file is compiled at run time with the MPI Fortran wrapper
+unset CMAKE_GENERATOR   # nekRS' run-time UDF build configures with CMake and then calls `make okl.i`; the conda env's Ninja default breaks it
+export CUDA_CACHE_DISABLE=1
+# The Nek5000 side of the case (ethier.usr) keeps per-field work arrays of size lx1^3*lelt on the stack;
+# upstream's job scripts raise the stack limit for that reason. With the default 8 MB the h-refined
+# cases segfault in useric. Applies to mpirun's children (inherited rlimit). Class C.
+ulimit -s unlimited 2>/dev/null || ulimit -s "$(ulimit -H -s)"
+
+echo "# nekRS $BACKEND: mode=$MODE ranks=$N_RANKS case=ethier hrefine=$H elements=$ELEMS (~$((ELEMS / N_RANKS))/rank) N=$ORDER points=$POINTS steps=$STEPS gpu_mpi=$NEKRS_GPU_MPI run_dir=$RUN_DIR"
+cd "$RUN_DIR"
+RUN_ID="$(l3_run_id)"
+"$L3_LAUNCHER" --gpus "$N_RANKS" --bind wrapper -- "$EXE" --setup ethier --backend "$BACKEND" --device-id 0 "$@" 2>&1 | tee "$RUN_DIR/stdout.log"
+rc=${PIPESTATUS[0]}
+if [ -z "${HPCPERF_DRY_RUN:-}" ]; then
+    l3_manifest "$RUN_DIR" "run_id=$RUN_ID" "app=nekrs" "variant=$VARIANT" "backend=$BACKEND" "mode=$MODE" "ranks=$N_RANKS" \
+        "elements=$ELEMS" "order=$ORDER" "points=$POINTS" "steps=$STEPS" "gpu_mpi=$NEKRS_GPU_MPI" \
+        "osc=$OMPI_MCA_osc" "extra_args=$*" "exit_code=$rc" "binary=$EXE" "binary_sha256=$(l3_sha_file "$EXE")" \
+        "par_sha256=$(l3_sha_file "$RUN_DIR/ethier.par")" "stdout=$RUN_DIR/stdout.log" "utc=$(date -u +%FT%TZ)"
+fi
+exit "$rc"
diff --git a/level3/nekrs/validate.sh b/level3/nekrs/validate.sh
new file mode 100755
index 0000000..8010265
--- /dev/null
+++ b/level3/nekrs/validate.sh
@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# Correctness check for nekRS: upstream's own CI test of the ethier case
+# (analytic Ethier-Steinman solution) run on N GPUs.
+#
+#   ./validate.sh [CUDA|HIP]        HPCPERF_GPUS=N (default 1)
+#     HPCPERF_NEKRS_CIMODE=C        upstream CI mode (default 2)
+#
+# `nekrs --cimode C` fixes the solver settings for CI and, at the last step,
+# checks the L2 errors of velocity/pressure/scalars against the exact solution
+# (references in examples/ethier/ci.inc, EPS = 0.3) plus solver iteration
+# counts. nekRS prints "CI test <name> passed|failed" per check and exits
+# non-zero on any failure. This validator requires ALL of:
+#   * run exited 0 (no timeout, no abort);
+#   * the COMPLETE expected set of CI checks was produced (count matches the
+#     per-cimode table below) -- not merely "some passed";
+#   * zero failed checks;
+#   * the coarse-solver LOCATION recorded in the log matches what the cimode
+#     selects (CPU for mode 2; DEVICE for mode 3 -- i.e. the run must really
+#     have exercised GPU HYPRE, not silently fallen back).
+# The coarse-solver location/precision are extracted and printed for the record.
+#
+# IMPORTANT SCOPE NOTE: cimode 2 runs the HYPRE BoomerAMG coarse solve on the
+# CPU (nekRS default). A PASS here validates the CUDA main application + a
+# CPU coarse solve; it does NOT validate GPU HYPRE. Use HPCPERF_NEKRS_CIMODE=3
+# (DEVICE coarse) to exercise the GPU HYPRE coarse solve.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+N="${HPCPERF_GPUS:-1}"
+CIMODE="${HPCPERF_NEKRS_CIMODE:-2}"
+TIMEOUT="${HPCPERF_VALIDATE_TIMEOUT:-2400}"
+VARIANT="${HPCPERF_NEKRS_VARIANT:-$([ "${HPCPERF_NEKRS_HYPRE_GPU:-ON}" = ON ] && echo hypregpu || echo cpucoarse)}"
+if [ "$VARIANT" = hypregpu ]; then VBD="$R/build/level3/nekrs/$MODEL"; else VBD="$R/build/level3/nekrs/$VARIANT.$MODEL"; fi
+OUT="$VBD/$L3_RUN_SUBDIR/validate.cimode$CIMODE.np$N.log"
+
+# complete CI-check counts and required coarse-solver location, per cimode
+case "$CIMODE" in
+    2) EXPECT_CHECKS=9; WANT_COARSE=CPU ;;
+    3) EXPECT_CHECKS=9; WANT_COARSE=DEVICE ;;
+    *) echo "validate.sh: expected CI-check count for cimode $CIMODE is not recorded here; add it after observing one run (refusing to guess)" >&2; exit 2 ;;
+esac
+
+export HPCPERF_GPUS="$N"
+mkdir -p "$(dirname "$OUT")"
+echo "validate.sh: nekRS $BACKEND variant=$VARIANT ethier --cimode $CIMODE (upstream CI mode; expect $EXPECT_CHECKS checks, coarse=$WANT_COARSE) on $N GPU(s)"
+rc=0
+HPCPERF_SCALE_MODE=smoke timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" --cimode "$CIMODE" > "$OUT" 2>&1 || rc=$?
+grep -aE '^#|hpcperf-launch: audit summary|CI test|L2 err|COARSE SOLVER LOCATION|COARSE SOLVER PRECISION|elapsedStepSum|ERROR|Abort|abort' "$OUT" | grep -aiv 'no error' | sed 's/^/  /' | tail -50
+if [ "$rc" -eq 124 ]; then echo "validate.sh: FAIL -- run timed out after ${TIMEOUT}s"; exit 1; fi
+
+COARSE="$(grep -a 'COARSE SOLVER LOCATION' "$OUT" | head -1 | sed 's/.*value: *//' | tr -d ' ' || true)"
+PASSED=$(grep -a -c 'CI test .* passed' "$OUT" || true)
+FAILED=$(grep -a -c 'CI test .* failed' "$OUT" || true)
+TOTAL=$((PASSED + FAILED))
+echo "  nekrs exit code $rc; CI checks passed=$PASSED failed=$FAILED total=$TOTAL (expected $EXPECT_CHECKS); coarse solver location=${COARSE:-UNKNOWN}"
+
+ok=1
+[ "$rc" -eq 0 ]                    || { echo "  run exited $rc"; ok=0; }
+[ "$FAILED" -eq 0 ]                || { echo "  $FAILED CI check(s) failed"; ok=0; }
+[ "$TOTAL" -eq "$EXPECT_CHECKS" ]  || { echo "  produced $TOTAL CI checks, expected the complete set of $EXPECT_CHECKS (incomplete run or changed CI)"; ok=0; }
+[ "${COARSE:-}" = "$WANT_COARSE" ] || { echo "  coarse solver location is '${COARSE:-UNKNOWN}', expected '$WANT_COARSE' -- the intended solve path did not run (no silent fallback allowed)"; ok=0; }
+if [ "$ok" -eq 1 ]; then
+    echo "nekRS $BACKEND validation ($N GPU, ethier --cimode $CIMODE, $EXPECT_CHECKS/$EXPECT_CHECKS checks, coarse=$COARSE): PASS"; exit 0
+fi
+echo "nekRS $BACKEND validation ($N GPU, ethier --cimode $CIMODE): FAIL (log: $OUT)"; exit 1
diff --git a/level3/nyx/README.md b/level3/nyx/README.md
new file mode 100644
index 0000000..f56ece4
--- /dev/null
+++ b/level3/nyx/README.md
@@ -0,0 +1,301 @@
+# Nyx (AMReX-Astro) -- Level 3 second batch
+
+Cosmological N-body + baryon hydrodynamics (dark-matter particles, Poisson
+gravity by AMReX MLMG multigrid, PPM hydro, comoving coordinates), full
+application workflow: IC read, gravity solve, hydro/particle advance, particle
+redistribution, plotfile/checkpoint I/O.
+
+## Provenance / versions
+
+| Item | Value |
+|---|---|
+| Nyx | tag `26.09`, `e06eabc1b9dbcad5612db9529aced682402daede` (2026-08-26), BSD-3-Clause-LBNL |
+| AMReX (used) | tag `26.09`, `a52ca73324ac2c7b65ec04f131e6df99eec9c576` (2026-09-01) -- **external, private build** |
+| AMReX (Nyx submodule pin) | `6e875b7cc1a4eec78e22ae4cdaa79f88acf5169e` (development 2026-08-12) -- **not used**, see below |
+| SUNDIALS (heatcool variant only) | Nyx submodule pin `5c53be85c88f63c5201c130b8cb2c686615cfb03` = v7.2.1 |
+| Build strategy | NATIVE (CMake): private AMReX 26.09 install + Nyx via `find_package(AMReX CONFIG)`; upstream's GPU CI options (`Nyx_HYDRO=YES Nyx_MPI=YES Nyx_OMP=NO`, C++17); double-precision particles as upstream's nightly regression builds |
+| Compiler / Toolkit / MPI | conda GCC 13.3.0 (host), CUDA 13.2.78 sm_100, conda Open MPI 5.0.10 (site profile gmu-hopper: `pml ob1 / btl self,sm,smcuda`) |
+| Profiles | `cuda132-gcc133-adiabatic` (Nyx_HEATCOOL=NO), `cpu-gcc133-adiabatic` (Nyx_GPU_BACKEND=NONE reference + AMReX plotfile tools + particle_compare), `cuda132-gcc133-heatcool` + `cpu-gcc133-heatcool` (SUNDIALS 7.2.1 CVODE, see "Heating/cooling variant") |
+| Source changes | **none** (class A: build options; class A derived decks written at run time) |
+
+Layout: `.deps/level3/nyx/<profile>/{src,build,install,logs,cache}`, application
+build tree `build/level3/nyx/<profile>`, run dirs
+`build/level3/nyx/<profile>/run/<case>.<mode>.np<N>` (dry-runs under `.dryrun/`).
+Fingerprint `.deps/level3/nyx/<profile>/install/.hpcperf-l3-fingerprint` (schema
+l3-2) + `BUILD_INFO.txt` (binary sha256, `AMREX_CUDA_ARCHS`, `cuobjdump` archs).
+
+### Why AMReX 26.09 and not the submodule pin
+
+The first build against the pin produced an **sm_86** binary: that AMReX's
+`convert_cuda_archs()` (Tools/CMake/AMReXUtils.cmake) drops every SM >= 10.0
+("CMake 3.30 does not support SM 10.0+ in cuda_select_nvcc_arch_flags"), the
+list becomes empty, autodetection runs and CMake 3.28's table maps this B200
+(compute capability 10.0, confirmed by the detection program itself) to
+`8.6+PTX`. No option of that AMReX yields sm_100. AMReX 26.09 rewrote the
+resolution (`AMReXCUDAArchs`, `nvcc --list-gpu-arch`) and produced
+`AMREX_CUDA_ARCHS=100` / `cuobjdump: sm_100` -- the same path the first-batch
+WarpX 26.09 build already used. 26.09 is 21 commits ahead of / 0 behind the
+pin (GitHub compare), Nyx's minimum is AMReX 20.11, and Nyx consumes an
+external AMReX through `find_package(AMReX CONFIG)` with the component set its
+own superbuild would request (3D, DOUBLE, PARTICLES/PDOUBLE, MPI, CUDA,
+LSOLVERS). The sm_86 attempt was removed
+(`.deps/level3/nyx/ATTEMPT-1-sm86-removed.txt`). WarpX's AMReX *binary* is not
+reused (different component set: EB, no linear solvers, FFT); only the same
+read-only source checkout tag.
+
+## Cases (all upstream decks, used from the read-only checkout)
+
+| Case | Deck | What | Mode |
+|---|---|---|---|
+| `minisb` | `Exec/MiniSB/inputs.32` + `nyx.ppm_type=0` | Santa Barbara cluster, 32^3 cells, 32,686 DM particles (shipped ASCII IC), 10 steps; **exactly upstream's nightly GPU regression test "MiniSB"** (2 ranks there) | smoke |
+| `lya_adiabatic` smoke | `Exec/LyA/inputs.rt.garuda` | upstream's GPU regression deck **"LyA-adiabatic"** (heat_cool_type=0, strang_split=1), 32^3, shipped `32.nyx` IC, z=100, 10 steps | smoke |
+| `lya_adiabatic` strong | `Exec/LyA/inputs` with heating/cooling OFF | the flagship 64^3 Lyman-alpha science deck (shipped `64sssss_20mpc.nyx` IC, z=159) as a **named adiabatic derivative**: `nyx.heat_cool_type=0 sdc_split=0 strang_split=1`. Does **not** cover the heating/cooling LyA workload | strong |
+| `scaling_synthetic` | `Exec/Scaling/inputs` (RandomPerCell) | upstream's scaling deck; `RandomPerCell` is documented as a testing-only initialisation -> **synthetic scaling/communication test, not a science IC** | strong (fixed G^3, default 256^3) / weak (64^3 cells per rank, box and total DM mass scaled with the tiles) |
+| `lya_heatcool` | `Exec/LyA/inputs.rt` as shipped (heat_cool_type=11, CVODE) | the heating/cooling LyA workload -- `heatcool` profile (SUNDIALS 7.2.1) | smoke (validation verdict STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING, see below) / strong (`Exec/LyA/inputs` 64^3, completeness record) |
+
+Decomposition: `amr.max_grid_size` is fixed per case (16 for 32^3/64^3 decks,
+64 for the synthetic decks) and `amr.refine_grid_layout=0` (as upstream's MiniSB
+deck), so the BoxArray is identical for every rank count and only the
+distribution changes; ranks > boxes is refused (never a silent idle rank),
+boxes % ranks != 0 is reported as imbalanced. One MPI rank per GPU; AMReX binds
+device 0 of the one GPU the launcher wrapper exposes; the launcher audits the
+expected/observed mapping (all runs below: every rank `verified`).
+
+Derived deck = upstream lines verbatim minus the I/O cadence / decomposition
+lines listed in the file header, plus `amrex.the_arena_init_size=0` (as the
+official test command) and checkpoints at step 0 and the final step
+(`chk00000`, `chk<final>`: needed for particle identity across rank counts;
+upstream's test command disables checkpoints -- I/O only).
+
+## Validation (`validate.sh`, pre-fixed criteria)
+
+Per case, for the N-GPU run:
+
+1. **Completeness**: exit code 0 (timeout -> FAIL), `plt00000` + final plotfile,
+   runlog reaches `max_step`, every plotfile variable finite (`amrex_fextrema`
+   through `l3_check.require_finite`), DM particle count == IC count (exact).
+2. **Official regression comparison** through `nyx_fcompare_check.py`, a strict
+   wrapper around AMReX's `fcompare -n 0 --rel_tol T --abort_if_not_all_found`
+   (both Headers parsed and compared -- variable set, dimension, levels, time,
+   domain, cell sizes, box arrays; `fextrema` on both plotfiles for raw finiteness;
+   exactly one parsed table row per variable and level, no message rows, no
+   duplicates, nothing dropped; a field that is identically zero in the reference
+   must be identically zero in the test; parser and tool exit status must agree):
+   N=1 vs a second independent 1-GPU run (same-configuration reproducibility, what
+   the nightly test measures); N>1 vs the 1-GPU plotfile of the same binary/deck.
+   Tolerance provenance (official GPU nightly reports at
+   `ccse.lbl.gov/pub/GpuRegressionTesting/Nyx/`, read 2026-09-05/06): MiniSB compares
+   with `--rel_tol 2e-10` (used as is); LyA-adiabatic compares with `--rel_tol 5e-09`
+   -- this validator applies the MiniSB value 2e-10 to both adiabatic decks, a
+   **project choice 25x stricter than upstream's** for LyA-adiabatic.
+   History: the first validator version parsed fcompare's table with a loose awk
+   filter that dropped `inf`/`nan` rows and message rows and accepted any `rc=1`
+   as "over tolerance"; a reviewer's CPU-only injection (Ne `abs=1 rel=inf`, a
+   missing-variable warning, a NaN row) passed it. Replaced on 2026-09-07 by the
+   strict wrapper (negative tests in `level3/tools/tests/test_nyx_validator.sh`);
+   every earlier verdict was re-derived offline from the saved plotfiles with the
+   strict wrapper (see "Results").
+   Box layouts (2026-09-08): the wrapper checks each plotfile's BoxArray geometrically
+   (physical extents -> cell boxes through `prob_lo`/`dx`: inside the level domain, no
+   overlap, level 0 covers the domain exactly). Identical box sets in any order are
+   compared normally; a *legal re-blocking* (the same cells partitioned differently,
+   e.g. another `amr.max_grid_size`) is reported as **UNSUPPORTED_LAYOUT** (comparator
+   exit 4, `validate.sh` exit 4: the comparison is not performed, the case is neither
+   PASS nor FAIL -- cross-BoxArray comparison would need `fcompare --allow_diff_grids`
+   and is not enabled here); a missing box, an overlap, a coverage deficit or different
+   covered cell sets at a level are STRUCTURAL failures (FAIL). All runs in this README
+   use the run.sh-fixed `max_grid_size`, so every real comparison so far had identical
+   box arrays (the 2026-09-08 offline re-check of the 2026-09-07 regression plotfiles
+   reports "identical box array" for every pair).
+   Particles: AMReX's `particle_compare` needs identical headers (incl.
+   `next_id` and per-file layout, i.e. the same rank count) and returns 0 even
+   when it prints "FAIL - Particle data headers do not agree" -> across rank
+   counts `nyx_particle_compare.py` matches particles by their exact t=0
+   position (chk00000 -> chk<final> by (id,cpu) within a run) and reports
+   particle_compare's abs/rel norms per component at the same tolerance.
+3. **Cross-backend reference**: CPU-profile binary (Nyx_GPU_BACKEND=NONE, same
+   Nyx/AMReX/deck), rel_tol **1e-8** fixed before any run (FMA/libm/reduction
+   differences host vs device over 10 steps).
+4. **Conservation**: comoving baryon mass `sum(density*dV)` (`amrex_fvolumesum`)
+   plt00000 -> final, |dM/M| <= 1e-9; DM count exact.
+
+For the two adiabatic decks no tolerance was changed after seeing results (2e-10 and
+1e-8 were in `validate.sh` before its first execution, 2026-09-05 23:58 UTC vs 2026-09-06
+00:02 UTC). For the heat/cool deck this is **not** true -- see the "I_R" section below.
+
+## Results (dgx003, 2026-09-06; logs under `build/level3/nyx/cuda132-gcc133-adiabatic/run/`)
+
+Build: AMReX 26.09 CUDA + Nyx = 121 s (nyx 115 s) at -j32; CPU profile 13 s
+(after AMReX); `cuobjdump` sm_100 only; cudart static (`l3_binary_backend_check`
+accepts embedded CUDA ELF).
+
+| GPUs | Case | fcompare vs reference (max rel err, tol 2e-10) | DM particles (rel, tol 2e-10) | vs CPU (tol 1e-8) | mass, count | audit | Result |
+|---|---|---|---|---|---|---|---|
+| 1 | minisb | rerun: 3.0e-11 (Temp) | 6.5e-16 | 3.9e-11 / 7.0e-16 | exact / 32,686 | 1 verified | **VALIDATED_PASS** |
+| 1 | lya_adiabatic | rerun: 6.5e-14 | 7.9e-16 | 9.6e-14 / 7.0e-16 | exact / 32,768 | 1 verified | **VALIDATED_PASS** |
+| 2 | minisb | vs 1 GPU: 3.3e-12 | 6.5e-16 | 3.6e-11 / 6.9e-16 | exact | 2 verified | **VALIDATED_PASS** |
+| 2 | lya_adiabatic | vs 1 GPU: 6.4e-14 | 8.8e-16 | 8.7e-14 / 7.0e-16 | exact | 2 verified | **VALIDATED_PASS** |
+| 4 | minisb | vs 1 GPU: 1.38e-10 | 1.3e-15 | 1.77e-10 / 1.7e-15 | exact | 4 verified | **VALIDATED_PASS** |
+| 4 | lya_adiabatic | vs 1 GPU: 6.7e-14 | 6.1e-16 | 9.3e-14 / 7.0e-16 | exact | 4 verified | **VALIDATED_PASS** |
+
+(The `Temp` field carries the largest relative differences; MiniSB at 4 GPUs is
+the closest to the official tolerance, 1.4e-10 vs 2e-10.)
+
+**Joint-HEAD regression, 2026-09-07 (code state `fc4d2a1`; run trees
+`build/level3/nyx/<profile>/run.regress-fc4d2a1/`, logs
+`build/level3/nyx/regress-fc4d2a1/validate_full.hc{NO,YES}.np{1,2,4}.log` +
+`queue_nyx.log`)**: the same decks re-run from scratch with the strict wrapper,
+new run ids, the historical `run/` trees untouched. Six `validate.sh` calls:
+
+| GPUs | Cases | vs rerun / 1-GPU (tol) | vs CPU (tol) | DM particles | audit | validate.sh exit / class |
+|---|---|---|---|---|---|---|
+| 1 | minisb, lya_adiabatic | 1.6e-11, 6.1e-14 (2e-10) | 5.2e-11, 8.7e-14 (1e-8) | <= 9.8e-16, counts exact | 1 verified | 0 **PASS** |
+| 2 | minisb, lya_adiabatic | 2.2e-11, 6.5e-14 (2e-10) | 7.4e-11, 8.7e-14 (1e-8) | exact | 2 verified | 0 **PASS** |
+| 4 | minisb, lya_adiabatic | 1.2e-11, 6.0e-14 (2e-10) | 6.5e-11, 9.6e-14 (1e-8) | exact | 4 verified | 0 **PASS** |
+| 1 | lya_heatcool | state 1.4e-13 (5e-5); I_R rel 1.24 (reported) | state 1.9e-13; I_R rel 1.26 | 8.8e-16, exact | 1 verified | 3 **STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING** |
+| 2 | lya_heatcool | state 1.4e-13; I_R rel 1.40 | state 1.9e-13; I_R rel 1.26 | exact | 2 verified | 3 **STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING** |
+| 4 | lya_heatcool | state 1.2e-13; I_R rel 1.40 | state 1.9e-13; I_R rel 1.46 | exact | 4 verified | 3 **STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING** |
+
+The three heat/cool calls are counted as PENDING, not as passes (exit-code classes:
+`level3/tools/l3_verdict.py`).
+
+Scaling runs (COMPLETED, 10 steps, not correctness-validated; timing = Nyx
+"Run time", includes IC read and I/O; too short for a performance statement):
+
+| Mode | Case | 1 GPU | 2 GPU | 4 GPU |
+|---|---|---|---|---|
+| strong | lya_adiabatic 64^3 (science IC, adiabatic) | 2.06 s | 2.35 s | 1.93 s |
+| strong | scaling_synthetic 256^3 (synthetic) | 39.5 s | 41.5 s | 29.7 s |
+| weak | scaling_synthetic 64^3 cells/rank (synthetic) | 1.63 s | 2.20 s | 3.52 s |
+
+No science IC larger than 64^3 is shipped (256^3/1024^3 exist only at OLCF
+paths); meaningful strong scaling of a science case needs such an IC
+(**blocked by data availability**, documented, not worked around).
+
+Dry-runs (`HPCPERF_DRY_RUN=1`, `HPCPERF_NODES` hypothetical): 8 GPUs (2 nodes)
+planned for all three decks; 40/80 GPUs planned for `scaling_synthetic` weak
+(40/80 boxes) and 40 GPUs for `lya_adiabatic` strong (64 boxes over 40 ranks =
+imbalanced, reported); **refused** for `minisb` (8 boxes) and for 80 ranks on
+the 64-box strong deck -- as designed. Multi-node remains BLOCKED/UNVERIFIED
+on this site (launcher note); HIP untested (no ROCm).
+
+## Heating/cooling variant (profile `cuda132-gcc133-heatcool`)
+
+Staged build: SUNDIALS 7.2.1 (Nyx's pinned submodule commit; `ENABLE_CUDA`, index
+size 32, `SUNDIALS_BUILD_PACKAGE_FUSED_KERNELS=ON` as Nyx's own
+`NyxSetupSUNDIALS.cmake`; CVODE **and ARKODE** because AMReX 26.09's
+`find_package(SUNDIALS)` requires the `arkode` component) -> AMReX 26.09 with
+`AMReX_SUNDIALS=ON` -> Nyx `Nyx_HEATCOOL=YES` (CVODE vectorized, `heat_cool_type
+11`). 310 s at -j16 (SUNDIALS 14 s, AMReX 134 s, Nyx 157 s); `cuobjdump` sm_100;
+CPU reference profile `cpu-gcc133-heatcool` built the same way (CPU SUNDIALS).
+
+**SUNDIALS independent probe first** (`sundials_probe.sh`: SUNDIALS' own example
+regression tests, `SUNDIALS_TEST_ENABLE_DEV_TESTS`, CUDA sm_100, CUDA 13.2):
+6 CUDA examples/tests, **5 pass** (cvAdvDiff_kry_cuda, _managed,
+cvAdvDiff_diag_cuda x3); `cvRoberts_block_cusolversp_batchqr` **fails only in the
+integrator statistics** printed with a 10 % integer allowance (nni 805 vs 823,
+ncfn 5 vs 4, netf 33 vs 31; the solution values agree at the test's 4-digit
+precision). That example uses the cuSolverSp batched-QR linear solver; **Nyx's
+HEATCOOL path uses CVODE with `CVDiag`** (`Source/HeatCool/integrate_state_vec_3d.cpp`),
+not cuSolverSp, so the failing example is off Nyx's path. Recorded as-is
+(`.deps/level3/nyx/cuda132-gcc133-heatcool/logs/sundials-ctest.log`).
+
+**Workload `lya_heatcool`**: `Exec/LyA/inputs.rt` exactly as shipped
+(heat_cool_type 11, UVB table TREECOOL_middle, 32^3, 32.nyx IC), 10 steps; "Integrating
+heating/cooling method ... Vectorized CVODE" in the log; 1.3 s on 1 GPU.
+
+**Validation** (same script): same-config rerun (1 GPU), 1-GPU reference (2/4 GPU)
+and CPU-heatcool reference all at **5e-5**. Provenance and history, stated plainly:
+upstream's GPU nightly "LyA" test compares `plt00354` (354 steps of `inputs.rt`) with
+`fcompare --rel_tol 5e-05` (its own maximum 2.8e-5 in Temp; it also compares `I_R`,
+relative error 4.2e-8 at that point where ||I_R|| ~ 6e18). The project's 10-step
+smoke deck (`plt00010`, z ~ 100 -> 90) is a different configuration. The first
+heat/cool validation here (2026-09-06 01:20 UTC) was run with the adiabatic
+tolerances 2e-10 / 1e-8 and FAILED; the official 5e-5 was looked up (01:25 UTC),
+adopted (01:27 UTC) and the runs repeated -> PASS for every state field. The
+`I_R` treatment (excluded from the gate) was introduced at the same time and is a
+project decision with no upstream counterpart. **Every state variable (density,
+momenta, rho_E, rho_e, Temp, Ne, phi_grav, grav_*, pressure, particle_*) agrees to
+<= 1.6e-13 relative** (rerun, 2-GPU, 4-GPU and vs CPU alike; `Ne` is identically
+zero in every run and is checked exactly), DM particles agree to ~1e-15, baryon
+mass exact, counts exact, all ranks' GPUs verified. Verdict for the heating/cooling
+LyA deck: **STATE_AND_PARTICLES_PASS at 1/2/4 GPUs, I_R_CHECK_PENDING** -- not a full
+regression PASS (validate.sh exits 3 for this case). (Run dirs:
+`build/level3/nyx/cuda132-gcc133-heatcool/run/`.) The adiabatic results above and
+this section are separate claims; neither is presented as the other.
+
+### I_R -- what it is, why it is not accepted by a tolerance yet
+
+From the fixed source (Nyx `e06eabc1`, `Source/HeatCool/f_rhs_struct.H`,
+`Source/Hydro/sdc_hydro.cpp`, `Source/Initialization/Nyx_setup.cpp`):
+
+- `I_R` is the single component of the `SDC_IR_Type` state (`Nyx_setup.cpp:304`),
+  initialised to 0 (`Nyx_initdata.cpp:335`), written to plotfiles like any state.
+  It is computed once per cell per step in `ode_eos_finalize_struct` after the
+  CVODE solve, as the reaction integral net of the hydro source:
+  `I_R = [a_end^2 (rho e)_out - (a^2 (rho e)_orig + dt A_rhoe)] / (dt a_half)`
+  (comoving energy density per unit time). It is used again: at the next step it
+  is added to the external source for the hydro predictor (`sdc_hydro.cpp`, `IR_tmp`
+  added to `ext_src_old` before `construct_hydro_source`, subtracted afterwards),
+  and in the SDC branch it is the increment applied to (rho e)/(rho E). So it is a
+  state-carried coupling term, recomputable from the state before/after the ODE and
+  the hydro source -- not an independent physical observable.
+- Magnitude in the 10-step deck (offline reading of the saved plotfiles, 2026-09-07):
+  max|I_R| = 0.28, mean|I_R| = 0.044 at plt00010 (0 at plt00000); the terms it is
+  built from are O(a^2 rho e / (dt a_half)) ~ 3.7e2, i.e. `I_R` is a ~1e-3 residual of
+  nearly cancelling terms (z ~ 90: net heating/cooling negligible). Upstream's
+  354-step comparison is in a regime where ||I_R|| ~ 6e18 (physical reaction term
+  dominant) -- there a relative tolerance is meaningful; at step 10 it is not.
+- Reproducibility: between the 1-GPU run and its identical rerun, and vs the 2-GPU,
+  4-GPU and CPU runs, max|dI_R| = 0.27-0.33 (relative 0.97-1.46 over the two
+  campaigns), spread over 78-100 % of the cells; within the 10 steps the differences
+  are uncorrelated with the (1e-13 relative) state differences (corr(dI_R, d rho_e)
+  = -0.006; the observed dI_R would correspond, through the predictor coupling
+  dt a_half dI_R / a_end^2, to a change of rho_E of order 8e6 code units if it acted
+  fully, and nothing of the kind is present in the final state: rho_E agrees to
+  1e-15). This is an observation over 10 steps of a deck in which the net
+  heating/cooling is negligible. **It does not show that the I_R variation has no
+  effect on later steps**: I_R is consumed by the next step's hydro predictor and in
+  the SDC branch it is the increment applied to (rho e)/(rho E), so it is a
+  state-carried coupling term with an unresolved reproducibility question -- not a
+  diagnostic field that has been shown to be irrelevant. Whether the variability
+  comes from the CVODE per-cell solution at its tolerance (`sundials_reltol/abstol`
+  1e-4 defaults), from the batched host-device buffers (`rho_init_vode`,
+  `rhoe_src_vode`, `e_src_vode`) or from something else is **not identified** from
+  the plotfiles; a race or an uninitialised read is neither shown nor excluded.
+- Diagnostic estimate (not a verified error bound): if one asks how large a
+  per-cell change of I_R could be before the next step's energy update moves out of
+  the state tolerance, the linear one-step coupling gives
+  `|dI_R|_max ~ tol_state * max_cells|a^2 rho e| / (dt * a_half)`
+  with tol_state = 5e-5 (the heat/cool state tolerance), `max|a^2 rho e|/(dt a_half)`
+  = 3.7e2 read from the saved plt00010 of the 1-GPU run (Nyx code units: rho e in
+  comoving M_sun/Mpc^3 (km/s)^2, dt the runlog `dt` of step 10 = 2.99e5 code time
+  units, a = 0.01094 at z = 90.4, a_half the mid-step value), i.e. **~1.9e-2 in the
+  units of I_R (energy density per code time)**. Assumptions: linear one-step
+  coupling only, no amplification over later steps, the maximum-|a^2 rho e| cell
+  as the worst case, and a single tolerance for all cells. The observed
+  |dI_R| = 0.28-0.33 exceeds this estimate ~15x, so no criterion derived from the
+  state tolerance accepts the field; and "exclude it and PASS" is not a criterion.
+  The estimate is recorded to show why no tolerance is set, not as a proven bound.
+- Consequence: the validator reports `I_R` (parsed, finite, not gated) and
+  downgrades the case to I_R_CHECK_PENDING (exit 3). The heat/cool deck is
+  therefore **not** in the count of full correctness passes and its timings do not
+  enter a performance summary. Resolving it needs, at the original configuration,
+  CPU-vs-CPU and GPU-vs-GPU repeat runs of I_R, a run with per-step plotfiles (or
+  the SDC_IR/hydro-source intermediates) to attribute the variation to a term, an
+  independent tolerance comparison, and a regime (more steps, lower z) where the
+  reaction term is physical. None of this was run in the 2026-09-07/08 rounds;
+  historical run directories are untouched.
+
+Heat/cool strong scaling (`Exec/LyA/inputs` 64^3, `amr.max_grid_size=16` -> 64
+boxes, 10 steps) and the 8/40/80-GPU dry-runs: 8 ranks planned (2 hypothetical
+nodes, balanced); 40 ranks planned and flagged **IMBALANCED (64 boxes over 40
+ranks)**; 80 ranks **refused** ("80 ranks requested but ... only 64 boxes -- a rank
+without work is never launched silently"). Timings of the 1/2/4-GPU strong runs:
+`SECOND_BATCH_STATUS.md`.
+
+## Files
+
+`fetch.sh` (pinned clones + SHA checks), `build.sh` (staged, profile-aware,
+fingerprinted), `run.sh` (cases/modes, decomposition guard, manifest),
+`validate.sh` + `nyx_particle_compare.py` (criteria above), this README.
diff --git a/level3/nyx/build.sh b/level3/nyx/build.sh
new file mode 100755
index 0000000..1b66492
--- /dev/null
+++ b/level3/nyx/build.sh
@@ -0,0 +1,192 @@
+#!/usr/bin/env bash
+# Build Nyx 26.09 natively (CMake) against a PRIVATE AMReX 26.09 install and, for
+# the heating/cooling variant, a private SUNDIALS 7.2.1 (the commit Nyx pins).
+# Staged build, all inside one profile tree -- nothing shared with WarpX/Level 2:
+#   [1] SUNDIALS (heatcool only)  .deps/level3/nyx/<profile>/install/sundials
+#   [2] AMReX 26.09               .deps/level3/nyx/<profile>/install/amrex
+#   [3] Nyx                       build/level3/nyx/<profile>  -> install/bin/nyx_*
+#
+#   ./build.sh [CUDA|HIP|CPU]          (default CUDA)
+#   HPCPERF_NYX_HEATCOOL=NO|YES        (default NO -> adiabatic variant; YES -> SUNDIALS CVODE + CUDA fused kernels)
+#   HPCPERF_NYX_PROFILE=<name>         (override the derived profile name)
+#   HPCPERF_BUILD_JOBS=N               (default 32)
+#
+# Profiles: <backend><toolkit>-gcc<ver>-<variant>, e.g.
+#   cuda132-gcc133-adiabatic   CUDA 13.2 / conda GCC 13.3 / Nyx_HEATCOOL=NO, sm_100
+#   cuda132-gcc133-heatcool    same + Nyx_HEATCOOL=YES (SUNDIALS 7.2.1, ENABLE_CUDA, fused kernels)
+#   cpu-gcc133-adiabatic       Nyx_GPU_BACKEND=NONE reference build; its AMReX also builds the
+#                              plotfile tools (amrex_fcompare/fnan/fvolumesum/...) and this script
+#                              adds particle_compare -- the official comparison tools validate.sh uses
+#
+# Why an external AMReX 26.09 instead of Nyx's submodule pin (6e875b7c): the
+# pin's CMake drops SM >= 10.0 (convert_cuda_archs) and autodetects 8.6+PTX on
+# this node -> an sm_86 binary (first attempt, removed; see README). AMReX 26.09
+# (a52ca73, 21 commits ahead / 0 behind the pin) resolves sm_100 correctly. Nyx
+# requires AMReX >= 20.11 and consumes it through find_package(AMReX CONFIG).
+# AMReX options = exactly what Nyx's superbuild would set for the same Nyx options
+# (cmake/NyxSetupAMReX.cmake: 3D, DOUBLE, PARTICLES(PDOUBLE), MPI, no OMP, no
+# Fortran/PROBINIT, LINEAR_SOLVERS, SUNDIALS iff HEATCOOL, GPU backend); SUNDIALS
+# options = cmake/NyxSetupSUNDIALS.cmake (CVODE + ARKODE -- AMReX 26.09 requires the arkode component -- index 32, fused kernels).
+# Nyx options mirror upstream's GPU CI (Nyx_HYDRO=YES Nyx_MPI=YES Nyx_OMP=NO,
+# CMAKE_CXX_STANDARD=17). Modification class: A (build options; out-of-source
+# builds; no file of any checkout is modified).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+l3_isolate_build_env    # never see Level 2 .deps/install prefixes
+
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+HC="$(echo "${HPCPERF_NYX_HEATCOOL:-NO}" | tr '[:lower:]' '[:upper:]')"
+case "$HC" in YES|NO) ;; *) echo "build.sh: HPCPERF_NYX_HEATCOOL must be YES or NO" >&2; exit 2;; esac
+VARIANT=adiabatic; [ "$HC" = YES ] && VARIANT=heatcool
+SRC="$R/_upstream/level3/Nyx"; AMREX_SRC="$R/_upstream/level3/amrex"
+[ -f "$SRC/CMakeLists.txt" ] && [ -f "$AMREX_SRC/CMakeLists.txt" ] || { echo "build.sh: sources missing -- run $HERE/fetch.sh first" >&2; exit 1; }
+[ "$HC" = NO ] || [ -f "$SRC/subprojects/sundials/CMakeLists.txt" ] || { echo "build.sh: sundials submodule missing -- run $HERE/fetch.sh" >&2; exit 1; }
+SHA="$(git -C "$SRC" rev-parse HEAD)"; AMREX_SHA="$(git -C "$AMREX_SRC" rev-parse HEAD)"
+SUNDIALS_SHA="$(git -C "$SRC/subprojects/sundials" rev-parse HEAD 2>/dev/null || echo none)"
+GCC_MM="$(l3_version_mm "$("$CXX" -dumpfullversion 2>/dev/null || "$CXX" -dumpversion)")"
+
+case "$BACKEND" in
+    CUDA)
+        ARCH="${HPCPERF_CUDA_ARCH:-$(l3_gpu_arch)}"; [ -n "$ARCH" ] || { echo "build.sh: cannot determine GPU arch (no GPU?) -- set HPCPERF_CUDA_ARCH" >&2; exit 1; }
+        MODEL=cuda; ARCHNOTE="sm_$ARCH"; AMREX_GPU=CUDA
+        ARCH_FLAGS=("-DCMAKE_CUDA_ARCHITECTURES=$ARCH" "-DCMAKE_CUDA_HOST_COMPILER=$CXX")
+        PROFILE_DEFAULT="cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-${VARIANT}" ;;
+    HIP)
+        command -v hipcc >/dev/null 2>&1 || { echo "build.sh: HIP requested but hipcc not found -- HIP build is UNTESTED on this machine (no ROCm)" >&2; exit 1; }
+        ARCH="${HPCPERF_HIP_ARCH:-gfx950}"; MODEL=hip; ARCHNOTE="$ARCH"; AMREX_GPU=HIP
+        ARCH_FLAGS=("-DAMReX_AMD_ARCH=$ARCH" -DCMAKE_CXX_COMPILER=hipcc)
+        PROFILE_DEFAULT="hip-${ARCH}-${VARIANT}" ;;
+    CPU|NONE)
+        MODEL=cpu; ARCHNOTE=host; AMREX_GPU=NONE; ARCH_FLAGS=()
+        PROFILE_DEFAULT="cpu-gcc${GCC_MM}-${VARIANT}" ;;
+    *) echo "usage: $0 [CUDA|HIP|CPU]" >&2; exit 2 ;;
+esac
+PROFILE="${HPCPERF_NYX_PROFILE:-$PROFILE_DEFAULT}"
+l3_paths_profile nyx "$PROFILE"
+BUILD_DIR="$L3_BUILD"
+JOBS="${HPCPERF_BUILD_JOBS:-32}"
+AMREX_PREFIX="$L3_INSTALL/amrex"; SUND_PREFIX="$L3_INSTALL/sundials"
+TOOLS=OFF; [ "$MODEL" = cpu ] && TOOLS=ON
+
+AMREX_OPTS="AMReX_SPACEDIM=3 AMReX_PRECISION=DOUBLE AMReX_PARTICLES=ON AMReX_PARTICLES_PRECISION=DOUBLE AMReX_MPI=ON AMReX_OMP=OFF AMReX_FORTRAN=OFF AMReX_PROBINIT=OFF AMReX_LINEAR_SOLVERS=ON AMReX_EB=OFF AMReX_FFT=OFF AMReX_SUNDIALS=$( [ "$HC" = YES ] && echo ON || echo OFF) AMReX_GPU_BACKEND=$AMREX_GPU arch=$ARCHNOTE AMReX_PLOTFILE_TOOLS=$TOOLS"
+CMAKE_OPTS="Nyx_GPU_BACKEND=$AMREX_GPU arch=$ARCHNOTE Nyx_HYDRO=YES Nyx_HEATCOOL=$HC Nyx_MPI=YES Nyx_OMP=NO Nyx_SINGLE_PRECISION_PARTICLES=NO CMAKE_CXX_STANDARD=17 CMAKE_BUILD_TYPE=Release amrex=external($AMREX_OPTS) sundials=$( [ "$HC" = YES ] && echo "external(ENABLE_CUDA=$( [ "$MODEL" = cuda ] && echo ON || echo OFF) INDEX_SIZE=32 FUSED_KERNELS=$( [ "$MODEL" = cuda ] && echo ON || echo OFF) CVODE+ARKODE(AMReX 26.09 SUNDIALS component set))" || echo off)"
+DEPS="amrex=26.09($AMREX_SHA) [Nyx submodule pin $(git -C "$SRC/subprojects/amrex" rev-parse HEAD 2>/dev/null || echo unknown) not used: no sm_100 through CMake] sundials=$( [ "$HC" = YES ] && echo "7.2.1($SUNDIALS_SHA)" || echo off) profile=$PROFILE"
+FP="$(l3_fingerprint_text nyx "$SHA" "$MODEL" "$DEPS" "$CMAKE_OPTS" "runtime(amrex.use_gpu_aware_mpi default)")"
+l3_fingerprint_check "$L3_INSTALL" "$FP" || exit 1
+
+COMMON=(-G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17 -DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX")
+[ "$MODEL" = hip ] && COMMON=(-G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17 -DCMAKE_C_COMPILER="$CC")
+echo "# Nyx $BACKEND profile=$PROFILE: Nyx $SHA (26.09), AMReX $AMREX_SHA (26.09, external), sundials $( [ "$HC" = YES ] && echo "$SUNDIALS_SHA (7.2.1)" || echo off), arch $ARCHNOTE, host $CXX, MPI $(mpirun --version 2>/dev/null | head -1)"
+echo "# resources: -j$JOBS; expected AMReX 5-10 min + Nyx 2-5 min (+ SUNDIALS ~3 min); trees $L3_BUILD_DEPS, $BUILD_DIR"
+t0=$(date +%s)
+
+stage() { # stage <name> <src> <build> <log-prefix> [cmake options...]
+    local name=$1 src=$2 bld=$3 log=$4; shift 4
+    mkdir -p "$bld"
+    cmake -S "$src" -B "$bld" "${COMMON[@]}" "$@" > "$L3_LOGS/$log-configure.log" 2>&1 \
+        || { tail -40 "$L3_LOGS/$log-configure.log"; echo "build.sh: $name configure failed (log: $L3_LOGS/$log-configure.log)" >&2; exit 1; }
+    cmake --build "$bld" -j "$JOBS" > "$L3_LOGS/$log-build.log" 2>&1 \
+        || { tail -40 "$L3_LOGS/$log-build.log"; echo "build.sh: $name build failed (log: $L3_LOGS/$log-build.log)" >&2; exit 1; }
+}
+
+# [1] SUNDIALS 7.2.1 (Nyx's pinned submodule commit), options from cmake/NyxSetupSUNDIALS.cmake
+if [ "$HC" = YES ] && [ ! -f "$SUND_PREFIX/.hpcperf-stage-done" ]; then
+    SUND_GPU=(-DENABLE_CUDA=OFF)
+    [ "$MODEL" = cuda ] && SUND_GPU=(-DENABLE_CUDA=ON -DSUNDIALS_INDEX_SIZE=32 -DSUNDIALS_BUILD_PACKAGE_FUSED_KERNELS=ON "${ARCH_FLAGS[@]}")
+    [ "$MODEL" = hip ] && SUND_GPU=(-DENABLE_HIP=ON -DSUNDIALS_BUILD_PACKAGE_FUSED_KERNELS=ON)
+    stage SUNDIALS "$SRC/subprojects/sundials" "$L3_BUILD_DEPS/sundials" sundials \
+        -DCMAKE_INSTALL_PREFIX="$SUND_PREFIX" -DBUILD_SHARED_LIBS=OFF -DBUILD_STATIC_LIBS=ON \
+        -DEXAMPLES_ENABLE_C=OFF -DEXAMPLES_ENABLE_CXX=OFF -DEXAMPLES_INSTALL=OFF -DENABLE_MPI=OFF -DENABLE_OPENMP=OFF \
+        -DBUILD_ARKODE=ON -DBUILD_KINSOL=OFF -DBUILD_IDA=OFF -DBUILD_IDAS=OFF -DBUILD_CVODES=OFF -DBUILD_TESTING=OFF \
+        "${SUND_GPU[@]}"
+    cmake --install "$L3_BUILD_DEPS/sundials" > "$L3_LOGS/sundials-install.log" 2>&1 || { echo "build.sh: SUNDIALS install failed" >&2; exit 1; }
+    touch "$SUND_PREFIX/.hpcperf-stage-done"
+fi
+t1=$(date +%s)
+
+# [2] AMReX 26.09, component set identical to what Nyx's superbuild sets for these Nyx options
+if [ ! -f "$AMREX_PREFIX/.hpcperf-stage-done" ]; then
+    AMREX_SUND=(-DAMReX_SUNDIALS=OFF); [ "$HC" = YES ] && AMREX_SUND=(-DAMReX_SUNDIALS=ON "-DSUNDIALS_ROOT=$SUND_PREFIX")
+    stage AMReX "$AMREX_SRC" "$L3_BUILD_DEPS/amrex" amrex \
+        -DCMAKE_INSTALL_PREFIX="$AMREX_PREFIX" -DBUILD_SHARED_LIBS=OFF \
+        -DAMReX_SPACEDIM=3 -DAMReX_PRECISION=DOUBLE -DAMReX_PARTICLES=ON -DAMReX_PARTICLES_PRECISION=DOUBLE \
+        -DAMReX_MPI=ON -DAMReX_OMP=OFF -DAMReX_FORTRAN=OFF -DAMReX_PROBINIT=OFF -DAMReX_LINEAR_SOLVERS=ON \
+        -DAMReX_EB=OFF -DAMReX_FFT=OFF -DAMReX_AMRDATA=OFF -DAMReX_BUILD_TUTORIALS=OFF -DAMReX_INSTALL=ON \
+        "-DAMReX_GPU_BACKEND=$AMREX_GPU" -DAMReX_PLOTFILE_TOOLS="$TOOLS" "${ARCH_FLAGS[@]}" "${AMREX_SUND[@]}"
+    cmake --install "$L3_BUILD_DEPS/amrex" > "$L3_LOGS/amrex-install.log" 2>&1 || { echo "build.sh: AMReX install failed" >&2; exit 1; }
+    if [ "$MODEL" = cuda ]; then
+        archs="$(/usr/bin/grep -h 'AMREX_CUDA_ARCHS:INTERNAL' "$L3_BUILD_DEPS/amrex/CMakeCache.txt" | cut -d= -f2)"
+        [ "$archs" = "$ARCH" ] || { echo "build.sh: AMReX resolved CUDA archs '$archs', expected '$ARCH' -- refusing" >&2; exit 1; }
+    fi
+    touch "$AMREX_PREFIX/.hpcperf-stage-done"
+fi
+t2=$(date +%s)
+
+# [3] Nyx against the private AMReX (+SUNDIALS). ENABLE_CUDA=ON makes Nyx include
+# AMReXTargetHelpers (setup_target_for_cuda_compilation) in the external-AMReX branch.
+NYX_GPU=(-DNyx_GPU_BACKEND="$AMREX_GPU")
+[ "$MODEL" = cuda ] && NYX_GPU+=(-DENABLE_CUDA=ON "${ARCH_FLAGS[@]}")
+[ "$MODEL" = hip ] && NYX_GPU+=("${ARCH_FLAGS[@]}")
+NYX_SUND=(); [ "$HC" = YES ] && NYX_SUND=("-DSUNDIALS_ROOT=$SUND_PREFIX")
+# Nyx_SINGLE_PRECISION_PARTICLES=NO: double-precision particles, as upstream's nightly
+# regression builds (GNU make default; LyA-adiabatic passes USE_SINGLE_PRECISION_PARTICLES=FALSE
+# explicitly) -- the CMake default would be single precision (PSINGLE).
+stage Nyx "$SRC" "$BUILD_DIR" nyx \
+    "-DAMReX_ROOT=$AMREX_PREFIX" "-DCMAKE_PREFIX_PATH=$AMREX_PREFIX" \
+    -DNyx_HYDRO=YES -DNyx_HEATCOOL="$HC" -DNyx_MPI=YES -DNyx_OMP=NO -DNyx_SINGLE_PRECISION_PARTICLES=NO \
+    "${NYX_GPU[@]}" "${NYX_SUND[@]}"
+/usr/bin/grep -q 'AMReX found: configuration file located at' "$L3_LOGS/nyx-configure.log" \
+    || { echo "build.sh: Nyx did not pick up the external AMReX (would have fallen back to the submodule)" >&2; exit 1; }
+t3=$(date +%s)
+
+mkdir -p "$L3_INSTALL/bin"
+for exe in Exec/MiniSB/nyx_MiniSB Exec/LyA/nyx_LyA Exec/AMR-density/nyx_AMR-density; do
+    [ -x "$BUILD_DIR/$exe" ] && cp -f "$BUILD_DIR/$exe" "$L3_INSTALL/bin/"
+done
+[ -x "$L3_INSTALL/bin/nyx_MiniSB" ] && [ -x "$L3_INSTALL/bin/nyx_LyA" ] || { echo "build.sh: nyx_MiniSB / nyx_LyA not produced under $BUILD_DIR/Exec" >&2; exit 1; }
+if [ "$MODEL" = cuda ]; then
+    for exe in "$L3_INSTALL"/bin/nyx_*; do
+        got="$(cuobjdump --list-elf "$exe" 2>/dev/null | /usr/bin/grep -o 'sm_[0-9a-z]*' | sort -u | paste -sd,)"
+        [ "$got" = "sm_$ARCH" ] || { echo "build.sh: $(basename "$exe") embeds '$got', expected sm_$ARCH -- refusing to install" >&2; exit 1; }
+    done
+fi
+if [ "$MODEL" = cpu ]; then
+    for t in fcompare fnan fvolumesum fextrema fvarnames ftime; do
+        [ -x "$AMREX_PREFIX/bin/amrex_$t" ] && cp -f "$AMREX_PREFIX/bin/amrex_$t" "$L3_INSTALL/bin/"
+    done
+    [ -x "$L3_INSTALL/bin/amrex_fcompare" ] || { echo "build.sh: amrex_fcompare not installed by AMReX (AMReX_PLOTFILE_TOOLS)" >&2; exit 1; }
+    # particle_compare (AMReX Tools/Postprocessing/C_Src) against the installed AMReX
+    PC_SRC="$L3_SRC/particle_compare"; rm -rf "$PC_SRC"; mkdir -p "$PC_SRC"
+    cp "$AMREX_SRC/Tools/Postprocessing/C_Src/particle_compare.cpp" "$PC_SRC/"
+    cat > "$PC_SRC/CMakeLists.txt" <<'EOF'
+cmake_minimum_required(VERSION 3.24)
+project(particle_compare C CXX)   # AMReXConfig's find_dependency(MPI) needs the C language enabled
+find_package(AMReX REQUIRED CONFIG)
+add_executable(particle_compare particle_compare.cpp)
+target_link_libraries(particle_compare PRIVATE AMReX::amrex_3d)
+EOF
+    PC_EXTRA=(); [ "$HC" = YES ] && PC_EXTRA=("-DSUNDIALS_ROOT=$SUND_PREFIX")   # AMReXConfig's find_dependency(SUNDIALS) for the heatcool AMReX
+    stage particle_compare "$PC_SRC" "$L3_BUILD_DEPS/particle_compare" particle_compare "-DAMReX_ROOT=$AMREX_PREFIX" "${PC_EXTRA[@]}"
+    cp -f "$L3_BUILD_DEPS/particle_compare/particle_compare" "$L3_INSTALL/bin/"
+fi
+l3_fingerprint_write "$L3_INSTALL" "$FP"
+{
+    echo "profile=$PROFILE backend=$MODEL variant=$VARIANT jobs=$JOBS utc=$(date -u +%FT%TZ)"
+    echo "seconds: sundials=$((t1 - t0)) amrex=$((t2 - t1)) nyx=$((t3 - t2)) total=$(( $(date +%s) - t0 ))"
+    echo "amrex=$AMREX_SHA (26.09) options: $AMREX_OPTS"
+    [ "$HC" = YES ] && echo "sundials=$SUNDIALS_SHA (7.2.1)"
+    for b in "$L3_INSTALL"/bin/*; do echo "$(basename "$b") sha256=$(l3_sha_file "$b")"; done
+    if [ "$MODEL" = cuda ]; then
+        echo "AMREX_CUDA_ARCHS=$(/usr/bin/grep -h 'AMREX_CUDA_ARCHS:INTERNAL' "$L3_BUILD_DEPS/amrex/CMakeCache.txt" | cut -d= -f2)"
+        echo "cuobjdump(nyx_MiniSB): $(cuobjdump --list-elf "$L3_INSTALL/bin/nyx_MiniSB" 2>/dev/null | /usr/bin/grep -o 'sm_[0-9a-z]*' | sort -u | paste -sd,) (cudart static; ldd shows $(ldd "$L3_INSTALL/bin/nyx_MiniSB" | /usr/bin/grep -oE 'lib(cudart|cuda|cusparse|cublas|curand)[^ ]*' | sort -u | paste -sd, || echo none))"
+    fi
+} > "$L3_INSTALL/BUILD_INFO.txt"
+echo "# built in $(( $(date +%s) - t0 )) s (sundials $((t1 - t0)), amrex $((t2 - t1)), nyx $((t3 - t2))): $(ls "$L3_INSTALL/bin" | paste -sd' ') (installed under $L3_INSTALL)"
+echo "# compiler warning lines (nyx): $(/usr/bin/grep -c 'warning' "$L3_LOGS/nyx-build.log" || true)"
+[ "$MODEL" = cuda ] && /usr/bin/grep -E 'AMREX_CUDA_ARCHS|cuobjdump' "$L3_INSTALL/BUILD_INFO.txt"
+exit 0
diff --git a/level3/nyx/fetch.sh b/level3/nyx/fetch.sh
new file mode 100755
index 0000000..3f23972
--- /dev/null
+++ b/level3/nyx/fetch.sh
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# Fetch Nyx (AMReX-Astro) at the pinned release plus the AMReX release it is
+# built against here. Read-only checkouts under _upstream/level3/; nothing is
+# committed to this repository.
+#
+#   Nyx 26.09            e06eabc1b9dbcad5612db9529aced682402daede  (tag 26.09, 2026-08-26)
+#     subprojects/amrex    6e875b7cc1a4eec78e22ae4cdaa79f88acf5169e  (development 2026-08-12; the pin)
+#     subprojects/sundials 5c53be85c88f63c5201c130b8cb2c686615cfb03  (v7.2.1; used by the heatcool profile)
+#   AMReX 26.09          a52ca73324ac2c7b65ec04f131e6df99eec9c576  (tag 26.09, 2026-09-01; 21 commits
+#                        ahead of / 0 behind the Nyx pin -- a strict descendant)
+#
+# Why AMReX 26.09 and not the submodule pin: the pinned AMReX resolves CUDA
+# architectures through CMake's legacy cuda_select_nvcc_arch_flags and its
+# convert_cuda_archs() drops every SM >= 10.0 ("CMake 3.30 does not support SM
+# 10.0+"), then autodetects 8.6+PTX on this B200 node -> an sm_86 binary (first
+# attempt, removed). AMReX 26.09 resolves the architecture through nvcc
+# --list-gpu-arch into CMAKE_CUDA_ARCHITECTURES (the path WarpX 26.09 already
+# verified here with sm_100). Nyx's own minimum is AMREX_MINIMUM_VERSION 20.11
+# and it consumes an external AMReX through find_package(AMReX CONFIG). This is a
+# private build for Nyx: WarpX's AMReX *binary* is not reused (WarpX's superbuild
+# builds AMReX with a different component set: EB, no linear solvers, FFT ...).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+UP="$R/_upstream/level3"
+NYX_TAG=26.09;   NYX_SHA=e06eabc1b9dbcad5612db9529aced682402daede
+AMREX_PIN_SHA=6e875b7cc1a4eec78e22ae4cdaa79f88acf5169e
+SUNDIALS_SHA=5c53be85c88f63c5201c130b8cb2c686615cfb03
+AMREX_TAG=26.09; AMREX_SHA=a52ca73324ac2c7b65ec04f131e6df99eec9c576
+
+mkdir -p "$UP"
+if [ ! -d "$UP/Nyx/.git" ]; then git clone -q --branch "$NYX_TAG" --depth 1 https://github.com/AMReX-Astro/Nyx.git "$UP/Nyx"; fi
+got="$(git -C "$UP/Nyx" rev-parse HEAD)"
+[ "$got" = "$NYX_SHA" ] || { echo "fetch.sh: $UP/Nyx is at $got, expected $NYX_SHA (tag $NYX_TAG)" >&2; exit 1; }
+git -C "$UP/Nyx" submodule update --init --depth 1 subprojects/amrex subprojects/sundials
+a="$(git -C "$UP/Nyx/subprojects/amrex" rev-parse HEAD)"; s="$(git -C "$UP/Nyx/subprojects/sundials" rev-parse HEAD)"
+[ "$a" = "$AMREX_PIN_SHA" ] || { echo "fetch.sh: Nyx amrex submodule at $a, expected $AMREX_PIN_SHA" >&2; exit 1; }
+[ "$s" = "$SUNDIALS_SHA" ] || { echo "fetch.sh: Nyx sundials submodule at $s, expected $SUNDIALS_SHA" >&2; exit 1; }
+
+if [ ! -d "$UP/amrex/.git" ]; then git clone -q --branch "$AMREX_TAG" --depth 1 https://github.com/AMReX-Codes/amrex.git "$UP/amrex"; fi
+x="$(git -C "$UP/amrex" rev-parse HEAD)"
+[ "$x" = "$AMREX_SHA" ] || { echo "fetch.sh: $UP/amrex is at $x, expected $AMREX_SHA (tag $AMREX_TAG)" >&2; exit 1; }
+echo "# Nyx $NYX_TAG $got (submodules amrex $a, sundials $s); AMReX $AMREX_TAG $x -> $UP"
diff --git a/level3/nyx/nyx_fcompare_check.py b/level3/nyx/nyx_fcompare_check.py
new file mode 100755
index 0000000..9d87a54
--- /dev/null
+++ b/level3/nyx/nyx_fcompare_check.py
@@ -0,0 +1,286 @@
+#!/usr/bin/env python3
+"""Strict plotfile comparison for the Nyx validator (wraps AMReX fcompare, never trusts it blindly).
+
+    nyx_fcompare_check.py <ref_plotfile> <plotfile> --rel_tol R [--abs_tol_zero_ref A] [--diagnostic VAR ...]
+                          --fcompare PATH --fextrema PATH [--out FILE] [--label L]
+
+Steps (every one must succeed; a structural problem is exit 2, a tolerance violation exit 1, agreement exit 0,
+an unsupported-but-legal box layout exit 4):
+ 1. Header check (independent of fcompare): both plotfile Headers are parsed; the variable SET, the
+    dimension, the number of levels, the simulation time (relative difference <= 1e-12), the domain box
+    and the cell sizes must be identical. The box array of every level is checked GEOMETRICALLY (physical
+    extents -> cell-index boxes via prob_lo/dx): in each plotfile the boxes must lie inside the level
+    domain, must not overlap, and at level 0 must cover the domain exactly (a missing box, an overlap or a
+    coverage deficit is STRUCTURAL, exit 2). Between the two plotfiles: identical box sets (any order)
+    proceed; different partitions of the SAME cell set (a legal re-blocking, e.g. another max_grid_size)
+    are reported as UNSUPPORTED_LAYOUT (exit 4) -- this validator does not compare across BoxArrays
+    (fcompare would need --allow_diff_grids), it never treats that as a science FAIL and never skips
+    it silently; different covered cell sets at a level (missing/extra refinement) are STRUCTURAL.
+ 2. Raw finiteness: `fextrema` on BOTH plotfiles; every variable of the header set must appear exactly once
+    with finite min/max. A reference variable whose min == max == 0 is a ZERO-REFERENCE field.
+ 3. fcompare (`-n 0 --rel_tol R --abort_if_not_all_found`): stdout is parsed strictly -- one `level = L`
+    block per level, exactly one row per variable of the header set, no duplicates, no extra rows, no
+    `< ... >` message rows (missing variable, NaN), no WARNING/ERROR lines; abs and rel must parse as
+    numbers (inf/nan are parsed, never dropped).
+ 4. Decision per variable (all levels):
+      diagnostic variables : parsed and reported only (still must be finite in step 2 and parseable in 3);
+      zero-reference field : abs error <= abs_tol_zero_ref (default 0 = exact) -- the relative error is
+                             undefined there (fcompare prints inf or 0) and is never used;
+      otherwise            : rel must be finite and <= rel_tol. A non-finite rel on a non-zero reference
+                             is a violation, not a skip.
+ 5. Consistency with the tool: if the parsed verdict is AGREE while fcompare exited non-zero, or DISAGREE
+    while it exited 0, the run is a STRUCTURAL failure (the parser and the tool must agree).
+The full report (headers, extrema, fcompare output, per-variable decisions) is written to --out.
+"""
+import argparse, math, os, re, subprocess, sys
+
+STRUCT, TOL, OK, UNSUPPORTED = 2, 1, 0, 4
+
+
+class Fail(Exception):
+    def __init__(self, code, msg):
+        super().__init__(msg); self.code = code
+
+
+def read_header(plt):
+    """Parse an AMReX plotfile Header: names, dim, time, finest level, domain, cell sizes, per-level boxes."""
+    p = os.path.join(plt, "Header")
+    if not os.path.isfile(p): raise Fail(STRUCT, f"{p} missing")
+    L = [l.rstrip("\n") for l in open(p)]
+    try:
+        i = 0; version = L[i].strip(); i += 1
+        nvars = int(L[i]); i += 1
+        names = [L[i + k].strip() for k in range(nvars)]; i += nvars
+        dim = int(L[i]); i += 1
+        time = float(L[i]); i += 1
+        finest = int(L[i]); i += 1
+        prob_lo = [float(x) for x in L[i].split()]; i += 1
+        prob_hi = [float(x) for x in L[i].split()]; i += 1
+        i += 1  # refinement ratios (empty line for a single level)
+        dom = re.findall(r"\(\(([-\d, ]+)\) \(([-\d, ]+)\) \(([-\d, ]+)\)\)", L[i]); i += 1
+        steps = L[i].split(); i += 1
+        dx = [[float(x) for x in L[i + k].split()] for k in range(finest + 1)]; i += finest + 1
+        coord = int(L[i]); i += 1
+        i += 1  # bwidth
+        levels = []
+        for lev in range(finest + 1):
+            hdr = L[i].split(); i += 1
+            if int(hdr[0]) != lev: raise ValueError(f"level header {hdr} for level {lev}")
+            ngrids = int(hdr[1]); ltime = float(hdr[2])
+            lstep = int(L[i]); i += 1
+            boxes = []
+            for g in range(ngrids):
+                ext = []
+                for d in range(dim):
+                    lo, hi = [float(x) for x in L[i].split()]; i += 1
+                    ext.append((lo, hi))
+                boxes.append(tuple(ext))
+            path = L[i].strip(); i += 1
+            levels.append(dict(ngrids=ngrids, time=ltime, step=lstep, boxes=boxes, path=path))
+    except (IndexError, ValueError) as ex:
+        raise Fail(STRUCT, f"{p}: cannot parse AMReX plotfile Header ({ex})")
+    if len(set(names)) != nvars: raise Fail(STRUCT, f"{p}: duplicate variable names in Header")
+    if len(dom) != finest + 1: raise Fail(STRUCT, f"{p}: {len(dom)} domain boxes for {finest + 1} level(s)")
+    return dict(version=version, nvars=nvars, names=names, dim=dim, time=time, finest=finest,
+                prob_lo=prob_lo, prob_hi=prob_hi, domain=dom, dx=dx, coord=coord, levels=levels)
+
+
+def index_boxes(h, lev, tag):
+    """Physical box extents of one level -> cell-index boxes [(lo, hi)] through prob_lo and dx (hi inclusive)."""
+    out = []
+    for ext in h["levels"][lev]["boxes"]:
+        lo, hi = [], []
+        for d, (plo, phi) in enumerate(ext):
+            x0 = (plo - h["prob_lo"][d]) / h["dx"][lev][d]; x1 = (phi - h["prob_lo"][d]) / h["dx"][lev][d]
+            i0, i1 = round(x0), round(x1)
+            if abs(x0 - i0) > 1e-6 or abs(x1 - i1) > 1e-6:
+                raise Fail(STRUCT, f"level {lev}: {tag} box {ext} is not aligned to the cell grid (dx={h['dx'][lev][d]!r})")
+            if i1 <= i0: raise Fail(STRUCT, f"level {lev}: {tag} box {ext} is empty")
+            lo.append(i0); hi.append(i1 - 1)
+        out.append((tuple(lo), tuple(hi)))
+    return out
+
+
+def box_volume(lo, hi):
+    v = 1
+    for l, h in zip(lo, hi): v *= (h - l + 1)
+    return v
+
+
+def box_intersection(a, b):
+    lo = tuple(max(x, y) for x, y in zip(a[0], b[0])); hi = tuple(min(x, y) for x, y in zip(a[1], b[1]))
+    return 0 if any(l > h for l, h in zip(lo, hi)) else box_volume(lo, hi)
+
+
+def check_layout(a, b, lev):
+    """Geometric BoxArray check of one level. Returns a report line for identical layouts; raises
+    Fail(STRUCT) for a missing box / overlap / coverage deficit / different covered cell sets, and
+    Fail(UNSUPPORTED) for a legal re-blocking (same cells, different partition)."""
+    dom = [tuple(int(x) for x in s.split(",")) for s in a["domain"][lev][:2]]
+    A, B = index_boxes(a, lev, "ref"), index_boxes(b, lev, "test")
+    totals = {}
+    for tag, X in (("ref", A), ("test", B)):
+        for (lo, hi) in X:
+            if any(l < dl or h > dh for l, h, dl, dh in zip(lo, hi, dom[0], dom[1])):
+                raise Fail(STRUCT, f"level {lev}: {tag} box {lo}-{hi} lies outside the level domain {dom[0]}-{dom[1]}")
+        for i in range(len(X)):
+            for j in range(i + 1, len(X)):
+                if box_intersection(X[i], X[j]) > 0:
+                    raise Fail(STRUCT, f"level {lev}: {tag} boxes {X[i][0]}-{X[i][1]} and {X[j][0]}-{X[j][1]} overlap")
+        totals[tag] = sum(box_volume(lo, hi) for lo, hi in X)
+        if lev == 0 and totals[tag] != box_volume(*dom):
+            raise Fail(STRUCT, f"level 0: {tag} boxes cover {totals[tag]} of {box_volume(*dom)} domain cells "
+                               f"({len(X)} boxes) -- missing box / coverage deficit")
+    if sorted(A) == sorted(B):
+        return f"    level {lev}: identical box array ({len(A)} boxes, {totals['ref']} cells)"
+    common = sum(box_intersection(x, y) for x in A for y in B)
+    if totals["ref"] == totals["test"] == common:
+        raise Fail(UNSUPPORTED, f"level {lev}: legal re-blocking (ref {len(A)} boxes, test {len(B)} boxes, the same "
+                                f"{common} cells covered) -- comparison across different BoxArrays is not supported by this "
+                                f"validator (fcompare would need --allow_diff_grids); comparison NOT performed")
+    raise Fail(STRUCT, f"level {lev}: box arrays cover different cell sets (ref {totals['ref']} cells, test {totals['test']}, "
+                       f"common {common}) -- missing/extra refinement, not a re-blocking")
+
+
+def check_headers(a, b):
+    out = []
+    if set(a["names"]) != set(b["names"]):
+        raise Fail(STRUCT, f"variable sets differ: only-in-ref={sorted(set(a['names'])-set(b['names']))} only-in-test={sorted(set(b['names'])-set(a['names']))}")
+    if a["names"] != b["names"]: out.append(f"    note: variable ORDER differs (same set); matched by name")
+    for k in ("dim", "finest", "prob_lo", "prob_hi", "domain", "dx", "coord"):
+        if a[k] != b[k]: raise Fail(STRUCT, f"Header field {k} differs: ref={a[k]} test={b[k]}")
+    if abs(a["time"] - b["time"]) > 1e-12 * max(abs(a["time"]), 1e-300):
+        raise Fail(STRUCT, f"simulation time differs: ref={a['time']!r} test={b['time']!r}")
+    for lev, (la, lb) in enumerate(zip(a["levels"], b["levels"])):
+        if la["step"] != lb["step"]: raise Fail(STRUCT, f"level {lev}: step differs {la['step']} vs {lb['step']}")
+        if la["ngrids"] != len(la["boxes"]) or lb["ngrids"] != len(lb["boxes"]):
+            raise Fail(STRUCT, f"level {lev}: declared grid count differs from the listed boxes")
+        out.append(check_layout(a, b, lev))
+    out.append(f"    headers: {a['nvars']} variables, dim {a['dim']}, {a['finest']+1} level(s), time {a['time']!r}, "
+               f"{sum(l['ngrids'] for l in a['levels'])} grids -- identical structure")
+    return out
+
+
+def run(cmd):
+    p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
+    return p.returncode, p.stdout
+
+
+def parse_float(tok):
+    try:
+        return float(tok)
+    except ValueError:
+        raise Fail(STRUCT, f"cannot parse number {tok!r}")
+
+
+def extrema(fextrema, plt, names):
+    rc, out = run([fextrema, plt])
+    if rc != 0: raise Fail(STRUCT, f"fextrema failed on {plt} (rc={rc}): {out.strip()[-300:]}")
+    got = {}
+    for line in out.splitlines():
+        f = line.split()
+        if len(f) >= 3 and f[0] in names:
+            if f[0] in got: raise Fail(STRUCT, f"fextrema: variable {f[0]} listed twice for {plt}")
+            lo, hi = parse_float(f[-2]), parse_float(f[-1])
+            if not (math.isfinite(lo) and math.isfinite(hi)): raise Fail(STRUCT, f"non-finite raw values in {plt}: {f[0]} min={f[-2]} max={f[-1]}")
+            got[f[0]] = (lo, hi)
+    missing = [n for n in names if n not in got]
+    if missing: raise Fail(STRUCT, f"fextrema output for {plt} lacks variables {missing}")
+    return got, out
+
+
+def parse_fcompare(text, names, nlevels):
+    """Return {(level, name): (abs, rel)}; raise Fail(STRUCT) on any structural anomaly."""
+    rows, level, seen_levels = {}, None, []
+    msg_re = re.compile(r"^\s*(\S+)\s+<\s*(.*?)\s*>\s*$")
+    num_re = re.compile(r"^\s*(\S+)\s+(\S+)\s+(\S+)\s*$")
+    for line in text.splitlines():
+        s = line.strip()
+        if not s: continue
+        if re.search(r"\b(ERROR|WARNING)\b", s):
+            raise Fail(STRUCT, f"fcompare reported: {s}")
+        m = re.match(r"^level\s*=\s*(\d+)$", s)
+        if m:
+            level = int(m.group(1)); seen_levels.append(level); continue
+        if s.startswith("variable name") or s.startswith("(||A - B||") or set(s) <= set("-"): continue
+        if s.startswith("PLOTFILE AGREE"): continue
+        m = msg_re.match(line)
+        if m and m.group(1) in names:
+            raise Fail(STRUCT, f"fcompare row for {m.group(1)} is a message, not a value: <{m.group(2)}>")
+        m = num_re.match(line)
+        if m and m.group(1) in names:
+            if level is None: raise Fail(STRUCT, f"table row before any 'level =' line: {s}")
+            key = (level, m.group(1))
+            if key in rows: raise Fail(STRUCT, f"duplicate fcompare row for {m.group(1)} at level {level}")
+            rows[key] = (parse_float(m.group(2)), parse_float(m.group(3)))
+            continue
+        # anything else that looks like a table row but is not a known variable
+        if m and re.match(r"^[-+0-9.eEinfa]+$", m.group(2)) and re.match(r"^[-+0-9.eEinfa]+$", m.group(3)):
+            raise Fail(STRUCT, f"fcompare row for unexpected variable {m.group(1)!r}")
+    if seen_levels != list(range(nlevels)): raise Fail(STRUCT, f"fcompare levels {seen_levels} != expected {list(range(nlevels))}")
+    for lev in range(nlevels):
+        missing = [n for n in names if (lev, n) not in rows]
+        if missing: raise Fail(STRUCT, f"fcompare table incomplete at level {lev}: missing rows for {missing}")
+    return rows
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("ref"); ap.add_argument("test")
+    ap.add_argument("--rel_tol", type=float, required=True)
+    ap.add_argument("--abs_tol_zero_ref", type=float, default=0.0, help="absolute tolerance for fields whose reference is identically zero (default 0 = exact)")
+    ap.add_argument("--diagnostic", nargs="*", default=[], help="variables reported but not gated (must still be present and finite)")
+    ap.add_argument("--fcompare", required=True); ap.add_argument("--fextrema", required=True)
+    ap.add_argument("--out"); ap.add_argument("--label", default="")
+    a = ap.parse_args()
+    report = [f"# nyx_fcompare_check {a.label}", f"ref={a.ref}", f"test={a.test}", f"rel_tol={a.rel_tol!r} abs_tol_zero_ref={a.abs_tol_zero_ref!r} diagnostic={a.diagnostic}"]
+    code, summary = OK, ""
+    try:
+        ha, hb = read_header(a.ref), read_header(a.test)
+        report += check_headers(ha, hb)
+        names = ha["names"]; nlev = ha["finest"] + 1
+        unknown = [d for d in a.diagnostic if d not in names]
+        if unknown: raise Fail(STRUCT, f"diagnostic variables not in the plotfile: {unknown}")
+        exa, txa = extrema(a.fextrema, a.ref, names); exb, txb = extrema(a.fextrema, a.test, names)
+        report += ["## fextrema ref", txa.rstrip(), "## fextrema test", txb.rstrip()]
+        zero_ref = {n for n in names if exa[n] == (0.0, 0.0)}
+        rc, out = run([a.fcompare, "-n", "0", "--rel_tol", repr(a.rel_tol), "--abort_if_not_all_found", a.ref, a.test])
+        report += [f"## fcompare rc={rc}", out.rstrip()]
+        rows = parse_fcompare(out, names, nlev)
+        bad, worst, diag = [], (None, 0.0), []
+        for (lev, n), (ab, rel) in sorted(rows.items()):
+            if n in a.diagnostic:
+                diag.append(f"{n}@L{lev}: abs={ab:.6e} rel={rel:.6e}")
+                if not (math.isfinite(ab)): raise Fail(STRUCT, f"diagnostic {n} at level {lev}: non-finite absolute error {ab}")
+                continue
+            if n in zero_ref:
+                ok = math.isfinite(ab) and ab <= a.abs_tol_zero_ref
+                report.append(f"    L{lev} {n:24s} ZERO-REFERENCE abs={ab:.6e} (tol {a.abs_tol_zero_ref:g}) {'ok' if ok else 'BAD'}")
+                if not ok: bad.append(f"{n}@L{lev}(abs={ab:.3e}, zero reference)")
+                continue
+            ok = math.isfinite(rel) and rel <= a.rel_tol
+            report.append(f"    L{lev} {n:24s} abs={ab:.6e} rel={rel:.6e} {'ok' if ok else 'BAD'}")
+            if not ok: bad.append(f"{n}@L{lev}(rel={rel:.3e})")
+            if math.isfinite(rel) and rel > worst[1]: worst = (f"{n}@L{lev}", rel)
+        verdict = OK if not bad else TOL
+        if verdict == OK and rc != 0 and not (a.diagnostic and rc == 1):
+            # fcompare found something we did not (its own rc=1 for an over-tolerance diagnostic variable is expected)
+            raise Fail(STRUCT, f"parsed verdict AGREE but fcompare exited {rc} -- parser/tool inconsistency")
+        if verdict == TOL and rc == 0:
+            raise Fail(STRUCT, "parsed verdict DISAGREE but fcompare exited 0 -- parser/tool inconsistency")
+        code = verdict
+        summary = (f"RESULT: {'AGREE' if code == OK else 'DISAGREE'} nvars={len(names)} levels={nlev} rel_tol={a.rel_tol:g} "
+                   f"worst={worst[0]} {worst[1]:.3e} zero_ref={sorted(zero_ref) or 'none'} bad={bad or 'none'} diagnostic={diag or 'none'}")
+    except Fail as ex:
+        code, summary = ex.code, f"RESULT: {'UNSUPPORTED_LAYOUT' if ex.code == UNSUPPORTED else 'STRUCTURAL_FAIL'} {ex}"
+    report.append(summary)
+    text = "\n".join(report) + "\n"
+    if a.out:
+        os.makedirs(os.path.dirname(os.path.abspath(a.out)), exist_ok=True)
+        open(a.out, "w").write(text)
+    print(summary)
+    return code
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/level3/nyx/nyx_particle_compare.py b/level3/nyx/nyx_particle_compare.py
new file mode 100755
index 0000000..ece5433
--- /dev/null
+++ b/level3/nyx/nyx_particle_compare.py
@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+"""Compare Nyx dark-matter particles between two runs that may have used a
+DIFFERENT number of MPI ranks.
+
+AMReX's own particle_compare (Tools/Postprocessing/C_Src) compares particle
+files chunk by chunk and requires identical headers, including `next_id` and the
+per-file layout -- i.e. the same process count. Across rank counts Nyx assigns
+particle (id, cpu) pairs per reading rank, so ids differ as well. Particle
+identity is therefore established through the initial checkpoint: every particle
+of a run is keyed by its exact (bit-identical) t=0 position, read from the
+checkpoint written at step 0 with the same (id, cpu) as the final checkpoint.
+
+    nyx_particle_compare.py <run_dir_A> <run_dir_B> <final_step> [--rel_tol R] [--abs_tol A]
+
+Reads <run>/chk00000/DM and <run>/chk<final>/DM (checkpoint format: ids present).
+Reports, per real component (position_x/y/z, mass, xvel, yvel, zvel), the maximum
+absolute difference and the relative difference max|a-b| / max|a| (the same
+definition AMReX's particle_compare prints), and exits 0 only if every component
+satisfies abs <= abs_tol or rel <= rel_tol (particle_compare semantics: with
+abs_tol 0 the relative tolerance alone decides). Exit 2 on any structural problem
+(count mismatch, unmatched particle, unreadable file, non-finite data).
+"""
+import argparse, math, os, struct, sys
+import numpy as np
+
+
+def read_particle_dir(pdir):
+    """Return (header dict, ints[n, num_int] int32, reals[n, num_real] float64)."""
+    lines = [l.strip() for l in open(os.path.join(pdir, "Header")) if l.strip()]
+    i = 0
+    version = lines[i]; i += 1
+    ndim = int(lines[i]); i += 1
+    nreal_extra = int(lines[i]); i += 1
+    real_names = lines[i:i + nreal_extra]; i += nreal_extra
+    nint_extra = int(lines[i]); i += 1
+    int_names = lines[i:i + nint_extra]; i += nint_extra
+    is_chk = int(lines[i]); i += 1
+    nparticles = int(lines[i]); i += 1
+    next_id = int(lines[i]); i += 1
+    finest = int(lines[i]); i += 1
+    ngrids = [int(lines[i + l]) for l in range(finest + 1)]; i += finest + 1
+    grids = []  # (level, which, count, where)
+    for lev in range(finest + 1):
+        for g in range(ngrids[lev]):
+            w, c, o = lines[i].split(); i += 1
+            grids.append((lev, int(w), int(c), int(o)))
+    if "single" in version:
+        raise SystemExit(f"{pdir}: single-precision particle files are not expected here")
+    num_int = 2 * is_chk + nint_extra
+    num_real = ndim + nreal_extra
+    ints = np.zeros((nparticles, num_int), dtype=np.int32)
+    reals = np.zeros((nparticles, num_real), dtype=np.float64)
+    pos = 0
+    for lev, which, count, where in grids:
+        if count == 0:
+            continue
+        fn = os.path.join(pdir, f"Level_{lev}", f"DATA_{which:05d}")
+        with open(fn, "rb") as f:
+            f.seek(where)
+            ib = f.read(4 * num_int * count)
+            rb = f.read(8 * num_real * count)
+        if len(ib) != 4 * num_int * count or len(rb) != 8 * num_real * count:
+            raise SystemExit(f"{fn}: short read at offset {where} (grid count {count})")
+        if num_int:
+            ints[pos:pos + count] = np.frombuffer(ib, dtype="<i4").reshape(count, num_int)
+        reals[pos:pos + count] = np.frombuffer(rb, dtype="<f8").reshape(count, num_real)
+        pos += count
+    if pos != nparticles:
+        raise SystemExit(f"{pdir}: header says {nparticles} particles, grids hold {pos}")
+    names = [f"DM_position_{'xyz'[d]}" for d in range(ndim)] + [f"DM_{n}" for n in real_names]
+    hdr = dict(version=version, ndim=ndim, is_checkpoint=is_chk, nparticles=nparticles, next_id=next_id,
+               num_int=num_int, num_real=num_real, names=names, int_names=(["id", "cpu"] if is_chk else []) + int_names)
+    return hdr, ints, reals
+
+
+def keyed_final(run, final_step):
+    """Map each particle's exact initial position -> its final real components (via id,cpu)."""
+    h0, i0, r0 = read_particle_dir(os.path.join(run, "chk00000", "DM"))
+    h1, i1, r1 = read_particle_dir(os.path.join(run, f"chk{final_step:05d}", "DM"))
+    for h, tag in ((h0, "chk00000"), (h1, f"chk{final_step:05d}")):
+        if not h["is_checkpoint"]:
+            raise SystemExit(f"{run}/{tag}: not a checkpoint (no particle ids) -- run.sh must write checkpoints")
+    if h0["nparticles"] != h1["nparticles"]:
+        raise SystemExit(f"{run}: particle count changed {h0['nparticles']} -> {h1['nparticles']}")
+    if not (np.isfinite(r0).all() and np.isfinite(r1).all()):
+        raise SystemExit(f"{run}: non-finite particle data")
+    # (id, cpu) -> row, then order the final data by the initial-position key
+    key0 = i0[:, 0].astype(np.int64) * (1 << 32) + i0[:, 1].astype(np.int64)
+    key1 = i1[:, 0].astype(np.int64) * (1 << 32) + i1[:, 1].astype(np.int64)
+    if len(np.unique(key0)) != len(key0) or len(np.unique(key1)) != len(key1):
+        raise SystemExit(f"{run}: duplicate (id,cpu) pairs")
+    order1 = np.argsort(key1); k1s = key1[order1]
+    idx = np.searchsorted(k1s, key0)
+    if not np.array_equal(k1s[idx], key0):
+        raise SystemExit(f"{run}: a particle of chk00000 is missing in the final checkpoint")
+    final_by_initial_row = r1[order1[idx]]          # row j: final data of the particle that was row j initially
+    init_pos = r0[:, :h0["ndim"]]
+    # canonical order: lexicographic on the exact initial position (identical across runs by construction)
+    canon = np.lexsort([init_pos[:, d] for d in reversed(range(h0["ndim"]))])
+    return h1, init_pos[canon], final_by_initial_row[canon]
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("run_a"); ap.add_argument("run_b"); ap.add_argument("final_step", type=int)
+    ap.add_argument("--rel_tol", type=float, default=0.0); ap.add_argument("--abs_tol", type=float, default=0.0)
+    a = ap.parse_args()
+    ha, pa, fa = keyed_final(a.run_a, a.final_step)
+    hb, pb, fb = keyed_final(a.run_b, a.final_step)
+    if ha["names"] != hb["names"] or ha["nparticles"] != hb["nparticles"]:
+        raise SystemExit(f"component/count mismatch: {ha['names']} ({ha['nparticles']}) vs {hb['names']} ({hb['nparticles']})")
+    if not np.array_equal(pa, pb):
+        raise SystemExit("initial (t=0) particle positions differ between the runs -- cannot establish identity")
+    print(f" {ha['nparticles']} particles matched through their exact initial positions; comparing chk{a.final_step:05d}")
+    print(f" {'component':<20} {'abs error':>16} {'rel error':>16}")
+    ok = True
+    for j, name in enumerate(ha["names"]):
+        d = np.abs(fa[:, j] - fb[:, j]).max()
+        ref = np.abs(fa[:, j]).max()
+        rel = d / ref if ref > 0 else d
+        print(f" {name:<20} {d:16.8e} {rel:16.8e}")
+        if d > a.abs_tol and rel > a.rel_tol:
+            ok = False
+    if ok:
+        print(f" PARTICLES AGREE to relative tolerance {a.rel_tol:g}" + (f" and/or absolute tolerance {a.abs_tol:g}" if a.abs_tol > 0 else ""))
+        return 0
+    print(f" PARTICLES DISAGREE to relative tolerance {a.rel_tol:g}")
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/level3/nyx/run.sh b/level3/nyx/run.sh
new file mode 100755
index 0000000..5168eb5
--- /dev/null
+++ b/level3/nyx/run.sh
@@ -0,0 +1,189 @@
+#!/usr/bin/env bash
+# Run Nyx (full cosmological N-body + baryon hydrodynamics workflow: dark-matter
+# particle IC read, Poisson gravity via AMReX MLMG, PPM hydro, particle push,
+# periodic redistribution, plotfile I/O) on N GPUs, one MPI rank per GPU.
+#
+#   ./run.sh [CUDA|HIP|CPU] [extra Nyx inputs overrides...]
+#
+# Cases (HPCPERF_NYX_CASE), all upstream decks from the read-only checkout:
+#   minisb (default)  Exec/MiniSB/inputs.32 -- Santa Barbara cluster, 32^3 cells,
+#                     32,686 DM particles from the shipped ASCII IC, 10 steps;
+#                     exactly upstream's nightly GPU regression test "MiniSB"
+#                     (run there as `inputs.32 nyx.ppm_type=0`, 2 MPI ranks).
+#                     Adiabatic (no heating/cooling in the deck). smoke only.
+#   lya_adiabatic     smoke : Exec/LyA/inputs.rt.garuda -- upstream's GPU regression
+#                             test "LyA-adiabatic" deck (heat_cool_type=0,
+#                             strang_split=1, 32^3 cells, shipped 32.nyx IC, z=100).
+#                     strong: Exec/LyA/inputs (the flagship 64^3 Lyman-alpha science
+#                             deck with the shipped 64sssss_20mpc.nyx IC, z=159) with
+#                             heating/cooling turned OFF (heat_cool_type=0, sdc_split=0,
+#                             strang_split=1, UVB table unused). This is a NAMED
+#                             ADIABATIC DERIVATIVE of LyA; it does not cover the
+#                             heating/cooling LyA workload (see lya_heatcool).
+#   lya_heatcool      Exec/LyA/inputs as shipped (heat_cool_type=11, CVODE via
+#                     SUNDIALS) -- requires a *heatcool* profile (HPCPERF_NYX_HEATCOOL=YES build).
+#   scaling_synthetic weak: Exec/Scaling/inputs physics with nyx.particle_init_type=RandomPerCell
+#                     (upstream labels this initialisation as testing-only): a
+#                     SYNTHETIC scaling/communication test, NOT a science IC.
+#
+# Decomposition: `amr.max_grid_size` fixes the box size so the BoxArray is the
+# same for every rank count (amr.refine_grid_layout=0, as upstream's MiniSB deck);
+# ranks are refused when boxes < ranks (a rank without a box is never launched
+# silently) and reported when boxes % ranks != 0 (imbalance). Nothing changes N.
+# Rank -> GPU: AMReX binds device 0 of what the launcher's per-rank wrapper
+# exposes (one visible GPU per rank, audited by the launcher).
+#
+# Controls:
+#   HPCPERF_GPUS=N|all            ranks (= GPUs, default 1)
+#   HPCPERF_SCALE_MODE            smoke | strong | weak (default smoke)
+#   HPCPERF_NYX_STEPS             max_step (default: 10 = upstream's MiniSB test length)
+#   HPCPERF_NYX_PROFILE           build profile (default: cuda<tk>-gcc<v>-adiabatic|heatcool)
+#   HPCPERF_NYX_HEATCOOL=YES      select the heatcool profile
+#   HPCPERF_NYX_GPU_AWARE=0|1     amrex.use_gpu_aware_mpi (default: AMReX auto)
+#   HPCPERF_NYX_MGS               override amr.max_grid_size (decomposition control only)
+#   HPCPERF_NYX_WEAK_CELLS        weak: cells per rank per dimension (default 64)
+# Derived deck (class A): upstream lines kept verbatim except the parameters listed
+# in the header of the written file (I/O cadence, decomposition, and -- for the named
+# adiabatic derivative -- heat_cool_type/sdc_split/strang_split); the official test
+# command's `amrex.the_arena_init_size=0 amr.checkpoint_files_output=0` are applied.
+# stdout is copied to <run_dir>/stdout.log; a run_manifest.txt records provenance.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"; [ $# -gt 0 ] && shift
+HC="$(echo "${HPCPERF_NYX_HEATCOOL:-NO}" | tr '[:lower:]' '[:upper:]')"
+VARIANT=adiabatic; [ "$HC" = YES ] && VARIANT=heatcool
+GCC_MM="$(l3_version_mm "$("$CXX" -dumpfullversion 2>/dev/null || "$CXX" -dumpversion)")"
+case "$BACKEND" in
+    CUDA) MODEL=cuda; PROFILE_DEFAULT="cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-${VARIANT}" ;;
+    HIP)  MODEL=hip;  PROFILE_DEFAULT="hip-${HPCPERF_HIP_ARCH:-gfx950}-${VARIANT}" ;;
+    CPU)  MODEL=cpu;  PROFILE_DEFAULT="cpu-gcc${GCC_MM}-${VARIANT}" ;;
+    *) echo "usage: $0 [CUDA|HIP|CPU] [inputs overrides]" >&2; exit 2 ;;
+esac
+PROFILE="${HPCPERF_NYX_PROFILE:-$PROFILE_DEFAULT}"
+l3_paths_profile nyx "$PROFILE"
+SRC="$R/_upstream/level3/Nyx"
+CASE="${HPCPERF_NYX_CASE:-minisb}"
+MODE="$(l3_scale_mode nyx)" || exit 2
+N_RANKS="$(hpcperf_ranks nyx yes)" || exit 2
+STEPS="${HPCPERF_NYX_STEPS:-10}"
+hpcperf_forbid_args nyx max_step amr.n_cell amr.max_grid_size amr.refine_grid_layout amr.plot_int amr.plot_file amr.check_int \
+    amr.checkpoint_files_output nyx.heat_cool_type nyx.particle_init_type nyx.binary_particle_file nyx.ascii_particle_file \
+    geometry.prob_hi nyx.particle_initrandom_mass_total -- "$@" || exit 2
+
+# ----------------------------------------------------------------------------- case -> deck
+EXTRA=()          # deck lines appended (documented in the derived file header)
+DROP='^\s*(max_step|amr\.plot_int|amr\.check_int|amr\.checkpoint_files_output|amr\.plot_file|amr\.check_file|amr\.max_grid_size|amr\.refine_grid_layout)\s*='
+LINKS=()          # upstream data files the deck references by relative name
+case "$CASE" in
+    minisb)
+        [ "$MODE" = smoke ] || { echo "run.sh: case minisb is the 32^3 official regression deck: smoke only (use lya_adiabatic for strong, scaling_synthetic for weak)" >&2; exit 2; }
+        EXE="$L3_INSTALL/bin/nyx_MiniSB"; DECK="$SRC/Exec/MiniSB/inputs.32"; LINKS=(ic_sb_32.ascii)
+        NX=32; MGS="${HPCPERF_NYX_MGS:-16}"
+        DROP="$DROP|^\s*nyx\.ppm_type\s*="; EXTRA+=("nyx.ppm_type = 0   # as upstream's nightly GPU test command") ;;
+    lya_adiabatic)
+        EXE="$L3_INSTALL/bin/nyx_LyA"
+        [ "$HC" = NO ] || { echo "run.sh: lya_adiabatic is defined for the adiabatic profile (HPCPERF_NYX_HEATCOOL=NO)" >&2; exit 2; }
+        case "$MODE" in
+            smoke)  DECK="$SRC/Exec/LyA/inputs.rt.garuda"; LINKS=(32.nyx); NX=32; MGS="${HPCPERF_NYX_MGS:-16}" ;;
+            strong) DECK="$SRC/Exec/LyA/inputs"; LINKS=(64sssss_20mpc.nyx); NX=64; MGS="${HPCPERF_NYX_MGS:-16}"
+                    DROP="$DROP|^\s*nyx\.(heat_cool_type|sdc_split|strang_split|uvb_rates_file)\s*="
+                    EXTRA+=("nyx.heat_cool_type = 0   # NAMED ADIABATIC DERIVATIVE of Exec/LyA/inputs (heating/cooling OFF)" \
+                            "nyx.sdc_split = 0" "nyx.strang_split = 1   # required by Nyx without SDC/HEATCOOL (as inputs.rt.garuda)") ;;
+            weak)   echo "run.sh: weak scaling with a science IC is not defined for Nyx (the shipped ICs are two different cosmologies); use HPCPERF_NYX_CASE=scaling_synthetic (labelled synthetic)" >&2; exit 2 ;;
+        esac ;;
+    lya_heatcool)
+        EXE="$L3_INSTALL/bin/nyx_LyA"
+        [ "$HC" = YES ] || { echo "run.sh: lya_heatcool needs the heatcool profile: HPCPERF_NYX_HEATCOOL=YES (build.sh + run.sh)" >&2; exit 2; }
+        [ "$MODE" != weak ] || { echo "run.sh: weak not defined for lya_heatcool" >&2; exit 2; }
+        case "$MODE" in
+            smoke)  DECK="$SRC/Exec/LyA/inputs.rt"; LINKS=(32.nyx TREECOOL_middle); NX=32; MGS="${HPCPERF_NYX_MGS:-16}" ;;
+            strong) DECK="$SRC/Exec/LyA/inputs"; LINKS=(64sssss_20mpc.nyx TREECOOL_middle); NX=64; MGS="${HPCPERF_NYX_MGS:-16}" ;;
+        esac ;;
+    scaling_synthetic)
+        EXE="$L3_INSTALL/bin/nyx_LyA"; DECK="$SRC/Exec/Scaling/inputs"; LINKS=(TREECOOL_middle)
+        [ "$MODE" != smoke ] || { echo "run.sh: scaling_synthetic is a scaling deck (HPCPERF_SCALE_MODE=strong|weak); correctness cases are minisb / lya_adiabatic" >&2; exit 2; }
+        L="${HPCPERF_NYX_WEAK_CELLS:-64}"
+        # upstream deck: 64^3 cells in a 28.49002849 Mpc box, RandomPerCell, total DM mass 869658119634944.0
+        BOX0=28.49002849; MASS0=869658119634944.0
+        if [ "$MODE" = weak ]; then
+            TOPO="$(hpcperf_topology nyx "$N_RANKS")" || exit 2; read -r PX PY PZ <<< "$TOPO"
+            NX=$((L * PX)); NY=$((L * PY)); NZ=$((L * PZ)); MGS="${HPCPERF_NYX_MGS:-$L}"; WHAT="weak: ${L}^3 cells per rank, ${PX}x${PY}x${PZ} tiles"
+        else
+            G="${HPCPERF_NYX_GLOBAL:-256}"; [ $((G % 64)) -eq 0 ] || { echo "run.sh: HPCPERF_NYX_GLOBAL=$G must be a multiple of 64" >&2; exit 2; }
+            PX=$((G / 64)); PY=$PX; PZ=$PX; NX=$G; NY=$G; NZ=$G; MGS="${HPCPERF_NYX_MGS:-64}"; WHAT="strong: fixed ${G}^3 global grid (upstream 64^3 deck scaled x$PX per dimension)"
+        fi
+        PHX="$(python3 -c "print(f'{$BOX0*$PX:.8f}')")"; PHY="$(python3 -c "print(f'{$BOX0*$PY:.8f}')")"; PHZ="$(python3 -c "print(f'{$BOX0*$PZ:.8f}')")"
+        MASS="$(python3 -c "print(f'{$MASS0*$PX*$PY*$PZ:.1f}')")"
+        DROP="$DROP|^\s*(amr\.n_cell|geometry\.prob_hi|nyx\.particle_initrandom_mass_total)\s*="
+        EXTRA+=("amr.n_cell = $NX $NY $NZ   # $WHAT" \
+                "geometry.prob_hi = $PHX $PHY $PHZ   # box scaled with the tiles (mean density unchanged)" \
+                "nyx.particle_initrandom_mass_total = $MASS   # total DM mass scaled with the volume")
+        if [ "$HC" = NO ]; then
+            DROP="$DROP|^\s*nyx\.(heat_cool_type|sdc_split|strang_split|uvb_rates_file)\s*="
+            EXTRA+=("nyx.heat_cool_type = 0   # adiabatic profile" "nyx.sdc_split = 0" "nyx.strang_split = 1")
+        fi ;;
+    *) echo "run.sh: HPCPERF_NYX_CASE must be minisb | lya_adiabatic | lya_heatcool | scaling_synthetic (got '$CASE')" >&2; exit 2 ;;
+esac
+[ -x "$EXE" ] || { echo "run.sh: $EXE not found -- run ./build.sh $BACKEND (profile $PROFILE) first" >&2; exit 1; }
+[ -f "$DECK" ] || { echo "run.sh: deck $DECK missing (run fetch.sh)" >&2; exit 1; }
+[ "$MODEL" = cpu ] || l3_binary_backend_check "$EXE" "$MODEL" || exit 1
+: "${NY:=$NX}"; : "${NZ:=$NX}"
+# boxes with the fixed layout (n_cell / max_grid_size per dimension)
+for d in "$NX" "$NY" "$NZ"; do [ $((d % MGS)) -eq 0 ] || { echo "run.sh: amr.n_cell $d not divisible by amr.max_grid_size $MGS" >&2; exit 2; }; done
+BOXES=$(( (NX / MGS) * (NY / MGS) * (NZ / MGS) ))
+if [ "$N_RANKS" -gt "$BOXES" ]; then
+    echo "run.sh: $N_RANKS ranks requested but the ${NX}x${NY}x${NZ} grid with amr.max_grid_size=$MGS has only $BOXES boxes -- a rank without work is never launched silently; lower HPCPERF_GPUS or set HPCPERF_NYX_MGS (e.g. $((MGS / 2)) -> $((BOXES * 8)) boxes)" >&2; exit 2
+fi
+BALANCE=balanced; [ $((BOXES % N_RANKS)) -eq 0 ] || BALANCE="IMBALANCED ($BOXES boxes over $N_RANKS ranks)"
+
+RUN_DIR="$(l3_rundir "$L3_BUILD/$L3_RUN_SUBDIR/$CASE.$MODE.np$N_RANKS")" || exit 2
+IN="$RUN_DIR/inputs"
+{
+    echo "# derived from upstream $(realpath --relative-to="$SRC" "$DECK") (HPC-Performance-AI level3/nyx/run.sh; profile $PROFILE)"
+    echo "# upstream lines removed and replaced below: max_step, amr.plot_int/plot_file/check_int/check_file/checkpoint_files_output, amr.max_grid_size, amr.refine_grid_layout$( [ ${#EXTRA[@]} -gt 0 ] && echo ', plus the case-specific lines marked with a comment')"
+    grep -vE "$DROP" "$DECK"
+    echo ""
+    echo "# --- level3/nyx/run.sh ---"
+    echo "max_step = $STEPS"
+    echo "amr.max_grid_size = $MGS   # fixed BoxArray ($BOXES boxes) for every rank count"
+    echo "amr.refine_grid_layout = 0"
+    echo "amr.plot_file = plt"
+    echo "amr.plot_int = $STEPS      # plt00000 (initial) and plt$(printf '%05d' "$STEPS") (final)"
+    echo "amr.plot_vars = ALL"
+    echo "amr.check_file = chk"
+    echo "amr.check_int = $STEPS      # chk00000 and chk$(printf '%05d' "$STEPS"): particle ids for the cross-rank-count"
+    echo "amr.checkpoint_files_output = 1   # particle comparison (nyx_particle_compare.py); upstream's test command disables checkpoints"
+    echo "amrex.the_arena_init_size = 0     # as upstream's nightly test command"
+    for e in "${EXTRA[@]}"; do echo "$e"; done
+    [ -n "${HPCPERF_NYX_GPU_AWARE:-}" ] && echo "amrex.use_gpu_aware_mpi = $HPCPERF_NYX_GPU_AWARE"
+} > "$IN"
+for f in "${LINKS[@]}"; do
+    src="$(dirname "$DECK")/$f"; [ -f "$src" ] || { echo "run.sh: upstream data file $src missing" >&2; exit 1; }
+    ln -sfn "$src" "$RUN_DIR/$f"
+done
+
+BIND=wrapper; [ "$MODEL" = cpu ] && BIND=none
+echo "# Nyx $BACKEND profile=$PROFILE case=$CASE mode=$MODE ranks=$N_RANKS grid=${NX}x${NY}x${NZ} max_grid_size=$MGS boxes=$BOXES ($BALANCE) steps=$STEPS exe=$(basename "$EXE") deck=$(realpath --relative-to="$SRC" "$DECK") run_dir=$RUN_DIR"
+cd "$RUN_DIR"
+RUN_ID="$(l3_run_id)"
+# real exit code of the launcher/application (pipefail: tee's 0 never masks it; set +e so it is recorded, not aborted on)
+set +e; set -o pipefail
+"$L3_LAUNCHER" --gpus "$N_RANKS" --bind "$BIND" -- "$EXE" "$IN" "$@" 2>&1 | tee "$RUN_DIR/stdout.log"
+rc=$?
+set +o pipefail; set -e
+if [ -z "${HPCPERF_DRY_RUN:-}" ]; then
+    ICS=""; for f in "${LINKS[@]}"; do ICS="$ICS $f=$(l3_sha_file "$(dirname "$DECK")/$f")"; done
+    l3_manifest "$RUN_DIR" "run_id=$RUN_ID" "app=nyx" "backend=$BACKEND" "profile=$PROFILE" "case=$CASE" "mode=$MODE" \
+        "ranks=$N_RANKS" "grid=${NX}x${NY}x${NZ}" "max_grid_size=$MGS" "boxes=$BOXES" "steps=$STEPS" \
+        "exit_code=$rc" "binary=$EXE" "binary_sha256=$(l3_sha_file "$EXE")" "deck=$DECK" "deck_sha256=$(l3_sha_file "$DECK")" \
+        "input=$IN" "input_sha256=$(l3_sha_file "$IN")" "ic_sha256=${ICS# }" \
+        "fingerprint=$L3_INSTALL/.hpcperf-l3-fingerprint" "fingerprint_sha256=$(l3_sha_file "$L3_INSTALL/.hpcperf-l3-fingerprint")" \
+        "stdout=$RUN_DIR/stdout.log" "utc=$(date -u +%FT%TZ)"
+fi
+exit "$rc"
diff --git a/level3/nyx/sundials_probe.sh b/level3/nyx/sundials_probe.sh
new file mode 100755
index 0000000..af92959
--- /dev/null
+++ b/level3/nyx/sundials_probe.sh
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+# Independent SUNDIALS 7.2.1 + CUDA 13.2 check BEFORE it is wired into Nyx's
+# heating/cooling build: build the commit Nyx pins (subprojects/sundials,
+# 5c53be85 = v7.2.1) with ENABLE_CUDA and its CUDA examples/tests enabled, then
+# run SUNDIALS' own CUDA test set (ctest -R cuda: each example compares its
+# output with the shipped .out answer file through SUNDIALS' testRunner).
+#
+#   ./sundials_probe.sh          HPCPERF_BUILD_JOBS=N (default 16)
+# Build tree: .deps/level3/nyx/cuda132-gcc133-heatcool/build/sundials-test; log logs/sundials-ctest.log
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+l3_isolate_build_env
+SRC="$R/_upstream/level3/Nyx/subprojects/sundials"; [ -f "$SRC/CMakeLists.txt" ] || { echo "sundials_probe.sh: run fetch.sh" >&2; exit 1; }
+ARCH="${HPCPERF_CUDA_ARCH:-$(l3_gpu_arch)}"; GCC_MM="$(l3_version_mm "$("$CXX" -dumpfullversion)")"
+l3_paths_profile nyx "cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-heatcool"
+B="$L3_BUILD_DEPS/sundials-test"; JOBS="${HPCPERF_BUILD_JOBS:-16}"
+echo "# SUNDIALS $(git -C "$SRC" rev-parse HEAD) (v$(/usr/bin/grep -oE 'PACKAGE_VERSION_(MAJOR|MINOR|PATCH) "[0-9]+"' "$SRC/CMakeLists.txt" | /usr/bin/grep -oE '[0-9]+' | paste -sd.)) CUDA sm_$ARCH test build -> $B"
+mkdir -p "$B"
+cmake -S "$SRC" -B "$B" -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX" -DCMAKE_CXX_STANDARD=17 \
+    -DENABLE_CUDA=ON "-DCMAKE_CUDA_ARCHITECTURES=$ARCH" "-DCMAKE_CUDA_HOST_COMPILER=$CXX" -DSUNDIALS_INDEX_SIZE=32 -DSUNDIALS_BUILD_PACKAGE_FUSED_KERNELS=ON \
+    -DENABLE_MPI=OFF -DENABLE_OPENMP=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_STATIC_LIBS=ON \
+    -DBUILD_ARKODE=ON -DBUILD_CVODE=ON -DBUILD_KINSOL=OFF -DBUILD_IDA=OFF -DBUILD_IDAS=OFF -DBUILD_CVODES=OFF \
+    -DEXAMPLES_ENABLE_C=ON -DEXAMPLES_ENABLE_CXX=ON -DEXAMPLES_ENABLE_CUDA=ON -DEXAMPLES_INSTALL=OFF -DBUILD_TESTING=ON \
+    -DSUNDIALS_TEST_ENABLE_DEV_TESTS=ON -DSUNDIALS_TEST_ENABLE_DIFF_OUTPUT=ON \
+    > "$L3_LOGS/sundials-test-configure.log" 2>&1 || { tail -30 "$L3_LOGS/sundials-test-configure.log"; echo "sundials_probe.sh: configure failed" >&2; exit 1; }
+cmake --build "$B" -j "$JOBS" > "$L3_LOGS/sundials-test-build.log" 2>&1 || { tail -30 "$L3_LOGS/sundials-test-build.log"; echo "sundials_probe.sh: build failed" >&2; exit 1; }
+# SUNDIALS_TEST_ENABLE_DEV_TESTS registers every example as a test whose output is compared
+# with the shipped answer file (examples/*/*.out) -- this is SUNDIALS' own regression check.
+( cd "$B" && ctest -N | /usr/bin/grep -iE 'cuda|cusolver' ) > "$L3_LOGS/sundials-ctest-list.log" 2>&1 || true
+( cd "$B" && CUDA_VISIBLE_DEVICES=0 ctest -R 'cuda|cusolver' --timeout 600 --output-on-failure -j 1 ) > "$L3_LOGS/sundials-ctest.log" 2>&1 || true
+/usr/bin/grep -E 'tests passed|Failed|Timeout|Not Run' "$L3_LOGS/sundials-ctest.log" | head -12
+if /usr/bin/grep -qE '100% tests passed, 0 tests failed out of [1-9]' "$L3_LOGS/sundials-ctest.log"; then
+    echo "sundials_probe.sh: PASS -- $(/usr/bin/grep -oE 'out of [0-9]+' "$L3_LOGS/sundials-ctest.log" | head -1) SUNDIALS CUDA examples/tests on sm_$ARCH with CUDA $(l3_cuda_version)"
+    echo "PASS $(date -u +%FT%TZ) $(/usr/bin/grep -E 'tests passed' "$L3_LOGS/sundials-ctest.log")" > "$B/.hpcperf-probe-pass"; exit 0
+fi
+echo "sundials_probe.sh: FAIL -- see $L3_LOGS/sundials-ctest.log"; exit 1
diff --git a/level3/nyx/validate.sh b/level3/nyx/validate.sh
new file mode 100755
index 0000000..84ab279
--- /dev/null
+++ b/level3/nyx/validate.sh
@@ -0,0 +1,310 @@
+#!/usr/bin/env bash
+# Correctness check for Nyx on N GPUs with upstream's own comparison tools and
+# pre-fixed criteria (no tolerance is derived from the results).
+#
+#   ./validate.sh [CUDA|HIP]        HPCPERF_GPUS=N (default 1); HPCPERF_NYX_CASES="minisb lya_adiabatic"
+#
+# For every case (default: the two official GPU-regression decks, MiniSB and
+# LyA-adiabatic, both 10 steps as upstream) the N-GPU run must satisfy ALL of:
+#  [1] completeness: run.sh exit code 0 (timeout -> FAIL), plt00000 and the final
+#      plotfile exist, the runlog reaches max_step, every plotfile variable is
+#      finite (amrex_fextrema min/max through l3_check.require_finite), the DM
+#      particle count in the final plotfile equals the IC count.
+#  [2] official regression comparison at a pre-fixed tolerance, through the strict
+#      wrapper nyx_fcompare_check.py around AMReX's fcompare (Header structure, raw
+#      finiteness of BOTH plotfiles via fextrema, one parsed row per variable and level,
+#      zero-reference fields judged by an absolute rule, non-finite errors never dropped,
+#      parser/tool consistency) plus nyx_particle_compare.py on the DM particles:
+#        N=1 : against a second, independent 1-GPU run of the same binary and deck
+#              (same-configuration reproducibility, exactly what the nightly test measures);
+#        N>1 : against the 1-GPU plotfile of the same binary/deck (rank-count
+#              independence; the BoxArray is fixed by run.sh, only the distribution changes).
+#      Tolerance provenance (official GPU nightly reports, ccse.lbl.gov/pub/GpuRegressionTesting/Nyx):
+#        MiniSB       : fcompare --rel_tol 2e-10 (UPSTREAM, used as is);
+#        LyA-adiabatic: upstream uses 5e-09; this validator applies the MiniSB value 2e-10
+#                       to both adiabatic decks -- a PROJECT choice, 25x stricter than upstream;
+#        LyA heat/cool: upstream compares plt00354 with --rel_tol 5e-05 (their max 2.8e-05 in
+#                       Temp; their report also compares I_R, rel 4.2e-08 there); this project's
+#                       10-step smoke deck (plt00010) is a different configuration. The 5e-5 was
+#                       adopted on 2026-09-06 after a first heat/cool validation at 2e-10 FAILED
+#                       (recorded in the review package).
+#  [3] cross-backend reference: the CPU-profile binary (Nyx_GPU_BACKEND=NONE, same
+#      Nyx/AMReX/deck) run on 1 rank; same comparison with rel_tol 1e-8 for the adiabatic
+#      decks, a PROJECT tolerance FIXED before any run (two orders above the same-platform
+#      tolerance to allow for FMA contraction, libm and reduction-order differences
+#      between host and device code over 10 steps); 5e-5 for heat/cool.
+#  [4] conservation: total comoving baryon mass sum(density*dV) (amrex_fvolumesum)
+#      between plt00000 and the final plotfile: |dM/M| <= 1e-9 (adiabatic, periodic:
+#      no mass source); DM particle count exact.
+#  I_R (heat/cool only): the SDC reaction-integral field is NOT accepted or rejected by a
+#      relative tolerance -- it is a small residual of cancelling O(a^2 rho e / dt) terms
+#      whose run-to-run variation is O(1) relative at this stage of the deck while every
+#      state field agrees to 1e-13 over the 10 steps (README, section "I_R"). I_R is a
+#      state-carried term that the NEXT step's hydro predictor consumes, so the agreement of
+#      the state fields over this short horizon does not establish that its variation has no
+#      effect later; it is a coupling field with an unresolved reproducibility question, not a
+#      diagnostic that has been shown to be irrelevant. It is checked for presence and
+#      finiteness, reported, and the case verdict is downgraded to I_R_CHECK_PENDING
+#      (exit 3): state and particle checks PASS, the full heat/cool acceptance does not.
+#  Box layouts: the comparator checks each plotfile's BoxArray geometrically (inside the
+#      domain, no overlap, level 0 covers the domain). A different but legal partition of the
+#      same cells (re-blocking) is reported as UNSUPPORTED_LAYOUT (comparator exit 4): the
+#      comparison is not performed, the case is neither PASS nor FAIL, validate.sh exits 4.
+# Exit codes: 0 PASS; 1 FAIL; 3 STATE_AND_PARTICLES_PASS with I_R_CHECK_PENDING; 4 UNSUPPORTED_LAYOUT
+# (precedence FAIL > UNSUPPORTED_LAYOUT > PENDING > PASS). Only 0 is a PASS; a queue must record
+# 3/4 and continue (level3/tools/l3_verdict.py classifies them, never as PASS).
+# Exit codes of run.sh/launcher/application and every tool are captured; a
+# missing tool, plotfile or reference is FAIL, never skipped silently.
+#
+# Offline / harness hooks (no GPU): HPCPERF_NYX_OFFLINE=1 compares EXISTING run
+# directories without executing run.sh (reports go to HPCPERF_NYX_REPORT_DIR, default
+# <runs>/offline_recheck.<utc>/, never into the historical run directories);
+# HPCPERF_NYX_TOOLS_DIR overrides the AMReX tool directory; HPCPERF_NYX_SKIP_PARTICLES=1
+# skips the DM particle comparison (test harness only; printed loudly).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+[ "$BACKEND" != CPU ] || { echo "validate.sh: validates a GPU backend against the CPU reference; use CUDA or HIP" >&2; exit 2; }
+N="${HPCPERF_GPUS:-1}"
+CASES="${HPCPERF_NYX_CASES:-minisb lya_adiabatic}"
+STEPS="${HPCPERF_NYX_STEPS:-10}"
+TIMEOUT="${HPCPERF_VALIDATE_TIMEOUT:-1800}"
+REL_TOL_SAME=2e-10     # MiniSB's official tolerance, applied to both adiabatic decks (see header)
+REL_TOL_XBACKEND=1e-8  # PROJECT CPU-vs-GPU tolerance for the adiabatic decks (see header)
+HC_REL_TOL=5e-5        # upstream's LyA heat/cool tolerance (plt00354); adopted for the 10-step deck, see header
+HC_DIAGNOSTIC="I_R"    # reported, not gated; forces the I_R_CHECK_PENDING verdict (see header)
+ABS_TOL_ZERO_REF=0     # a field that is identically zero in the reference must be identically zero in the test
+case_tols() { # case_tols <case> -> sets TOL_SAME TOL_X DIAG
+    case "$1" in
+        lya_heatcool) TOL_SAME=$HC_REL_TOL; TOL_X=$HC_REL_TOL; DIAG="$HC_DIAGNOSTIC" ;;
+        *)            TOL_SAME=$REL_TOL_SAME; TOL_X=$REL_TOL_XBACKEND; DIAG="" ;;
+    esac
+}
+MASS_TOL=1e-9          # pre-fixed baryon mass conservation tolerance
+python3 -c 'import numpy' 2>/dev/null || { echo "validate.sh: python3 with numpy required" >&2; exit 1; }
+export HPCPERF_GPUS="$N" HPCPERF_SCALE_MODE=smoke HPCPERF_NYX_STEPS="$STEPS"
+OFFLINE="${HPCPERF_NYX_OFFLINE:-}"; SKIP_PARTICLES="${HPCPERF_NYX_SKIP_PARTICLES:-}"
+[ -z "$SKIP_PARTICLES" ] || echo "validate.sh: WARNING -- HPCPERF_NYX_SKIP_PARTICLES set: DM particle comparison skipped (test harness mode, not a validation)"
+
+HC="$(echo "${HPCPERF_NYX_HEATCOOL:-NO}" | tr '[:lower:]' '[:upper:]')"; VARIANT=adiabatic; [ "$HC" = YES ] && VARIANT=heatcool
+GCC_MM="$(l3_version_mm "$("$CXX" -dumpfullversion 2>/dev/null || "$CXX" -dumpversion)")"
+case "$BACKEND" in
+    CUDA) GPU_PROFILE="${HPCPERF_NYX_PROFILE:-cuda$(l3_version_mm "$(l3_cuda_version)")-gcc${GCC_MM}-${VARIANT}}" ;;
+    HIP)  GPU_PROFILE="${HPCPERF_NYX_PROFILE:-hip-${HPCPERF_HIP_ARCH:-gfx950}-${VARIANT}}" ;;
+esac
+CPU_PROFILE="${HPCPERF_NYX_CPU_PROFILE:-cpu-gcc${GCC_MM}-${VARIANT}}"
+GPU_ROOT="$R/.deps/level3/nyx/$GPU_PROFILE"; CPU_ROOT="$R/.deps/level3/nyx/$CPU_PROFILE"
+GPU_RUNS="$R/build/level3/nyx/$GPU_PROFILE/$L3_RUN_SUBDIR"; CPU_RUNS="$R/build/level3/nyx/$CPU_PROFILE/$L3_RUN_SUBDIR"
+TOOLS="${HPCPERF_NYX_TOOLS_DIR:-$CPU_ROOT/install/bin}"
+for t in amrex_fcompare amrex_fextrema amrex_fvolumesum particle_compare; do
+    [ -x "$TOOLS/$t" ] || { echo "validate.sh: FAIL -- $TOOLS/$t missing: build the CPU reference profile first (./build.sh CPU)"; exit 1; }
+done
+[ -n "$OFFLINE" ] || [ -x "$GPU_ROOT/install/bin/nyx_MiniSB" ] || { echo "validate.sh: FAIL -- GPU profile $GPU_PROFILE not built (./build.sh $BACKEND)"; exit 1; }
+mkdir -p "$GPU_RUNS" "$CPU_RUNS"
+if [ -n "$OFFLINE" ]; then
+    REPORT_DIR="${HPCPERF_NYX_REPORT_DIR:-$GPU_RUNS/offline_recheck.$(date -u +%Y%m%dT%H%M%SZ)}"; mkdir -p "$REPORT_DIR"
+    echo "validate.sh: OFFLINE mode -- no run.sh execution; comparing existing run directories; reports under $REPORT_DIR"
+fi
+ok=1; pending=""; unsupported=""
+fail() { echo "validate.sh: FAIL -- $*"; ok=0; }
+report_dir() { # report_dir <run_dir>: where this run's comparison outputs go (never a historical dir in offline mode)
+    if [ -n "$OFFLINE" ]; then local d="$REPORT_DIR/$(basename "$1")"; mkdir -p "$d"; echo "$d"; else echo "$1"; fi
+}
+
+run_gpu() { # run_gpu <case> <n> <stdout-file>
+    local case=$1 n=$2 out=$3 rc=0
+    if [ -n "$OFFLINE" ]; then [ -f "$GPU_RUNS/$case.smoke.np$n/run_manifest.txt" ] && return 0; echo "validate.sh: OFFLINE -- run directory $GPU_RUNS/$case.smoke.np$n missing" >&2; return 66; fi
+    HPCPERF_NYX_CASE="$case" HPCPERF_GPUS="$n" timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" > "$out" 2>&1 || rc=$?
+    return $rc
+}
+run_cpu() {
+    local case=$1 out=$2 rc=0
+    if [ -n "$OFFLINE" ]; then [ -f "$CPU_RUNS/$case.smoke.np1/run_manifest.txt" ] && return 0; echo "validate.sh: OFFLINE -- CPU reference $CPU_RUNS/$case.smoke.np1 missing" >&2; return 66; fi
+    HPCPERF_NYX_CASE="$case" HPCPERF_GPUS=1 HPCPERF_NYX_PROFILE="$CPU_PROFILE" timeout "$TIMEOUT" "$HERE/run.sh" CPU > "$out" 2>&1 || rc=$?
+    return $rc
+}
+manifest_val() { /usr/bin/grep -m1 "^$2=" "$1/run_manifest.txt" 2>/dev/null | cut -d= -f2- || true; }
+final_plt() { printf 'plt%05d' "$STEPS"; }
+
+# [1] completeness + finiteness + particle count
+check_complete() { # check_complete <run_dir> <label> <expected particle count or ''>
+    local d=$1 label=$2 want_np=$3 plt   # (set -u: expansions of one `local` happen before its assignments)
+    plt="$d/$(final_plt)"
+    [ -f "$d/plt00000/Header" ] || { fail "$label: initial plotfile plt00000 missing"; return 1; }
+    [ -f "$plt/Header" ] || { fail "$label: final plotfile $(final_plt) missing (run did not reach step $STEPS)"; return 1; }
+    [ -f "$d/runlog" ] || { fail "$label: runlog missing"; return 1; }
+    local last; last="$(awk 'NF>=2 && $1 ~ /^[0-9]+$/ {s=$1} END{print s+0}' "$d/runlog")"
+    [ "$last" -eq "$STEPS" ] || { fail "$label: runlog last step $last != $STEPS"; return 1; }
+    # finite extrema for every variable (fextrema prints: name min max per variable)
+    local ext; ext="$("$TOOLS/amrex_fextrema" "$plt" 2>&1)" || { fail "$label: amrex_fextrema failed on $plt: $ext"; return 1; }
+    python3 - "$label" <<PY || { fail "$label: non-finite plotfile extrema"; return 1; }
+import sys, os
+sys.path.insert(0, os.environ["L3_TOOLS"])
+from l3_check import require_finite, ValidationError
+label = sys.argv[1]; n = 0
+try:
+    for line in """$ext""".splitlines():
+        f = line.split()
+        if len(f) >= 3:
+            try: lo, hi = float(f[-2]), float(f[-1])
+            except ValueError: continue
+            require_finite(f"{label} {f[0]} min", lo); require_finite(f"{label} {f[0]} max", hi); n += 1
+    if n < 5: raise ValidationError(f"only {n} variables parsed from fextrema output")
+    print(f"    {label}: {n} plotfile variables finite (fextrema)")
+except ValidationError as ex:
+    print(f"  VALIDATION ERROR: {ex}"); sys.exit(1)
+PY
+    # DM particle count from the particle Header (version, dim, nreal, names..., nint, names..., is_checkpoint, nparticles)
+    local np; np="$(python3 - "$plt/DM/Header" <<'PY'
+import sys
+L = [l.strip() for l in open(sys.argv[1]) if l.strip()]
+i = 2; nreal = int(L[i]); i += 1 + nreal; nint = int(L[i]); i += 1 + nint; i += 1  # is_checkpoint
+print(int(L[i]))
+PY
+)" || { fail "$label: cannot read $plt/DM/Header"; return 1; }
+    if [ -n "$want_np" ]; then
+        [ "$np" -eq "$want_np" ] || { fail "$label: DM particle count $np != IC count $want_np"; return 1; }
+        echo "    $label: DM particles $np == IC count (exact)"
+    fi
+    echo "$np" > "$d/.np_final"
+    return 0
+}
+
+# [4] baryon mass conservation between plt00000 and the final plotfile
+check_mass() { # check_mass <run_dir> <label>
+    local d=$1 label=$2 m0 m1
+    m0="$("$TOOLS/amrex_fvolumesum" -v density "$d/plt00000" 2>&1)" || { fail "$label: fvolumesum failed: $m0"; return 1; }
+    m1="$("$TOOLS/amrex_fvolumesum" -v density "$d/$(final_plt)" 2>&1)" || { fail "$label: fvolumesum failed: $m1"; return 1; }
+    python3 - "$label" "$MASS_TOL" "$m0" "$m1" <<'PY' || { fail "$label: baryon mass not conserved"; return 1; }
+import sys, re, os
+sys.path.insert(0, os.environ["L3_TOOLS"])
+from l3_check import require_finite, ValidationError
+label, tol = sys.argv[1], float(sys.argv[2])
+def val(txt):
+    nums = re.findall(r"[-+]?\d+\.\d+e[-+]?\d+|[-+]?\d+\.\d+|[-+]?\d+e[-+]?\d+", txt.split("density")[-1])
+    if not nums: raise ValidationError(f"cannot parse fvolumesum output: {txt!r}")
+    return float(nums[-1])
+try:
+    m0 = require_finite("mass(t0)", val(sys.argv[3])); m1 = require_finite("mass(t1)", val(sys.argv[4]))
+    rel = abs(m1 - m0) / abs(m0)
+    print(f"    {label}: baryon mass {m0:.12e} -> {m1:.12e}, |dM/M| = {rel:.3e} (tol {tol:.0e}) {'ok' if rel <= tol else 'BAD'}")
+    sys.exit(0 if rel <= tol else 1)
+except ValidationError as ex:
+    print(f"  VALIDATION ERROR: {ex}"); sys.exit(1)
+PY
+}
+
+# [2]/[3] strict plotfile comparison (nyx_fcompare_check.py around fcompare/fextrema) + DM particles
+compare() { # compare <ref_dir> <dir> <rel_tol> <label>
+    local ref=$1 d=$2 tol=$3 label=$4 out rc=0 plt rep; plt="$(final_plt)"; rep="$(report_dir "$d")"
+    local diag_args=(); [ -n "${DIAG:-}" ] && diag_args=(--diagnostic $DIAG)
+    # exit 0 = every gated variable within tolerance; 1 = a gated variable outside it; 2 = structural
+    # (header mismatch, box overlap/missing box/coverage deficit, non-finite raw values, missing/duplicate/
+    # message rows, tool errors, parser/tool inconsistency); 4 = legal but different box layout
+    # (UNSUPPORTED_LAYOUT: not compared, neither PASS nor FAIL). The full report (headers, fextrema,
+    # fcompare output, decisions) is in fcompare.<label>.txt.
+    out="$(python3 "$HERE/nyx_fcompare_check.py" "$ref/$plt" "$d/$plt" --rel_tol "$tol" --abs_tol_zero_ref "$ABS_TOL_ZERO_REF" "${diag_args[@]}" \
+            --fcompare "$TOOLS/amrex_fcompare" --fextrema "$TOOLS/amrex_fextrema" --out "$rep/fcompare.$label.txt" --label "$label" 2>&1)" || rc=$?
+    case "$rc" in
+        0) echo "    $label: plotfile comparison rel_tol $tol: ${out#RESULT: }" ;;
+        1) fail "$label: plotfile comparison (rel_tol $tol): ${out#RESULT: }"; return 1 ;;
+        4) unsupported="${unsupported:+$unsupported }$label"
+           echo "    $label: plotfile comparison NOT performed: ${out#RESULT: }"
+           echo "    $label: UNSUPPORTED_LAYOUT -- a different but legal box layout is outside this validator's scope; case verdict UNSUPPORTED_LAYOUT (not PASS, not FAIL)"
+           return 1 ;;
+        *) fail "$label: plotfile comparison STRUCTURAL failure (rc=$rc): ${out#RESULT: }"; return 1 ;;
+    esac
+    if [ -n "${DIAG:-}" ]; then
+        # the diagnostic field was parsed and is finite but is not accepted by any tolerance here
+        pending="${pending:+$pending }$label:${DIAG}(${out##*diagnostic=})"
+        echo "    $label: diagnostic field(s) $DIAG reported, NOT accepted -> case verdict I_R_CHECK_PENDING"
+    fi
+    if [ -n "$SKIP_PARTICLES" ]; then echo "    $label: DM particle comparison SKIPPED (harness mode)"; return 0; fi
+    # DM particles: AMReX's particle_compare needs identical headers (incl. next_id and the per-file
+    # layout), i.e. the same rank count -- and it exits 0 even when it prints "FAIL - Particle data
+    # headers do not agree". Across rank counts the particles are matched through their exact t=0
+    # positions (chk00000 -> chk<final>, ids within a run) by nyx_particle_compare.py, which prints
+    # per-component abs/rel errors with particle_compare's definitions and exits nonzero on disagreement.
+    rc=0; out="$(python3 "$HERE/nyx_particle_compare.py" "$ref" "$d" "$STEPS" --rel_tol "$tol" 2>&1)" || rc=$?
+    echo "$out" > "$rep/particle_compare.$label.txt"
+    if [ "$rc" -ne 0 ] || ! echo "$out" | /usr/bin/grep -q 'PARTICLES AGREE'; then fail "$label: DM particle comparison (rel_tol $tol) rc=$rc: $(echo "$out" | /usr/bin/grep -E 'AGREE|DISAGREE|rror|missing|differ' | head -2 | tr '\n' ' ')"; return 1; fi
+    local pw; pw="$(echo "$out" | awk '$1 ~ /^DM_/ {if ($3+0 > m) {m=$3+0; v=$1}} END{printf "%s %.3e", v, m}')"
+    echo "    $label: DM particles (t=0-identity matched, chk$(printf '%05d' "$STEPS")) rel_tol $tol: PARTICLES AGREE (max rel err $pw)"
+    return 0
+}
+
+ic_count() { # expected DM particle count from the deck's IC file
+    case "$1" in
+        minisb) head -1 "$R/_upstream/level3/Nyx/Exec/MiniSB/ic_sb_32.ascii" | tr -d ' ' ;;
+        lya_adiabatic|lya_heatcool) python3 -c "import struct; f=open('$R/_upstream/level3/Nyx/Exec/LyA/32.nyx','rb'); print(struct.unpack('<q', f.read(8))[0])" ;;
+        *) echo "" ;;
+    esac
+}
+
+for CASE in $CASES; do
+    echo "validate.sh: === Nyx $BACKEND case=$CASE, $STEPS steps, $N GPU(s) [profile $GPU_PROFILE; CPU reference $CPU_PROFILE] ==="
+    WANT_NP="$(ic_count "$CASE")"; case_tols "$CASE"; REL_TOL_SAME_CASE=$TOL_SAME; REL_TOL_X_CASE=$TOL_X
+    D="$GPU_RUNS/$CASE.smoke.np$N"
+    rc=0; run_gpu "$CASE" "$N" "$GPU_RUNS/validate.$CASE.np$N.stdout" || rc=$?
+    /usr/bin/grep -aE '^# Nyx|hpcperf-launch: audit summary|Run time =|Total Time|ERROR|Error|abort' "$GPU_RUNS/validate.$CASE.np$N.stdout" | head -8 || true
+    if [ "$rc" -eq 124 ]; then fail "$CASE np$N timed out after ${TIMEOUT}s"; continue; fi
+    if [ "$rc" -ne 0 ]; then fail "$CASE np$N run.sh exited $rc (see $GPU_RUNS/validate.$CASE.np$N.stdout)"; continue; fi
+    [ "$(manifest_val "$D" exit_code)" = 0 ] || { fail "$CASE np$N manifest exit_code=$(manifest_val "$D" exit_code)"; continue; }
+    check_complete "$D" "$CASE np$N" "$WANT_NP" || continue
+    check_mass "$D" "$CASE np$N" || true
+
+    # reference for [2]
+    if [ "$N" -eq 1 ]; then
+        REF="$GPU_RUNS/$CASE.smoke.np1.rerun"
+        if [ -n "$OFFLINE" ]; then
+            [ -f "$REF/$(final_plt)/Header" ] || { fail "$CASE np1: OFFLINE -- rerun directory $REF missing"; continue; }
+        else
+            rm -rf "$REF"; rc=0; run_gpu "$CASE" 1 "$GPU_RUNS/validate.$CASE.np1.rerun.stdout" || rc=$?
+            # run.sh wrote into the canonical np1 dir again: move that fresh result aside as the rerun,
+            # keeping the first run as the one under test (manifests record both run_ids)
+            if [ "$rc" -ne 0 ]; then fail "$CASE np1 rerun exited $rc"; continue; fi
+            mv "$GPU_RUNS/$CASE.smoke.np1" "$REF"
+            # re-run the case under test so that $D holds a real result again (third execution)
+            rc=0; run_gpu "$CASE" 1 "$GPU_RUNS/validate.$CASE.np1.stdout" || rc=$?
+            [ "$rc" -eq 0 ] || { fail "$CASE np1 (re-execution) exited $rc"; continue; }
+            check_complete "$D" "$CASE np1" "$WANT_NP" || continue
+        fi
+        LABEL2="same-config-rerun"
+    else
+        REF="$GPU_RUNS/$CASE.smoke.np1"
+        BIN="$(manifest_val "$D" binary_sha256)"
+        if [ ! -f "$REF/$(final_plt)/Header" ] || [ "$(manifest_val "$REF" binary_sha256)" != "$BIN" ] || [ "$(manifest_val "$REF" deck_sha256)" != "$(manifest_val "$D" deck_sha256)" ] || [ "$(manifest_val "$REF" steps)" != "$STEPS" ]; then
+            echo "    (1-GPU reference for $CASE missing or from a different binary/deck -- producing it now)"
+            rc=0; run_gpu "$CASE" 1 "$GPU_RUNS/validate.$CASE.np1.stdout" || rc=$?
+            [ "$rc" -eq 0 ] || { fail "$CASE np1 reference run exited $rc"; continue; }
+            check_complete "$REF" "$CASE np1(ref)" "$WANT_NP" || continue
+        fi
+        LABEL2="vs-1GPU"
+    fi
+    compare "$REF" "$D" "$REL_TOL_SAME_CASE" "$CASE.np$N.$LABEL2" || true
+
+    # [3] CPU reference (independent backend, same Nyx/AMReX sources and deck)
+    C="$CPU_RUNS/$CASE.smoke.np1"
+    if [ ! -f "$C/$(final_plt)/Header" ] || [ "$(manifest_val "$C" deck_sha256)" != "$(manifest_val "$D" deck_sha256)" ] || [ "$(manifest_val "$C" steps)" != "$STEPS" ]; then
+        echo "    (CPU reference for $CASE missing -- running the CPU profile now, 1 rank)"
+        rc=0; run_cpu "$CASE" "$CPU_RUNS/validate.$CASE.cpu.stdout" || rc=$?
+        [ "$rc" -eq 0 ] || { fail "$CASE CPU reference run exited $rc (see $CPU_RUNS/validate.$CASE.cpu.stdout)"; continue; }
+    fi
+    check_complete "$C" "$CASE cpu-ref" "$WANT_NP" || continue
+    compare "$C" "$D" "$REL_TOL_X_CASE" "$CASE.np$N.vs-CPU" || true
+done
+
+CRIT="fcompare/particle rel_tol adiabatic $REL_TOL_SAME / heatcool $HC_REL_TOL vs $( [ "$N" -eq 1 ] && echo rerun || echo 1-GPU), CPU reference rel_tol adiabatic $REL_TOL_XBACKEND / heatcool $HC_REL_TOL, zero-reference fields exact, baryon mass |dM/M|<=$MASS_TOL, DM count exact, finite"
+if [ "$ok" -ne 1 ]; then
+    echo "Nyx $BACKEND validation ($N GPU, cases: $CASES): FAIL${unsupported:+ (and UNSUPPORTED_LAYOUT [$unsupported])}${pending:+ (and I_R_CHECK_PENDING [$pending])}"; exit 1
+elif [ -n "$unsupported" ]; then
+    echo "Nyx $BACKEND validation ($N GPU, cases: $CASES; $CRIT): UNSUPPORTED_LAYOUT [$unsupported] -- plotfile comparison not performed for these pairs (different but legal box layouts are not supported by this validator); not a PASS${pending:+; I_R_CHECK_PENDING [$pending]}"; exit 4
+elif [ -n "$pending" ]; then
+    echo "Nyx $BACKEND validation ($N GPU, cases: $CASES; $CRIT): STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING [$pending] -- heat/cool acceptance incomplete, not a full regression PASS"; exit 3
+fi
+echo "Nyx $BACKEND validation ($N GPU, cases: $CASES; $CRIT): PASS"; exit 0
diff --git a/level3/qmcpack/README.md b/level3/qmcpack/README.md
new file mode 100644
index 0000000..46c5131
--- /dev/null
+++ b/level3/qmcpack/README.md
@@ -0,0 +1,122 @@
+# QMCPACK -- Level 3 second batch
+
+Real-space quantum Monte Carlo (VMC/DMC) with upstream's recommended NVIDIA GPU
+configuration: OpenMP target offload for the batched drivers plus CUDA
+(cuBLAS/cuSOLVER) -- `QMC_GPU="openmp;cuda"` -- which requires an LLVM/Clang
+compiler with NVPTX offload ("For NVIDIA GPUs, LLVM clang", upstream docs).
+
+## Provenance / versions
+
+| Item | Value |
+|---|---|
+| QMCPACK | tag `v4.4.0`, `2601d62e353934f1526cab1f67f30b6672b7c76f` (2026-08-31), NCSA/Illinois licence |
+| Compiler (private) | **LLVM 23.1.0** (llvmorg-23.1.0, 2026-08-25) built from `llvm-project-23.1.0.src.tar.xz` (sha256 `ab1f0e3e...479ff`): clang;lld + runtimes openmp;offload + GPU runtimes target `nvptx64-nvidia-cuda` (`libc;openmp` -> libompdevice/libomptarget-nvptx.bc); host GCC 14.2.1; `.deps/level3/qmcpack/clang231-cuda132-offload/install/llvm` |
+| CUDA | 13.2.78 (clang 23 lists CUDA 13.2 as FULLY_SUPPORTED), sm_100; nvcc for the CUDA parts with clang as host compiler (QMCPACK adds `--allow-unsupported-compiler`) |
+| MPI | conda Open MPI 5.0.10 wrappers redirected to clang (`OMPI_CC/OMPI_CXX`) |
+| Dependencies (private, same profile) | OpenBLAS 0.3.30 (TARGET=SAPPHIRERAPIDS, single-threaded), HDF5 1.14.5 (parallel; upstream-tested version), Boost 1.90.0 headers; FFTW3/libxml2 from the conda env / system |
+| Build strategy | NATIVE + private toolchain (Spack not used: the local Spack is 2025-05 and its `qmcpack` recipe is inert for 4.x GPU options -- audit) |
+| Profile | `clang231-cuda132-offload` |
+
+### Why a private LLVM, and why built from source
+
+- The node has no offload-capable clang: `/home/bcui2/clang+llvm` is an old
+  install (nvptx bitcode up to sm_60) that does not even start (`libtinfo.so.5`).
+- The official binary release `LLVM-23.1.0-Linux-X64.tar.xz` (2.0 GB, sha256
+  `18da30f7...fec8c`) contains `libomp.so` and the clang-offload tools but **no
+  libomptarget / device runtime** (recorded in
+  `.deps/level3/qmcpack/downloads/LLVM-binary-release-check.txt`) -- upstream's
+  FAQ says the same ("pre-packaged LLVM releases: most likely no").
+- First source build (LLVM_ENABLE_RUNTIMES=openmp;offload only) produced
+  `libomptarget.so` but no NVPTX device runtime: since LLVM 21 the device runtime
+  is built through `LLVM_RUNTIME_TARGETS=...;nvptx64-nvidia-cuda` and
+  `LIBOMPTARGET_DEVICE_ARCHITECTURES` is unused (openmp/docs/ReleaseNotes);
+  clang failed with "no library 'libomptarget-nvptx.bc' found". Reconfigured with
+  the GPU runtimes target: `lib/nvptx64-nvidia-cuda/libompdevice.a` +
+  `libomptarget-nvptx.bc` (62 s incremental). Build time host part: 573 s at -j24
+  on local scratch (`/tmp/hpcperf-l3-b2-scratch/qmcpack-llvm`; extracting the
+  150k-file tree on the project NFS alone took >1 h and was abandoned).
+
+### Offload probe (toolchain/probe_offload.sh) -- PASS before any QMCPACK build
+
+`OMP_TARGET_OFFLOAD=MANDATORY` (no silent host fallback), `--offload-arch=sm_100`:
+`num_devices=1`, default device != initial device, `omp_is_initial_device()==0`
+inside the target region, axpy/reduction vs host `max|d|=4.4e-16`,
+`rel(sum)=6.4e-13`; MPI 2 and 4 ranks through the common launcher: every rank
+exactly one visible GPU, target region on the device; launcher audit 4 ranks
+verified (2-rank run too short to be sampled: unverified, expected mapping
+logged). Record: `install/llvm/OFFLOAD_PROBE.txt`.
+
+## Build / run / validate
+
+`build.sh` (deps + QMCPACK real; `HPCPERF_QMCPACK_COMPLEX=1` adds the complex
+build) refuses to run unless the probe PASSED. Stages: OpenBLAS 0.3.30
+(`TARGET=SAPPHIRERAPIDS`, single-threaded) -> HDF5 1.14.5 (parallel, clang MPI
+wrappers; `-DCMAKE_IGNORE_PATH=/usr/lib64/cmake/ZLIB;/lib64/cmake/ZLIB` because
+the node's zlib-ng CMake package references a `libz.a` that is not installed) ->
+Boost 1.90.0 headers -> QMCPACK (`QMC_GPU="openmp;cuda" QMC_GPU_ARCHS=sm_100
+QMC_MPI=ON QMC_COMPLEX=OFF QMC_MIXED_PRECISION=OFF ENABLE_PHDF5=ON
+BUILD_UNIT_TESTS=ON QMC_GPU_VISIBILITY_VARIABLE=CUDA_VISIBLE_DEVICES`). Build
+2026-09-06: deps 310 s, QMCPACK 672 s at -j12; `cuobjdump` archs of the
+executable: sm_100 only; `libomptarget.so` resolved from the private LLVM
+(checked by run.sh before every run).
+
+`run.sh`: `tests/solids/diamondC_2x1x1_pp/qmc_short_vmcbatch_dmcbatch.in.xml`
+(upstream's batched VMC+DMC test deck, 16 electrons, B-spline orbitals from
+`pwscf.pwscf.h5`, BFD pseudopotential, J1+J2) -- smoke verbatim (256 walkers),
+strong = `total_walkers` fixed over N (default 256 = the verbatim population, see
+the population limit below), weak = `walkers_per_rank` (default 256); one rank per
+GPU, `OMP_NUM_THREADS` crowds, `OMP_TARGET_OFFLOAD=MANDATORY` (no silent
+host fallback). NiO performance decks are not used (orbital files only behind an
+anl.box.com link).
+
+`validate.sh` (criteria in its header): [1] upstream ctests on the built tree
+(1 GPU): **unit 64/64 passed** (219 s), **deterministic diamond subset 526/526
+passed** (260 s; `deterministic-diamondC_{1x1x1,2x1x1}_pp-{vmcbatch,dmcbatch,
+sdbatch,vmc_sdj,vmc_dmc}*`, upstream's exact-scalar references with upstream
+tolerances); [2] the science case on N GPUs through `qmc_check.py`:
+completeness, offload/CUDA banners, "Running OpenMP offload code path" /
+"Running on a GPU via CUDA/HIP acceleration" reported by the wavefunction and
+driver components, device memory allocated through the offload runtime, 25 DMC
+blocks finite, upstream's `check_scalars.py --ns 3 --series 1 -e 2 --le
+"-21.844975 0.02"` (DIAMOND2_DMC_SCALARS), and for N > 1 statistical consistency
+with the 1-GPU run (3 sqrt(sigma_1^2 + sigma_N^2)). First 1-GPU run: DMC
+LocalEnergy -21.84924 +- 0.01517 Ha vs -21.844975 +- 0.02 (deviation -0.21 sigma),
+534 s wall, launcher audit 1 verified.
+
+### Results (2026-09-06, profile `clang231-cuda132-offload`, 8 crowds per rank)
+
+| Run | GPUs | Walkers (total / per GPU) | DMC LocalEnergy [Ha] (vs ref -21.844975 +- 0.02) | Wall [s] | Audit |
+|---|---|---|---|---|---|
+| validate / smoke = strong np1 | 1 | 256 / 256 | -21.8492 +- 0.0152 (-0.21 sigma) | 534 | 1 verified |
+| validate | 2 | 256 / 128 | -21.8566 +- 0.0126 (-0.58 sigma; 0.37 sigma vs 1 GPU) | 316 | 2 verified, 0 mismatch |
+| validate | 4 | 256 / 64 | -21.8313 +- 0.0105 (+0.68 sigma; 0.97 sigma vs 1 GPU) | 254 | 4 verified, 0 mismatch |
+| strong | 2 | 256 / 128 | -21.8365 +- 0.0088 | 332 | 2 verified |
+| strong | 4 | 256 / 64 | -21.8511 +- 0.0100 | 211 | 4 verified |
+| weak | 2 | 512 / 256 | -21.8466 +- 0.0077 | 632 | 2 verified |
+| weak | 4 | 1024 / 256 | -21.8433 +- 0.0034 | 656 | 4 verified |
+
+Unit ctests 64/64 and deterministic diamond ctests 526/526 (1 GPU) precede the
+science case. Strong scaling 1.61x / 2.53x on 2 / 4 GPUs (64 walkers per B200 is
+far below saturation), weak efficiency 84 % / 81 %; the error bars of the weak
+series shrink as 1/sqrt(walkers), as they should. Every run passes upstream's
+`check_scalars.py` window (`qmc_summary.txt` in each run directory). First
+attempts at 4096 total / 1024 per-rank walkers FAILED (next section).
+
+### Walker-population limit found on this build (open issue, documented, not worked around)
+
+Device memory grows by ~320 MB per walker although QMCPACK's own allocators report
+~27 MiB: `Free memory on the default device` drops from 181.8 GB to 99.5 GB with 256
+walkers and to 17.1 GB with 512 (VMC-only probes, 8 crowds); with 1024 walkers on
+one B200 (128 per crowd) every crowd aborts in `cuSolverInverter.hpp:55`
+(`CUSOLVER_STATUS_INTERNAL_ERROR`, i.e. cuSOLVER cannot get workspace), and so
+did the first strong/weak attempts (4096 total / 1024 per rank), recorded as
+FAILED in `SECOND_BATCH_STATUS.md`. Ruled out: allocation granularity (4096 x 1
+KiB `cudaMalloc`/`omp_target_alloc` cost 1.0/0.5 KiB each, microbenchmark), the
+libomptarget memory manager (`LIBOMPTARGET_MEMORY_MANAGER_THRESHOLD=0` changes
+nothing). Not identified: which component of the LLVM 23.1 offload runtime + CUDA
+13.2 + QMCPACK 4.4.0 stack holds the memory (a `LIBOMPTARGET_INFO=-1` trace of the
+256-walker probe produced 3.5e8 lines and was abandoned). Consequences: run.sh
+refuses more than `HPCPERF_QMCPACK_MAX_WALKERS_PER_GPU` (default 300) walkers per
+GPU unless forced; strong scaling = the verbatim 256-walker deck over 1/2/4 GPUs,
+weak scaling = 256 walkers per GPU. The 256-walker DMC runs use ~120 GB of the
+183 GB per GPU.
diff --git a/level3/qmcpack/build.sh b/level3/qmcpack/build.sh
new file mode 100755
index 0000000..aa9a092
--- /dev/null
+++ b/level3/qmcpack/build.sh
@@ -0,0 +1,102 @@
+#!/usr/bin/env bash
+# Build QMCPACK v4.4.0 with upstream's recommended NVIDIA GPU configuration:
+# LLVM/Clang OpenMP target offload + CUDA (QMC_GPU="openmp;cuda", QMC_GPU_ARCHS=sm_100),
+# using the private LLVM 23.1.0 toolchain from toolchain/build_llvm.sh (probe:
+# toolchain/probe_offload.sh must have PASSED first -- checked here).
+#
+#   ./build.sh                       HPCPERF_BUILD_JOBS=N (default 32)
+#   HPCPERF_QMCPACK_COMPLEX=1        also build the complex-valued executable
+#
+# Stages (profile clang231-cuda132-offload, .deps/level3/qmcpack/<profile>/):
+#   openblas 0.3.30 (gcc/gfortran, single-threaded)  -> install/openblas
+#   HDF5 1.14.5 (parallel, built with the clang MPI wrappers) -> install/hdf5
+#   Boost 1.90.0 headers                              -> install/boost-1.90.0
+#   QMCPACK real (+complex)                           -> build/level3/qmcpack/<profile>/{real,complex}, install/qmcpack
+# Compilers: clang/clang++ 23.1.0 through the conda Open MPI wrappers (OMPI_CC/OMPI_CXX);
+# CUDA parts through nvcc 13.2 with clang as host compiler (QMCPACK adds
+# --allow-unsupported-compiler itself); FFTW3 and libxml2 from the project conda env
+# / system (header/library only); BLAS/LAPACK = private OpenBLAS. Full precision,
+# MPI on, unit tests built. Modification class: A/C (build options, environment).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+l3_isolate_build_env
+l3_clean_conda_build_env
+PROFILE="${HPCPERF_QMCPACK_PROFILE:-clang231-cuda132-offload}"
+l3_paths_profile qmcpack "$PROFILE"
+SRC="$R/_upstream/level3/qmcpack"; DL="$R/.deps/level3/qmcpack/downloads"
+[ -f "$SRC/CMakeLists.txt" ] || { echo "build.sh: run $HERE/fetch.sh" >&2; exit 1; }
+LLVM="$L3_INSTALL/llvm"; CLANG="$LLVM/bin/clang"; CLANGXX="$LLVM/bin/clang++"
+[ -x "$CLANGXX" ] || { echo "build.sh: private LLVM missing ($LLVM) -- run toolchain/build_llvm.sh" >&2; exit 1; }
+[ -f "$LLVM/OFFLOAD_PROBE.txt" ] && /usr/bin/grep -q '^PASS' "$LLVM/OFFLOAD_PROBE.txt" || { echo "build.sh: offload probe has not PASSED (toolchain/probe_offload.sh) -- refusing to build QMCPACK on an unverified offload toolchain" >&2; exit 1; }
+SHA="$(git -C "$SRC" rev-parse HEAD)"
+ARCH="${HPCPERF_CUDA_ARCH:-$(l3_gpu_arch)}"; CUDA_ROOT="${CUDA_HOME:-/usr/local/cuda}"
+JOBS="${HPCPERF_BUILD_JOBS:-32}"; INST="$L3_INSTALL"; BLD="$L3_BUILD_DEPS"
+export OMPI_CC="$CLANG" OMPI_CXX="$CLANGXX" OMPI_FC=/usr/bin/gfortran
+export LD_LIBRARY_PATH="$LLVM/lib/x86_64-unknown-linux-gnu:$LLVM/lib:${LD_LIBRARY_PATH:-}"
+unset CMAKE_GENERATOR CUDAARCHS
+CLANG_V="$("$CLANG" --version | head -1 | /usr/bin/grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)"
+OMPI_V="$(mpirun --version | head -1 | /usr/bin/grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)"
+DEPS="llvm=$CLANG_V(offload sm_$ARCH, $(cat "$LLVM/OFFLOAD_PROBE.txt" | cut -c1-30)) hdf5=1.14.5(parallel) boost=1.90.0(headers) openblas=0.3.30 fftw3=conda libxml2=system openmpi=$OMPI_V cuda=$(l3_cuda_version) profile=$PROFILE"
+OPTS="QMC_GPU=openmp;cuda QMC_GPU_ARCHS=sm_$ARCH QMC_MPI=ON QMC_COMPLEX=OFF(+ON on request) QMC_MIXED_PRECISION=OFF ENABLE_PHDF5=ON BUILD_UNIT_TESTS=ON CMAKE_BUILD_TYPE=Release CXX=mpicxx(clang++)"
+FP="$(l3_fingerprint_text qmcpack "$SHA" cuda "$DEPS" "$OPTS" "not-used(walker-parallel; MPI on host buffers)")"
+l3_fingerprint_check "$INST" "$FP" || exit 1
+echo "# QMCPACK profile=$PROFILE: qmcpack $SHA (v4.4.0), clang $CLANG_V, $(mpirun --version | head -1), CUDA $(l3_cuda_version) sm_$ARCH, -j$JOBS; expected 30-60 min + deps ~15 min"
+T0=$(date +%s)
+run() { local log=$1; shift; "$@" > "$L3_LOGS/$log" 2>&1 || { tail -40 "$L3_LOGS/$log"; echo "build.sh: FAILED: $* (log $L3_LOGS/$log)" >&2; exit 1; }; }
+untar() { # untar <tarball> <dest-parent> -> prints the extracted top-level directory (tolerates a leading ./ entry)
+    local tb=$1 dst=$2 top; mkdir -p "$dst"
+    top="$(tar -tf "$tb" | sed 's|^\./||' | /usr/bin/grep -v '^$' | head -1 | cut -d/ -f1)"
+    [ -n "$top" ] && [ "$top" != . ] || { echo "untar: cannot determine top-level directory of $tb" >&2; return 1; }
+    [ -d "$dst/$top" ] || tar -xf "$tb" -C "$dst"; echo "$dst/$top"; }
+
+if [ ! -f "$INST/openblas/.hpcperf-stage-done" ]; then
+    d="$(untar "$DL/OpenBLAS-0.3.30.tar.gz" "$L3_SRC")"
+    run openblas-make.log make -C "$d" -j "$JOBS" USE_THREAD=0 USE_OPENMP=0 DYNAMIC_ARCH=0 NO_AFFINITY=1 TARGET="${HPCPERF_OPENBLAS_TARGET:-SAPPHIRERAPIDS}" CC=/usr/bin/gcc FC=/usr/bin/gfortran
+    run openblas-install.log make -C "$d" PREFIX="$INST/openblas" install
+    touch "$INST/openblas/.hpcperf-stage-done"
+fi
+BLAS="$INST/openblas/lib/libopenblas.so"
+if [ ! -f "$INST/hdf5/.hpcperf-stage-done" ]; then
+    d="$(untar "$DL/hdf5-1.14.5.tar.gz" "$L3_SRC")"; mkdir -p "$BLD/hdf5"
+    run hdf5-configure.log cmake -S "$d" -B "$BLD/hdf5" -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="$(command -v mpicc)" \
+        -DHDF5_ENABLE_PARALLEL=ON -DHDF5_BUILD_CPP_LIB=OFF -DHDF5_BUILD_FORTRAN=OFF -DHDF5_BUILD_JAVA=OFF -DHDF5_BUILD_EXAMPLES=OFF -DBUILD_TESTING=OFF \
+        -DHDF5_BUILD_TOOLS=ON -DBUILD_SHARED_LIBS=ON -DHDF5_ENABLE_Z_LIB_SUPPORT=ON -DHDF5_ENABLE_SZIP_SUPPORT=OFF -DCMAKE_INSTALL_PREFIX="$INST/hdf5" \
+        "-DCMAKE_IGNORE_PATH=/usr/lib64/cmake/ZLIB;/lib64/cmake/ZLIB"   # the node's zlib-ng-compat CMake package references a libz.a that is not installed; use FindZLIB (/usr/lib64/libz.so + /usr/include/zlib.h)
+    run hdf5-build.log cmake --build "$BLD/hdf5" -j "$JOBS"; run hdf5-install.log cmake --install "$BLD/hdf5"
+    touch "$INST/hdf5/.hpcperf-stage-done"
+fi
+if [ ! -f "$INST/boost-1.90.0/.hpcperf-stage-done" ]; then
+    tar -xf "$DL/boost-1.90.0-b2-nodocs.tar.xz" -C "$INST" boost-1.90.0/boost boost-1.90.0/LICENSE_1_0.txt 2>/dev/null || tar -xf "$DL/boost-1.90.0-b2-nodocs.tar.xz" -C "$INST"
+    [ -f "$INST/boost-1.90.0/boost/version.hpp" ] || { echo "build.sh: Boost headers not found after extraction" >&2; exit 1; }
+    touch "$INST/boost-1.90.0/.hpcperf-stage-done"
+fi
+T1=$(date +%s)
+build_qmc() { # build_qmc <real|complex>
+    local kind=$1 cplx=OFF; [ "$kind" = complex ] && cplx=ON
+    mkdir -p "$L3_BUILD/$kind"
+    run "qmcpack-$kind-configure.log" cmake -S "$SRC" -B "$L3_BUILD/$kind" -G Ninja -DCMAKE_BUILD_TYPE=Release \
+        -DCMAKE_C_COMPILER="$(command -v mpicc)" -DCMAKE_CXX_COMPILER="$(command -v mpicxx)" "-DCMAKE_CUDA_COMPILER=$CUDA_ROOT/bin/nvcc" \
+        -DQMC_GPU="openmp;cuda" "-DQMC_GPU_ARCHS=sm_$ARCH" -DQMC_MPI=ON -DQMC_COMPLEX="$cplx" -DQMC_MIXED_PRECISION=OFF \
+        -DENABLE_PHDF5=ON "-DHDF5_ROOT=$INST/hdf5" "-DBOOST_ROOT=$INST/boost-1.90.0" \
+        "-DBLAS_LIBRARIES=$BLAS" "-DLAPACK_LIBRARIES=$BLAS" -DBUILD_UNIT_TESTS=ON -DQMC_GPU_VISIBILITY_VARIABLE=CUDA_VISIBLE_DEVICES \
+        "-DCMAKE_INSTALL_PREFIX=$INST/qmcpack-$kind"
+    run "qmcpack-$kind-build.log" cmake --build "$L3_BUILD/$kind" -j "$JOBS"
+    run "qmcpack-$kind-install.log" cmake --install "$L3_BUILD/$kind"
+    local exe; exe="$(ls "$INST/qmcpack-$kind"/bin/qmcpack* | head -1)"
+    local archs; archs="$(cuobjdump --list-elf "$exe" 2>/dev/null | /usr/bin/grep -o 'sm_[0-9]*' | sort -u | paste -sd,)"
+    echo "qmcpack_$kind exe=$exe sha256=$(l3_sha_file "$exe") cuobjdump_archs=${archs:-none} offload_libs=$(ldd "$exe" | /usr/bin/grep -oE 'lib(omptarget|omp|cudart|cublas|cusolver)[^ ]*' | sort -u | paste -sd,)" >> "$INST/BUILD_INFO.txt"
+    [ -n "$archs" ] && [ "$archs" != "sm_$ARCH" ] && { echo "build.sh: $exe embeds '$archs' != sm_$ARCH" >&2; exit 1; }
+    return 0
+}
+: > "$INST/BUILD_INFO.txt"
+build_qmc real
+[ -n "${HPCPERF_QMCPACK_COMPLEX:-}" ] && build_qmc complex
+T2=$(date +%s)
+l3_fingerprint_write "$INST" "$FP"
+{ echo "profile=$PROFILE qmcpack=$SHA utc=$(date -u +%FT%TZ) jobs=$JOBS seconds: deps=$((T1 - T0)) qmcpack=$((T2 - T1))"; echo "deps: $DEPS"; echo "options: $OPTS"; } >> "$INST/BUILD_INFO.txt"
+cat "$INST/BUILD_INFO.txt"; echo "# QMCPACK built in $((T2 - T0)) s"
diff --git a/level3/qmcpack/fetch.sh b/level3/qmcpack/fetch.sh
new file mode 100755
index 0000000..28c904d
--- /dev/null
+++ b/level3/qmcpack/fetch.sh
@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# Fetch QMCPACK v4.4.0 and the private dependency sources (read-only).
+#
+#   QMCPACK v4.4.0   2601d62e353934f1526cab1f67f30b6672b7c76f (2026-08-31), NCSA/Illinois licence
+#   LLVM 23.1.0      llvm-project-23.1.0.src.tar.xz (toolchain/build_llvm.sh; sha256 in that script)
+#   HDF5 1.14.5      hdf5-1.14.5.tar.gz   (QMCPACK's tested HDF5; the conda 2.x is outside upstream's test range)
+#   Boost 1.90.0     boost-1.90.0-b2-nodocs.tar.xz (headers only are used; upstream nightlies use 1.90/1.84)
+#   OpenBLAS 0.3.30  shared with the other second-batch profiles' downloads (own build per profile)
+# Datasets: QMCPACK's in-repo tests/ (178 MB, deterministic + short statistical tests) are used;
+# the NiO performance splines (external Box link, 43 MB-8.8 GB each) are NOT downloaded in this round.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+UP="$R/_upstream/level3"; DL="$R/.deps/level3/qmcpack/downloads"
+SHA=2601d62e353934f1526cab1f67f30b6672b7c76f
+mkdir -p "$UP" "$DL"
+if [ ! -d "$UP/qmcpack/.git" ]; then
+    git clone -q --branch v4.4.0 --depth 1 https://github.com/QMCPACK/qmcpack.git "$UP/qmcpack"
+fi
+got="$(git -C "$UP/qmcpack" rev-parse HEAD)"; [ "$got" = "$SHA" ] || { echo "fetch.sh: qmcpack at $got, expected $SHA (v4.4.0)" >&2; exit 1; }
+declare -A URLS=(
+  [hdf5-1.14.5.tar.gz]=https://github.com/HDFGroup/hdf5/releases/download/hdf5_1.14.5/hdf5-1.14.5.tar.gz
+  [boost-1.90.0-b2-nodocs.tar.xz]=https://github.com/boostorg/boost/releases/download/boost-1.90.0/boost-1.90.0-b2-nodocs.tar.xz
+  [OpenBLAS-0.3.30.tar.gz]=https://github.com/OpenMathLib/OpenBLAS/releases/download/v0.3.30/OpenBLAS-0.3.30.tar.gz
+)
+# sha256 recorded at first download (2026-09-06); OpenBLAS matches the dftfe/geos downloads
+declare -A SHAS=(
+  [OpenBLAS-0.3.30.tar.gz]=27342cff518646afb4c2b976d809102e368957974c250a25ccc965e53063c95d
+)
+for f in "${!URLS[@]}"; do
+    [ -s "$DL/$f" ] || curl -sSL -o "$DL/$f" "${URLS[$f]}"
+    got_sha="$(sha256sum "$DL/$f" | cut -d' ' -f1)"
+    if [ -n "${SHAS[$f]:-}" ]; then [ "$got_sha" = "${SHAS[$f]}" ] || { echo "fetch.sh: $f sha256 $got_sha != expected ${SHAS[$f]}" >&2; exit 1; }; fi
+    echo "$got_sha  $f"
+done | sort -k2 > "$DL/SHA256SUMS"
+echo "# qmcpack $got; downloads under $DL:"; sed 's/^/#   /' "$DL/SHA256SUMS"
diff --git a/level3/qmcpack/qmc_check.py b/level3/qmcpack/qmc_check.py
new file mode 100755
index 0000000..2ecb384
--- /dev/null
+++ b/level3/qmcpack/qmc_check.py
@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+"""Completeness / GPU-evidence / statistical check of one QMCPACK run for validate.sh.
+
+usage: qmc_check.py <run_dir> <prefix> "<ref_mean> <ref_sigma>" <nsigma> <equil_blocks> <check_scalars.py>
+
+Prints key=value summary lines; exit 1 (after printing VALIDATION ERROR) on any violation:
+  * qmc.out contains "QMCPACK execution completed successfully" and no "QMCPACK ERROR"; the manifest
+    records exit_code=0;
+  * banner lines "OpenMP target offload to accelerators build option is enabled" and
+    "CUDA acceleration build option is enabled" are present (the binary's GPU paths are compiled in and
+    initialised; the run had OMP_TARGET_OFFLOAD=MANDATORY, so a host fallback would have aborted);
+  * <prefix>.s001.scalar.dat (DMC) exists with 25 blocks, every column finite; s000 (VMC) exists;
+  * upstream's check_scalars.py (--ns nsigma --series 1 -e equil --le "<ref>") passes; its computed
+    mean/error bar are exported as dmc_mean/dmc_err for the cross-rank consistency check.
+"""
+import os
+import re
+import subprocess
+import sys
+
+sys.path.insert(0, os.environ["L3_TOOLS"])
+from l3_check import ValidationError, require_finite  # noqa: E402
+
+run_dir, prefix, ref, nsigma, equil, checker = sys.argv[1:7]
+run_dir, checker = os.path.abspath(run_dir), os.path.abspath(checker)   # check_scalars.py runs with cwd = run_dir
+out = os.path.join(run_dir, "qmc.out")
+txt = open(out, errors="replace").read() if os.path.exists(out) else ""
+manifest = open(os.path.join(run_dir, "run_manifest.txt"), errors="replace").read() if os.path.exists(os.path.join(run_dir, "run_manifest.txt")) else ""
+
+
+def scalar_table(path):
+    rows, cols = [], None
+    for line in open(path):
+        if line.startswith("#"):
+            cols = line[1:].split()
+            continue
+        if line.strip():
+            rows.append([require_finite(f"{os.path.basename(path)} value", x) for x in line.split()])
+    return cols, rows
+
+
+try:
+    if not txt:
+        raise ValidationError("qmc.out missing or empty")
+    m = re.search(r"^exit_code=(\d+)$", manifest, re.M)
+    if not m or int(m.group(1)) != 0:
+        raise ValidationError(f"manifest exit_code = {m.group(1) if m else 'missing'}")
+    if "QMCPACK ERROR" in txt:
+        raise ValidationError("'QMCPACK ERROR' in output")
+    if "QMCPACK execution completed successfully" not in txt:
+        raise ValidationError("no 'QMCPACK execution completed successfully' -- run incomplete")
+    offload = "OpenMP target offload to accelerators build option is enabled" in txt
+    cuda = "CUDA acceleration build option is enabled" in txt
+    if not (offload and cuda):
+        raise ValidationError(f"GPU banner missing (offload={offload}, cuda={cuda})")
+    # the drivers/wavefunction components must report the device code paths, and device memory must
+    # actually have been allocated through the offload runtime during the run
+    n_offload_paths = len(re.findall(r"Running OpenMP offload code path", txt))
+    n_cuda_paths = len(re.findall(r"Running on a GPU via CUDA/HIP acceleration", txt))
+    dev_mib = [int(x) for x in re.findall(r"Device memory allocated via OpenMP offload\s*:\s*(\d+) MiB", txt)]
+    if n_offload_paths == 0 or n_cuda_paths == 0:
+        raise ValidationError(f"no device code paths reported (offload paths={n_offload_paths}, cuda paths={n_cuda_paths})")
+    if not dev_mib or max(dev_mib) <= 0:
+        raise ValidationError(f"no device memory was allocated through the offload runtime (records: {dev_mib})")
+    m_wpr = re.search(r"walkers_per_rank\s*=\s*\[([^\]]*)\]", txt)
+    walkers_per_rank = m_wpr.group(1).strip() if m_wpr else "?"
+    s0, s1 = (os.path.join(run_dir, f"{prefix}.s00{i}.scalar.dat") for i in (0, 1))
+    for p in (s0, s1):
+        if not os.path.exists(p):
+            raise ValidationError(f"{os.path.basename(p)} missing")
+    cols, rows = scalar_table(s1)
+    if len(rows) != 25:
+        raise ValidationError(f"DMC scalar file has {len(rows)} blocks, expected 25")
+    if "LocalEnergy" not in cols:
+        raise ValidationError("DMC scalar file lacks a LocalEnergy column")
+    vcols, vrows = scalar_table(s0)
+    pop = None   # the batched drivers print no per-block walker count; the population is taken from walkers_per_rank in qmc.out
+    res = subprocess.run([sys.executable, checker, "--ns", nsigma, "--series", "1", "-p", prefix, "-e", equil, "--le", ref],
+                         cwd=run_dir, capture_output=True, text=True)
+    rep = res.stdout + res.stderr
+    mean = re.search(r"computed\s+mean value\s*:\s*(-?[0-9.]+)", rep)
+    err = re.search(r"computed\s+error bar\s*:\s*(-?[0-9.]+)", rep)
+    print(f"completed=True offload_banner={offload} cuda_banner={cuda} offload_code_paths={n_offload_paths} cuda_code_paths={n_cuda_paths} "
+          f"max_device_mib_via_offload={max(dev_mib)} walkers_per_rank=[{walkers_per_rank}] vmc_blocks={len(vrows)} dmc_blocks={len(rows)}")
+    if mean and err:
+        print(f"dmc_mean={float(mean.group(1))!r} dmc_err={float(err.group(1))!r} reference=\"{ref}\" nsigma={nsigma} equilibration_blocks={equil}")
+    for line in rep.splitlines():
+        if re.search(r"reference|computed|tolerance|deviation|status of this test|Testing quantity", line):
+            print("check_scalars: " + line.strip())
+    if res.returncode != 0 or "status of this test      :   pass" not in rep:
+        raise ValidationError(f"check_scalars.py exit {res.returncode}: DMC total energy outside the upstream reference window")
+    if not (mean and err):
+        raise ValidationError("could not parse check_scalars.py computed mean/error")
+except ValidationError as ex:
+    print(f"  VALIDATION ERROR: {ex}")
+    sys.exit(1)
diff --git a/level3/qmcpack/run.sh b/level3/qmcpack/run.sh
new file mode 100755
index 0000000..04b4731
--- /dev/null
+++ b/level3/qmcpack/run.sh
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+# Run QMCPACK (real build, OpenMP target offload + CUDA) on N GPUs, one MPI rank per GPU,
+# OpenMP threads per rank = walker crowds (upstream's recommended GPU layout).
+#
+#   ./run.sh [CUDA]
+#
+# Case (HPCPERF_QMCPACK_CASE):
+#   diamond2 (default)  tests/solids/diamondC_2x1x1_pp/qmc_short_vmcbatch_dmcbatch.in.xml -- upstream's
+#       batched-driver VMC (256 walkers, 100 warm-up steps, 1 block) + DMC (256 walkers, 25 blocks x 100
+#       steps, tau 0.005, no non-local moves) test for the 2x1x1 diamond supercell: 16 electrons, B-spline
+#       single-particle orbitals from pwscf.pwscf.h5 (Quantum ESPRESSO), BFD pseudopotential, J1+J2
+#       Jastrows. Upstream runs this deck on 1 rank x 4 threads and on 4 ranks (qmc_short_vmcbatch_dmcbatch_4r,
+#       identical <qmc> blocks) and checks the DMC total energy against -21.844975 +- 0.02 Ha
+#       (DIAMOND2_DMC_SCALARS, check_scalars.py --ns 3), which is exactly what validate.sh does here.
+#     smoke : the deck verbatim (total_walkers 256 -> split over the N ranks)
+#     strong: derived deck, VMC+DMC total_walkers = HPCPERF_QMCPACK_WALKERS (default 256 = the verbatim
+#             population, fixed over N)
+#     weak  : derived deck, VMC+DMC walkers_per_rank = HPCPERF_QMCPACK_WALKERS_PER_RANK (default 256),
+#             i.e. total = W x N (QMC weak scaling = more statistics at constant work per GPU)
+#   POPULATION LIMIT on this build (measured 2026-09-06, see README): device memory grows by ~320 MB per
+#   walker (82 GB at 256 walkers, 165 GB at 512; 1024 walkers on one B200 exhaust the 183 GB and cuSOLVER
+#   aborts with CUSOLVER_STATUS_INTERNAL_ERROR) although QMCPACK's own allocators report ~27 MiB -- so more
+#   than ~300 walkers per GPU are refused by the run, not silently reduced. Larger requests are honoured
+#   only when explicitly asked for and are expected to FAIL until the cause (LLVM 23.1 offload runtime on
+#   CUDA 13.2/B200) is understood.
+#   The derived decks change ONLY the walker-population parameter (recorded in the manifest with the diff);
+#   the DMC energy estimate does not depend on the population beyond the population-control bias (which
+#   shrinks with more walkers), so upstream's reference check applies in every mode.
+#   NiO (tests/performance/NiO, S1-S256) is NOT used: its orbital files exist only behind an anl.box.com link.
+#
+# Controls: HPCPERF_GPUS=N|all, HPCPERF_CPUS_PER_RANK=T (OpenMP threads per rank, default 8),
+#           HPCPERF_SCALE_MODE=smoke|strong|weak, HPCPERF_QMCPACK_PROFILE, HPCPERF_DRY_RUN=1,
+#           HPCPERF_QMCPACK_MAX_WALKERS_PER_GPU (default 300), HPCPERF_QMCPACK_FORCE_POPULATION=1
+# OMP_TARGET_OFFLOAD=MANDATORY: a failed offload aborts the run instead of silently falling back to the host.
+# Output: <run_dir>/qmc.out (stdout+stderr incl. the launcher audit), <prefix>.s00N.scalar.dat, run_manifest.txt.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"; [ $# -gt 0 ] && shift
+[ "$BACKEND" = CUDA ] || { echo "run.sh: only CUDA (OpenMP offload + CUDA) is built for QMCPACK here" >&2; exit 2; }
+PROFILE="${HPCPERF_QMCPACK_PROFILE:-clang231-cuda132-offload}"
+l3_paths_profile qmcpack "$PROFILE"
+SRC="$R/_upstream/level3/qmcpack"
+LLVM="$L3_INSTALL/llvm"
+EXE="$L3_INSTALL/qmcpack-real/bin/qmcpack"
+[ -x "$EXE" ] || { echo "run.sh: $EXE not found -- run ./build.sh first (profile $PROFILE)" >&2; exit 1; }
+export LD_LIBRARY_PATH="$LLVM/lib:$LLVM/lib/x86_64-unknown-linux-gnu:$L3_INSTALL/hdf5/lib:$L3_INSTALL/openblas/lib:${LD_LIBRARY_PATH:-}"
+l3_binary_backend_check "$EXE" cuda || exit 1
+# the offload runtime must be the private LLVM's (never a stray system libomptarget)
+ldd "$EXE" | /usr/bin/grep -q "libomptarget.so.*$LLVM" || { echo "run.sh: $EXE does not resolve libomptarget from $LLVM:"; ldd "$EXE" | /usr/bin/grep -E 'omptarget|libomp' >&2; exit 1; }
+CASE="${HPCPERF_QMCPACK_CASE:-diamond2}"
+MODE="$(l3_scale_mode qmcpack)" || exit 2
+N_RANKS="$(hpcperf_ranks qmcpack yes)" || exit 2
+THREADS="${HPCPERF_CPUS_PER_RANK:-8}"
+case "$CASE" in
+    diamond2) CASE_DIR="$SRC/tests/solids/diamondC_2x1x1_pp"; INP="$CASE_DIR/qmc_short_vmcbatch_dmcbatch.in.xml"; AUX="C.BFD.xml pwscf.pwscf.h5"; PREFIX=qmc_short_vmcbatch_dmcbatch ;;
+    *) echo "run.sh: HPCPERF_QMCPACK_CASE must be diamond2" >&2; exit 2 ;;
+esac
+[ -f "$INP" ] || { echo "run.sh: $INP missing (run fetch.sh)" >&2; exit 1; }
+case "$MODE" in
+    smoke)  LABEL="$CASE"; DERIV="verbatim" ;;
+    strong) W="${HPCPERF_QMCPACK_WALKERS:-256}"; LABEL="$CASE.w$W"; DERIV="total_walkers=$W" ;;
+    weak)   W="${HPCPERF_QMCPACK_WALKERS_PER_RANK:-256}"; LABEL="$CASE.wpr$W"; DERIV="walkers_per_rank=$W" ;;
+esac
+# population guard (see the header): walkers per GPU above the measured device-memory limit are refused unless forced
+MAXW="${HPCPERF_QMCPACK_MAX_WALKERS_PER_GPU:-300}"
+case "$MODE" in smoke) WPR=$(( (256 + N_RANKS - 1) / N_RANKS )) ;; strong) WPR=$(( (W + N_RANKS - 1) / N_RANKS )) ;; weak) WPR=$W ;; esac
+if [ "$WPR" -gt "$MAXW" ] && [ -z "${HPCPERF_QMCPACK_FORCE_POPULATION:-}" ]; then
+    echo "run.sh: $WPR walkers per GPU requested, above the measured limit of $MAXW for this build (device memory ~320 MB/walker; 1024 walkers on one B200 abort in cuSOLVER) -- refusing; set HPCPERF_QMCPACK_FORCE_POPULATION=1 to try anyway" >&2; exit 2
+fi
+RUN_DIR="$(l3_rundir "$L3_BUILD/$L3_RUN_SUBDIR/$LABEL.$MODE.np$N_RANKS.t$THREADS")" || exit 2
+DECK="$RUN_DIR/$(basename "$INP")"
+if [ "$MODE" = smoke ]; then
+    cp "$INP" "$DECK"
+else
+    python3 - "$INP" "$DECK" "$MODE" "$W" <<'PY'
+import re, sys
+src, dst, mode, w = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4]
+txt = open(src).read()
+pat = re.compile(r'<parameter name="total_walkers">\s*\d+\s*</parameter>')
+if len(pat.findall(txt)) != 2:
+    sys.exit("run.sh: expected exactly two total_walkers parameters (VMC, DMC) in the upstream deck")
+name = "total_walkers" if mode == "strong" else "walkers_per_rank"
+open(dst, "w").write(pat.sub(f'<parameter name="{name}"> {w} </parameter>', txt))
+PY
+    diff -u "$INP" "$DECK" > "$RUN_DIR/deck.diff" || true
+fi
+for f in $AUX; do ln -sfn "$CASE_DIR/$f" "$RUN_DIR/$f"; done
+export OMP_NUM_THREADS="$THREADS" OMP_TARGET_OFFLOAD=MANDATORY OMP_PROC_BIND=false
+echo "# QMCPACK $BACKEND profile=$PROFILE case=$CASE mode=$MODE ($DERIV) ranks=$N_RANKS threads/rank=$THREADS deck=$(realpath --relative-to="$SRC" "$INP") run_dir=$RUN_DIR"
+cd "$RUN_DIR"
+RUN_ID="$(l3_run_id)"
+set +e; set -o pipefail
+"$L3_LAUNCHER" --gpus "$N_RANKS" --cpus-per-rank "$THREADS" --bind wrapper -- "$EXE" "$(basename "$DECK")" 2>&1 | tee "$RUN_DIR/qmc.out"
+rc=$?
+set +o pipefail; set -e
+if [ -z "${HPCPERF_DRY_RUN:-}" ]; then
+    l3_manifest "$RUN_DIR" "run_id=$RUN_ID" "app=qmcpack" "backend=$BACKEND" "profile=$PROFILE" "case=$CASE" "label=$LABEL" "mode=$MODE" "deck_derivation=$DERIV" \
+        "ranks=$N_RANKS" "threads_per_rank=$THREADS" "walkers_per_gpu=$WPR" "exit_code=$rc" "binary=$EXE" "binary_sha256=$(l3_sha_file "$EXE")" \
+        "upstream_input=$INP" "upstream_input_sha256=$(l3_sha_file "$INP")" "deck_sha256=$(l3_sha_file "$DECK")" \
+        "orbitals_h5_sha256=$(l3_sha_file "$CASE_DIR/pwscf.pwscf.h5")" "pseudo_sha256=$(l3_sha_file "$CASE_DIR/C.BFD.xml")" \
+        "fingerprint_sha256=$(l3_sha_file "$L3_INSTALL/.hpcperf-l3-fingerprint")" "prefix=$PREFIX" "omp_target_offload=MANDATORY" "utc=$(date -u +%FT%TZ)"
+    /usr/bin/grep -aE 'offload to accelerators|CUDA acceleration|OpenMP device|devices|MPI Nodes|MPI ranks|OMP 1st level|Rank.*device' "$RUN_DIR/qmc.out" 2>/dev/null | head -12 | sed 's/^/gpu_evidence: /' >> "$RUN_DIR/run_manifest.txt" || true
+fi
+exit "$rc"
diff --git a/level3/qmcpack/toolchain/build_llvm.sh b/level3/qmcpack/toolchain/build_llvm.sh
new file mode 100755
index 0000000..5b81611
--- /dev/null
+++ b/level3/qmcpack/toolchain/build_llvm.sh
@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# Private LLVM/Clang OpenMP-offload toolchain for QMCPACK (upstream's documented
+# GPU compiler for NVIDIA: "For NVIDIA GPUs, LLVM clang"; QMC_GPU="openmp;cuda").
+#
+#   LLVM 23.1.0 (llvmorg-23.1.0, 2026-08-25; llvm-project-23.1.0.src.tar.xz
+#   sha256 ab1f0e3ec52448c33e8782eaf0422504b87c7b016b22514653ee0d8fcee479ff)
+#   projects clang;lld, runtimes openmp;offload (host) + the GPU runtimes target
+#   nvptx64-nvidia-cuda with libc;openmp (LLVM >= 21 builds the OpenMP device runtime
+#   libompdevice through LLVM_RUNTIME_TARGETS; LIBOMPTARGET_DEVICE_ARCHITECTURES is
+#   unused since LLVM 21 -- the first attempt without the GPU runtime target produced no
+#   device runtime and clang failed with "no library 'libomptarget-nvptx.bc' found"),
+#   CUDA 13.2 (clang 23 lists CUDA_132 as FULLY_SUPPORTED),
+#   host compiler system GCC 14.2.1 (clang's default GCC toolchain on this node).
+#
+# Why a source build: the official binary release LLVM-23.1.0-Linux-X64.tar.xz
+# (sha256 18da30f7..., 2.0 GB) ships libomp.so and the clang-offload-* tools but
+# NO libomptarget / device runtime (.deps/level3/qmcpack/downloads/LLVM-binary-release-check.txt);
+# the node's own /home clang+llvm is an old, broken (libtinfo.so.5) install with
+# nvptx bitcode up to sm_60 only. Spack's llvm+cuda would also build from source.
+#
+#   ./build_llvm.sh            profile clang231-cuda132-offload, -j${HPCPERF_BUILD_JOBS:-24}
+# Install: .deps/level3/qmcpack/clang231-cuda132-offload/install/llvm  (~2-3 GB, ~1 h)
+# Nothing under the project conda env or the system is modified.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../../.." && pwd)"
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+PROFILE="${HPCPERF_QMCPACK_PROFILE:-clang231-cuda132-offload}"
+l3_paths_profile qmcpack "$PROFILE"
+VER=23.1.0; TARBALL="$R/.deps/level3/qmcpack/downloads/llvm-project-$VER.src.tar.xz"
+EXPECT_SHA=ab1f0e3ec52448c33e8782eaf0422504b87c7b016b22514653ee0d8fcee479ff
+PREFIX="$L3_INSTALL/llvm"
+# Source + build trees on the node's LOCAL disk (the project filesystem is NFS and
+# extracting/compiling the ~150k-file LLVM tree there took >1 h just to unpack); the
+# install, logs and BUILD_INFO go to the profile tree on the project filesystem.
+SCRATCH="${HPCPERF_LLVM_SCRATCH:-/tmp/hpcperf-l3-b2-scratch/qmcpack-llvm}"; mkdir -p "$SCRATCH"
+SRC="$SCRATCH/llvm-project-$VER.src"; BUILD="$SCRATCH/build"
+echo "$SCRATCH" > "$L3_SRC/LLVM_SCRATCH_LOCATION.txt"
+JOBS="${HPCPERF_BUILD_JOBS:-24}"
+CUDA_ROOT="${CUDA_HOME:-/usr/local/cuda-13.2}"
+ARCH="${HPCPERF_CUDA_ARCH:-$(l3_gpu_arch)}"; [ -n "$ARCH" ] || ARCH=100
+# clean host environment: system GCC 14 as host compiler; cmake/ninja from the project conda env
+# (tools only -- conda is NOT on CMAKE_PREFIX_PATH so no conda libraries are linked)
+CONDA_BIN="$R/.conda_env/bin"
+export PATH="/usr/bin:/usr/sbin:/bin:$CUDA_ROOT/bin"; unset CMAKE_PREFIX_PATH LD_LIBRARY_PATH CPATH LIBRARY_PATH CMAKE_GENERATOR CUDAARCHS CC CXX
+CMAKE="$CONDA_BIN/cmake"; NINJA="$CONDA_BIN/ninja"
+[ -x "$CMAKE" ] && [ -x "$NINJA" ] || { echo "build_llvm.sh: cmake/ninja not found under $CONDA_BIN" >&2; exit 1; }
+
+if [ -f "$PREFIX/.hpcperf-stage-done" ]; then echo "# LLVM $VER already installed under $PREFIX"; exit 0; fi
+[ -f "$TARBALL" ] || { echo "build_llvm.sh: $TARBALL missing (download llvm-project-$VER.src.tar.xz from the llvmorg-$VER release)" >&2; exit 1; }
+got="$(sha256sum "$TARBALL" | cut -d' ' -f1)"; [ "$got" = "$EXPECT_SHA" ] || { echo "build_llvm.sh: tarball sha256 $got != $EXPECT_SHA" >&2; exit 1; }
+if [ ! -f "$SRC/.hpcperf-extracted" ]; then rm -rf "$SRC"; tar -xJf "$TARBALL" -C "$SCRATCH"; touch "$SRC/.hpcperf-extracted"; fi
+# the device-architecture variable name changed across LLVM releases; use the one this tree defines
+DEVARCH_VAR=LIBOMPTARGET_DEVICE_ARCHITECTURES
+/usr/bin/grep -rqs 'OFFLOAD_DEVICE_ARCHITECTURES' "$SRC/offload/CMakeLists.txt" "$SRC/offload/DeviceRTL/CMakeLists.txt" && \
+    ! /usr/bin/grep -rqs 'LIBOMPTARGET_DEVICE_ARCHITECTURES' "$SRC/offload/CMakeLists.txt" "$SRC/offload/DeviceRTL/CMakeLists.txt" && DEVARCH_VAR=OFFLOAD_DEVICE_ARCHITECTURES
+
+echo "# LLVM $VER -> $PREFIX: host gcc $(/usr/bin/gcc -dumpfullversion), CUDA $CUDA_ROOT, device arch sm_$ARCH ($DEVARCH_VAR), -j$JOBS (expect ~1 h, ~15 GB build tree)"
+mkdir -p "$BUILD" "$L3_LOGS"
+t0=$(date +%s)
+"$CMAKE" -S "$SRC/llvm" -B "$BUILD" -G Ninja "-DCMAKE_MAKE_PROGRAM=$NINJA" \
+    -DCMAKE_BUILD_TYPE=Release "-DCMAKE_INSTALL_PREFIX=$PREFIX" \
+    -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
+    -DLLVM_ENABLE_PROJECTS="clang;lld" -DLLVM_ENABLE_RUNTIMES="openmp;offload" \
+    -DLLVM_TARGETS_TO_BUILD="X86;NVPTX" -DLLVM_ENABLE_ASSERTIONS=OFF \
+    -DLLVM_INCLUDE_TESTS=OFF -DLLVM_INCLUDE_BENCHMARKS=OFF -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_DOCS=OFF \
+    -DCLANG_INCLUDE_TESTS=OFF -DLLVM_ENABLE_ZSTD=OFF -DLLVM_ENABLE_LIBXML2=OFF -DLLVM_ENABLE_TERMINFO=OFF -DLLVM_ENABLE_LIBEDIT=OFF \
+    -DLLVM_PARALLEL_LINK_JOBS=4 -DLLVM_INSTALL_UTILS=ON \
+    "-DCUDAToolkit_ROOT=$CUDA_ROOT" "-D${DEVARCH_VAR}=sm_$ARCH" \
+    -DOPENMP_ENABLE_LIBOMPTARGET=ON -DLIBOMP_OMPT_SUPPORT=ON \
+    -DLLVM_RUNTIME_TARGETS="default;nvptx64-nvidia-cuda" \
+    -DRUNTIMES_nvptx64-nvidia-cuda_LLVM_ENABLE_RUNTIMES="libc;openmp" \
+    > "$L3_LOGS/llvm-configure.log" 2>&1 || { tail -40 "$L3_LOGS/llvm-configure.log"; echo "build_llvm.sh: configure failed" >&2; exit 1; }
+"$CMAKE" --build "$BUILD" -j "$JOBS" > "$L3_LOGS/llvm-build.log" 2>&1 || { tail -40 "$L3_LOGS/llvm-build.log"; echo "build_llvm.sh: build failed (log $L3_LOGS/llvm-build.log)" >&2; exit 1; }
+"$CMAKE" --install "$BUILD" > "$L3_LOGS/llvm-install.log" 2>&1 || { echo "build_llvm.sh: install failed" >&2; exit 1; }
+t1=$(date +%s)
+# evidence that the offload runtime and the sm_100 device runtime exist
+OMPT="$(find "$PREFIX/lib" -name 'libomptarget.so*' | head -1)"; DEVRT="$(find "$PREFIX/lib" -name 'libomptarget*nvptx*' -o -name 'libomptarget.devicertl.a' -o -name 'libompdevice*' 2>/dev/null | head -3 | paste -sd,)"
+[ -n "$OMPT" ] || { echo "build_llvm.sh: libomptarget.so not installed -- offload runtime missing" >&2; exit 1; }
+{
+    echo "llvm=$VER tarball_sha256=$EXPECT_SHA host_gcc=$(/usr/bin/gcc -dumpfullversion) cuda=$CUDA_ROOT device_arch=sm_$ARCH ($DEVARCH_VAR) jobs=$JOBS seconds=$((t1 - t0)) utc=$(date -u +%FT%TZ)"
+    echo "clang: $("$PREFIX/bin/clang" --version | head -1)"
+    echo "libomptarget: $OMPT"; echo "device runtime: ${DEVRT:-<none found>}"
+    echo "offload-arch: $("$PREFIX/bin/offload-arch" 2>&1 | head -2 | paste -sd,)"
+} > "$PREFIX/BUILD_INFO.txt"; cat "$PREFIX/BUILD_INFO.txt"
+touch "$PREFIX/.hpcperf-stage-done"
+echo "# LLVM built in $((t1 - t0)) s"
diff --git a/level3/qmcpack/toolchain/probe_offload.sh b/level3/qmcpack/toolchain/probe_offload.sh
new file mode 100755
index 0000000..81c3b26
--- /dev/null
+++ b/level3/qmcpack/toolchain/probe_offload.sh
@@ -0,0 +1,92 @@
+#!/usr/bin/env bash
+# Minimal OpenMP target-offload probe for the private LLVM toolchain, run BEFORE
+# QMCPACK is built (spec: no silent host fallback; prove the target region does
+# not run on the initial device; check numerics; then MPI with one rank per GPU).
+#
+#   ./probe_offload.sh            (1 GPU probe + 2- and 4-rank MPI probe via the common launcher)
+#
+# Checks (all must hold, else exit 1):
+#   [1] omp_get_num_devices() >= 1 and the default device is not the host
+#   [2] inside `#pragma omp target`: omp_is_initial_device() == 0 (executed on the GPU)
+#   [3] OMP_TARGET_OFFLOAD=MANDATORY set for every run (a failed offload aborts instead of
+#       falling back to the host); LIBOMPTARGET_INFO summary recorded
+#   [4] numerics: y = a*x + y on 2^20 doubles and a reduction, compared with the host result
+#       to 1e-12 relative (require_finite on the sums)
+#   [5] MPI (2 and 4 ranks, one GPU each through the launcher wrapper): every rank reports
+#       exactly 1 visible device, offload succeeds, and the nvidia-smi audit verifies the mapping
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+PROFILE="${HPCPERF_QMCPACK_PROFILE:-clang231-cuda132-offload}"
+l3_paths_profile qmcpack "$PROFILE"
+LLVM="$L3_INSTALL/llvm"; CLANG="$LLVM/bin/clang"; CLANGXX="$LLVM/bin/clang++"
+[ -x "$CLANG" ] || { echo "probe_offload.sh: $CLANG missing -- run build_llvm.sh" >&2; exit 1; }
+ARCH="${HPCPERF_CUDA_ARCH:-$(l3_gpu_arch)}"
+CUDA_ROOT="${CUDA_HOME:-/usr/local/cuda}"
+export OMPI_CC="$CLANG" OMPI_CXX="$CLANGXX" OMP_TARGET_OFFLOAD=MANDATORY
+export LD_LIBRARY_PATH="$LLVM/lib/x86_64-unknown-linux-gnu:$LLVM/lib:${LD_LIBRARY_PATH:-}"
+W="$L3_BUILD_DEPS/offload-probe"; rm -rf "$W"; mkdir -p "$W" "$L3_LOGS"
+cat > "$W/probe.cpp" <<'EOF'
+#include <omp.h>
+#include <cstdio>
+#include <cmath>
+#include <vector>
+#include <cstdlib>
+#ifdef USE_MPI
+#include <mpi.h>
+#endif
+int main(int argc, char** argv) {
+  int rank = 0, size = 1;
+#ifdef USE_MPI
+  MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size);
+#endif
+  const int ndev = omp_get_num_devices();
+  const int def = omp_get_default_device(), init = omp_get_initial_device();
+  int on_initial = 1;                       // must become 0 inside the target region
+  #pragma omp target map(tofrom: on_initial)
+  { on_initial = omp_is_initial_device(); }
+  const size_t n = 1u << 20; const double a = 3.0;
+  std::vector<double> x(n), y(n), yh(n);
+  for (size_t i = 0; i < n; ++i) { x[i] = std::sin(0.001 * i); y[i] = std::cos(0.002 * i); yh[i] = a * x[i] + y[i]; }
+  double* xp = x.data(); double* yp = y.data(); double sum = 0.0;
+  #pragma omp target teams distribute parallel for map(to: xp[0:n]) map(tofrom: yp[0:n]) reduction(+: sum)
+  for (size_t i = 0; i < n; ++i) { yp[i] = a * xp[i] + yp[i]; sum += yp[i]; }
+  double sumh = 0.0, maxdiff = 0.0;
+  for (size_t i = 0; i < n; ++i) { sumh += yh[i]; maxdiff = std::fmax(maxdiff, std::fabs(yh[i] - y[i])); }
+  const double rel = std::fabs(sum - sumh) / std::fabs(sumh);
+  const char* cvd = std::getenv("CUDA_VISIBLE_DEVICES");
+  std::printf("probe rank %d/%d: num_devices=%d default_device=%d initial_device=%d target_ran_on_initial_device=%d "
+              "axpy_maxabs=%.3e sum_rel=%.3e sum_finite=%d CUDA_VISIBLE_DEVICES=%s\n",
+              rank, size, ndev, def, init, on_initial, maxdiff, rel, std::isfinite(sum) && std::isfinite(sumh), cvd ? cvd : "<unset>");
+  int ok = (ndev >= 1) && (def != init) && (on_initial == 0) && (maxdiff < 1e-12) && (rel < 1e-12) && std::isfinite(sum);
+#ifdef USE_MPI
+  int allok = 0; MPI_Allreduce(&ok, &allok, 1, MPI_INT, MPI_MIN, MPI_COMM_WORLD); ok = allok; MPI_Finalize();
+#endif
+  return ok ? 0 : 1;
+}
+EOF
+FLAGS=(-O2 -std=c++17 -fopenmp "--offload-arch=sm_$ARCH" "--cuda-path=$CUDA_ROOT")
+echo "# probe: clang $("$CLANG" --version | head -1); --offload-arch=sm_$ARCH; CUDA $CUDA_ROOT; OMP_TARGET_OFFLOAD=MANDATORY"
+"$CLANGXX" "${FLAGS[@]}" -o "$W/probe" "$W/probe.cpp" 2>&1 | tee "$L3_LOGS/probe-compile.log" | /usr/bin/grep -iE 'error|warning: .*(cuda|offload)' || true
+[ -x "$W/probe" ] || { echo "probe_offload.sh: FAIL -- single-process probe did not compile (log $L3_LOGS/probe-compile.log)"; exit 1; }
+echo "# [1-4] single process, GPU 0"
+rc=0; CUDA_VISIBLE_DEVICES=0 LIBOMPTARGET_INFO=16 "$W/probe" > "$L3_LOGS/probe-1gpu.log" 2>&1 || rc=$?
+/usr/bin/grep -E '^probe rank|Libomptarget|omptarget' "$L3_LOGS/probe-1gpu.log" | head -8
+[ "$rc" -eq 0 ] || { echo "probe_offload.sh: FAIL -- single-process offload probe exited $rc (see $L3_LOGS/probe-1gpu.log)"; exit 1; }
+echo "# [5] MPI + one rank per GPU (conda Open MPI wrappers with OMPI_CXX=clang++)"
+mpicxx "${FLAGS[@]}" -DUSE_MPI -o "$W/probe_mpi" "$W/probe.cpp" > "$L3_LOGS/probe-mpi-compile.log" 2>&1 || { tail -20 "$L3_LOGS/probe-mpi-compile.log"; echo "probe_offload.sh: FAIL -- MPI probe did not compile"; exit 1; }
+for n in 2 4; do
+    rc=0; "$L3_LAUNCHER" --gpus "$n" --bind wrapper -- "$W/probe_mpi" > "$L3_LOGS/probe-mpi-np$n.log" 2>&1 || rc=$?
+    /usr/bin/grep -aE 'probe rank [0-9]|audit summary|hpcperf-bind:' "$L3_LOGS/probe-mpi-np$n.log" | head -12
+    [ "$rc" -eq 0 ] || { echo "probe_offload.sh: FAIL -- MPI probe with $n ranks exited $rc"; exit 1; }
+    [ "$(/usr/bin/grep -ac 'probe rank [0-9]' "$L3_LOGS/probe-mpi-np$n.log")" -eq "$n" ] || { echo "probe_offload.sh: FAIL -- expected $n rank reports"; exit 1; }
+    /usr/bin/grep -aq 'audit summary: '"$n"' verified, 0 mismatch' "$L3_LOGS/probe-mpi-np$n.log" || echo "probe_offload.sh: NOTE -- GPU mapping not fully verified by sampling for $n ranks (see log)"
+    good="$(/usr/bin/grep -acE 'probe rank [0-9]+/[0-9]+: num_devices=1 default_device=0 initial_device=1 target_ran_on_initial_device=0 ' "$L3_LOGS/probe-mpi-np$n.log")"
+    [ "$good" -eq "$n" ] || { echo "probe_offload.sh: FAIL -- only $good of $n ranks report exactly one device with the target region on it"; exit 1; }
+done
+echo "probe_offload.sh: PASS -- clang $("$CLANG" --version | head -1 | /usr/bin/grep -oE '[0-9]+\.[0-9]+\.[0-9]+') offload to sm_$ARCH: target regions run on the device (not the initial device), numerics match the host to <1e-12, MANDATORY offload, 2 and 4 MPI ranks each with exactly one GPU"
+echo "PASS $(date -u +%FT%TZ) clang=$("$CLANG" --version | head -1) arch=sm_$ARCH" > "$L3_INSTALL/llvm/OFFLOAD_PROBE.txt"
diff --git a/level3/qmcpack/validate.sh b/level3/qmcpack/validate.sh
new file mode 100755
index 0000000..fbe8bec
--- /dev/null
+++ b/level3/qmcpack/validate.sh
@@ -0,0 +1,111 @@
+#!/usr/bin/env bash
+# Correctness check for QMCPACK (real, OpenMP offload + CUDA) on N GPUs.
+#
+#   ./validate.sh [CUDA]      HPCPERF_GPUS=N (default 1), HPCPERF_CPUS_PER_RANK=T (default 8)
+#
+# [1] (N = 1 only, skip with HPCPERF_QMCPACK_SKIP_CTEST=1) upstream's own ctest suites on this binary:
+#     the Catch2 unit tests (label `unit`, they exercise the offload/CUDA kernels, batched BLAS,
+#     determinant updates, ...) and the deterministic integration tests of the diamond cases
+#     (fixed seeds, exact scalar references with upstream tolerances). Every failure is a FAIL.
+# [2] Science case (run.sh diamond2, smoke = upstream deck verbatim) on N GPUs:
+#     completeness ("QMCPACK execution completed successfully", no "QMCPACK ERROR", exit 0,
+#     25 DMC blocks in the s001 scalar file, every scalar finite), GPU evidence from QMCPACK's
+#     own banner ("OpenMP target offload to accelerators build option is enabled" AND "CUDA
+#     acceleration build option is enabled"; the run itself has OMP_TARGET_OFFLOAD=MANDATORY so a
+#     host fallback would have aborted), and the upstream statistical check: check_scalars.py
+#     --ns 3 --series 1 -e 2 --le "-21.844975 0.02" (DIAMOND2_DMC_SCALARS -- the very test upstream
+#     runs on this deck at 1 and 4 ranks). For N > 1 additionally: the N-GPU DMC energy must agree
+#     with the 1-GPU run of the same binary/deck within 3 sqrt(sigma_1^2 + sigma_N^2) (block error
+#     bars from check_scalars -- a pre-fixed statistical consistency criterion; QMC results are
+#     stochastic, bitwise agreement across rank counts is not expected).
+# Every exit code is captured; timeout / nonzero / missing files -> FAIL; nothing is read from old runs
+# except the 1-GPU reference, which is re-run when its binary hash differs.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+N="${HPCPERF_GPUS:-1}"; T="${HPCPERF_CPUS_PER_RANK:-8}"
+TIMEOUT="${HPCPERF_VALIDATE_TIMEOUT:-3600}"
+PROFILE="${HPCPERF_QMCPACK_PROFILE:-clang231-cuda132-offload}"
+l3_paths_profile qmcpack "$PROFILE"
+SRC="$R/_upstream/level3/qmcpack"; RUNS="$L3_BUILD/$L3_RUN_SUBDIR"; BLD="$L3_BUILD/real"
+REF_LE="-21.844975 0.02"   # upstream DIAMOND2_DMC_SCALARS totenergy (mean sigma) for qmc_short_vmcbatch_dmcbatch, series 1
+NSIGMA=3                    # upstream check_scalars default used by QMC_RUN_AND_CHECK
+EQUIL=2                     # upstream: -e 2 blocks of equilibration
+DET_REGEX='deterministic-diamondC_(1x1x1|2x1x1)_pp-(vmcbatch|dmcbatch|sdbatch|vmc_sdj|vmc_dmc)'
+export HPCPERF_GPUS="$N" HPCPERF_CPUS_PER_RANK="$T" HPCPERF_SCALE_MODE=smoke
+mkdir -p "$RUNS"; ok=1
+fail() { echo "validate.sh: FAIL -- $*"; ok=0; }
+manifest_val() { /usr/bin/grep -m1 "^$2=" "$1/run_manifest.txt" 2>/dev/null | cut -d= -f2- || true; }
+LLVM="$L3_INSTALL/llvm"
+export LD_LIBRARY_PATH="$LLVM/lib:$LLVM/lib/x86_64-unknown-linux-gnu:$L3_INSTALL/hdf5/lib:$L3_INSTALL/openblas/lib:${LD_LIBRARY_PATH:-}"
+
+if [ "$N" -eq 1 ] && [ -z "${HPCPERF_QMCPACK_SKIP_CTEST:-}" ]; then
+    echo "validate.sh: [1] QMCPACK $BACKEND upstream ctest suites on 1 GPU [profile $PROFILE, build $BLD]"
+    [ -f "$BLD/CTestTestfile.cmake" ] || { fail "build tree $BLD has no CTestTestfile.cmake"; }
+    if [ -f "$BLD/CTestTestfile.cmake" ]; then
+        # ctest launches mpiexec itself for the MPI tests: same slot-accounting relaxation as the common launcher
+        # (this Slurm allocation advertises 1 task slot), one GPU visible, no oversubscription of that GPU.
+        export PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe OMPI_MCA_rmaps_base_oversubscribe=true
+        export OMPI_MCA_pml=ob1 OMPI_MCA_btl=self,sm,smcuda OMP_TARGET_OFFLOAD=MANDATORY CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0}"
+        for suite in "unit:-L unit" "deterministic:-R $DET_REGEX"; do
+            name="${suite%%:*}"; sel="${suite#*:}"
+            log="$RUNS/ctest.$name.np1.log"
+            # shellcheck disable=SC2086
+            ( cd "$BLD" && timeout "$TIMEOUT" ctest $sel --timeout 900 --output-on-failure -j "${HPCPERF_CTEST_JOBS:-4}" ) > "$log" 2>&1 || rc=$?
+            rc=${rc:-0}
+            summary="$(/usr/bin/grep -aE '^[0-9]+% tests passed|tests passed,' "$log" | tail -1)"
+            ntests="$(/usr/bin/grep -aoE 'out of [0-9]+' "$log" | tail -1 | /usr/bin/grep -oE '[0-9]+' || echo 0)"
+            echo "    ctest $name: ${summary:-no summary} (exit $rc, log $log)"
+            [ "$rc" -eq 124 ] && fail "ctest $name timed out after ${TIMEOUT}s"
+            [ "${ntests:-0}" -gt 0 ] || fail "ctest $name selected no tests"
+            /usr/bin/grep -aq '^100% tests passed' "$log" || { fail "ctest $name did not pass 100%"; /usr/bin/grep -aE '\*\*\*Failed|Timeout|Not Run' "$log" | head -20 | sed 's/^/      /'; }
+            unset rc
+        done
+        unset PRTE_MCA_rmaps_default_mapping_policy OMPI_MCA_rmaps_base_oversubscribe
+    fi
+fi
+
+echo "validate.sh: [2] QMCPACK $BACKEND diamondC_2x1x1_pp VMC+DMC (batched drivers, upstream deck) on $N GPU(s) x $T threads"
+D="$RUNS/diamond2.smoke.np$N.t$T"
+rc=0; HPCPERF_QMCPACK_CASE=diamond2 timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" > "$RUNS/validate.diamond2.np$N.stdout" 2>&1 || rc=$?
+/usr/bin/grep -aE '^# QMCPACK|hpcperf-launch: audit summary' "$RUNS/validate.diamond2.np$N.stdout" || true
+if [ "$rc" -eq 124 ]; then fail "diamond2 timed out after ${TIMEOUT}s"; elif [ "$rc" -ne 0 ]; then fail "diamond2 run.sh exited $rc (see $RUNS/validate.diamond2.np$N.stdout)"; fi
+check_run() { # check_run <run_dir> -> writes <run_dir>/qmc_summary.txt; returns 1 on any violation
+    local d=$1
+    python3 "$HERE/qmc_check.py" "$d" "$(manifest_val "$d" prefix)" "$REF_LE" "$NSIGMA" "$EQUIL" "$SRC/tests/scripts/check_scalars.py" > "$d/qmc_summary.txt"
+}
+if [ -f "$D/qmc.out" ]; then
+    check_run "$D" || fail "diamond2 np$N: completeness/GPU-evidence/statistical check failed"
+    sed 's/^/    /' "$D/qmc_summary.txt"
+    if [ "$N" -gt 1 ]; then
+        REF="$RUNS/diamond2.smoke.np1.t$T"
+        if [ ! -f "$REF/qmc_summary.txt" ] || [ "$(manifest_val "$REF" binary_sha256)" != "$(manifest_val "$D" binary_sha256)" ]; then
+            echo "    (1-GPU reference missing/stale -- running diamond2 on 1 GPU now)"
+            rc=0; HPCPERF_GPUS=1 HPCPERF_QMCPACK_CASE=diamond2 timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" > "$RUNS/validate.diamond2.np1.stdout" 2>&1 || rc=$?
+            [ "$rc" -eq 0 ] || fail "diamond2 1-GPU reference run exited $rc"
+            if [ -f "$REF/qmc.out" ]; then check_run "$REF" || fail "diamond2 1-GPU reference run failed its own checks"; fi
+        fi
+        python3 - "$REF/qmc_summary.txt" "$D/qmc_summary.txt" "$NSIGMA" <<'PY' || fail "diamond2 np$N: DMC energy inconsistent with the 1-GPU run"
+import math, re, sys
+def load(p):
+    d = {}
+    for l in open(p):
+        for k, v in re.findall(r"(\w+)=(-?[0-9.]+(?:e[-+]?\d+)?)", l): d[k] = float(v)
+    return d
+a, b, ns = load(sys.argv[1]), load(sys.argv[2]), float(sys.argv[3])
+for k in ("dmc_mean", "dmc_err"):
+    if k not in a or k not in b: print(f"    missing {k} in a summary"); sys.exit(1)
+sig = math.sqrt(a["dmc_err"] ** 2 + b["dmc_err"] ** 2); d = abs(a["dmc_mean"] - b["dmc_mean"])
+print(f"    cross-rank: 1-GPU {a['dmc_mean']:.6f} +- {a['dmc_err']:.6f}  N-GPU {b['dmc_mean']:.6f} +- {b['dmc_err']:.6f}  |diff| = {d:.6f} Ha = {d / sig if sig > 0 else float('inf'):.2f} sigma (limit {ns:.0f} sigma) {'ok' if d <= ns * sig else 'BAD'}")
+sys.exit(0 if d <= ns * sig else 1)
+PY
+    fi
+fi
+
+if [ "$ok" -eq 1 ]; then echo "QMCPACK $BACKEND validation ($N GPU x $T threads; $( [ "$N" -eq 1 ] && [ -z "${HPCPERF_QMCPACK_SKIP_CTEST:-}" ] && echo "upstream unit + deterministic ctests 100%, " || true)diamondC_2x1x1 VMC+DMC complete/finite, offload+CUDA active, DMC energy within upstream ${NSIGMA}-sigma reference$( [ "$N" -gt 1 ] && echo ", consistent with 1-GPU" || true)): PASS"; exit 0; fi
+echo "QMCPACK $BACKEND validation ($N GPU): FAIL"; exit 1
diff --git a/level3/sparta/README.md b/level3/sparta/README.md
new file mode 100644
index 0000000..0cdef29
--- /dev/null
+++ b/level3/sparta/README.md
@@ -0,0 +1,143 @@
+# SPARTA (Level 3)
+
+Direct Simulation Monte Carlo (DSMC) for rarefied gas dynamics: particle
+move/sort, collisions, grid-cell decomposition, MPI migration -- the full
+application driven by its own input scripts, KOKKOS package on the GPU.
+
+## Provenance
+
+- Official repository: https://github.com/sparta/sparta (docs
+  https://sparta.github.io/doc/Manual.html; Kokkos section
+  https://sparta.github.io/doc/Section_accelerate.html)
+- Release policy: one stream of dated tags (no stable/feature split).
+- Selected: **`27Aug2026`** (2026-08-28), commit
+  `95b9abaa8bd548991cc3c3f1c58b34722f7ade74`, fetched by `fetch.sh` into
+  `_upstream/level3/sparta` (shallow, read-only). This release moved the
+  bundled Kokkos to 5.0.2 and made KOKKOS builds CMake-only / C++20.
+- License: GPL-2.0 (`LICENSE`).
+- Application-owned LOC (cloc 2.06, code lines): `src/` **131,181** (C++
+  104,683; headers 25,195; incl. `src/KOKKOS` 36,909 in 194 files). Bundled
+  `lib/kokkos` (Kokkos 5.0.2, 223,495 lines) counted separately, not modified.
+
+## Build strategy: NATIVE (upstream CMake preset + bundled Kokkos 5.0.2)
+
+`build.sh CUDA` = the documented recipe: `cmake -S sparta/cmake -C
+cmake/presets/kokkos_common.cmake` with `nvcc_wrapper` (host conda GCC 13.3.0)
+as CXX, `Kokkos_ENABLE_CUDA`, `Kokkos_ARCH_BLACKWELL100` (the docs list "GB200
+(Blackwell) -> BLACKWELL100" explicitly), `Kokkos_ENABLE_SERIAL=ON`,
+`Kokkos_ENABLE_OPENMP=OFF`, `FFT_KOKKOS=CUFFT`, C++20, `BUILD_MPI` (conda Open
+MPI 5.0.10, CUDA-aware), `SPARTA_MACHINE=kokkos_cuda`. Build time on dgx003:
+**579 s** at `-j32` (278 targets); 108 warning lines (nvcc_wrapper multiple
+`-O` flags, a few upstream notes), no errors. Executable
+`build/level3/sparta/cuda/src/spa_kokkos_cuda` (302 MB, static Kokkos);
+install prefix `.deps/level3/sparta/install` with fingerprint (upstream
+commit, Kokkos 5.0.2, compiler, CUDA 13.2.78, MPI, CMake options).
+
+Why not the others: **there is no Spack package for this SPARTA** -- the
+`sparta` recipe in Spack (local and upstream) is the unrelated bioinformatics
+tool sPARTA; upstream documents only CMake presets. No Apptainer on the node
+and no upstream image. Site modules broken. Level 2's Kokkos 5.2.1 is not used
+because the bundled 5.0.2 is the version the release was tested with (SPARTA
+does not pin an external Kokkos, so `USE_EXTERNAL_KOKKOS=ON` remains a
+documented fallback).
+
+HIP: `build.sh HIP` carries the upstream `kokkos_hip` recipe (`hipcc`,
+`Kokkos_ARCH_AMD_GFX950`, `FFT_KOKKOS=HIPFFT`) and exits with a clear message
+here (no ROCm). Note the bundled Kokkos 5.0.2 has no `AMD_GFX950`
+architecture (added in Kokkos 5.1); an MI355X build would need
+`USE_EXTERNAL_KOKKOS`. **Untested.**
+
+## Changes from upstream
+
+Class **A -- none.** `bench/in.collide` is run unmodified from `bench/` with
+its documented `-var x y z` size variables; the log goes to the build tree.
+
+## Execution model
+
+One MPI rank per GPU (upstream: "the -np setting ... should set the number of
+MPI tasks/node to be equal to the # of physical GPUs on the node"), `-k on g 1
+-sf kk -pk kokkos gpu/aware yes`. The common launcher's per-rank wrapper gives
+each rank exactly one visible GPU and audits the mapping. GPU-aware MPI
+defaults to `yes` -- unlike LAMMPS, SPARTA does **not** auto-detect
+CUDA-awareness, so `HPCPERF_SPARTA_GPU_AWARE=no` must be used with a
+non-CUDA-aware MPI. Any rank count is legal: the deck uses `balance_grid rcb
+part` (recursive coordinate bisection), no processor-grid constraint.
+
+## Inputs (`HPCPERF_SCALE_MODE`)
+
+| Mode | Grid cells | Particles (10/cell) | Per rank @4 GPU | Topology | Steps | Memory/GPU (est.) | Runtime on B200 | Validation quantity |
+|---|---|---|---|---|---|---|---|---|
+| smoke (default) | 10x10x10 (upstream default) | 10,000 | 2,500 | RCB | 30 + 100 | < 0.1 GB | 0.03-0.05 s | Np, temp, Natt vs upstream reference log |
+| strong | S^3, S=`HPCPERF_SPARTA_STRONG` (100) | 10,000,000 | 2,500,000 | RCB | 30 + 100 | ~2 GB | 0.87 s (1 GPU) / 0.38 s (4 GPU) | same stats |
+| weak | (L*PX)x(L*PY)x(L*PZ), L=`HPCPERF_SPARTA_LOCAL` (50) | 1,250,000 x N | 1,250,000 | grid from `hpcperf_topology.py` (RCB inside) | 30 + 100 | ~0.3 GB | 0.23 s (4 GPU) | same stats |
+
+The deck runs 30 equilibration steps followed by the 100-step benchmark
+(`run 30` / `run 100`, upstream). Memory estimate ~100 B/particle plus
+per-cell data; all sizes above are far below a B200's 180 GB. The strong
+default is a *correctness* size: 10M particles per 100 steps take under a
+second, so 4-GPU vs 1-GPU timings (0.38 s vs 0.87 s) indicate the run is
+already partially communication/launch bound and are not a scaling result.
+
+## Validation (`validate.sh`, upstream mechanism)
+
+Upstream's own guidance (`examples/README`, `tools/testing/regression.py`)
+is statistical: DSMC is stochastic and "should get statistically similar
+answers ... on different numbers of processors, but not identical answers".
+`validate.sh` therefore compares the stats table of the unmodified
+`bench/in.collide` with the reference log SPARTA ships,
+`bench/log.7Jul14.collide.icc.10K.1`, on three quantities:
+
+1. particle count `Np` == 10,000 at every stats row (closed box, no
+   chemistry: exact conservation);
+2. gas temperature (`compute temp`, printed as `c_temp`; the 2014 log labels
+   it `temp`): mean over the benchmark steps within 2 % of the reference.
+   Elastic VSS collisions conserve energy exactly, so within a run the
+   temperature is constant; its value is set by the Maxwellian sampling of the
+   initial velocities, whose statistical scatter for 10^4 particles is
+   sqrt(2/3N) ~ 0.8 %. 2 % is ~2.5 sigma of that noise and far below any
+   unit/physics error;
+3. mean collision attempts per step `Natt` within 15 % (fixed by density,
+   temperature and cross-section; run-to-run scatter is a few %).
+
+With N > 1 GPUs the same three criteria are applied between the N-rank run and
+this build's 1-rank run. Observed on dgx003 (2026-09-04): **PASS at 1, 2 and 4
+GPUs**:
+
+| GPUs | Np | mean temp (ref 274.41 K) | rel | mean Natt (ref 943.7) | rel | vs 1-GPU temp / Natt |
+|---|---|---|---|---|---|---|
+| 1 | 10,000 at every row | 275.43 | 3.7e-3 | 946.1 | 2.5e-3 | -- |
+| 2 | 10,000 | 271.65 | 1.0e-2 | 942.4 | 1.4e-3 | 1.4e-2 / 3.9e-3 |
+| 4 | 10,000 | 274.64 | 8.5e-4 | 947.7 | 4.2e-3 | 2.9e-3 / 1.7e-3 |
+
+(All temperature deviations are within ~1.7 sigma of the sampling noise; the
+nominal gas temperature is 273.15 K.)
+
+## Results on dgx003 (4x B200, CUDA 13.2.78, Slurm job 9552083)
+
+| Run | Ranks x GPUs | rank->GPU | CPU binding | Topology | Problem | Loop time (100-step benchmark) | Validation |
+|---|---|---|---|---|---|---|---|
+| smoke | 1 x 1 | wrapper; audit unverified (0.03 s run, too short to sample) | runtime default | RCB | 10k particles | 0.026 s | PASS |
+| smoke | 2 x 2 | wrapper; audit 1 verified / 1 unverified (short run) | runtime default | RCB | 10k particles | 0.041 s | PASS |
+| smoke | 4 x 4 | wrapper; audit 4/4 verified | runtime default | RCB | 10k particles | 0.053 s | PASS |
+| strong | 1 x 1 | wrapper; 1/1 verified | runtime default | RCB | 100^3 cells, 10M particles | 0.869 s | run completes, Np conserved |
+| strong | 4 x 4 | wrapper; 4/4 verified | runtime default | RCB | 100^3 cells, 10M particles | 0.381 s | run completes |
+| weak | 4 x 4 | wrapper; 4/4 verified | runtime default | 2x2x1 -> 100x100x50 | 5M particles (1.25M/rank) | 0.229 s | run completes |
+
+Dry-runs (`HPCPERF_DRY_RUN=1`, hypothetical allocations) -- **DRY-RUN /
+UNVALIDATED**, nothing executed:
+
+| GPUs | Nodes x GPUs/node | Mode | Grid | Particles | Per rank | Launch |
+|---|---|---|---|---|---|---|
+| 8 | 1 x 8 | strong | 100^3 | 10M | 1.25M | `mpirun -np 8 --host dgx003:8 --map-by ppr:8:node ...` (single node) |
+| 40 | 5 x 8 | weak | 250x200x100 | 50M | 1.25M | 5 nodes x 8 -- multi-node BLOCKED on this site |
+| 80 | 10 x 8 | weak | 250x200x200 | 100M | 1.25M | 10 nodes x 8 -- multi-node BLOCKED on this site |
+
+## Limitations
+
+- Multi-node: BLOCKED/UNVERIFIED on this site; 40/80-GPU shapes are plans.
+- HIP: recipe present, untested; bundled Kokkos lacks gfx950.
+- Only `bench/in.collide` is wrapped; `in.free` and `in.sphere` (surface
+  collisions, `fix balance`) build with this configuration but have no
+  wrappers yet.
+- Reference logs are from 2014 (Intel CPU) and only for 1 and 8 ranks; the
+  comparison is statistical by design (upstream policy), not bitwise.
diff --git a/level3/sparta/build.sh b/level3/sparta/build.sh
new file mode 100755
index 0000000..5a338a3
--- /dev/null
+++ b/level3/sparta/build.sh
@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+# Build SPARTA (KOKKOS package, CUDA or HIP) with upstream's native CMake and
+# the Kokkos it bundles (lib/kokkos) -- upstream's own kokkos_cuda/kokkos_hip
+# preset recipe, with the GPU architecture chosen for this node instead of the
+# preset's hard-coded HOPPER90.
+#
+#   ./build.sh [CUDA|HIP]        (default CUDA)
+#
+# Layout (Level 3 isolation): source _upstream/level3/sparta, build
+# build/level3/sparta/<cuda|hip>, install .deps/level3/sparta/install
+# (+ .hpcperf-l3-fingerprint), logs .deps/level3/sparta/logs.
+#
+# Recipe = upstream cmake/presets/kokkos_common.cmake (PKG_KOKKOS, BUILD_MPI,
+# -O3) loaded with -C, plus the settings of cmake/presets/kokkos_cuda.cmake
+# given explicitly: nvcc_wrapper as CXX (host compiler = conda GCC 13.3.0),
+# Kokkos_ENABLE_CUDA, Kokkos_ENABLE_SERIAL, FFT_KOKKOS=CUFFT, and
+# Kokkos_ARCH_BLACKWELL100 (sm_100) instead of HOPPER90. SPARTA requires
+# C++20 with the KOKKOS package (cmake/CMakeLists.txt).
+#
+# Modification class: A (no upstream file modified; the arch differs from the
+# shipped preset only through command-line cache entries).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+l3_isolate_build_env    # Level 3 builds must not see Level 2 .deps/install prefixes
+
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+SRC="$R/_upstream/level3/sparta"
+[ -f "$SRC/cmake/CMakeLists.txt" ] || { echo "build.sh: SPARTA source missing -- run $HERE/fetch.sh first" >&2; exit 1; }
+SHA="$(git -C "$SRC" rev-parse HEAD)"
+KOKKOS_VER="$(sed -n 's/^set(Kokkos_VERSION_\(MAJOR\|MINOR\|PATCH\) \([0-9]*\))/\2/p' "$SRC/lib/kokkos/CMakeLists.txt" | paste -sd.)"
+l3_paths sparta
+BUILD_DIR="$R/build/level3/sparta/$MODEL"
+JOBS="${HPCPERF_BUILD_JOBS:-32}"
+
+case "$BACKEND" in
+    CUDA)
+        ARCH="${HPCPERF_CUDA_ARCH:-$(l3_gpu_arch)}"
+        case "$ARCH" in
+            100) KARCH=BLACKWELL100;; 120) KARCH=BLACKWELL120;; 90) KARCH=HOPPER90;; 80) KARCH=AMPERE80;;
+            *) echo "build.sh: no Kokkos arch mapping for compute capability '$ARCH' (set HPCPERF_CUDA_ARCH)" >&2; exit 2;;
+        esac
+        export NVCC_WRAPPER_DEFAULT_COMPILER="$CXX"
+        GPU_FLAGS=(-DCMAKE_CXX_COMPILER="$SRC/lib/kokkos/bin/nvcc_wrapper"
+                   -DKokkos_ENABLE_CUDA=ON "-DKokkos_ARCH_$KARCH=ON" -DFFT_KOKKOS=CUFFT)
+        FFTK=CUFFT ;;
+    HIP)
+        command -v hipcc >/dev/null 2>&1 || { echo "build.sh: HIP requested but hipcc not found -- HIP build is UNTESTED on this machine (no ROCm)" >&2; exit 1; }
+        KARCH="${HPCPERF_HIP_ARCH:-AMD_GFX950}"
+        GPU_FLAGS=(-DCMAKE_CXX_COMPILER=hipcc -DKokkos_ENABLE_HIP=ON "-DKokkos_ARCH_$KARCH=ON" -DFFT_KOKKOS=HIPFFT)
+        FFTK=HIPFFT ;;
+    *) echo "usage: $0 [CUDA|HIP]" >&2; exit 2 ;;
+esac
+
+CMAKE_OPTS="preset=kokkos_common BUILD_MPI=ON PKG_KOKKOS=ON CXX_STANDARD=20 Kokkos_ENABLE_${BACKEND}=ON Kokkos_ARCH_${KARCH} Kokkos_ENABLE_SERIAL=ON Kokkos_ENABLE_OPENMP=OFF FFT_KOKKOS=$FFTK"
+FP="$(l3_fingerprint_text sparta "$SHA" "$MODEL" "kokkos(bundled)=$KOKKOS_VER" "$CMAKE_OPTS" "runtime(-pk kokkos gpu/aware)")"
+l3_fingerprint_check "$L3_INSTALL" "$FP" || exit 1
+
+echo "# SPARTA $BACKEND: upstream $SHA, bundled Kokkos $KOKKOS_VER, arch $KARCH, MPI $(mpirun --version 2>/dev/null | head -1)"
+mkdir -p "$BUILD_DIR"
+cmake -S "$SRC/cmake" -B "$BUILD_DIR" -G Ninja -C "$SRC/cmake/presets/kokkos_common.cmake" \
+    -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="$L3_INSTALL" -DCMAKE_CXX_STANDARD=20 \
+    -DSPARTA_MACHINE="kokkos_$MODEL" -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=OFF \
+    "${GPU_FLAGS[@]}" > "$L3_LOGS/configure-$MODEL.log" 2>&1 \
+    || { tail -30 "$L3_LOGS/configure-$MODEL.log"; echo "build.sh: configure failed (log: $L3_LOGS/configure-$MODEL.log)" >&2; exit 1; }
+t0=$(date +%s)
+cmake --build "$BUILD_DIR" -j "$JOBS" > "$L3_LOGS/build-$MODEL.log" 2>&1 \
+    || { tail -30 "$L3_LOGS/build-$MODEL.log"; echo "build.sh: build failed (log: $L3_LOGS/build-$MODEL.log)" >&2; exit 1; }
+cmake --install "$BUILD_DIR" > "$L3_LOGS/install-$MODEL.log" 2>&1 || { echo "build.sh: install failed" >&2; exit 1; }
+l3_fingerprint_write "$L3_INSTALL" "$FP"
+EXE="$(find "$BUILD_DIR" -maxdepth 2 -name "spa_kokkos_$MODEL" -type f | head -1)"
+echo "# built in $(( $(date +%s)-t0 )) s: ${EXE:-<spa_kokkos_$MODEL not found under $BUILD_DIR>} (installed under $L3_INSTALL)"
+echo "# compiler warning lines: $(grep -c 'warning' "$L3_LOGS/build-$MODEL.log" || true)"
diff --git a/level3/sparta/fetch.sh b/level3/sparta/fetch.sh
new file mode 100755
index 0000000..38116ab
--- /dev/null
+++ b/level3/sparta/fetch.sh
@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# Fetch SPARTA at the recorded release into _upstream/level3/sparta
+# (gitignored, read-only reference). Idempotent; a checkout at another commit
+# is an error, never silently reused.
+#
+#   ./fetch.sh
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+UPSTREAM_URL="https://github.com/sparta/sparta.git"
+UPSTREAM_TAG="27Aug2026"
+UPSTREAM_SHA="95b9abaa8bd548991cc3c3f1c58b34722f7ade74"
+DEST="$R/_upstream/level3/sparta"
+if [ -d "$DEST/.git" ]; then
+    have="$(git -C "$DEST" rev-parse HEAD)"
+    [ "$have" = "$UPSTREAM_SHA" ] && { echo "fetch.sh: $DEST already at $UPSTREAM_TAG ($UPSTREAM_SHA)"; exit 0; }
+    echo "fetch.sh: $DEST is at $have, not the recorded $UPSTREAM_SHA ($UPSTREAM_TAG); remove it to re-fetch" >&2; exit 1
+fi
+mkdir -p "$(dirname "$DEST")"
+echo "fetch.sh: cloning $UPSTREAM_URL @ $UPSTREAM_TAG (shallow)"
+git clone --quiet --depth 1 --branch "$UPSTREAM_TAG" "$UPSTREAM_URL" "$DEST"
+have="$(git -C "$DEST" rev-parse HEAD)"
+[ "$have" = "$UPSTREAM_SHA" ] || { echo "fetch.sh: tag $UPSTREAM_TAG resolved to $have, expected $UPSTREAM_SHA" >&2; exit 1; }
+echo "fetch.sh: ok -> $DEST ($UPSTREAM_SHA)"
diff --git a/level3/sparta/run.sh b/level3/sparta/run.sh
new file mode 100755
index 0000000..e628336
--- /dev/null
+++ b/level3/sparta/run.sh
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Run the SPARTA collisional-flow benchmark (upstream bench/in.collide) on N GPUs.
+#
+#   ./run.sh [CUDA|HIP] [extra spa args...]
+#
+# Execution model (upstream Section_accelerate): one MPI rank per GPU, KOKKOS
+# package on the device (`-k on g 1 -sf kk`), particles/grid distributed by
+# SPARTA's own `balance_grid rcb part` -- any rank count is legal. Ranks go
+# through the common launcher with the per-rank GPU wrapper (each rank sees
+# exactly one GPU; mapping audited). GPU-aware MPI (`-pk kokkos gpu/aware
+# yes`, SPARTA's default on GPUs) matches this repository's CUDA-aware
+# Open MPI; HPCPERF_SPARTA_GPU_AWARE=no disables it.
+#
+# Resource / size controls (common Level 3 parameters):
+#   HPCPERF_GPUS=N|all        ranks = GPUs (default 1)
+#   HPCPERF_SCALE_MODE        smoke | strong | weak   (default smoke)
+#     smoke  : upstream deck as shipped: 10x10x10 cells, 10 particles/cell =
+#              10,000 particles; 30 equilibration + 100 benchmark steps
+#              (reference log bench/log.7Jul14.collide.icc.10K.1)
+#     strong : ONE fixed global grid, S^3 cells (S=HPCPERF_SPARTA_STRONG,
+#              default 100: 1,000,000 cells = 10,000,000 particles), split by
+#              SPARTA over the ranks
+#     weak   : fixed work per rank, L^3 cells per rank (L=HPCPERF_SPARTA_LOCAL,
+#              default 50: 125,000 cells = 1,250,000 particles/rank); grid
+#              L*PX x L*PY x L*PZ with PXxPYxPZ from hpcperf_topology.py
+#   HPCPERF_SPARTA_GPU_AWARE  yes|no (default yes)
+# The deck is upstream's bench/in.collide, unmodified; sizes enter through its
+# own -var x/y/z variables (particles = 10 * cells by construction).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"; [ $# -gt 0 ] && shift
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+BUILD_DIR="$R/build/level3/sparta/$MODEL"
+EXE="$(find "$BUILD_DIR" -maxdepth 2 -name "spa_kokkos_$MODEL" -type f 2>/dev/null | head -1)"
+[ -n "$EXE" ] && [ -x "$EXE" ] || { echo "run.sh: spa_kokkos_$MODEL not found under $BUILD_DIR -- run ./build.sh $BACKEND first" >&2; exit 1; }
+SRC="$R/_upstream/level3/sparta"
+
+N_RANKS="$(hpcperf_ranks sparta yes)" || exit 2
+hpcperf_forbid_args sparta -in -i -var -v -k -kokkos -sf -suffix -pk -package -log -- "$@" || exit 2
+MODE="$(l3_scale_mode sparta)" || exit 2
+GAM="${HPCPERF_SPARTA_GPU_AWARE:-yes}"
+
+case "$MODE" in
+    smoke)  X=10; Y=10; Z=10 ;;
+    strong) S="${HPCPERF_SPARTA_STRONG:-100}"; X=$S; Y=$S; Z=$S ;;
+    weak)   L="${HPCPERF_SPARTA_LOCAL:-50}"
+            TOPO="$(hpcperf_topology sparta "$N_RANKS")" || exit 2
+            read -r PX PY PZ <<< "$TOPO"
+            X=$((L * PX)); Y=$((L * PY)); Z=$((L * PZ)) ;;
+esac
+CELLS=$((X * Y * Z)); PARTS=$((10 * CELLS))
+RUN_DIR="$BUILD_DIR/$L3_RUN_SUBDIR"
+[ -n "${HPCPERF_DRY_RUN:-}" ] && RUN_DIR="$RUN_DIR/.dryrun"   # dry-run never overwrites real results
+mkdir -p "$RUN_DIR"
+LOG="$RUN_DIR/log.$MODE.np$N_RANKS.sparta"
+rm -f "$LOG"      # validate only against THIS run's output; never a stale log
+echo "# SPARTA $BACKEND: mode=$MODE ranks=$N_RANKS grid=${X}x${Y}x${Z} = $CELLS cells, $PARTS particles ($((PARTS / N_RANKS))/rank), gpu-aware=$GAM, log=$LOG"
+cd "$SRC/bench"   # ar.species / ar.vss are referenced relative to the deck
+RUN_ID="$(l3_run_id)"
+rc=0
+"$L3_LAUNCHER" --gpus "$N_RANKS" --bind wrapper -- \
+    "$EXE" -k on g 1 -sf kk -pk kokkos gpu/aware "$GAM" \
+    -in in.collide -var x "$X" -var y "$Y" -var z "$Z" -log "$LOG" -echo none "$@" || rc=$?
+if [ -z "${HPCPERF_DRY_RUN:-}" ]; then
+    l3_manifest "$RUN_DIR" "run_id=$RUN_ID" "app=sparta" "backend=$BACKEND" "mode=$MODE" \
+        "ranks=$N_RANKS" "cells=$CELLS" "particles=$PARTS" "gpu_aware=$GAM" "exit_code=$rc" \
+        "binary=$EXE" "binary_sha256=$(l3_sha_file "$EXE")" "input=$SRC/bench/in.collide" \
+        "input_sha256=$(l3_sha_file "$SRC/bench/in.collide")" "log=$LOG" "utc=$(date -u +%FT%TZ)"
+fi
+exit "$rc"
diff --git a/level3/sparta/validate.sh b/level3/sparta/validate.sh
new file mode 100755
index 0000000..81ac566
--- /dev/null
+++ b/level3/sparta/validate.sh
@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+# Correctness check for the SPARTA Kokkos build, using upstream's benchmark
+# deck bench/in.collide as shipped (10,000 particles) and the reference log
+# SPARTA ships for it, bench/log.7Jul14.collide.icc.10K.1.
+#
+#   ./validate.sh [CUDA|HIP]        HPCPERF_GPUS=N (default 1) selects the rank count
+#
+# DSMC is stochastic, so per-step collision counts cannot match exactly. What
+# is checked (adapted subset of upstream's tolerance-based regression):
+#   * the benchmark stats block is COMPLETE -- it must span the equilibration
+#     boundary (step 30) through the final step (130); a truncated run FAILs;
+#   * particle count Np == 10,000 at every stats row (exact conservation);
+#   * mean gas temperature over steps >= 40 within 2 % of the reference
+#     (statistical noise ~0.8 % for 10^4 particles);
+#   * mean collision attempts (Natt) within 15 %.
+# With HPCPERF_GPUS>1 the same criteria compare the N-rank run with this build's
+# 1-rank run. Reproducibility: the run's real exit code is captured (nonzero /
+# timeout / missing log / non-finite -> FAIL), run.sh removes its target log
+# first so only this run's output is used, and NaN/Inf is rejected explicitly.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+N="${HPCPERF_GPUS:-1}"
+REF="$R/_upstream/level3/sparta/bench/log.7Jul14.collide.icc.10K.1"
+RUN_DIR="$R/build/level3/sparta/$MODEL/$L3_RUN_SUBDIR"
+TIMEOUT="${HPCPERF_VALIDATE_TIMEOUT:-900}"
+[ -f "$REF" ] || { echo "validate.sh: reference log $REF missing (run fetch.sh)" >&2; exit 1; }
+
+export HPCPERF_GPUS="$N"
+echo "validate.sh: SPARTA $BACKEND smoke (bench/in.collide 10x10x10, 10,000 particles) on $N GPU(s)"
+run_once() { local ng=$1 out=$2 rc=0; HPCPERF_GPUS="$ng" HPCPERF_SCALE_MODE=smoke timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" > "$out" 2>&1 || rc=$?; return $rc; }
+mkdir -p "$RUN_DIR"
+VOUT="$RUN_DIR/validate.smoke.np$N.stdout"
+rc=0; run_once "$N" "$VOUT" || rc=$?
+grep -aE '^#|hpcperf-launch: audit summary|Loop time|ERROR|abort' "$VOUT" || true
+if [ "$rc" -eq 124 ]; then echo "validate.sh: FAIL -- run timed out after ${TIMEOUT}s"; exit 1; fi
+[ "$rc" -eq 0 ] || { echo "validate.sh: FAIL -- run.sh exited $rc (see $VOUT)"; exit 1; }
+LOG="$RUN_DIR/log.smoke.np$N.sparta"
+[ -f "$LOG" ] || { echo "validate.sh: FAIL -- no log produced ($LOG)"; exit 1; }
+if [ "$N" -gt 1 ] && [ ! -f "$RUN_DIR/log.smoke.np1.sparta" ]; then
+    echo "validate.sh: producing the 1-GPU run for rank-count comparison"
+    r1=0; run_once 1 "$RUN_DIR/validate.smoke.np1.stdout" || r1=$?
+    [ "$r1" -eq 0 ] && [ -f "$RUN_DIR/log.smoke.np1.sparta" ] || { echo "validate.sh: FAIL -- 1-GPU run failed (rc=$r1)"; exit 1; }
+fi
+
+python3 - "$LOG" "$REF" "$N" "$RUN_DIR/log.smoke.np1.sparta" <<'PY'
+import re, sys, os
+sys.path.insert(0, os.environ["L3_TOOLS"])
+from l3_check import require_finite, ValidationError
+def stats(path):
+    blocks, cur, cols = [], [], None
+    for ln in open(path).read().splitlines():
+        p = ln.split()
+        if p[:2] == ["Step", "CPU"]:
+            cols = ["temp" if c == "c_temp" else c for c in p]; cur = []; blocks.append(cur); continue
+        if cols and p and re.match(r'^\d+$', p[0]):
+            cur.append(dict(zip(cols, p))); continue
+        if cols and cur and not p: cols = None
+    return blocks[-1] if blocks else []
+EXPECT_FIRST, EXPECT_LAST = 30, 130     # benchmark block: run 30 (equilibrate) then run 100
+FIELDS = ["Np", "temp", "Natt"]
+def mean(rows, k, minstep=40):
+    v = [require_finite(f"{k}@{int(r['Step'])}", r[k]) for r in rows if int(r["Step"]) >= minstep]
+    if not v: raise ValidationError(f"no rows with Step>={minstep} for {k}")
+    return sum(v) / len(v)
+log, ref, n, log1 = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
+NPART = 10.0 * 10 * 10 * 10
+ok = True
+try:
+    got, want = stats(log), stats(ref)
+    for label, rows in (("run", got), ("reference", want)):
+        if not rows: raise ValidationError(f"{label} log has no benchmark stats block")
+        for f in FIELDS:
+            if f not in rows[0]: raise ValidationError(f"{label} log missing field '{f}' (have {list(rows[0])})")
+    steps = [int(r["Step"]) for r in got]
+    if steps[0] != EXPECT_FIRST or steps[-1] != EXPECT_LAST:
+        raise ValidationError(f"run stats block spans steps {steps[0]}..{steps[-1]}, expected {EXPECT_FIRST}..{EXPECT_LAST} (truncated/incomplete run)")
+    def check(rows, base, label):
+        global ok
+        bad_np = [int(r["Step"]) for r in rows if require_finite("Np", r["Np"]) != NPART]
+        print(f"  {label}: Np == {NPART:.0f} at every row: {'ok' if not bad_np else 'BAD at ' + str(bad_np)}"); ok &= not bad_np
+        for k, tol in (("temp", 0.02), ("Natt", 0.15)):
+            a, b = mean(rows, k), mean(base, k)
+            rel = abs(a - b) / abs(b)
+            print(f"  {label}: mean {k:<5} {a:12.4f} vs {b:12.4f} rel {rel:.3e} (tol {tol}) {'ok' if rel <= tol else 'BAD'}")
+            ok &= rel <= tol
+    print(f"[1] {n}-GPU run vs upstream reference log (icc, 1 proc, 2014):")
+    check(got, want, "vs-ref")
+    if n > 1:
+        one = stats(log1)
+        if not one: raise ValidationError("1-GPU log has no benchmark stats block")
+        print(f"[2] {n}-GPU run vs this build's 1-GPU run:")
+        check(got, one, "vs-1gpu")
+except ValidationError as ex:
+    print(f"  VALIDATION ERROR: {ex}")
+    print(f"SPARTA CUDA validation ({n} GPU): FAIL"); sys.exit(1)
+print(f"SPARTA CUDA validation ({n} GPU, bench/in.collide vs log.7Jul14.collide.icc.10K.1): {'PASS' if ok else 'FAIL'}")
+sys.exit(0 if ok else 1)
+PY
diff --git a/level3/specfem3d/README.md b/level3/specfem3d/README.md
new file mode 100644
index 0000000..0b8e125
--- /dev/null
+++ b/level3/specfem3d/README.md
@@ -0,0 +1,130 @@
+# SPECFEM3D Cartesian (Level 3)
+
+Spectral-element seismic wave propagation: the complete workflow -- mesh
+(CUBIT mesh + SCOTCH partitioner, or the in-house `xmeshfem3D`), per-slice
+database generation, and the GPU time loop of `xspecfem3D` (stiffness,
+MPI halo assembly, absorbing boundaries, seismogram output).
+
+## Provenance
+
+- Official repository: https://github.com/SPECFEM/specfem3d (moved from
+  geodynamics/specfem3d); docs https://specfem3d.readthedocs.io/
+- Release policy: tagged releases (v4.1.1 is the latest, 2024-03-15);
+  development happens on `devel`, ~2.5 years ahead of the release.
+- Selected: **v4.1.1**, commit `c67d3ae7d4bfc5ac75cb9e5601d93afa262d3d8d`,
+  fetched by `fetch.sh` into `_upstream/level3/specfem3d` (shallow, read-only;
+  the `m4`/`flexwin`/`pyCMT3D` submodules are not needed). Two source
+  back-ports come from `devel` `cc2e9ffa7e7cb5338e05f5a7df81cfbe60e00683`
+  (2026-07-24).
+- License: GPL-3.0.
+- Application-owned LOC (cloc 2.06, code lines): `src/` **142,398**
+  (Fortran 90 119,057; CUDA 12,210; `src/gpu` 15,058 in 63 `.cu` files).
+  `utils/` (163,847) and the bundled `external_libs/` (SCOTCH 5.1.12b,
+  METIS, PaToH; 137,463) are counted separately.
+
+## Build strategy: NATIVE (upstream autotools + bundled SCOTCH)
+
+`build.sh CUDA` copies the sources into `.deps/level3/specfem3d/src`
+(autotools builds in-tree), applies the two patches below and runs upstream's
+recipe: `./configure --with-mpi --with-cuda=cuda12 FC=/usr/bin/gfortran
+CC=<conda gcc 13.3> MPIFC=mpif90 MPI_INC=<conda mpi.h dir> CUDA_INC/CUDA_LIB
+USE_BUNDLED_SCOTCH=1`, then `make -j all GENCODE="-gencode=arch=compute_100,
+code=sm_100 -gencode=arch=compute_100,code=compute_100 -DGPU_DEVICE_Blackwell"`.
+Toolchain: conda GCC 13.3.0 for C and as nvcc's host compiler, system
+gfortran 14.2.1 for Fortran (the conda env has no gfortran), conda Open MPI
+5.0.10 with `OMPI_FC=/usr/bin/gfortran`. Build time on dgx003: **21 s** at
+`-j16` (428 objects; SCOTCH, 1,000+ Fortran units, 63 CUDA units), 7
+warning lines. 24 executables installed under `.deps/level3/specfem3d/install/bin`
+with the fingerprint (upstream commit, SCOTCH 5.1.12b, compilers, CUDA
+13.2.78, MPI, configure/make options, patch list).
+
+Why not the others: no Spack package exists for SPECFEM3D Cartesian (only
+`specfem3d-globe`); no Apptainer on the node and no upstream image; site
+modules broken. HIP: `build.sh HIP` carries `--with-hip` but v4.1.1 knows only
+MI8..MI250 (gfx803..gfx90a); devel added MI300/MI350. **Untested.**
+
+## Changes from upstream (all recorded in `patches/` and `build.sh`)
+
+| Class | Change | Size / provenance |
+|---|---|---|
+| D | `0001-cuda13-deviceOverlap-guard.patch`: `src/gpu/initialize_gpu.cu` reads `cudaDeviceProp.deviceOverlap`, removed in CUDA 13 -> guarded, `asyncEngineCount` printed instead (diagnostic text only) | 10 lines, back-port of upstream devel |
+| D | `0002-blackwell-device-block.patch`: `GPU_DEVICE_Blackwell` block in `src/gpu/mesh_constants_cuda.h` (`#undef USE_LAUNCH_BOUNDS`, identical to Hopper's) | 8 lines, back-port of upstream devel |
+| B | make-time `GENCODE` override = devel's `--with-cuda=cuda13` value (sm_100 SASS + compute_100 PTX); v4.1.1's `configure` stops at `cuda12` and cannot be regenerated here (no autoreconf, empty `m4/`) | no file edited |
+| B | bundled SCOTCH built without gzip support (generated `Makefile.inc`: `-DCOMMON_FILE_COMPRESS_GZ`/`-lz` removed; conda GCC has no `zlib.h`) | generated file only |
+| C | `OMPI_FC=/usr/bin/gfortran`, `MPI_INC` (configure's `mpif90 -showme:incdirs` detection returns nothing with the conda wrapper) | environment |
+
+No numerics, physics or algorithm changed; `flags.guess`'s gfortran flags
+(`-std=f2008 -pedantic-errors -ffpe-trap=invalid,zero,overflow`) are used as
+shipped and compile cleanly with gfortran 14.
+
+## Execution model
+
+One MPI rank per GPU (upstream: `device = myrank % device_count`). The common
+launcher's per-rank wrapper gives each rank one visible GPU (device 0), and
+audits the mapping (4/4 verified). `NPROC` in `Par_file` is the number of mesh
+slices and must equal the rank count -- it is fixed when the mesh is
+partitioned, so every rank count gets its own mesh + databases (SPECFEM3D's
+normal workflow; `run.sh` does all three stages through the launcher with the
+same N). Halo exchange in v4.1.1 is host-staged (no GPU-aware MPI). CPU
+binding: runtime default.
+
+## Inputs (`HPCPERF_SCALE_MODE`)
+
+| Mode | Mesh | Elements | Per rank @4 GPU | Topology | NSTEP / DT | GPU memory (est.) | Time loop on B200 | Validation quantity |
+|---|---|---|---|---|---|---|---|---|
+| smoke (default) | upstream `homogeneous_halfspace` CUBIT mesh, SCOTCH partition | 20,736 | 5,184 | SCOTCH (any N) | 5000 / 0.05 s | ~0.2 GB | 0.8-0.9 s | seismograms vs upstream `REF_SEIS` |
+| strong | `xmeshfem3D`, same 134x134x60 km domain refined G x (G=`HPCPERF_SPECFEM_STRONG`, 2): 72x72x32 | 165,888 | 41,472 | `NPROC_XI x NPROC_ETA` from `hpcperf_topology.py --dims 2` (NEX divisible) | 1000 / 0.025 s | ~1.7 GB | 0.975 s (1 GPU) / 0.412 s (4 GPU) | run completes; seismograms written |
+| weak | per-rank block (36F)x(36F)x(16F), F=`HPCPERF_SPECFEM_LOCAL` (2), domain extended PX x PY at fixed resolution | 165,888 x N | 165,888 | 2-D grid | 1000 / 0.025 s | ~1.7 GB | 1.035 s (4 GPU, 663,552 elements) | run completes |
+
+Memory estimate ~10 KB per element on the GPU (NGLL 5, single precision
+fields). The weak deck keeps element shape, DT and per-rank work identical for
+every N (the domain grows, source and stations stay in the first block).
+Preprocessing cost is CPU-side and per slice: `xgenerate_databases` for
+165,888 elements on one rank takes ~12 s; at G=4 (1.33M elements on one rank)
+its serial neighbour search exceeded 20 minutes and was abandoned, which is
+why the strong default is G=2.
+
+## Validation (`validate.sh`, upstream mechanism)
+
+Upstream ships reference seismograms for the homogeneous half-space
+(`EXAMPLES/applications/homogeneous_halfspace/REF_SEIS`, 12 traces: 4 stations
+x 3 components) and compares runs with them using
+`utils/scripts/compare_seismogram_correlations.py` (per trace: correlation
+coefficient, L2 misfit normalised by the reference energy, cross-correlation
+time shift; upstream thresholds corr >= 0.8, misfit <= 1 %, shift <= 0.01 s).
+The references are CPU/double-precision results; the GPU solver is single
+precision, so upstream's tolerance-based comparison is the appropriate
+criterion and is used unchanged. Observed on dgx003 (2026-09-04): **PASS at 1,
+2 and 4 GPUs** -- correlation 1.00000 on all 12 traces, worst misfit 2.7e-4 /
+2.3e-4 / 2.7e-4, worst time shift 2.5e-5 / 2.4e-5 / 2.7e-5 s.
+
+## Results on dgx003 (4x B200, CUDA 13.2.78, Slurm job 9552083)
+
+| Run | Ranks x GPUs | rank->GPU | CPU binding | Topology | Problem | Time loop (`output_solver.txt`) | Wall incl. mesh+databases | Validation |
+|---|---|---|---|---|---|---|---|---|
+| smoke | 1 x 1 | wrapper; audit 1/1 verified | runtime default | 1 slice | 20,736 el., 5000 steps | 0.860 s | 4 s | PASS |
+| smoke | 2 x 2 | wrapper; 2/2 verified | runtime default | SCOTCH 2 | same | 0.779 s | 6 s | PASS |
+| smoke | 4 x 4 | wrapper; 4/4 verified | runtime default | SCOTCH 4 | same | 0.806 s | 7 s | PASS |
+| strong | 1 x 1 | wrapper; 1/1 verified | runtime default | 1x1 | 165,888 el., 1000 steps | 0.975 s | 15 s | completes |
+| strong | 4 x 4 | wrapper; 4/4 verified | runtime default | 2x2 | 165,888 el. (41,472/rank) | 0.412 s | 17 s | completes |
+| weak | 4 x 4 | wrapper; 4/4 verified | runtime default | 2x2 | 663,552 el. (165,888/rank) | 1.035 s | 32 s | completes |
+
+Dry-runs (`HPCPERF_DRY_RUN=1`, hypothetical allocations) -- **DRY-RUN /
+UNVALIDATED**, nothing executed (each of the three stages prints its plan):
+
+| GPUs | Nodes x GPUs/node | Mode | Mesh | Elements | Per rank | NPROC_XI x NPROC_ETA | Launch |
+|---|---|---|---|---|---|---|---|
+| 8 | 1 x 8 | strong | 72x72x32 on 134x134x60 km | 165,888 | 20,736 | 4x2 | `mpirun -np 8 --host dgx003:8 --map-by ppr:8:node ...` (single node) |
+| 40 | 5 x 8 | weak | 576x360x32 on 1072x670x60 km | 6,635,520 | 165,888 | 8x5 | 5 nodes x 8 -- multi-node BLOCKED on this site |
+| 80 | 10 x 8 | weak | 720x576x32 on 1340x1072x60 km | 13,271,040 | 165,888 | 10x8 | 10 nodes x 8 -- multi-node BLOCKED on this site |
+
+## Limitations
+
+- Multi-node: BLOCKED/UNVERIFIED on this site; 40/80-GPU shapes are plans.
+- HIP: untested; v4.1.1 has no MI300/MI350 configure option.
+- v4.1.1 + CUDA 13.2 + sm_100 is not an upstream-validated combination; it
+  needs the two back-ports and the make-time GENCODE above (all upstream
+  devel content).
+- Only the homogeneous half-space family is wrapped; layered_halfspace,
+  Mount_StHelens, CPML and fault examples build with this configuration but
+  have no wrappers yet.
diff --git a/level3/specfem3d/build.sh b/level3/specfem3d/build.sh
new file mode 100755
index 0000000..ecee2e6
--- /dev/null
+++ b/level3/specfem3d/build.sh
@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+# Build SPECFEM3D Cartesian (meshfem3D, decompose_mesh, generate_databases,
+# specfem3D) with upstream's autotools build, CUDA-enabled, from a private
+# copy of the source tree.
+#
+#   ./build.sh [CUDA|HIP]        (default CUDA)
+#
+# Layout (Level 3 isolation): read-only clone _upstream/level3/specfem3d;
+# patched private source copy .deps/level3/specfem3d/src (autotools builds
+# in-tree: obj/ and bin/ live there); install .deps/level3/specfem3d/install/bin;
+# logs .deps/level3/specfem3d/logs. Only dependency besides MPI/CUDA is SCOTCH,
+# bundled (external_libs/scotch_5.1.12b) and built by the same make.
+#
+# Toolchain: CC = conda GCC 13.3.0 (also nvcc's host compiler, first `gcc` on
+# PATH as upstream's Makefile expects), FC = system gfortran 14.2.1 (the conda
+# env has no gfortran), MPIFC = conda Open MPI's mpif90 pointed at that
+# gfortran via OMPI_FC (class C). The conda MPI Fortran module loads under
+# gfortran 14 (.mod format 15; verified with a 2-rank MPI Fortran test).
+#
+# Modifications (all recorded in patches/, provenance upstream devel
+# cc2e9ffa, 2026-07-24):
+#   class D  0001-cuda13-deviceOverlap-guard.patch -- v4.1.1 reads
+#            cudaDeviceProp.deviceOverlap, a field removed in CUDA 13; upstream
+#            devel guards it (CUDA_VERSION < 13000) and prints asyncEngineCount
+#            instead. Diagnostic output only, no numerics. 10 lines.
+#   class D  0002-blackwell-device-block.patch -- upstream devel's
+#            GPU_DEVICE_Blackwell block (same content as Hopper's:
+#            #undef USE_LAUNCH_BOUNDS). 8 lines.
+#   class B  v4.1.1's configure knows --with-cuda=cuda4..cuda12 (cuda12 =
+#            sm_90 + GPU_DEVICE_Hopper); devel added cuda13 = sm_100 +
+#            GPU_DEVICE_Blackwell. Regenerating configure needs autoreconf and
+#            the m4 submodule (absent), so the same result is obtained by
+#            configuring with --with-cuda=cuda12 and overriding the GENCODE
+#            make variable on the command line with upstream devel's cuda13
+#            value. No upstream build file is edited.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+l3_isolate_build_env    # Level 3 builds must not see Level 2 .deps/install prefixes
+
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+UP="$R/_upstream/level3/specfem3d"
+[ -f "$UP/configure" ] || { echo "build.sh: $UP missing -- run $HERE/fetch.sh first" >&2; exit 1; }
+SHA="$(git -C "$UP" rev-parse HEAD)"
+l3_paths specfem3d
+JOBS="${HPCPERF_BUILD_JOBS:-32}"
+SYS_FC="${HPCPERF_SYSTEM_GFORTRAN:-/usr/bin/gfortran}"
+[ -x "$SYS_FC" ] || { echo "build.sh: no gfortran at $SYS_FC (set HPCPERF_SYSTEM_GFORTRAN); the conda env has none" >&2; exit 1; }
+export OMPI_FC="$SYS_FC"
+PATCHES=("$HERE/patches/0001-cuda13-deviceOverlap-guard.patch" "$HERE/patches/0002-blackwell-device-block.patch")
+
+case "$BACKEND" in
+    CUDA)
+        command -v nvcc >/dev/null || { echo "build.sh: nvcc not on PATH" >&2; exit 1; }
+        ARCH="${HPCPERF_CUDA_ARCH:-$(l3_gpu_arch)}"
+        CONF_GPU=(--with-cuda=cuda12)
+        # = upstream devel's cuda13 GENCODE (sm_100 SASS + compute_100 PTX), written as two -gencode flags so
+        # that no shell quoting travels through the make command line
+        GENCODE="-gencode=arch=compute_${ARCH},code=sm_${ARCH} -gencode=arch=compute_${ARCH},code=compute_${ARCH} -DGPU_DEVICE_Blackwell"
+        ARCHNOTE="sm_$ARCH" ;;
+    HIP)
+        command -v hipcc >/dev/null 2>&1 || { echo "build.sh: HIP requested but hipcc not found -- HIP build is UNTESTED on this machine (no ROCm)" >&2; exit 1; }
+        # v4.1.1 knows --with-hip=MI8..MI250 (gfx803..gfx90a) only; devel added MI300/MI350 (gfx942/gfx950)
+        CONF_GPU=(--with-hip=MI250); GENCODE=""; ARCHNOTE="gfx950 (UNTESTED; v4.1.1 has no MI350 option)" ;;
+    *) echo "usage: $0 [CUDA|HIP]" >&2; exit 2 ;;
+esac
+CMAKE_OPTS="configure: FC=$SYS_FC CC=$CC MPIFC=mpif90(OMPI_FC=$SYS_FC) --with-mpi ${CONF_GPU[*]} USE_BUNDLED_SCOTCH=1; make GENCODE=${GENCODE:-default}"
+PATCHNAMES=(); for p in "${PATCHES[@]}"; do PATCHNAMES+=("$(basename "$p")"); done
+FP="$(l3_fingerprint_text specfem3d "$SHA" "$MODEL" "scotch=5.1.12b (bundled)" "$CMAKE_OPTS" "no (host-staged halo exchange in v4.1.1)" "${PATCHNAMES[@]}")"
+l3_fingerprint_check "$L3_INSTALL" "$FP" || exit 1
+
+echo "# SPECFEM3D $BACKEND: upstream $SHA (v4.1.1), arch $ARCHNOTE, MPI $(mpirun --version 2>/dev/null | head -1), FC $($SYS_FC --version | head -1), CC $($CC --version | head -1)"
+rm -rf "$L3_SRC"; mkdir -p "$L3_SRC"
+# private source copy (configure needs the top-level DATA/ defaults, which point into EXAMPLES/); doc/ stays in the clone
+rsync -a --exclude .git --exclude doc "$UP/" "$L3_SRC/"
+for p in "${PATCHES[@]}"; do
+    (cd "$L3_SRC" && patch -p1 --forward --silent < "$p") || { echo "build.sh: patch $(basename "$p") failed to apply" >&2; exit 1; }
+    echo "# applied $(basename "$p")"
+done
+cd "$L3_SRC"
+CUDA_ROOT="${CUDA_HOME:-$(dirname "$(dirname "$(command -v nvcc)")")}"
+# mpi.h for the nvcc-compiled GPU sources (configure's auto-detection via `mpif90 -showme:incdirs` did not
+# reach the nvcc command line here); MPI_INC is the documented configure variable for this
+MPI_INC_DIR="$(mpicc -showme:incdirs 2>/dev/null | awk '{print $1}')"; [ -f "$MPI_INC_DIR/mpi.h" ] || MPI_INC_DIR="$(dirname "$(dirname "$(command -v mpicc)")")/include"
+[ -f "$MPI_INC_DIR/mpi.h" ] || { echo "build.sh: cannot locate mpi.h (looked in $MPI_INC_DIR)" >&2; exit 1; }
+FC="$SYS_FC" CC="$CC" MPIFC=mpif90 MPI_INC="$MPI_INC_DIR" CUDA_INC="$CUDA_ROOT/include" CUDA_LIB="$CUDA_ROOT/lib64" USE_BUNDLED_SCOTCH=1 \
+    ./configure --with-mpi "${CONF_GPU[@]}" > "$L3_LOGS/configure-$MODEL.log" 2>&1 \
+    || { tail -40 "$L3_LOGS/configure-$MODEL.log"; echo "build.sh: configure failed (log: $L3_LOGS/configure-$MODEL.log)" >&2; exit 1; }
+# bundled SCOTCH: the generated Makefile.inc enables gzip-compressed mesh files (-DCOMMON_FILE_COMPRESS_GZ,
+# -lz); the conda GCC has no zlib.h in its sysroot and the meshes here are uncompressed -> build SCOTCH
+# without that optional feature (class B, generated file only; upstream sources untouched)
+sed -i -e 's/ -DCOMMON_FILE_COMPRESS_GZ//' -e 's/ -lz\b//' external_libs/scotch/src/Makefile.inc
+t0=$(date +%s)
+if [ -n "$GENCODE" ]; then MAKEVARS=("GENCODE=$GENCODE"); else MAKEVARS=(); fi
+make -j "$JOBS" "${MAKEVARS[@]}" all > "$L3_LOGS/build-$MODEL.log" 2>&1 \
+    || { grep -n -i 'error' "$L3_LOGS/build-$MODEL.log" | head -20 || true; echo "build.sh: build failed (log: $L3_LOGS/build-$MODEL.log)" >&2; exit 1; }
+mkdir -p "$L3_INSTALL/bin"; cp -f bin/x* "$L3_INSTALL/bin/"
+l3_fingerprint_write "$L3_INSTALL" "$FP"
+echo "# built in $(( $(date +%s)-t0 )) s: $(ls "$L3_INSTALL/bin" | tr '\n' ' ')"
+echo "# compiler warning lines: $(grep -c -i 'warning' "$L3_LOGS/build-$MODEL.log" || true)"
diff --git a/level3/specfem3d/fetch.sh b/level3/specfem3d/fetch.sh
new file mode 100755
index 0000000..cd10663
--- /dev/null
+++ b/level3/specfem3d/fetch.sh
@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+# Fetch SPECFEM3D Cartesian at the latest release tag into
+# _upstream/level3/specfem3d (gitignored, read-only). Idempotent; a checkout
+# at another commit is an error, never silently reused.
+#
+#   ./fetch.sh
+#
+# The two source patches in patches/ are backports of upstream `devel` commits
+# (CUDA 13 `deviceOverlap` guard, Blackwell device block); their provenance is
+# the devel snapshot recorded in DEVEL_SNAPSHOT below (not checked out here).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+UPSTREAM_URL="https://github.com/SPECFEM/specfem3d.git"
+UPSTREAM_TAG="v4.1.1"
+UPSTREAM_SHA="c67d3ae7d4bfc5ac75cb9e5601d93afa262d3d8d"
+DEVEL_SNAPSHOT="cc2e9ffa7e7cb5338e05f5a7df81cfbe60e00683"   # devel @ 2026-07-24, source of the patches
+DEST="$R/_upstream/level3/specfem3d"
+
+if [ -d "$DEST/.git" ]; then
+    have="$(git -C "$DEST" rev-parse HEAD)"
+    if [ "$have" = "$UPSTREAM_SHA" ]; then echo "fetch.sh: $DEST already at $UPSTREAM_TAG ($UPSTREAM_SHA)"; exit 0; fi
+    echo "fetch.sh: $DEST is at $have, not the recorded $UPSTREAM_SHA ($UPSTREAM_TAG); remove it to re-fetch" >&2; exit 1
+fi
+mkdir -p "$(dirname "$DEST")"
+echo "fetch.sh: cloning $UPSTREAM_URL @ $UPSTREAM_TAG (shallow; submodules m4/flexwin/pyCMT3D not needed)"
+git clone --quiet --depth 1 --branch "$UPSTREAM_TAG" "$UPSTREAM_URL" "$DEST"
+have="$(git -C "$DEST" rev-parse HEAD)"
+[ "$have" = "$UPSTREAM_SHA" ] || { echo "fetch.sh: tag $UPSTREAM_TAG resolved to $have, expected $UPSTREAM_SHA" >&2; exit 1; }
+echo "fetch.sh: ok -> $DEST ($UPSTREAM_SHA); patches backported from devel $DEVEL_SNAPSHOT"
diff --git a/level3/specfem3d/patches/0001-cuda13-deviceOverlap-guard.patch b/level3/specfem3d/patches/0001-cuda13-deviceOverlap-guard.patch
new file mode 100644
index 0000000..ed5eecd
--- /dev/null
+++ b/level3/specfem3d/patches/0001-cuda13-deviceOverlap-guard.patch
@@ -0,0 +1,19 @@
+--- a/src/gpu/initialize_gpu.cu
++++ b/src/gpu/initialize_gpu.cu
+@@ -241,11 +241,16 @@
+       }else{
+         fprintf(fp,"  canMapHostMemory: FALSE\n");
+       }
++#if CUDA_VERSION < 13000 || (defined (__CUDACC_VER_MAJOR__) && (__CUDACC_VER_MAJOR__ < 13))
+       if (deviceProp.deviceOverlap){
+         fprintf(fp,"  deviceOverlap: TRUE\n");
+       }else{
+         fprintf(fp,"  deviceOverlap: FALSE\n");
+       }
++#else
++      // CUDA version >= 13, deviceOverlap deprecated, replaced by asyncEngineCount
++      fprintf(fp,"  asyncEngineCount: %d\n", deviceProp.asyncEngineCount);
++#endif
+       if (deviceProp.concurrentKernels){
+         fprintf(fp,"  concurrentKernels: TRUE\n");
+       }else{
diff --git a/level3/specfem3d/patches/0002-blackwell-device-block.patch b/level3/specfem3d/patches/0002-blackwell-device-block.patch
new file mode 100644
index 0000000..ef77cf4
--- /dev/null
+++ b/level3/specfem3d/patches/0002-blackwell-device-block.patch
@@ -0,0 +1,17 @@
+--- a/src/gpu/mesh_constants_cuda.h
++++ b/src/gpu/mesh_constants_cuda.h
+@@ -107,6 +107,14 @@
+ #undef USE_LAUNCH_BOUNDS
+ #endif
+ 
++#ifdef GPU_DEVICE_Blackwell
++// specifics see: https://docs.nvidia.com/cuda/blackwell-tuning-guide/index.html
++// register file size 64k 32-bit registers per SM
++// shared memory size 228KB per SM (for compute capability 10.0) or 128KB per SM (for compute capability 12.0)
++// maximum registers 255 per thread
++#undef USE_LAUNCH_BOUNDS
++#endif
++
+ /* ----------------------------------------------------------------------------------------------- */
+ 
+ // CUDA specifics
diff --git a/level3/specfem3d/run.sh b/level3/specfem3d/run.sh
new file mode 100755
index 0000000..f286d7a
--- /dev/null
+++ b/level3/specfem3d/run.sh
@@ -0,0 +1,141 @@
+#!/usr/bin/env bash
+# Run the full SPECFEM3D Cartesian workflow (mesh -> databases -> solver) for
+# the homogeneous-halfspace problem on N GPUs.
+#
+#   ./run.sh [CUDA|HIP]
+#
+# Workflow (upstream EXAMPLES/*/run_this_example.sh): the mesh is partitioned
+# into NPROC slices, `xgenerate_databases` (NPROC MPI ranks, CPU) builds the
+# per-slice databases, `xspecfem3D` (NPROC MPI ranks, GPU_MODE) runs the
+# spectral-element time loop. NPROC is fixed at mesh time, so every rank count
+# gets its own run directory and its own mesh/databases.
+#
+# Execution model: one MPI rank per GPU (upstream: device = myrank %
+# device_count; the common launcher's per-rank wrapper gives each rank one
+# visible GPU, so device 0 is that rank's GPU; audited). Halo exchange in
+# v4.1.1 is host-staged (no GPU-aware MPI requirement). All three stages are
+# launched through the common launcher with the same rank count.
+#
+# Resource / size controls:
+#   HPCPERF_GPUS=N|all        ranks = GPUs = NPROC (default 1)
+#   HPCPERF_SCALE_MODE        smoke | strong | weak  (default smoke)
+#     smoke  : upstream case as shipped: CUBIT mesh MESH-default (36x36x16 =
+#              20,736 HEX8 elements, 134x134x60 km, Vp 2.8 km/s), partitioned
+#              with the bundled SCOTCH by xdecompose_mesh (any NPROC), NSTEP
+#              5000, DT 0.05 s -- the case whose reference seismograms ship
+#     strong : in-house mesher xmeshfem3D on the SAME domain refined G times
+#              (G=HPCPERF_SPECFEM_STRONG, default 2): (36G)x(36G)x(16G) =
+#              165,888 elements, DT 0.05/G, NPROC = PX*PY from
+#              hpcperf_topology.py (2-D grid; NEX must be divisible by PX/PY).
+#              Larger G is legal but the serial per-slice neighbour search in
+#              xgenerate_databases dominates (G=4, 1.33M elements on one rank:
+#              > 20 min of CPU preprocessing before the GPU solver starts)
+#     weak   : fixed per-rank block (36F)x(36F)x(16F) elements (F =
+#              HPCPERF_SPECFEM_LOCAL, default 2: 165,888/rank), the DOMAIN is
+#              extended PX x PY times at fixed resolution (element shape, DT
+#              and per-step work per rank identical for every N); source and
+#              stations stay in the first block
+#   HPCPERF_SPECFEM_STEPS     NSTEP for strong/weak (default 1000; smoke keeps 5000)
+#
+# Derived inputs (class A): Par_file gets NPROC / GPU_MODE / NSTEP / DT (and
+# SAVE_MESH_FILES=.false. for strong/weak to skip VTK mesh dumps); the mesher's
+# Mesh_Par_file / interfaces.txt get the sizes above. Upstream files untouched.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+l3_paths specfem3d
+BIN="$L3_INSTALL/bin"
+for x in xdecompose_mesh xmeshfem3D xgenerate_databases xspecfem3D; do
+    [ -x "$BIN/$x" ] || { echo "run.sh: $BIN/$x missing -- run ./build.sh $BACKEND first" >&2; exit 1; }
+done
+EX="$R/_upstream/level3/specfem3d/EXAMPLES/applications/homogeneous_halfspace"
+[ -f "$EX/DATA/Par_file" ] || { echo "run.sh: $EX missing (run fetch.sh)" >&2; exit 1; }
+
+N_RANKS="$(hpcperf_ranks specfem3d yes)" || exit 2
+MODE="$(l3_scale_mode specfem3d)" || exit 2
+BUILD_DIR="$R/build/level3/specfem3d/$MODEL"
+# l3_rundir: dry-run gets a throwaway dir (the old code rm -rf'd the real run dir before the
+# per-stage dry-run checks, deleting real seismograms/databases when only a plan was requested).
+RUN_DIR="$(l3_rundir "$BUILD_DIR/$L3_RUN_SUBDIR/$MODE.np$N_RANKS")" || exit 2
+mkdir -p "$RUN_DIR/OUTPUT_FILES/DATABASES_MPI"
+cp -r "$EX/DATA" "$RUN_DIR/DATA"
+PAR="$RUN_DIR/DATA/Par_file"
+sed -i -e "s/^NPROC  *=.*/NPROC                           = $N_RANKS/" -e "s/^GPU_MODE  *=.*/GPU_MODE                        = .true./" "$PAR"
+
+case "$MODE" in
+    smoke)
+        NEX=36; NZ=16; STEPS=5000; DT=0.05; PX=1; PY=1; DESC="CUBIT mesh MESH-default, SCOTCH partition into $N_RANKS" ;;
+    strong)
+        G="${HPCPERF_SPECFEM_STRONG:-2}"; NEX=$((36 * G)); NZ=$((16 * G)); STEPS="${HPCPERF_SPECFEM_STEPS:-1000}"
+        DT="$(python3 -c "print(0.05/$G)")"
+        TOPO="$(hpcperf_topology specfem3d "$N_RANKS" --dims 2 --divides "$NEX,$NEX,0")" || exit 2
+        read -r PX PY _ <<< "$TOPO"; NEX_XI=$NEX; NEX_ETA=$NEX; LX=134000.0; LY=134000.0
+        DESC="xmeshfem3D ${NEX_XI}x${NEX_ETA}x${NZ} on 134x134x60 km, NPROC_XI x NPROC_ETA = ${PX}x${PY}" ;;
+    weak)
+        F="${HPCPERF_SPECFEM_LOCAL:-2}"; NZ=$((16 * F)); STEPS="${HPCPERF_SPECFEM_STEPS:-1000}"
+        DT="$(python3 -c "print(0.05/$F)")"
+        TOPO="$(hpcperf_topology specfem3d "$N_RANKS" --dims 2)" || exit 2
+        read -r PX PY _ <<< "$TOPO"; NEX_XI=$((36 * F * PX)); NEX_ETA=$((36 * F * PY))
+        LX="$(python3 -c "print(134000.0*$PX)")"; LY="$(python3 -c "print(134000.0*$PY)")"
+        DESC="xmeshfem3D ${NEX_XI}x${NEX_ETA}x${NZ} on $((134 * PX))x$((134 * PY))x60 km, NPROC_XI x NPROC_ETA = ${PX}x${PY}" ;;
+esac
+if [ "$MODE" != smoke ]; then
+    ELEMS=$((NEX_XI * NEX_ETA * NZ))
+    sed -i -e "s/^NSTEP  *=.*/NSTEP                           = $STEPS/" -e "s/^DT  *=.*/DT                              = $DT/" \
+           -e "s/^SAVE_MESH_FILES  *=.*/SAVE_MESH_FILES                 = .false./" "$PAR"
+    mkdir -p "$RUN_DIR/DATA/meshfem3D_files"
+    cp "$EX/meshfem3D_files/interface1.txt" "$RUN_DIR/DATA/meshfem3D_files/"
+    sed -e "s/^ 16\$/ $NZ/" "$EX/meshfem3D_files/interfaces.txt" > "$RUN_DIR/DATA/meshfem3D_files/interfaces.txt"
+    sed -e "s/^LATITUDE_MAX  *=.*/LATITUDE_MAX                    = $LY/" -e "s/^LONGITUDE_MAX  *=.*/LONGITUDE_MAX                   = $LX/" \
+        -e "s/^NEX_XI  *=.*/NEX_XI                          = $NEX_XI/" -e "s/^NEX_ETA  *=.*/NEX_ETA                         = $NEX_ETA/" \
+        -e "s/^NPROC_XI  *=.*/NPROC_XI                        = $PX/" -e "s/^NPROC_ETA  *=.*/NPROC_ETA                       = $PY/" \
+        -e "s/^CREATE_VTK_FILES  *=.*/CREATE_VTK_FILES                = .false./" \
+        -e "s/^1  *36  *1  *36  *1  *16  *1\$/1 $NEX_XI 1 $NEX_ETA 1 $NZ 1/" \
+        "$EX/meshfem3D_files/Mesh_Par_file" > "$RUN_DIR/DATA/meshfem3D_files/Mesh_Par_file"
+    grep -q "^1 $NEX_XI 1 $NEX_ETA 1 $NZ 1" "$RUN_DIR/DATA/meshfem3D_files/Mesh_Par_file" || { echo "run.sh: failed to rewrite the mesh region line" >&2; exit 1; }
+else
+    ELEMS=20736
+fi
+
+echo "# SPECFEM3D $BACKEND: mode=$MODE ranks=$N_RANKS elements=$ELEMS (~$((ELEMS / N_RANKS))/rank) NSTEP=$STEPS DT=$DT mesh: $DESC run_dir=$RUN_DIR"
+cd "$RUN_DIR"
+LAUNCH=("$L3_LAUNCHER" --gpus "$N_RANKS" --bind wrapper --)
+if [ "$MODE" = smoke ]; then
+    if [ -z "${HPCPERF_DRY_RUN:-}" ]; then
+        echo "# stage 1/3: xdecompose_mesh $N_RANKS (serial, CPU, SCOTCH)"
+        "$BIN/xdecompose_mesh" "$N_RANKS" "$EX/MESH-default" OUTPUT_FILES/DATABASES_MPI > OUTPUT_FILES/output_decompose_mesh.txt 2>&1 \
+            || { tail -20 OUTPUT_FILES/output_decompose_mesh.txt; echo "run.sh: xdecompose_mesh failed" >&2; exit 1; }
+    else
+        echo "# stage 1/3: xdecompose_mesh $N_RANKS (serial, CPU) -- skipped in dry-run"
+    fi
+else
+    echo "# stage 1/3: xmeshfem3D on $N_RANKS ranks (CPU)"
+    "${LAUNCH[@]}" "$BIN/xmeshfem3D" > OUTPUT_FILES/output_meshfem3D.log 2>&1 || { tail -30 OUTPUT_FILES/output_meshfem3D.log; echo "run.sh: xmeshfem3D failed" >&2; exit 1; }
+    grep -E 'hpcperf-launch: (actual|HYPOTHETICAL|launch|command|dry-run|NOTE)' OUTPUT_FILES/output_meshfem3D.log || true
+fi
+echo "# stage 2/3: xgenerate_databases on $N_RANKS ranks (CPU)"
+"${LAUNCH[@]}" "$BIN/xgenerate_databases" > OUTPUT_FILES/output_generate_databases.log 2>&1 || { tail -30 OUTPUT_FILES/output_generate_databases.log; echo "run.sh: xgenerate_databases failed" >&2; exit 1; }
+grep -E 'hpcperf-launch: (dry-run)' OUTPUT_FILES/output_generate_databases.log || true
+echo "# stage 3/3: xspecfem3D on $N_RANKS ranks (GPU_MODE)"
+t0=$(date +%s)
+# Capture the solver's real exit code directly (a `... | tee | grep || true` pipeline resets
+# PIPESTATUS via the trailing `true` and would report success even when the solver aborted).
+rc=0
+"${LAUNCH[@]}" "$BIN/xspecfem3D" > OUTPUT_FILES/output_specfem3D.log 2>&1 || rc=$?
+grep -E 'hpcperf-launch|Error|ERROR|GPU|Time loop|Elapsed|End of' OUTPUT_FILES/output_specfem3D.log || true
+[ "$rc" -eq 0 ] || { echo "run.sh: xspecfem3D exited $rc (see $RUN_DIR/OUTPUT_FILES/output_specfem3D.log)" >&2; exit "$rc"; }
+[ -n "${HPCPERF_DRY_RUN:-}" ] && exit 0
+WALL=$(( $(date +%s)-t0 ))
+echo "# solver wall time $WALL s; $(grep -E 'Total elapsed time in seconds|Time loop finished' OUTPUT_FILES/output_solver.txt 2>/dev/null | tr -s ' ' | tr '\n' ';')"
+echo "# seismograms: $(ls OUTPUT_FILES/*.semd 2>/dev/null | wc -l) files in $RUN_DIR/OUTPUT_FILES"
+l3_manifest "$RUN_DIR" "run_id=$(l3_run_id)" "app=specfem3d" "backend=$BACKEND" "mode=$MODE" \
+    "ranks=$N_RANKS" "elements=$ELEMS" "nstep=$STEPS" "dt=$DT" "solver_exit_code=$rc" \
+    "solver_wall_s=$WALL" "binary=$BIN/xspecfem3D" "binary_sha256=$(l3_sha_file "$BIN/xspecfem3D")" \
+    "par_file_sha256=$(l3_sha_file "$PAR")" "seismograms=$(ls OUTPUT_FILES/*.semd 2>/dev/null | wc -l)" "utc=$(date -u +%FT%TZ)"
diff --git a/level3/specfem3d/validate.sh b/level3/specfem3d/validate.sh
new file mode 100755
index 0000000..496f742
--- /dev/null
+++ b/level3/specfem3d/validate.sh
@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+# Correctness check for SPECFEM3D Cartesian on N GPUs, using upstream's own
+# reference seismograms and comparison tool.
+#
+#   ./validate.sh [CUDA|HIP]        HPCPERF_GPUS=N (default 1)
+#
+# Case: EXAMPLES/applications/homogeneous_halfspace as shipped (20,736 HEX8
+# elements, NSTEP 5000, DT 0.05 s), GPU_MODE, NPROC = N. Upstream's
+# utils/scripts/compare_seismogram_correlations.py reports per trace the
+# correlation, the L2 misfit (normalised by the reference energy) and the
+# cross-correlation time shift; thresholds corr>=0.8, err<=1%, shift<=0.01 s.
+# The GPU solver is single precision, so tolerance comparison (not bitwise) is
+# the right criterion and is used unchanged. PASS requires: run.sh exited 0,
+# EVERY reference trace was compared (count == number of REF_SEIS traces), all
+# produced seismograms are finite, and the comparison reports no poor
+# correlation / no poor match / no significant time shift.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+N="${HPCPERF_GPUS:-1}"
+UP="$R/_upstream/level3/specfem3d"
+REF="$UP/EXAMPLES/applications/homogeneous_halfspace/REF_SEIS"
+CMP="$UP/utils/scripts/compare_seismogram_correlations.py"
+RUN_DIR="$R/build/level3/specfem3d/$MODEL/$L3_RUN_SUBDIR/smoke.np$N"
+TIMEOUT="${HPCPERF_VALIDATE_TIMEOUT:-1800}"
+[ -d "$REF" ] && [ -f "$CMP" ] || { echo "validate.sh: $REF or $CMP missing (run fetch.sh)" >&2; exit 1; }
+NREF="$(ls "$REF"/*.semd 2>/dev/null | wc -l)"
+[ "$NREF" -ge 1 ] || { echo "validate.sh: no reference traces in $REF" >&2; exit 1; }
+
+export HPCPERF_GPUS="$N"
+echo "validate.sh: SPECFEM3D $BACKEND homogeneous_halfspace (20,736 elements, 5000 steps) on $N GPU(s); $NREF reference traces"
+VOUT="$R/build/level3/specfem3d/$MODEL/$L3_RUN_SUBDIR/validate.smoke.np$N.stdout"; mkdir -p "$(dirname "$VOUT")"
+rc=0; HPCPERF_SCALE_MODE=smoke timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" > "$VOUT" 2>&1 || rc=$?
+grep -aE '^#|hpcperf-launch: audit summary|Time loop|Elapsed time|End of the simulation|Error|ERROR' "$VOUT" || true
+if [ "$rc" -eq 124 ]; then echo "validate.sh: FAIL -- run timed out after ${TIMEOUT}s"; exit 1; fi
+[ "$rc" -eq 0 ] || { echo "validate.sh: FAIL -- run.sh exited $rc (see $VOUT)"; exit 1; }
+OUT="$RUN_DIR/OUTPUT_FILES"
+NGOT="$(ls "$OUT"/*.semd 2>/dev/null | wc -l)"
+[ "$NGOT" -ge "$NREF" ] || { echo "validate.sh: FAIL -- produced $NGOT seismograms, need >= $NREF"; exit 1; }
+
+# reject non-finite samples in the produced traces before trusting the correlation
+python3 - "$OUT" <<'PY' || { echo "validate.sh: FAIL -- non-finite sample in a produced seismogram"; exit 1; }
+import sys, os, glob, math
+sys.path.insert(0, os.environ["L3_TOOLS"])
+from l3_check import ValidationError
+try:
+    for f in sorted(glob.glob(sys.argv[1] + "/*.semd")):
+        nrow = 0
+        for ln in open(f):
+            parts = ln.split()
+            if len(parts) < 2: continue
+            for v in parts[:2]:
+                if not math.isfinite(float(v)): raise ValidationError(f"{os.path.basename(f)}: non-finite sample {v}")
+            nrow += 1
+        if nrow == 0: raise ValidationError(f"{os.path.basename(f)}: no samples")
+except ValidationError as ex:
+    print(f"  VALIDATION ERROR: {ex}"); sys.exit(1)
+PY
+
+echo "validate.sh: comparing with upstream REF_SEIS (utils/scripts/compare_seismogram_correlations.py)"
+CMP_OUT="$RUN_DIR/compare_ref_seis.log"
+python3 "$CMP" "$OUT/" "$REF/" > "$CMP_OUT" 2>&1 || true
+grep -E '^\|' "$CMP_OUT" | sed 's/^/  /'
+grep -E 'seismograms compared|poor correlation|poor match|significant time shift|no poor|no significant' "$CMP_OUT" | sed 's/^/  /'
+NCMP="$(grep -oE '^[0-9]+ seismograms compared' "$CMP_OUT" | awk '{print $1}')"
+ok=1
+[ "${NCMP:-0}" -eq "$NREF" ] || { echo "  only ${NCMP:-0} of $NREF reference traces were compared"; ok=0; }
+grep -q 'no poor correlations found' "$CMP_OUT" || ok=0
+grep -q 'no poor matches found' "$CMP_OUT" || ok=0
+grep -q 'no significant time shifts found' "$CMP_OUT" || ok=0
+if [ "$ok" -eq 1 ]; then
+    echo "SPECFEM3D $BACKEND validation ($N GPU, homogeneous_halfspace vs REF_SEIS, $NREF/$NREF traces, corr>=0.8 err<=1% shift<=0.01s): PASS"; exit 0
+fi
+echo "SPECFEM3D $BACKEND validation ($N GPU, homogeneous_halfspace vs REF_SEIS): FAIL (see $CMP_OUT)"; exit 1
diff --git a/level3/tools/l3_check.py b/level3/tools/l3_check.py
new file mode 100644
index 0000000..08cf234
--- /dev/null
+++ b/level3/tools/l3_check.py
@@ -0,0 +1,42 @@
+"""l3_check -- shared helpers for the Level 3 validators.
+
+Every numeric quantity that enters a pass/fail decision goes through require_finite,
+so a NaN/Inf (a diverged solve, an uninitialised field, a truncated reference) is an
+explicit FAIL with a named quantity -- never a silently-swallowed comparison. The
+validators import this via  sys.path.insert(0, os.environ["L3_TOOLS"]).
+"""
+import math
+import sys
+
+
+class ValidationError(Exception):
+    pass
+
+
+def require_finite(name, x):
+    """Return float(x) if finite; raise ValidationError naming the quantity otherwise."""
+    try:
+        v = float(x)
+    except (TypeError, ValueError):
+        raise ValidationError(f"{name} is not a number ({x!r})")
+    if not math.isfinite(v):
+        raise ValidationError(f"{name} is not finite ({v})")
+    return v
+
+
+def require_finite_seq(name, xs):
+    xs = list(xs)
+    if not xs:
+        raise ValidationError(f"{name}: empty sequence (no data)")
+    return [require_finite(f"{name}[{i}]", x) for i, x in enumerate(xs)]
+
+
+def rel_error(name, got, ref):
+    g = require_finite(f"{name}(got)", got)
+    r = require_finite(f"{name}(ref)", ref)
+    return abs(g - r) / max(abs(r), 1e-300)
+
+
+def fail(msg):
+    print(f"  VALIDATION ERROR: {msg}")
+    sys.exit(1)
diff --git a/level3/tools/l3_clean_env.sh b/level3/tools/l3_clean_env.sh
new file mode 100755
index 0000000..2d5b16c
--- /dev/null
+++ b/level3/tools/l3_clean_env.sh
@@ -0,0 +1,56 @@
+#!/bin/bash
+# l3_clean_env.sh -- run a command under an allow-listed environment.
+#
+#   l3_clean_env.sh [--list] [--show] [--] <cmd> [args...]
+#
+# Why: upstream tools record the process environment (CP2K's install_cp2k_toolchain.sh
+# writes `declare -x` of the whole environment into toolchain.env; nsys/ncu store it in
+# their reports; some build systems log it). A login shell carries credentials and
+# agent/session variables that must never end up in such files. This wrapper re-executes
+# the command with `env -i` and only the variables an HPC build/run needs.
+#
+# Allow-list (exact names or prefixes; nothing else survives):
+#   session/locale : PATH HOME USER LOGNAME SHELL TERM TMPDIR TZ LANG LANGUAGE LC_* XDG_RUNTIME_DIR
+#   scheduler      : SLURM_* SLURMD_NODENAME SLURM_TOPOLOGY_ADDR* (Slurm), PBS_* LSB_* (other sites)
+#   MPI/PMIx       : OMPI_* PRTE_* PMIX_* OPAL_* MPI_* HWLOC_* UCX_* NCCL_* FI_*
+#   GPU            : CUDA_HOME CUDA_PATH CUDA_VISIBLE_DEVICES CUDA_DEVICE_ORDER CUDA_CACHE_* NVCC_* CUDACXX CUDAHOSTCXX CUDAARCHS CUDA_LAUNCH_BLOCKING
+#                    ROCM_PATH ROCM_HOME HIP_* HSA_* HCC_* ROCR_*
+#   compilers/build: CC CXX FC F77 F90 F95 CPP LD AR RANLIB NM STRIP CFLAGS CXXFLAGS FFLAGS FCFLAGS LDFLAGS CPPFLAGS
+#                    CMAKE_* NINJA_* MAKEFLAGS MAKELEVEL CPATH C_INCLUDE_PATH CPLUS_INCLUDE_PATH LIBRARY_PATH LD_LIBRARY_PATH
+#                    LD_RUN_PATH PKG_CONFIG_PATH MANPATH INFOPATH
+#   conda/python   : CONDA_* _CONDA_* PYTHONPATH PYTHONNOUSERSITE PYTHONHOME VIRTUAL_ENV
+#   runtime knobs  : OMP_* KMP_* GOMP_* MKL_* OPENBLAS_* BLIS_* KOKKOS_* RAJA_* AMREX_* CP2K_DATA_DIR
+#   project        : HPCPERF_* HPC_PERFORMANCE_AI_ROOT _HPCPERF_* L3_*
+# Everything else (API keys, tokens, CLAUDE_*, ANTHROPIC_*, OPENAI_*, GITHUB_TOKEN, HUGGING_FACE*,
+# SSH_*, GPG_*, DBUS_*, ...) is dropped. A deny-list is applied AFTER the allow-list, so a
+# credential-looking name never survives through an allow-listed prefix (e.g. HPCPERF_*_TOKEN).
+# No value is ever printed by this script.
+set -euo pipefail
+DENY_REGEX='(^|_)(API_KEY|KEY|TOKEN|SECRET|SECRETS|PASSWORD|PASSWD|CREDENTIAL|CREDENTIALS)(_|$)|^(CLAUDE|ANTHROPIC|OPENAI|DEEPSEEK|HUGGING_FACE|HF|GITHUB|GH|AWS|AZURE|GOOGLE|SSH|GPG)(_|$)'
+ALLOW_EXACT="PATH HOME USER LOGNAME SHELL TERM TMPDIR TZ LANG LANGUAGE XDG_RUNTIME_DIR SLURMD_NODENAME \
+CUDA_HOME CUDA_PATH CUDA_VISIBLE_DEVICES CUDA_DEVICE_ORDER CUDACXX CUDAHOSTCXX CUDAARCHS CUDA_LAUNCH_BLOCKING ROCM_PATH ROCM_HOME \
+CC CXX FC F77 F90 F95 CPP LD AR RANLIB NM STRIP CFLAGS CXXFLAGS FFLAGS FCFLAGS LDFLAGS CPPFLAGS MAKEFLAGS MAKELEVEL \
+CPATH C_INCLUDE_PATH CPLUS_INCLUDE_PATH LIBRARY_PATH LD_LIBRARY_PATH LD_RUN_PATH PKG_CONFIG_PATH MANPATH INFOPATH \
+PYTHONPATH PYTHONNOUSERSITE PYTHONHOME VIRTUAL_ENV CP2K_DATA_DIR HPC_PERFORMANCE_AI_ROOT"
+ALLOW_PREFIX="LC_ SLURM_ PBS_ LSB_ OMPI_ PRTE_ PMIX_ OPAL_ MPI_ HWLOC_ UCX_ NCCL_ FI_ CUDA_CACHE_ NVCC_ HIP_ HSA_ HCC_ ROCR_ \
+CMAKE_ NINJA_ CONDA_ _CONDA_ OMP_ KMP_ GOMP_ MKL_ OPENBLAS_ BLIS_ KOKKOS_ RAJA_ AMREX_ HPCPERF_ _HPCPERF_ L3_"
+list=0; show=0
+while [ $# -gt 0 ]; do case "$1" in --list) list=1; shift;; --show) show=1; shift;; --) shift; break;; -*) echo "l3_clean_env.sh: unknown option $1" >&2; exit 2;; *) break;; esac; done
+if [ "$list" -eq 1 ]; then echo "exact: $ALLOW_EXACT"; echo "prefixes: $ALLOW_PREFIX"; exit 0; fi
+[ $# -gt 0 ] || { echo "usage: l3_clean_env.sh [--list] [--show] [--] <cmd> [args...]" >&2; exit 2; }
+keep=(); denied=()
+while IFS= read -r name; do
+    ok=0
+    for e in $ALLOW_EXACT; do [ "$name" = "$e" ] && { ok=1; break; }; done
+    if [ "$ok" -eq 0 ]; then for p in $ALLOW_PREFIX; do case "$name" in "$p"*) ok=1; break;; esac; done; fi
+    if [ "$ok" -eq 1 ] && [[ "$name" =~ $DENY_REGEX ]]; then ok=0; denied+=("$name"); fi
+    [ "$ok" -eq 1 ] && keep+=("$name")
+done < <(env | sed -n 's/^\([A-Za-z_][A-Za-z0-9_]*\)=.*/\1/p' | sort -u)
+args=()
+for name in "${keep[@]}"; do args+=("$name=${!name}"); done
+if [ "$show" -eq 1 ]; then
+    printf 'l3_clean_env: %d variables kept (names only): %s\n' "${#keep[@]}" "${keep[*]}" >&2
+    printf 'l3_clean_env: %d variables dropped' "$(( $(env | /usr/bin/grep -c '^[A-Za-z_][A-Za-z0-9_]*=') - ${#keep[@]} ))" >&2
+    [ "${#denied[@]}" -eq 0 ] && printf '\n' >&2 || printf ' (%d allow-listed names denied by the credential rule: %s)\n' "${#denied[@]}" "${denied[*]}" >&2
+fi
+exec env -i "${args[@]}" "$@"
diff --git a/level3/tools/l3_common.sh b/level3/tools/l3_common.sh
new file mode 100755
index 0000000..514cb2c
--- /dev/null
+++ b/level3/tools/l3_common.sh
@@ -0,0 +1,288 @@
+#!/bin/bash
+# l3_common.sh -- shared helpers for Level 3 application wrappers. Source it.
+#
+# Dependency isolation: every Level 3 application owns a private tree
+#     $R/.deps/level3/<app>/{src,build,install,logs}
+# (never a shared install root, so Kokkos/AMReX/MPI/hypre versions of
+# different applications cannot pollute each other), plus its upstream
+# checkout under $R/_upstream/level3/<Name> and its own build tree under
+# $R/build/level3/<app>/<backend>. Nothing here touches the Level 2 tree.
+#
+# Fingerprint: an install is stamped with .hpcperf-l3-fingerprint recording
+# application, upstream commit, dependency versions, compiler, CUDA/ROCm, GPU
+# arch, MPI, CMake options, GPU-aware-MPI option, patch hashes, site profile,
+# Spack lock hash and container image hash where applicable. A recorded
+# fingerprint that does not match the requested configuration FAILS FAST
+# (l3_fingerprint_check) -- stale installs are never reused silently.
+#
+# Runtime: launches go through the common launcher. Until the shared runtime
+# tools move to tools/runtime/ (proposal in tools/runtime/README.md) the
+# location is a single variable, HPCPERF_RUNTIME_DIR, defaulting to
+# level2/tools -- nothing is copied or moved, so Level 2 is not disturbed.
+
+# Some interactive environments export a `grep` shell FUNCTION (a ugrep wrapper with
+# -I/--ignore-files) into child processes; it changes grep's semantics (binary-file
+# handling, exit codes) inside these scripts. Level 3 scripts want the real grep.
+unset -f grep 2>/dev/null || true
+
+L3_R="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+L3_TOOLS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; export L3_TOOLS   # for the validators' python (l3_check.py)
+
+# L3_RUN_SUBDIR: name of the run-directory tree under each application build tree
+# ($R/build/level3/<app>/<profile>/<L3_RUN_SUBDIR>/<case>.<mode>.np<N>). Default "run".
+# HPCPERF_L3_RUN_SUBDIR=run.regress-<sha> sends a regression campaign into a fresh
+# sibling tree so that historical results are never overwritten (a plain name, no
+# path separators). run.sh and validate.sh of every application use $L3_RUN_SUBDIR.
+L3_RUN_SUBDIR="${HPCPERF_L3_RUN_SUBDIR:-run}"
+case "$L3_RUN_SUBDIR" in ""|*/*|.|..|.*) echo "l3_common: invalid HPCPERF_L3_RUN_SUBDIR '$L3_RUN_SUBDIR' (plain directory name expected)" >&2; return 2 2>/dev/null || exit 2;; esac
+export L3_RUN_SUBDIR
+HPCPERF_RUNTIME_DIR="${HPCPERF_RUNTIME_DIR:-$L3_R/level2/tools}"
+L3_LAUNCHER="$HPCPERF_RUNTIME_DIR/hpcperf_mpi_launch.sh"
+L3_TOPOLOGY="$HPCPERF_RUNTIME_DIR/hpcperf_topology.py"
+# shellcheck disable=SC1091
+source "$HPCPERF_RUNTIME_DIR/hpcperf_launch_common.sh"
+
+# l3_paths <app>: exports L3_APP, L3_UPSTREAM, L3_DEPS, L3_SRC, L3_BUILD_DEPS, L3_INSTALL, L3_LOGS
+l3_paths() {
+    L3_APP="$1"
+    L3_DEPS="$L3_R/.deps/level3/$L3_APP"
+    L3_SRC="$L3_DEPS/src"; L3_BUILD_DEPS="$L3_DEPS/build"; L3_INSTALL="$L3_DEPS/install"; L3_LOGS="$L3_DEPS/logs"
+    mkdir -p "$L3_SRC" "$L3_BUILD_DEPS" "$L3_INSTALL" "$L3_LOGS"
+}
+
+# l3_paths_profile <app> <profile>: second-batch layout -- one private tree per
+# *configuration profile* (compiler/Toolkit/backend/key-dependency variant):
+#     $R/.deps/level3/<app>/<profile>/{src,build,install,logs,cache}
+# exports L3_APP, L3_PROFILE, L3_DEPS, L3_SRC, L3_BUILD_DEPS, L3_INSTALL, L3_LOGS,
+# L3_CACHE and L3_BUILD (= $R/build/level3/<app>/<profile>, the application build
+# tree). Different profiles never share a mutable source tree or an install.
+l3_paths_profile() {
+    L3_APP="$1"; L3_PROFILE="$2"
+    case "$L3_PROFILE" in ""|*/*|.*) echo "l3_paths_profile: invalid profile name '$L3_PROFILE'" >&2; return 2;; esac
+    L3_DEPS="$L3_R/.deps/level3/$L3_APP/$L3_PROFILE"
+    L3_SRC="$L3_DEPS/src"; L3_BUILD_DEPS="$L3_DEPS/build"; L3_INSTALL="$L3_DEPS/install"; L3_LOGS="$L3_DEPS/logs"; L3_CACHE="$L3_DEPS/cache"
+    L3_BUILD="$L3_R/build/level3/$L3_APP/$L3_PROFILE"
+    mkdir -p "$L3_SRC" "$L3_BUILD_DEPS" "$L3_INSTALL" "$L3_LOGS" "$L3_CACHE"
+}
+
+# l3_clean_conda_build_env: the project conda env exports its own compiler-driving
+# variables (CFLAGS/CXXFLAGS/LDFLAGS with -march=nocona -mtune=haswell and conda
+# -isystem/-rpath paths, AR/RANLIB/NM/LD = conda binutils, CMAKE_ARGS, ...). They are
+# right for the conda GCC, wrong for a build that deliberately uses the system GCC
+# 14 toolchain (they made OpenBLAS fail with "target specific option mismatch" on
+# its AVX512 kernels and would pin -march=nocona on everything). Call after
+# hpcperf_env.sh in build scripts that select the system compilers; PATH, CUDA and
+# the MPI wrappers are left alone.
+l3_clean_conda_build_env() {
+    unset CFLAGS CXXFLAGS FFLAGS FCFLAGS FORTRANFLAGS CPPFLAGS LDFLAGS \
+          DEBUG_CFLAGS DEBUG_CXXFLAGS DEBUG_CPPFLAGS DEBUG_FFLAGS DEBUG_FORTRANFLAGS \
+          AR RANLIB NM LD STRIP AS CPP OBJCOPY OBJDUMP READELF SIZE STRINGS ADDR2LINE ELFEDIT GPROF CXXFILT LD_GOLD \
+          HOST BUILD CMAKE_ARGS MESON_ARGS GCC_AR GCC_NM GCC_RANLIB GXX GCC GFORTRAN F77 F90 F95 \
+          CONDA_BUILD_SYSROOT CC_FOR_BUILD CXX_FOR_BUILD \
+          C_INCLUDE_PATH CPLUS_INCLUDE_PATH CPATH LIBRARY_PATH 2>/dev/null || true   # login-shell leftovers pointing at foreign conda envs
+    echo "# l3: conda build variables (CFLAGS/LDFLAGS/AR/... ) cleared for a system-toolchain build"
+}
+
+# l3_version_mm <version string>: "13.2.78" -> "132", "13.3.0" -> "133" (profile-name component)
+l3_version_mm() { echo "$1" | awk -F. '{printf "%s%s", $1, $2}'; }
+
+# l3_clean_env_exec [--] <cmd...>
+#   Runs a command under an environment reduced to an explicit allow-list (scheduler,
+#   MPI, CUDA, compilers/build flags, conda/tool paths, HPCPERF_*, OpenMP/BLAS/UCX
+#   knobs, locale). Everything else from the login shell -- in particular API keys,
+#   tokens, agent/session variables -- is NOT passed on. Use it around every tool
+#   that dumps or records its process environment (upstream toolchain installers
+#   that write `declare -x` files, profilers such as nsys/ncu, `env`-recording
+#   build systems). The allow-list is printed by `l3_clean_env_exec --list`.
+#   Implementation: level3/tools/l3_clean_env.sh (also usable stand-alone).
+l3_clean_env_exec() { "$L3_TOOLS/l3_clean_env.sh" "$@"; }
+
+# l3_run_recorded <rc-file> <label> [--] <cmd...>
+#   Queue step: runs the command, appends "<label> <exit code>" to <rc-file> and returns 0, so a
+#   validation queue (even under `set -e`) records every step's real exit code and continues.
+#   Exit code 3 (Nyx: STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING) and 4 (UNSUPPORTED_LAYOUT) are
+#   recorded like any other and classified later by level3/tools/l3_verdict.py -- never as PASS,
+#   never as a reason to stop the queue.
+l3_run_recorded() {
+    local f=$1 label=$2; shift 2; [ "${1:-}" = "--" ] && shift
+    local rc=0; "$@" || rc=$?
+    echo "$label $rc" >> "$f"
+    return 0
+}
+
+# l3_isolate_build_env: remove the Level 2 dependency prefixes (everything under
+# $L3_R/.deps/install/, the validated Level 2 tree) from CMAKE_PREFIX_PATH and
+# LD_LIBRARY_PATH before a Level 3 configure, so a Level 3 build can never pick
+# up a Level 2 Kokkos/RAJA/hypre. Each Level 3 app owns its dependencies
+# (bundled, or under .deps/level3/<app>). Call once in build.sh after sourcing
+# hpcperf_env.sh. Does not touch the launcher (level2/tools, not .deps/install).
+l3_isolate_build_env() {
+    local before_c="${CMAKE_PREFIX_PATH:-}" before_l="${LD_LIBRARY_PATH:-}"
+    [ -n "$before_c" ] && export CMAKE_PREFIX_PATH="$(tr ':' '\n' <<<"$before_c" | grep -v "$L3_R/.deps/install/" | paste -sd:)"
+    [ -n "$before_l" ] && export LD_LIBRARY_PATH="$(tr ':' '\n' <<<"$before_l" | grep -v "$L3_R/.deps/install/" | paste -sd:)"
+    local n; n="$(tr ':' '\n' <<<"${CMAKE_PREFIX_PATH:-}" | grep -c "$L3_R/.deps/install/" || true)"
+    echo "# l3: build env isolated from Level 2 prefixes (CMAKE_PREFIX_PATH .deps/install entries remaining: $n)"
+}
+
+l3_first_line() { "$@" 2>/dev/null | head -n 1 || true; }
+l3_cuda_version() { nvcc --version 2>/dev/null | sed -n 's/^Cuda compilation tools, release [^,]*, V\([0-9][0-9.]*\).*$/\1/p' | head -n 1; }
+l3_gpu_arch() { # numeric compute capability of GPU 0, e.g. 100
+    nvidia-smi --query-gpu=compute_cap --format=csv,noheader 2>/dev/null | head -1 | tr -d ' .'
+}
+l3_site_profile() {
+    if [ -n "${HPCPERF_SITE_PROFILE:-}" ]; then echo "$HPCPERF_SITE_PROFILE"; return; fi
+    case "$(hostname -s)" in dgx003|hopper*|gpu0*) echo gmu-hopper ;; *) echo generic ;; esac
+}
+
+# l3_fingerprint_text <app> <upstream_sha> <backend> "<deps versions>" "<cmake options>" <gpu_aware_mpi> [patch files...]
+#   Prints the fingerprint for the configuration about to be built. Patches are
+#   recorded IN THE ORDER GIVEN with their content sha256 (the source-cache key
+#   depends on both the upstream SHA and this ordered patch-content hash, so a
+#   same-named patch whose bytes change invalidates the cache). A patch path
+#   that does not exist, or whose hash cannot be taken, is a hard error -- the
+#   fingerprint is never written with a silently-missing patch.
+l3_fingerprint_text() {
+    local app=$1 sha=$2 backend=$3 deps=$4 cmakeopts=$5 gam=$6; shift 6
+    local p h
+    echo "schema=l3-2"
+    echo "application=$app"
+    echo "upstream_commit=$sha"
+    echo "backend=$backend arch=sm_$(l3_gpu_arch)"
+    echo "dependencies=$deps"
+    echo "compiler=${CXX:-c++} ($(l3_first_line "${CXX:-c++}" --version))"
+    echo "fortran=${FC:-gfortran} ($(l3_first_line "${FC:-gfortran}" --version))"
+    echo "cuda=$(l3_cuda_version)"
+    echo "rocm=${ROCM_VERSION:-none}"
+    echo "mpi=$(l3_first_line mpirun --version)"
+    echo "cmake_options=$cmakeopts"
+    echo "gpu_aware_mpi=$gam"
+    echo "site_profile=$(l3_site_profile)"
+    echo "spack_lock_sha256=${L3_SPACK_LOCK_SHA:-none}"
+    echo "container_image_sha256=${L3_CONTAINER_SHA:-none}"
+    local idx=0 ordered=""
+    for p in "$@"; do
+        idx=$((idx+1))
+        if [ ! -f "$p" ]; then
+            echo "l3: patch file '$p' not found -- refusing to fingerprint a build with a missing patch" >&2
+            return 1
+        fi
+        h="$(sha256sum "$p" 2>/dev/null | cut -d' ' -f1)"
+        [ -n "$h" ] || { echo "l3: could not hash patch '$p'" >&2; return 1; }
+        echo "patch[$idx]=$(basename "$p") sha256=$h"
+        ordered="$ordered$h"
+    done
+    # ordered content hash of the whole patch series (empty series -> the literal 'none')
+    echo "patch_series_sha256=$( [ -n "$ordered" ] && printf '%s' "$ordered" | sha256sum | cut -d' ' -f1 || echo none )"
+}
+
+# l3_fingerprint_check <install_dir> <expected_text>
+#   0 = no fingerprint yet (fresh build) or identical; 1 = mismatch (prints diff, caller must fail).
+l3_fingerprint_check() {
+    local dir=$1 expected=$2 fp="$1/.hpcperf-l3-fingerprint" diffout
+    [ -f "$fp" ] || return 0
+    diffout="$(diff <(grep -v '^built=' "$fp") <(printf '%s\n' "$expected") || true)"
+    if [ -n "$diffout" ]; then
+        echo "l3: fingerprint mismatch for $dir (recorded < vs requested >):" >&2
+        echo "$diffout" >&2
+        echo "l3: refusing to reuse a differently-configured install; remove $dir or change the request" >&2
+        return 1
+    fi
+    return 0
+}
+
+# l3_fingerprint_write <install_dir> <text>: written only after a successful build+install.
+l3_fingerprint_write() {
+    local dir=$1 text=$2
+    { printf '%s\n' "$text"; echo "built=$(date -u +%Y-%m-%dT%H:%MZ) (build-time record)"; } > "$dir/.hpcperf-l3-fingerprint.tmp" \
+        && mv -f "$dir/.hpcperf-l3-fingerprint.tmp" "$dir/.hpcperf-l3-fingerprint"
+}
+
+# l3_scale_mode <app>: validated HPCPERF_SCALE_MODE (smoke|strong|weak; default smoke)
+l3_scale_mode() {
+    local m="${HPCPERF_SCALE_MODE:-smoke}"
+    case "$m" in smoke|strong|weak) echo "$m";; *) echo "$1/run.sh: HPCPERF_SCALE_MODE must be smoke|strong|weak (got '$m')" >&2; return 2;; esac
+}
+
+# ---------------------------------------------------------------------------
+# Result management (correctness / reproducibility)
+# ---------------------------------------------------------------------------
+
+# l3_run_id: a unique id for one real execution (UTC, pid, random).
+l3_run_id() { echo "$(date -u +%Y%m%dT%H%M%SZ)-$$-${RANDOM}"; }
+
+# l3_rundir <intended-real-path>
+#   Echoes the directory run.sh should actually write into, and prepares it.
+#   In a real run: rm -rf the intended dir and recreate it (fresh output only).
+#   In a dry-run (HPCPERF_DRY_RUN set): NEVER touch the real dir -- a throwaway
+#   sibling under .dryrun/ is used, so planning can never delete or overwrite a
+#   real result. Refuses to operate on a path that is not under a Level 3
+#   build/ tree (guards against an accidental rm of the wrong directory).
+l3_rundir() {
+    local real=$1 base parent
+    case "$real" in
+        "$L3_R"/build/level3/*) : ;;
+        *) echo "l3_rundir: refusing to manage '$real' (not under $L3_R/build/level3/)" >&2; return 2;;
+    esac
+    if [ -n "${HPCPERF_DRY_RUN:-}" ]; then
+        parent="$(dirname "$real")"; base="$(basename "$real")"
+        real="$parent/.dryrun/$base"
+        rm -rf "$real"; mkdir -p "$real"
+    else
+        rm -rf "$real"; mkdir -p "$real"
+    fi
+    printf '%s\n' "$real"
+}
+
+# l3_capture <logfile> -- <cmd...>
+#   Runs the command, copies combined stdout+stderr to <logfile>, and returns
+#   the command's real exit status (NOT tee's). Nothing is swallowed; callers
+#   check the status. Use this instead of `cmd | grep ... || true`.
+l3_capture() {
+    local log=$1; shift
+    [ "${1:-}" = -- ] && shift
+    mkdir -p "$(dirname "$log")"
+    set -o pipefail
+    "$@" 2>&1 | tee "$log"
+    local rc=${PIPESTATUS[0]}
+    set +o pipefail
+    return "$rc"
+}
+
+# l3_manifest <run_dir> key=value ...
+#   Appends structured provenance for one real run. Records the run_id once.
+l3_manifest() {
+    local dir=$1; shift
+    local f="$dir/run_manifest.txt"
+    { for kv in "$@"; do echo "$kv"; done; } >> "$f"
+}
+
+# l3_sha_file <path>: sha256 of a file, or the literal MISSING.
+l3_sha_file() { [ -f "$1" ] && sha256sum "$1" 2>/dev/null | cut -d' ' -f1 || echo MISSING; }
+
+# l3_binary_backend_check <exe> <expected: cuda|hip>
+#   Fails if the binary's GPU backend does not match what was requested (so a
+#   HIP request can never run a CUDA install and vice versa). Evidence: the
+#   linked runtime libraries (libcudart / libamdhip64); a binary that links the
+#   CUDA runtime statically (CMake's default CUDA_RUNTIME_LIBRARY=Static, e.g.
+#   AMReX-based apps) is accepted when cuobjdump finds embedded device code.
+l3_binary_backend_check() {
+    local exe=$1 want=$2 libs
+    [ -x "$exe" ] || { echo "l3: $exe not executable" >&2; return 1; }
+    libs="$(ldd "$exe" 2>/dev/null || true)"
+    case "$want" in
+        cuda) if ! grep -q 'libcudart' <<<"$libs"; then
+                  # capture first, then grep: under the callers' `set -o pipefail` a `cuobjdump | grep -q`
+                  # pipe reports cuobjdump's SIGPIPE (grep -q exits early) as a failure on large binaries
+                  local elf=""; command -v cuobjdump >/dev/null 2>&1 && elf="$(cuobjdump --list-elf "$exe" 2>/dev/null || true)"
+                  if grep -q 'sm_' <<<"$elf"; then
+                      : # static cudart with embedded CUDA device code
+                  else
+                      echo "l3: $exe is not a CUDA binary (no libcudart linked, no embedded CUDA ELF) but CUDA was requested" >&2; return 1
+                  fi
+              fi
+              grep -q 'libamdhip64' <<<"$libs" && { echo "l3: $exe links libamdhip64 (HIP) but CUDA was requested" >&2; return 1; } ;;
+        hip)  grep -q 'libamdhip64' <<<"$libs" || { echo "l3: $exe is not a HIP binary (no libamdhip64 linked) but HIP was requested" >&2; return 1; } ;;
+        *) return 0 ;;
+    esac
+    return 0
+}
diff --git a/level3/tools/l3_verdict.py b/level3/tools/l3_verdict.py
new file mode 100755
index 0000000..e40e36f
--- /dev/null
+++ b/level3/tools/l3_verdict.py
@@ -0,0 +1,113 @@
+#!/usr/bin/env python3
+"""l3_verdict -- verdict classes of a Level 3 validate.sh outcome, and a regression-campaign summary.
+
+Exit-code contract of level3/<app>/validate.sh (documented in each script):
+    0   PASS               -- the final verdict line ends with 'PASS' (e.g. '...): PASS')
+    1   FAIL
+    3   PENDING            -- Nyx heat/cool: 'STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING' (state and particle
+                              checks pass, the full acceptance is incomplete). NOT a PASS.
+    4   UNSUPPORTED_LAYOUT -- Nyx: different but legal box layouts, comparison not performed. NOT a PASS.
+    124 timeout -> FAIL; anything else -> FAIL
+A queue records the exit code of every step (l3_run_recorded in l3_common.sh) and continues; this module
+turns (rc, log) into exactly one of PASS / PENDING / UNSUPPORTED_LAYOUT / FAIL / MISSING. PENDING and
+UNSUPPORTED_LAYOUT are counted separately, never as PASS, and never enter a performance summary. An exit
+code that contradicts the log (rc 0 without a PASS line, rc 3 with a PASS line, a PASS line after a
+timeout) is FAIL.
+
+    l3_verdict.py classify --rc N --log FILE
+    l3_verdict.py summary --rc-file FILE --log FILE...      rc file: '<log basename> <rc>' per line
+"""
+import argparse
+import os
+import re
+import sys
+
+CLASSES = ("PASS", "PENDING", "UNSUPPORTED_LAYOUT", "FAIL", "MISSING")
+NOISE = re.compile(r"lua|posix|traceback|\[C\]|no file|no field")  # lmod noise on this site
+VERDICT_RE = re.compile(r"validation \(.*\): |I_R_CHECK_PENDING|UNSUPPORTED_LAYOUT|: FAIL\b|\bPASS$")
+AUDIT_RE = re.compile(r"audit summary: (\d+) verified, (\d+) mismatch, (\d+) unverified")
+
+
+def last_verdict_line(text):
+    lines = [l for l in text.splitlines() if l.strip() and not NOISE.search(l) and VERDICT_RE.search(l)]
+    return lines[-1] if lines else ""
+
+
+def is_pass_line(line):
+    line = line.rstrip()
+    return bool(re.search(r"\bPASS$", line)) and not re.search(r"PENDING|UNSUPPORTED|FAIL|not a ", line)
+
+
+def classify(rc, verdict_line):
+    """rc: int or None (log/step missing). verdict_line: the last verdict line of the log ('' if none)."""
+    if rc is None:
+        return "MISSING"
+    line = (verdict_line or "").rstrip()
+    if rc == 0:
+        return "PASS" if is_pass_line(line) else "FAIL"
+    if rc == 3:
+        return "PENDING" if "I_R_CHECK_PENDING" in line and not is_pass_line(line) else "FAIL"
+    if rc == 4:
+        return "UNSUPPORTED_LAYOUT" if "UNSUPPORTED_LAYOUT" in line and not is_pass_line(line) else "FAIL"
+    return "FAIL"
+
+
+def read_log(path):
+    if not path or not os.path.isfile(path):
+        return None
+    return open(path, errors="replace").read()
+
+
+def cmd_classify(a):
+    text = read_log(a.log)
+    if text is None:
+        print("MISSING")
+        return 0
+    print(classify(a.rc, last_verdict_line(text)))
+    return 0
+
+
+def cmd_summary(a):
+    rcs = {}
+    if a.rc_file:
+        for l in open(a.rc_file):
+            f = l.split()
+            if len(f) >= 2 and f[-1].lstrip("-").isdigit():
+                rcs[f[0]] = int(f[-1])
+    rows = []
+    for path in a.log:
+        name = os.path.basename(path)
+        text = read_log(path)
+        rc = rcs.get(name)
+        if text is None:
+            rows.append((name, rc, "MISSING", "(log missing)", []))
+            continue
+        line = last_verdict_line(text)
+        cls = classify(rc, line) if rc is not None else "MISSING"
+        rows.append((name, rc, cls, line.strip() or "(no verdict line)", AUDIT_RE.findall(text)))
+    print("| log | validate.sh rc | class | launcher audits verified/mismatch/unverified (one entry per launched run) | verdict line |")
+    print("|---|---|---|---|---|")
+    for name, rc, cls, line, au in rows:
+        print(f"| {name} | {'?' if rc is None else rc} | **{cls}** | {'; '.join('/'.join(x) for x in au) or 'none found'} | {line[:160]} |")
+    counts = {c: sum(1 for r in rows if r[2] == c) for c in CLASSES}
+    print()
+    print(f"{len(rows)} logs; " + ", ".join(f"{c}={counts[c]}" for c in CLASSES)
+          + "  (PASS counts only class PASS; PENDING and UNSUPPORTED_LAYOUT are not passes and are excluded from any performance summary)")
+    unv = [f"{name} [{'; '.join('/'.join(x) for x in au)}]" for name, _, _, _, au in rows if any(x[2] != "0" for x in au)]
+    mis = [name for name, _, _, _, au in rows if any(x[1] != "0" for x in au)]
+    print("Launcher audits with unverified ranks (GPU-binding evidence gap, not a correctness signal): " + (", ".join(unv) or "none"))
+    print("Launcher audits with MISMATCH (wrong GPU observed): " + (", ".join(mis) or "none"))
+    return 0
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    c = sub.add_parser("classify"); c.add_argument("--rc", type=int, required=True); c.add_argument("--log", required=True)
+    s = sub.add_parser("summary"); s.add_argument("--rc-file"); s.add_argument("--log", nargs="+", required=True)
+    a = ap.parse_args()
+    return cmd_classify(a) if a.cmd == "classify" else cmd_summary(a)
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/level3/tools/tests/run_all.sh b/level3/tools/tests/run_all.sh
new file mode 100755
index 0000000..398f06d
--- /dev/null
+++ b/level3/tools/tests/run_all.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+# Level 3 CPU-only regression tests (no GPU, no application, no build):
+#   test_l3_infra.sh -- correctness/reproducibility helpers (l3_common.sh, l3_check.py):
+#                       NaN/Inf rejection, real-exit-code capture, the failed-run
+#                       gate that prevents a stale-log false PASS, the dry-run
+#                       result-directory sentinel, and patch fingerprint / cache
+#                       invalidation.
+# Usage: level3/tools/tests/run_all.sh
+set -u
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+rc=0
+run() { echo "=== $1"; shift; bash "$@" || { echo "=== FAILED: $*"; rc=1; }; echo; }
+run "l3 infra (correctness/reproducibility)" "$HERE/test_l3_infra.sh"
+run "second-batch checkers (negative/positive)" "$HERE/test_l3_validators.sh"
+run "Nyx strict comparator + validate.sh chain" "$HERE/test_nyx_validator.sh"
+run "verdict classes / regression summary (rc 3 stays PENDING)" "$HERE/test_l3_verdict.sh"
+[ $rc -eq 0 ] && echo "ALL LEVEL3 TEST GROUPS PASSED" || echo "SOME LEVEL3 TEST GROUPS FAILED"
+exit $rc
diff --git a/level3/tools/tests/test_l3_infra.sh b/level3/tools/tests/test_l3_infra.sh
new file mode 100755
index 0000000..bb41b8c
--- /dev/null
+++ b/level3/tools/tests/test_l3_infra.sh
@@ -0,0 +1,96 @@
+#!/bin/bash
+# CPU-only negative/positive tests for the Level 3 correctness & reproducibility
+# helpers in level3/tools/l3_common.sh and l3_check.py. No GPU, no application,
+# no build -- these check that the mechanisms which decide PASS/FAIL behave.
+set -u
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../../.." && pwd)"
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+TOOLS="$R/level3/tools"
+pass=0; failn=0
+ok()  { echo "ok   $*"; pass=$((pass+1)); }
+bad() { echo "FAIL $*"; failn=$((failn+1)); }
+TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
+
+# 1. l3_check.require_finite rejects NaN/Inf/non-numbers, accepts finite
+py_check() { python3 - "$1" <<'PY'
+import os, sys
+sys.path.insert(0, os.environ["L3_TOOLS"])
+from l3_check import require_finite, ValidationError
+try:
+    require_finite("x", sys.argv[1]); print("ACCEPT")
+except ValidationError:
+    print("REJECT")
+PY
+}
+[ "$(L3_TOOLS=$TOOLS py_check nan)"  = REJECT ] && ok "1a: NaN rejected"            || bad "1a: NaN not rejected"
+[ "$(L3_TOOLS=$TOOLS py_check inf)"  = REJECT ] && ok "1b: Inf rejected"            || bad "1b: Inf not rejected"
+[ "$(L3_TOOLS=$TOOLS py_check abc)"  = REJECT ] && ok "1c: non-number rejected"     || bad "1c: non-number not rejected"
+[ "$(L3_TOOLS=$TOOLS py_check 1.5)"  = ACCEPT ] && ok "1d: finite accepted"         || bad "1d: finite rejected"
+
+# 2. l3_capture returns the COMMAND's exit code (not tee's), and saves output
+rc=0; l3_capture "$TMP/cap.log" -- bash -c 'echo hello; exit 7' >/dev/null 2>&1 || rc=$?
+[ "$rc" -eq 7 ] && ok "2a: l3_capture propagates real exit code (7)" || bad "2a: got rc=$rc, expected 7"
+grep -q hello "$TMP/cap.log" && ok "2b: l3_capture saved stdout" || bad "2b: output not saved"
+
+# 3. rc-gate pattern: a failed run must FAIL even if a stale log is present
+#    (this is the logic every validator relies on).
+echo "OLD PASS-looking log" > "$TMP/stale.log"
+fake_validate() { # simulates: run fails (rc=1) but an old log exists
+    local rc=0
+    bash -c 'exit 1' || rc=$?
+    [ "$rc" -eq 0 ] || return 1        # gate: nonzero run -> FAIL, regardless of stale log
+    return 0
+}
+if fake_validate; then bad "3: stale-log gate let a failed run pass"; else ok "3: failed run FAILs even with a stale log present"; fi
+
+# 4. l3_rundir: dry-run must not touch a real result dir (sentinel test)
+real="$R/build/level3/__selftest__/run/case.np1"
+mkdir -p "$real"; echo SENTINEL > "$real/keep.txt"
+d_real="$(l3_rundir "$real")"
+[ "$d_real" = "$real" ] && [ ! -e "$real/keep.txt" ] && ok "4a: real run recreates the dir fresh" || bad "4a: real run did not refresh ($d_real)"
+echo SENTINEL > "$real/keep.txt"
+d_dry="$(HPCPERF_DRY_RUN=1 l3_rundir "$real")"
+if [ "$d_dry" != "$real" ] && [ -f "$real/keep.txt" ] && [ "$(cat "$real/keep.txt")" = SENTINEL ]; then
+    ok "4b: dry-run used a scratch dir ($(basename "$(dirname "$d_dry")")/$(basename "$d_dry")) and left the real result untouched"
+else bad "4b: dry-run touched the real result dir (d_dry=$d_dry)"; fi
+d_bad=0; l3_rundir "/tmp/not-under-build" >/dev/null 2>&1 || d_bad=$?
+[ "$d_bad" -ne 0 ] && ok "4c: l3_rundir refuses a path outside build/level3/" || bad "4c: accepted an out-of-tree path"
+rm -rf "$R/build/level3/__selftest__"
+
+# 5. fingerprint patch handling: missing patch is an error; changed content
+#    changes the ordered series hash (cache-invalidation), same content stable.
+export CXX=/bin/true FC=/bin/true
+fp_missing=0; l3_fingerprint_text app sha cuda deps opts gam "$TMP/nope.patch" >/dev/null 2>&1 || fp_missing=$?
+[ "$fp_missing" -ne 0 ] && ok "5a: missing patch is a hard error" || bad "5a: missing patch accepted"
+printf 'A\n' > "$TMP/p.patch"
+h1="$(l3_fingerprint_text app sha cuda deps opts gam "$TMP/p.patch" 2>/dev/null | sed -n 's/^patch_series_sha256=//p')"
+printf 'B\n' > "$TMP/p.patch"   # same name, different content
+h2="$(l3_fingerprint_text app sha cuda deps opts gam "$TMP/p.patch" 2>/dev/null | sed -n 's/^patch_series_sha256=//p')"
+[ -n "$h1" ] && [ "$h1" != "$h2" ] && ok "5b: same-named patch with changed content changes the series hash (cache invalidated)" || bad "5b: series hash did not change ($h1 vs $h2)"
+hn="$(l3_fingerprint_text app sha cuda deps opts gam 2>/dev/null | sed -n 's/^patch_series_sha256=//p')"
+[ "$hn" = none ] && ok "5c: empty patch series -> 'none'" || bad "5c: empty series hash '$hn'"
+
+# 6. l3_clean_env.sh: allow-listed names pass, credential-looking names are dropped even under an
+#    allow-listed prefix, session/agent names are dropped, --show prints names only (never a value).
+CE="$TOOLS/l3_clean_env.sh"
+names="$(HPCPERF_SELFTEST_OK=keepme HPCPERF_SELFTEST_TOKEN=secretvalue CLAUDE_SELFTEST=secretvalue SELFTEST_API_KEY=secretvalue \
+         "$CE" -- env 2>/dev/null | sed -n 's/^\([A-Za-z_][A-Za-z0-9_]*\)=.*/\1/p')"
+echo "$names" | /usr/bin/grep -qx HPCPERF_SELFTEST_OK && ok "6a: clean env keeps an allow-listed project variable" || bad "6a: HPCPERF_SELFTEST_OK dropped"
+echo "$names" | /usr/bin/grep -qx HPCPERF_SELFTEST_TOKEN && bad "6b: credential-looking name survived through the HPCPERF_ prefix" || ok "6b: HPCPERF_*_TOKEN denied although its prefix is allow-listed"
+echo "$names" | /usr/bin/grep -qE '^(CLAUDE_SELFTEST|SELFTEST_API_KEY)$' && bad "6c: agent/credential names survived" || ok "6c: CLAUDE_* and *_API_KEY dropped"
+echo "$names" | /usr/bin/grep -qx PATH && ok "6d: PATH survives (the command can run)" || bad "6d: PATH dropped"
+show="$(HPCPERF_SELFTEST_TOKEN=secretvalue "$CE" --show -- true 2>&1)"
+echo "$show" | /usr/bin/grep -q secretvalue && bad "6e: --show printed a value" || ok "6e: --show prints names only"
+echo "$show" | /usr/bin/grep -q 'denied by the credential rule: .*HPCPERF_SELFTEST_TOKEN' && ok "6f: --show names the denied variable" || bad "6f: denied variable not reported"
+
+# 7. l3_run_recorded: records the real exit code of a queue step and never aborts the caller
+rcf="$TMP/rc.txt"; steps=0
+( set -e; for r in 0 3 0; do l3_run_recorded "$rcf" "step$r" -- bash -c "exit $r"; echo step >> "$TMP/steps"; done )
+[ "$(wc -l < "$TMP/steps")" -eq 3 ] && ok "7a: a queue under set -e continues past a step that exits 3" || bad "7a: queue stopped after $(wc -l < "$TMP/steps") step(s)"
+[ "$(tr '\n' ' ' < "$rcf")" = "step0 0 step3 3 step0 0 " ] && ok "7b: exit codes recorded verbatim (0 3 0)" || bad "7b: recorded '$(tr '\n' ' ' < "$rcf")'"
+
+echo
+echo "test_l3_infra: $pass passed, $failn failed"
+[ "$failn" -eq 0 ]
diff --git a/level3/tools/tests/test_l3_validators.sh b/level3/tools/tests/test_l3_validators.sh
new file mode 100755
index 0000000..05870e0
--- /dev/null
+++ b/level3/tools/tests/test_l3_validators.sh
@@ -0,0 +1,95 @@
+#!/bin/bash
+# Negative/positive tests for the second-batch application checkers (CPU only, no GPU, no build):
+# each checker must FAIL on a truncated run, a NaN/Inf, a non-converged/aborted solve, a missing GPU
+# banner and a nonzero exit code, and PASS on the genuine output it was written for. The genuine
+# outputs are taken from real runs under build/level3/<app>/<profile>/run when present; the CP2K
+# case falls back to a synthetic minimal output so the harness runs everywhere.
+set -u
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../../.." && pwd)"
+export L3_TOOLS="$R/level3/tools"
+pass=0; failn=0
+ok()  { echo "ok   $*"; pass=$((pass+1)); }
+bad() { echo "FAIL $*"; failn=$((failn+1)); }
+TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
+
+# ---------------------------------------------------------------- CP2K: cp2k_md_summary.py --check
+CP2K_CHK="$R/level3/cp2k/cp2k_md_summary.py"
+synth_cp2k() { # synth_cp2k <out> : minimal output with the features the checker gates on
+    local out=$1; {
+    echo " CP2K|            m dbcsr_acc spglib offload_cuda"
+    echo " GLOBAL| Run type                                                             MD"
+    echo " DBCSR| ACC: Number of devices/node                                            1"
+    echo "  Leaving inner SCF loop after reaching    50 steps."
+    echo " *** WARNING in qs_scf.F:700 :: SCF run NOT converged ***"
+    echo " ENERGY| Total FORCE_EVAL ( QS ) energy [hartree]          -1101.031005888854452"
+    echo " MD_INI| MD initialization"
+    for i in $(seq 1 10); do
+        echo "  *** SCF run converged in     5 steps ***"
+        echo " ENERGY| Total FORCE_EVAL ( QS ) energy [hartree]          -1101.0392895677$i"
+        echo " MD| Step number                                                         $i"
+        echo " MD| Potential energy [hartree]         -0.110103928957E+04  -0.110103928957E+04"
+        echo " MD| Conserved quantity [hartree]                            -0.110076554067E+04"
+    done
+    echo " GRID| ..."
+    echo " 1     collocate ortho    GPU                               15972172      21.34%"
+    echo " 1     integrate ortho    GPU                               15311000      20.46%"
+    echo "  **** **** ******  **  PROGRAM ENDED AT                2026-09-05 21:00:00.000"; } > "$out"
+}
+INP="$TMP/input.inp"; printf '&FORCE_EVAL\n  &DFT\n    &SCF\n      IGNORE_CONVERGENCE_FAILURE\n    &END SCF\n  &END DFT\n&END FORCE_EVAL\n' > "$INP"
+INP_STRICT="$TMP/input_strict.inp"; printf '&FORCE_EVAL\n  &DFT\n    &SCF\n      MAX_SCF 50\n    &END SCF\n  &END DFT\n&END FORCE_EVAL\n' > "$INP_STRICT"
+GOOD="$TMP/good.out"
+real="$(ls -d "$R"/build/level3/cp2k/*/run/h2o64.smoke.np1.t*/cp2k.out 2>/dev/null | head -1)"
+if [ -n "$real" ] && python3 "$CP2K_CHK" "$real" "$(dirname "$real")/input.inp" --check >/dev/null 2>&1; then cp "$real" "$GOOD"; cp "$(dirname "$real")/input.inp" "$INP"; echo "# cp2k: using real output $real"; else synth_cp2k "$GOOD"; echo "# cp2k: using synthetic output"; fi
+cp2k_check() { python3 "$CP2K_CHK" "$1" "${2:-$INP}" --check >/dev/null 2>&1; }
+cp2k_check "$GOOD" && ok "cp2k 1: genuine complete output accepted" || bad "cp2k 1: genuine output rejected"
+head -n $(( $(wc -l < "$GOOD") - 5 )) "$GOOD" | /usr/bin/grep -v 'PROGRAM ENDED' > "$TMP/trunc.out"
+cp2k_check "$TMP/trunc.out" && bad "cp2k 2: truncated output (no PROGRAM ENDED) accepted" || ok "cp2k 2: truncated output rejected"
+sed '0,/ENERGY| Total FORCE_EVAL/s/ENERGY| Total FORCE_EVAL ( QS ) energy \[hartree\] *-[0-9.]*/ENERGY| Total FORCE_EVAL ( QS ) energy [hartree]          NaN/' "$GOOD" > "$TMP/nan.out"
+cp2k_check "$TMP/nan.out" && bad "cp2k 3: NaN energy accepted" || ok "cp2k 3: NaN energy rejected"
+awk '/MD_INI\| MD initialization/{ini=1} ini && /SCF run converged in/ && !done {sub(/SCF run converged in +[0-9]+ steps/, "SCF run NOT converged"); done=1} {print}' "$GOOD" > "$TMP/nonconv.out"
+cp2k_check "$TMP/nonconv.out" && bad "cp2k 4: non-converged MD-step SCF accepted" || ok "cp2k 4: non-converged MD-step SCF rejected"
+cp2k_check "$GOOD" "$INP_STRICT" && bad "cp2k 5: non-converged initial SCF accepted although the deck lacks IGNORE_CONVERGENCE_FAILURE" || ok "cp2k 5: non-converged initial SCF rejected without IGNORE_CONVERGENCE_FAILURE"
+sed 's/DBCSR| ACC: Number of devices\/node *[0-9]*/DBCSR| ACC: Number of devices\/node                                            0/' "$GOOD" > "$TMP/nogpu.out"
+cp2k_check "$TMP/nogpu.out" && bad "cp2k 6: zero accelerator devices accepted" || ok "cp2k 6: zero accelerator devices rejected"
+sed 's/offload_cuda//' "$GOOD" > "$TMP/noflag.out"
+cp2k_check "$TMP/noflag.out" && bad "cp2k 7: missing offload_cuda cp2kflag accepted" || ok "cp2k 7: missing offload_cuda cp2kflag rejected"
+sed 's/ortho    GPU /ortho    CPU /' "$GOOD" > "$TMP/gridcpu.out"
+cp2k_check "$TMP/gridcpu.out" && bad "cp2k 8: GRID tasks all on CPU accepted" || ok "cp2k 8: GRID tasks all on CPU rejected"
+: > "$TMP/empty.out"; cp2k_check "$TMP/empty.out" && bad "cp2k 9: empty output accepted" || ok "cp2k 9: empty output rejected"
+
+# ---------------------------------------------------------------- QMCPACK: qmc_check.py
+QMC_CHK="$R/level3/qmcpack/qmc_check.py"; CS="$R/_upstream/level3/qmcpack/tests/scripts/check_scalars.py"
+realq="$(ls -d "$R"/build/level3/qmcpack/*/run/diamond2.smoke.np1.t*/qmc.out 2>/dev/null | head -1)"
+if [ -n "$realq" ] && [ -f "$CS" ]; then
+    RD="$(dirname "$realq")"; prefix="$(/usr/bin/grep -m1 '^prefix=' "$RD/run_manifest.txt" | cut -d= -f2)"
+    qmc_check() { python3 "$QMC_CHK" "$1" "$prefix" "-21.844975 0.02" 3 2 "$CS" >/dev/null 2>&1; }
+    mk() { rm -rf "$TMP/q"; cp -r "$RD" "$TMP/q"; }
+    mk; qmc_check "$TMP/q" && ok "qmc 1: genuine run accepted" || bad "qmc 1: genuine run rejected"
+    mk; sed -i '/QMCPACK execution completed successfully/d' "$TMP/q/qmc.out"; qmc_check "$TMP/q" && bad "qmc 2: incomplete run accepted" || ok "qmc 2: incomplete run rejected"
+    mk; sed -i 's/^exit_code=0$/exit_code=1/' "$TMP/q/run_manifest.txt"; qmc_check "$TMP/q" && bad "qmc 3: nonzero exit code accepted" || ok "qmc 3: nonzero exit code rejected"
+    mk; f="$TMP/q/$prefix.s001.scalar.dat"; awk '!/^#/ && ++n==2 {$2="nan"} {print}' "$f" > "$f.tmp" && mv "$f.tmp" "$f"; qmc_check "$TMP/q" && bad "qmc 4: NaN in DMC scalars accepted" || ok "qmc 4: NaN in DMC scalars rejected"
+    mk; f="$TMP/q/$prefix.s001.scalar.dat"; head -n 12 "$f" > "$f.tmp" && mv "$f.tmp" "$f"; qmc_check "$TMP/q" && bad "qmc 5: truncated DMC scalar file accepted" || ok "qmc 5: truncated DMC scalar file rejected"
+    mk; sed -i '/OpenMP target offload to accelerators build option is enabled/d' "$TMP/q/qmc.out"; qmc_check "$TMP/q" && bad "qmc 6: missing offload banner accepted" || ok "qmc 6: missing offload banner rejected"
+    mk; f="$TMP/q/$prefix.s001.scalar.dat"; awk '!/^#/ && NF>2 {$2=$2+1.0} {print}' "$f" > "$f.tmp" && mv "$f.tmp" "$f"; qmc_check "$TMP/q" && bad "qmc 7: energies shifted by +1 Ha accepted" || ok "qmc 7: energies shifted by +1 Ha rejected (statistical test)"
+    mk; echo "QMCPACK ERROR synthetic" >> "$TMP/q/qmc.out"; qmc_check "$TMP/q" && bad "qmc 8: 'QMCPACK ERROR' accepted" || ok "qmc 8: 'QMCPACK ERROR' rejected"
+else
+    echo "# qmcpack: no genuine run under build/level3/qmcpack/*/run yet -- qmc_check.py tests skipped"
+fi
+
+# ---------------------------------------------------------------- DFT-FE: dftfe_check.py
+DFT_CHK="$R/level3/dftfe/dftfe_check.py"
+reald="$(ls -d "$R"/build/level3/dftfe/*/run/al_md.smoke.np1*/dftfe.out 2>/dev/null | head -1)"
+if [ -f "$DFT_CHK" ] && [ -n "$reald" ]; then
+    RD="$(dirname "$reald")"; REFD="$R/_upstream/level3/dftfe/testsGPU/pseudopotential/real/accuracyBenchmarks/output_MD_0"
+    dft_check() { python3 "$DFT_CHK" "$1/dftfe.out" "$REFD" --check >/dev/null 2>&1; }
+    mk() { rm -rf "$TMP/d"; cp -r "$RD" "$TMP/d"; }
+    mk; dft_check "$TMP/d" && ok "dftfe 1: genuine run accepted" || bad "dftfe 1: genuine run rejected"
+    mk; sed -i '/MD run completed successfully/d' "$TMP/d/dftfe.out"; dft_check "$TMP/d" && bad "dftfe 2: incomplete MD accepted" || ok "dftfe 2: incomplete MD rejected"
+    mk; sed -i '0,/Total Energy in Ha at timeIndex/s/\(Total Energy in Ha at timeIndex\) *-[0-9.]*/\1 nan/' "$TMP/d/dftfe.out"; dft_check "$TMP/d" && bad "dftfe 3: NaN energy accepted" || ok "dftfe 3: NaN energy rejected"
+    mk; sed -i '0,/Total Energy in Ha at timeIndex/s/\(Total Energy in Ha at timeIndex\) *\(-[0-9]*\)\./\1 \2.9/' "$TMP/d/dftfe.out"; dft_check "$TMP/d" && bad "dftfe 4: energy off by ~0.5 Ha accepted" || ok "dftfe 4: energy off by ~0.5 Ha rejected"
+else
+    echo "# dftfe: checker or genuine run not present yet -- tests skipped"
+fi
+
+echo "== $pass ok, $failn FAIL =="; [ "$failn" -eq 0 ]
diff --git a/level3/tools/tests/test_l3_verdict.sh b/level3/tools/tests/test_l3_verdict.sh
new file mode 100755
index 0000000..5be0ab9
--- /dev/null
+++ b/level3/tools/tests/test_l3_verdict.sh
@@ -0,0 +1,45 @@
+#!/bin/bash
+# Verdict classes of level3/tools/l3_verdict.py (CPU only): exit code 3 (Nyx heat/cool I_R_CHECK_PENDING)
+# and 4 (UNSUPPORTED_LAYOUT) must stay their own classes in every summary -- never PASS -- and an exit
+# code that contradicts the log is FAIL. The queue-continuation half (a step exiting 3 does not abort
+# a queue) is test 7 of test_l3_infra.sh (l3_run_recorded).
+set -u
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../../.." && pwd)"
+V="$R/level3/tools/l3_verdict.py"
+pass=0; failn=0
+ok()  { echo "ok   $*"; pass=$((pass+1)); }
+bad() { echo "FAIL $*"; failn=$((failn+1)); }
+TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
+mk() { printf '%s\n' "$2" > "$TMP/$1"; }
+mk pass.log "hpcperf-launch: audit summary: 2 verified, 0 mismatch, 0 unverified (of 2 ranks)
+Nyx CUDA validation (2 GPU, cases: minisb lya_adiabatic; crit): PASS"
+mk pend.log "    lya_heatcool.np1.vs-CPU: diagnostic field(s) I_R reported, NOT accepted -> case verdict I_R_CHECK_PENDING
+Nyx CUDA validation (1 GPU, cases: lya_heatcool; crit): STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING [x] -- heat/cool acceptance incomplete, not a full regression PASS"
+mk unsup.log "    minisb.np2.vs-1GPU: UNSUPPORTED_LAYOUT -- a different but legal box layout is outside this validator's scope
+Nyx CUDA validation (2 GPU, cases: minisb; crit): UNSUPPORTED_LAYOUT [minisb.np2.vs-1GPU] -- plotfile comparison not performed for these pairs; not a PASS"
+mk fail.log "validate.sh: FAIL -- minisb.np2.vs-1GPU: plotfile comparison (rel_tol 2e-10): DISAGREE
+Nyx CUDA validation (2 GPU, cases: minisb): FAIL"
+mk noverdict.log "hpcperf-launch: audit summary: 0 verified, 0 mismatch, 1 unverified (of 1 ranks)
+some output that never reaches a verdict line"
+mk firstbatch.log "LAMMPS CUDA validation (4 GPU, in.lj thermo vs reference log): PASS"
+c() { python3 "$V" classify --rc "$1" --log "$TMP/$2"; }
+[ "$(c 0 pass.log)" = PASS ] && ok "1: rc 0 + ': PASS' line -> PASS" || bad "1: $(c 0 pass.log)"
+[ "$(c 0 firstbatch.log)" = PASS ] && ok "2: first-batch style PASS line -> PASS" || bad "2: $(c 0 firstbatch.log)"
+[ "$(c 3 pend.log)" = PENDING ] && ok "3: rc 3 + I_R_CHECK_PENDING -> PENDING (its own class)" || bad "3: $(c 3 pend.log)"
+[ "$(c 0 pend.log)" = FAIL ] && ok "4: rc 0 with a PENDING line is inconsistent -> FAIL, never PASS" || bad "4: $(c 0 pend.log)"
+[ "$(c 3 pass.log)" = FAIL ] && ok "5: rc 3 with a PASS line is inconsistent -> FAIL" || bad "5: $(c 3 pass.log)"
+[ "$(c 4 unsup.log)" = UNSUPPORTED_LAYOUT ] && ok "6: rc 4 + UNSUPPORTED_LAYOUT -> UNSUPPORTED_LAYOUT" || bad "6: $(c 4 unsup.log)"
+[ "$(c 1 fail.log)" = FAIL ] && ok "7: rc 1 -> FAIL" || bad "7: $(c 1 fail.log)"
+[ "$(c 0 noverdict.log)" = FAIL ] && ok "8: rc 0 without a verdict line -> FAIL" || bad "8: $(c 0 noverdict.log)"
+[ "$(c 124 pass.log)" = FAIL ] && ok "9: timeout (124) -> FAIL even with a stale PASS line" || bad "9: $(c 124 pass.log)"
+[ "$(c 0 absent.log)" = MISSING ] && ok "10: missing log -> MISSING" || bad "10: $(c 0 absent.log)"
+# summary: every class counted separately; the PENDING row is never marked PASS; unverified audits listed
+printf 'pass.log 0\npend.log 3\nunsup.log 4\nfail.log 1\nfirstbatch.log 0\n' > "$TMP/rc.txt"
+python3 "$V" summary --rc-file "$TMP/rc.txt" --log "$TMP"/{pass,pend,unsup,fail,noverdict,firstbatch}.log > "$TMP/sum.md" 2>&1; rc=$?
+[ "$rc" -eq 0 ] && /usr/bin/grep -q 'PASS=2, PENDING=1, UNSUPPORTED_LAYOUT=1, FAIL=1, MISSING=1' "$TMP/sum.md" && ok "11: summary counts PASS/PENDING/UNSUPPORTED_LAYOUT/FAIL/MISSING separately" || bad "11: rc=$rc: $(tail -3 "$TMP/sum.md")"
+/usr/bin/grep -E '^\| pend\.log ' "$TMP/sum.md" | /usr/bin/grep -q '\*\*PENDING\*\*' && ! /usr/bin/grep -E '^\| pend\.log ' "$TMP/sum.md" | /usr/bin/grep -q '\*\*PASS\*\*' && ok "12: the rc-3 row is labelled PENDING, not PASS" || bad "12: $(/usr/bin/grep -E '^\| pend\.log ' "$TMP/sum.md")"
+/usr/bin/grep -q 'unverified ranks.*noverdict.log \[0/0/1\]' "$TMP/sum.md" && ok "13: unverified launcher audit listed as a binding-evidence gap" || bad "13: $(/usr/bin/grep 'unverified ranks' "$TMP/sum.md")"
+echo
+echo "test_l3_verdict: $pass passed, $failn failed"
+[ "$failn" -eq 0 ]
diff --git a/level3/tools/tests/test_nyx_validator.sh b/level3/tools/tests/test_nyx_validator.sh
new file mode 100755
index 0000000..3297d59
--- /dev/null
+++ b/level3/tools/tests/test_nyx_validator.sh
@@ -0,0 +1,170 @@
+#!/bin/bash
+# Negative/positive tests for the Nyx validator (CPU only, no GPU, no application, no build).
+#
+# Part A: nyx_fcompare_check.py against STUB fcompare/fextrema tools whose output is injected
+#         (the reviewer's cases: rc=1 with Ne abs=1 rel=inf; rc=1 with a missing variable and the
+#         "not all variables present" warning; rc=1 with a NaN row -- all of which the previous awk
+#         parser accepted) plus normal control, legal zero/zero reference, zero reference vs nonzero
+#         test, duplicate row, truncated table, header variable-set mismatch, diagnostic field.
+# Part B: the REAL level3/nyx/validate.sh call chain in OFFLINE mode (HPCPERF_NYX_OFFLINE=1) on
+#         synthetic run directories, with the same stub tools: the outer verdict/exit code must
+#         follow the comparator (FAIL on every injected defect, PASS on the control, exit 3 for a
+#         diagnostic field).
+set -u
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../../.." && pwd)"
+CHK="$R/level3/nyx/nyx_fcompare_check.py"
+pass=0; failn=0
+ok()  { echo "ok   $*"; pass=$((pass+1)); }
+bad() { echo "FAIL $*"; failn=$((failn+1)); }
+TMP="$(mktemp -d)"; trap 'rm -rf "$TMP" "$R/build/level3/nyx/__selftest__"*' EXIT
+VARS="density xmom ymom zmom rho_E rho_e Temp Ne phi_grav grav_x grav_y grav_z I_R pressure particle_count particle_mass_density"
+
+# --- synthetic plotfile: an AMReX Header (single level, 8 grids) --------------------------------
+mk_header() { # mk_header <plotdir> [time] [vars...]; MK_BOXES="xlo xhi ylo yhi zlo zhi" lines (physical) override the default 2x2x2 layout
+    local d=$1 t=${2:-2.86455272510386e-06}; shift; [ $# -gt 0 ] && shift; local vars=("$@"); [ ${#vars[@]} -gt 0 ] || vars=($VARS)
+    local boxes=() l x0 x1 y0 y1 z0 z1
+    if [ -n "${MK_BOXES:-}" ]; then while IFS= read -r l; do [ -n "$l" ] && boxes+=("$l"); done <<< "$MK_BOXES"
+    else for b in "0 4" "4 8"; do for c in "0 4" "4 8"; do for e in "0 4" "4 8"; do boxes+=("$b $c $e"); done; done; done; fi
+    mkdir -p "$d/Level_0" "$d/DM"
+    { echo "HyperCLaw-V1.1"; echo "${#vars[@]}"; printf '%s\n' "${vars[@]}"; echo 3; echo "$t"; echo 0; echo "0 0 0 "; echo "8 8 8 "; echo ""
+      echo "((0,0,0) (31,31,31) (0,0,0)) "; echo "10 "; echo "0.25 0.25 0.25 "; echo 0; echo 0
+      echo "0 ${#boxes[@]} $t"; echo 10
+      for l in "${boxes[@]}"; do read -r x0 x1 y0 y1 z0 z1 <<< "$l"; echo "$x0 $x1"; echo "$y0 $y1"; echo "$z0 $z1"; done
+      echo "Level_0/Cell"; } > "$d/Header"
+    # DM particle header: version, dim, nreal, names, nint, names, is_checkpoint, nparticles
+    { echo "Version_Two_Dot_Zero_double"; echo 3; echo 4; printf 'mass\nxvel\nyvel\nzvel\n'; echo 0; echo 0; echo 32768; } > "$d/DM/Header"
+}
+# stub fextrema: prints "name min max" for every variable of the plotfile Header; VALUE can be overridden per var
+mk_tools() { # mk_tools <dir> ; env FEXTREMA_OVERRIDE="Ne:0:0,rho_e:nan:1"
+    local t=$1; mkdir -p "$t"
+    cat > "$t/amrex_fextrema" <<'EOF'
+#!/bin/bash
+plt=$1; n=$(sed -n 2p "$plt/Header"); i=0
+sed -n "3,$((2+n))p" "$plt/Header" | while read -r v; do lo=1.0; hi=2.0
+  IFS=, read -ra ov <<< "${FEXTREMA_OVERRIDE:-}"; for o in "${ov[@]}"; do IFS=: read -r name mn mx <<< "$o"; [ "$name" = "$v" ] && { lo=$mn; hi=$mx; }; done
+  printf ' %-24s %24s %24s\n' "$v" "$lo" "$hi"; done
+EOF
+    # stub fcompare: prints the canned table from $FCOMPARE_TABLE (file) and exits $FCOMPARE_RC
+    cat > "$t/amrex_fcompare" <<'EOF'
+#!/bin/bash
+cat "$FCOMPARE_TABLE"; exit "${FCOMPARE_RC:-0}"
+EOF
+    printf '#!/bin/bash\necho stub\n' > "$t/amrex_fvolumesum"; printf '#!/bin/bash\necho stub\n' > "$t/particle_compare"
+    chmod +x "$t"/*
+}
+table() { # table <rc-independent canned fcompare output>: one row per var with given abs/rel; overrides "var=abs:rel"
+    local extra="$1"; shift
+    { echo ""; echo "            variable name             absolute error            relative error"; echo "                                         (||A - B||)         (||A - B||/||A||)"
+      echo " ----------------------------------------------------------------------------"; echo " level = 0"
+      for v in $VARS; do a=1e-14; r=1e-14
+          for o in "$@"; do IFS='=' read -r name vals <<< "$o"; [ "$name" = "$v" ] && { IFS=: read -r a r <<< "$vals"; }; done
+          case "$a" in SKIP) continue;; MSG) printf ' %-24s  %-50s\n' "$v" "$r"; continue;; esac
+          printf ' %-24s  %24s  %24s\n' "$v" "$a" "$r"; done
+      [ -n "$extra" ] && echo "$extra"; true; }
+}
+T="$TMP/tools"; mk_tools "$T"; export FEXTREMA_OVERRIDE="Ne:0:0"   # Ne identically zero in the reference (as in the real deck)
+REF="$TMP/ref/plt00010"; TST="$TMP/tst/plt00010"; mk_header "$REF"; mk_header "$TST"
+run_chk() { # run_chk <table-file> <rc> [extra args]
+    local tbl=$1 rc=$2; shift 2
+    FCOMPARE_TABLE="$tbl" FCOMPARE_RC="$rc" python3 "$CHK" "$REF" "$TST" --rel_tol 2e-10 --fcompare "$T/amrex_fcompare" --fextrema "$T/amrex_fextrema" --out "$TMP/out.txt" "$@" >"$TMP/chk.out" 2>&1
+}
+# A1 control: everything tiny, Ne zero/zero (abs 0 rel 0 as fcompare prints), rc 0 -> AGREE
+table "" "Ne=0:0" > "$TMP/t1"; run_chk "$TMP/t1" 0; [ $? -eq 0 ] && ok "A1: control (all within 2e-10, Ne zero/zero) AGREE" || bad "A1: control rejected: $(cat "$TMP/chk.out")"
+# A2 reviewer case: rc=1, Ne abs=1 rel=inf (reference norm zero -> inf) -> must FAIL (zero-reference abs rule)
+table "" "Ne=1:inf" > "$TMP/t2"; run_chk "$TMP/t2" 1; rc=$?; [ $rc -eq 1 ] && ok "A2: Ne abs=1 rel=inf on a zero reference -> DISAGREE (rc 1)" || bad "A2: rc=$rc: $(cat "$TMP/chk.out")"
+# A3 reviewer case: rc=1, a variable missing + fcompare's warning line -> STRUCTURAL
+table " WARNING: not all variables present in both files" "Temp=SKIP:x" > "$TMP/t3"; run_chk "$TMP/t3" 1; rc=$?; [ $rc -eq 2 ] && ok "A3: missing variable + WARNING -> STRUCTURAL (rc 2)" || bad "A3: rc=$rc: $(cat "$TMP/chk.out")"
+# A4 reviewer case: rc=1, NaN message row -> STRUCTURAL
+table "" "rho_e=MSG:< NaN present in B > " > "$TMP/t4"; run_chk "$TMP/t4" 1; rc=$?; [ $rc -eq 2 ] && ok "A4: '< NaN present in B >' row -> STRUCTURAL" || bad "A4: rc=$rc: $(cat "$TMP/chk.out")"
+# A5 numeric nan in the rel column on a non-zero reference -> not finite -> DISAGREE or STRUCTURAL, never AGREE
+table "" "Temp=1e-3:nan" > "$TMP/t5"; run_chk "$TMP/t5" 1; rc=$?; [ $rc -ne 0 ] && ok "A5: rel=nan on a non-zero reference rejected (rc $rc)" || bad "A5: rel=nan accepted"
+# A6 inf rel on a NON-zero reference (fextrema says Temp is non-zero) -> DISAGREE
+table "" "Temp=1e-3:inf" > "$TMP/t6"; run_chk "$TMP/t6" 1; rc=$?; [ $rc -eq 1 ] && ok "A6: rel=inf on a non-zero reference -> DISAGREE" || bad "A6: rc=$rc: $(cat "$TMP/chk.out")"
+# A7 zero reference, nonzero test, fcompare says agree (rc 0, abs tiny) -> abs rule 0 -> DISAGREE (and parser/tool inconsistency is STRUCTURAL) => non-zero
+table "" "Ne=1e-20:0" > "$TMP/t7"; run_chk "$TMP/t7" 0; rc=$?; [ $rc -ne 0 ] && ok "A7: zero reference vs 1e-20 test -> rejected (rc $rc, exact rule)" || bad "A7: accepted"
+# A8 duplicate row -> STRUCTURAL
+{ table "" ; printf ' %-24s  %24s  %24s\n' Temp 1e-14 1e-14; } > "$TMP/t8"; run_chk "$TMP/t8" 0; rc=$?; [ $rc -eq 2 ] && ok "A8: duplicate row -> STRUCTURAL" || bad "A8: rc=$rc"
+# A9 truncated table (last 3 rows missing) -> STRUCTURAL
+table "" | head -n -3 > "$TMP/t9"; run_chk "$TMP/t9" 0; rc=$?; [ $rc -eq 2 ] && ok "A9: truncated table -> STRUCTURAL" || bad "A9: rc=$rc"
+# A10 header variable-set mismatch (test plotfile lacks pressure) -> STRUCTURAL before fcompare
+mk_header "$TMP/tst2/plt00010" 2.86455272510386e-06 $(echo $VARS | sed 's/ pressure//'); table "" > "$TMP/t10"
+FCOMPARE_TABLE="$TMP/t10" FCOMPARE_RC=0 python3 "$CHK" "$REF" "$TMP/tst2/plt00010" --rel_tol 2e-10 --fcompare "$T/amrex_fcompare" --fextrema "$T/amrex_fextrema" >"$TMP/chk.out" 2>&1; rc=$?
+[ $rc -eq 2 ] && ok "A10: header variable set differs -> STRUCTURAL" || bad "A10: rc=$rc"
+# A11 header time mismatch -> STRUCTURAL
+mk_header "$TMP/tst3/plt00010" 3.0e-06; FCOMPARE_TABLE="$TMP/t1" FCOMPARE_RC=0 python3 "$CHK" "$REF" "$TMP/tst3/plt00010" --rel_tol 2e-10 --fcompare "$T/amrex_fcompare" --fextrema "$T/amrex_fextrema" >"$TMP/chk.out" 2>&1; rc=$?
+[ $rc -eq 2 ] && ok "A11: simulation time differs -> STRUCTURAL" || bad "A11: rc=$rc"
+# A12 raw non-finite value in the reference (fextrema max=nan) -> STRUCTURAL even if fcompare is happy
+FEXTREMA_OVERRIDE="Ne:0:0,rho_e:1:nan" run_chk "$TMP/t1" 0; rc=$?; [ $rc -eq 2 ] && ok "A12: non-finite raw value (fextrema) -> STRUCTURAL" || bad "A12: rc=$rc"
+# A13 diagnostic field over tolerance is reported, others gate: rc from fcompare 1 (it counts I_R) -> AGREE
+table "" "Ne=0:0" "I_R=0.29:1.3" > "$TMP/t13"; run_chk "$TMP/t13" 1 --diagnostic I_R; rc=$?; [ $rc -eq 0 ] && /usr/bin/grep -q 'diagnostic=\[.I_R@L0: abs=2.9' "$TMP/chk.out" && ok "A13: diagnostic I_R (rel 1.3) reported, state gated -> AGREE" || bad "A13: rc=$rc: $(cat "$TMP/chk.out")"
+# A14 the same I_R row WITHOUT the diagnostic flag -> DISAGREE
+run_chk "$TMP/t13" 1; rc=$?; [ $rc -eq 1 ] && ok "A14: I_R gated when not declared diagnostic -> DISAGREE" || bad "A14: rc=$rc"
+# A15 a genuine over-tolerance state variable -> DISAGREE
+table "" "Ne=0:0" "Temp=1e-3:5e-9" > "$TMP/t15"; run_chk "$TMP/t15" 1; rc=$?; [ $rc -eq 1 ] && ok "A15: Temp rel 5e-9 > 2e-10 -> DISAGREE" || bad "A15: rc=$rc"
+# A16 parser/tool inconsistency: table says agree but fcompare rc=1 and no diagnostic -> STRUCTURAL
+run_chk "$TMP/t1" 1; rc=$?; [ $rc -eq 2 ] && ok "A16: AGREE table but fcompare rc=1 -> STRUCTURAL (inconsistency)" || bad "A16: rc=$rc"
+# --- box layouts: legal re-blocking is UNSUPPORTED_LAYOUT (exit 4, explicit), real defects are STRUCTURAL ----
+chk_layout() { # chk_layout <plotdir with another layout> -> rc, output in chk.out (control table, rc 0)
+    FCOMPARE_TABLE="$TMP/t1" FCOMPARE_RC=0 python3 "$CHK" "$REF" "$1" --rel_tol 2e-10 --fcompare "$T/amrex_fcompare" --fextrema "$T/amrex_fextrema" >"$TMP/chk.out" 2>&1
+}
+REBLOCK=$'0 8 0 4 0 4\n0 8 0 4 4 8\n0 8 4 8 0 4\n0 8 4 8 4 8'                       # 4 boxes, same 32^3 cells as the 8-box layout
+DEFAULT8="$(for b in "0 4" "4 8"; do for c in "0 4" "4 8"; do for e in "0 4" "4 8"; do echo "$b $c $e"; done; done; done)"
+MK_BOXES="$REBLOCK" mk_header "$TMP/lay1/plt00010"; chk_layout "$TMP/lay1/plt00010"; rc=$?
+[ $rc -eq 4 ] && /usr/bin/grep -q 'UNSUPPORTED_LAYOUT.*legal re-blocking' "$TMP/chk.out" && ok "A17: same cells, different partition (4 vs 8 boxes) -> UNSUPPORTED_LAYOUT (exit 4), reported explicitly" || bad "A17: rc=$rc: $(cat "$TMP/chk.out")"
+MK_BOXES="$(echo "$DEFAULT8" | head -n 7)" mk_header "$TMP/lay2/plt00010"; chk_layout "$TMP/lay2/plt00010"; rc=$?
+[ $rc -eq 2 ] && /usr/bin/grep -q 'coverage deficit' "$TMP/chk.out" && ok "A18: a missing box (7 of 8, level 0 not covered) -> STRUCTURAL" || bad "A18: rc=$rc: $(cat "$TMP/chk.out")"
+MK_BOXES="$(echo "$DEFAULT8" | head -n 7; echo "$DEFAULT8" | head -n 1)" mk_header "$TMP/lay3/plt00010"; chk_layout "$TMP/lay3/plt00010"; rc=$?
+[ $rc -eq 2 ] && /usr/bin/grep -q 'overlap' "$TMP/chk.out" && ok "A19: overlapping boxes -> STRUCTURAL" || bad "A19: rc=$rc: $(cat "$TMP/chk.out")"
+MK_BOXES="$(echo "$DEFAULT8" | head -n 7; echo "4 12 4 8 4 8")" mk_header "$TMP/lay4/plt00010"; chk_layout "$TMP/lay4/plt00010"; rc=$?
+[ $rc -eq 2 ] && /usr/bin/grep -q 'outside the level domain' "$TMP/chk.out" && ok "A20: a box outside the domain -> STRUCTURAL" || bad "A20: rc=$rc: $(cat "$TMP/chk.out")"
+MK_BOXES="$(echo "$DEFAULT8" | tac)" mk_header "$TMP/lay5/plt00010"; chk_layout "$TMP/lay5/plt00010"; rc=$?
+[ $rc -eq 0 ] && /usr/bin/grep -q 'RESULT: AGREE' "$TMP/chk.out" && ok "A21: the same boxes listed in another order -> identical layout, compared normally" || bad "A21: rc=$rc: $(cat "$TMP/chk.out")"
+
+# --- Part B: real validate.sh chain, offline, synthetic run dirs, stub tools -----------------------
+VAL="$R/level3/nyx/validate.sh"
+if [ -f "$VAL" ]; then
+    SUB="run.selftest-$$"; PROF="__selftest__"; export HPCPERF_L3_RUN_SUBDIR="$SUB" HPCPERF_NYX_PROFILE="${PROF}-adiabatic" HPCPERF_NYX_CPU_PROFILE="${PROF}-cpu"
+    GR="$R/build/level3/nyx/${PROF}-adiabatic/$SUB"; CR="$R/build/level3/nyx/${PROF}-cpu/$SUB"
+    mk_run() { # mk_run <dir> [dm particle count]
+        local d=$1 np=${2:-32768}; mkdir -p "$d"; mk_header "$d/plt00000" 0.0; mk_header "$d/plt00010"
+        sed -i "s/^32768$/$np/" "$d/plt00000/DM/Header" "$d/plt00010/DM/Header"
+        printf '#  nstep time dt z a\n%s\n' "$(for s in $(seq 0 10); do echo "       $s 1.7e7 2.6e5 100 0.0099"; done)" > "$d/runlog"
+        printf 'run_id=selftest\nexit_code=0\nbinary_sha256=deadbeef\ndeck_sha256=cafe\nsteps=10\n' > "$d/run_manifest.txt"
+    }
+    # MiniSB's IC count (Exec/MiniSB/ic_sb_32.ascii) is 32686 when the upstream checkout is present; the count check is skipped otherwise
+    NP_SB="$(head -1 "$R/_upstream/level3/Nyx/Exec/MiniSB/ic_sb_32.ascii" 2>/dev/null | tr -d ' ')"; NP_SB="${NP_SB:-32768}"
+    for d in "$GR/minisb.smoke.np1" "$GR/minisb.smoke.np1.rerun" "$GR/minisb.smoke.np2" "$CR/minisb.smoke.np1"; do mk_run "$d" "$NP_SB"; done
+    printf '#!/bin/bash\necho " density  1.0e+10"\n' > "$T/amrex_fvolumesum"; chmod +x "$T/amrex_fvolumesum"
+    export HPCPERF_NYX_OFFLINE=1 HPCPERF_NYX_TOOLS_DIR="$T" HPCPERF_NYX_SKIP_PARTICLES=1 HPCPERF_NYX_CASES=minisb HPCPERF_NYX_REPORT_DIR="$TMP/reports"
+    chain() { FCOMPARE_TABLE="$1" FCOMPARE_RC="$2" HPCPERF_GPUS="$3" bash "$VAL" CUDA > "$TMP/val.out" 2>&1; echo $?; }
+    # ic_count for minisb reads the upstream IC file; if the checkout is absent the count check is skipped by the script (empty want)
+    rc=$(chain "$TMP/t1" 0 2); /usr/bin/grep -q ': PASS$' "$TMP/val.out" && [ "$rc" -eq 0 ] && ok "B1: validate.sh offline chain, control -> PASS (exit 0)" || bad "B1: rc=$rc: $(tail -3 "$TMP/val.out")"
+    rc=$(chain "$TMP/t2" 1 2); [ "$rc" -eq 1 ] && /usr/bin/grep -q 'FAIL' "$TMP/val.out" && ok "B2: chain, Ne abs=1 rel=inf -> FAIL (exit 1)" || bad "B2: rc=$rc: $(tail -3 "$TMP/val.out")"
+    rc=$(chain "$TMP/t3" 1 2); [ "$rc" -eq 1 ] && /usr/bin/grep -q 'STRUCTURAL' "$TMP/val.out" && ok "B3: chain, missing variable warning -> FAIL/STRUCTURAL" || bad "B3: rc=$rc: $(tail -3 "$TMP/val.out")"
+    rc=$(chain "$TMP/t4" 1 2); [ "$rc" -eq 1 ] && /usr/bin/grep -q 'STRUCTURAL' "$TMP/val.out" && ok "B4: chain, NaN row -> FAIL/STRUCTURAL" || bad "B4: rc=$rc: $(tail -3 "$TMP/val.out")"
+    rc=$(chain "$TMP/t9" 0 2); [ "$rc" -eq 1 ] && ok "B5: chain, truncated table -> FAIL" || bad "B5: rc=$rc"
+    rc=$(chain "$TMP/t8" 0 2); [ "$rc" -eq 1 ] && ok "B6: chain, duplicate row -> FAIL" || bad "B6: rc=$rc"
+    rc=$(chain "$TMP/t15" 1 2); [ "$rc" -eq 1 ] && ok "B7: chain, Temp over tolerance -> FAIL" || bad "B7: rc=$rc"
+    # heat/cool case: diagnostic I_R -> exit 3 (STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING), never PASS
+    export HPCPERF_NYX_HEATCOOL=YES HPCPERF_NYX_PROFILE="${PROF}-heatcool" HPCPERF_NYX_CPU_PROFILE="${PROF}-cpuhc" HPCPERF_NYX_CASES=lya_heatcool
+    GH="$R/build/level3/nyx/${PROF}-heatcool/$SUB"; CH="$R/build/level3/nyx/${PROF}-cpuhc/$SUB"
+    for d in "$GH/lya_heatcool.smoke.np1" "$GH/lya_heatcool.smoke.np1.rerun" "$GH/lya_heatcool.smoke.np2" "$CH/lya_heatcool.smoke.np1"; do mk_run "$d"; done
+    rc=$(chain "$TMP/t13" 1 2); [ "$rc" -eq 3 ] && /usr/bin/grep -q 'I_R_CHECK_PENDING' "$TMP/val.out" && ! /usr/bin/grep -q '): PASS$' "$TMP/val.out" && ok "B8: chain, heat/cool with diagnostic I_R -> exit 3, I_R_CHECK_PENDING, no PASS" || bad "B8: rc=$rc: $(tail -2 "$TMP/val.out")"
+    table "" "Ne=0:0" "I_R=0.29:1.3" "Temp=1e-3:1e-3" > "$TMP/t15hc"   # Temp rel 1e-3 > the heat/cool tolerance 5e-5
+    rc=$(chain "$TMP/t15hc" 1 2); [ "$rc" -eq 1 ] && ok "B9: chain, heat/cool with a state variable over tolerance -> FAIL (not PENDING)" || bad "B9: rc=$rc"
+    # reports never land in the run directories in offline mode
+    [ -z "$(ls "$GH/lya_heatcool.smoke.np2" | /usr/bin/grep -E 'fcompare|particle_compare')" ] && [ -n "$(ls "$TMP/reports" 2>/dev/null)" ] && ok "B10: offline reports written under HPCPERF_NYX_REPORT_DIR, run dirs untouched" || bad "B10: report placement"
+    # a legally re-blocked run directory: validate.sh exits 4 with UNSUPPORTED_LAYOUT -- not PASS, not FAIL, not PENDING-as-PASS
+    MK_BOXES="$REBLOCK" mk_run "$GH/lya_heatcool.smoke.np2"
+    rc=$(chain "$TMP/t13" 0 2); [ "$rc" -eq 4 ] && /usr/bin/grep -q '): UNSUPPORTED_LAYOUT \[' "$TMP/val.out" && ! /usr/bin/grep -q '): PASS$' "$TMP/val.out" && ok "B11: chain, re-blocked layout -> exit 4, UNSUPPORTED_LAYOUT verdict, no PASS" || bad "B11: rc=$rc: $(tail -2 "$TMP/val.out")"
+    # a defective layout (missing box) is a FAIL of the case, not UNSUPPORTED
+    MK_BOXES="$(echo "$DEFAULT8" | head -n 7)" mk_run "$GH/lya_heatcool.smoke.np2"
+    rc=$(chain "$TMP/t13" 0 2); [ "$rc" -eq 1 ] && /usr/bin/grep -q 'STRUCTURAL.*coverage deficit' "$TMP/val.out" && ok "B12: chain, missing box -> FAIL (STRUCTURAL: coverage deficit)" || bad "B12: rc=$rc: $(tail -2 "$TMP/val.out")"
+    mk_run "$GH/lya_heatcool.smoke.np2"
+    unset HPCPERF_NYX_OFFLINE HPCPERF_NYX_TOOLS_DIR HPCPERF_NYX_SKIP_PARTICLES HPCPERF_NYX_CASES HPCPERF_NYX_REPORT_DIR HPCPERF_NYX_HEATCOOL HPCPERF_NYX_PROFILE HPCPERF_NYX_CPU_PROFILE HPCPERF_L3_RUN_SUBDIR
+else
+    echo "# validate.sh not found -- Part B skipped"
+fi
+echo "test_nyx_validator: $pass passed, $failn failed"
+[ "$failn" -eq 0 ]
diff --git a/level3/warpx/README.md b/level3/warpx/README.md
new file mode 100644
index 0000000..f14cbfe
--- /dev/null
+++ b/level3/warpx/README.md
@@ -0,0 +1,156 @@
+# WarpX (Level 3)
+
+Full 3D electromagnetic particle-in-cell: charge/current deposition,
+Yee/FDTD Maxwell solve, Boris push, particle and guard-cell exchange -- the
+complete application driven by its own inputs files, on the GPU through AMReX.
+
+## Provenance
+
+- Official repository: **https://github.com/BLAST-WarpX/warpx** (the former
+  ECP-WarpX/WarpX URL redirects there); docs https://warpx.readthedocs.io/
+- Release policy: monthly `YY.MM` tags; the pinned AMReX release is recorded
+  in `dependencies.json`.
+- Selected: **WarpX 26.09** (2026-09-03), commit
+  `0c62c75e53a9ad08241535444bd7e53fd1deba88`, with **AMReX 26.09**
+  `a52ca73324ac2c7b65ec04f131e6df99eec9c576` (the exact tag WarpX 26.09
+  pins). Both fetched by `fetch.sh` into `_upstream/level3/{WarpX,amrex}`
+  (shallow, read-only).
+- License: BSD-3-Clause-LBNL (`LICENSE.txt`, `LEGAL.txt`).
+- Application-owned LOC (cloc 2.06, code lines): `Source/` **112,459**
+  (C++ 69,266 in 247 files; headers 37,146). AMReX `Src/` 273,313 counted
+  separately (dependency, not modified). No third-party source is vendored
+  in-tree; with the build options below nothing is downloaded at configure
+  time.
+
+## Build strategy: NATIVE (upstream CMake superbuild, local AMReX source)
+
+`build.sh CUDA` = upstream's documented CMake route:
+`-DWarpX_COMPUTE=CUDA -DCMAKE_CUDA_ARCHITECTURES=100 -DWarpX_DIMS=3
+-DWarpX_MPI=ON -DWarpX_amrex_src=<AMReX 26.09 checkout> -DWarpX_OPENPMD=OFF
+-DWarpX_QED=OFF -DWarpX_PYTHON=OFF -DWarpX_FFT=OFF -DWarpX_APP=ON
+-DWarpX_LIB=OFF -DBUILD_TESTING=OFF`, host compiler conda GCC 13.3.0 (upstream
+requires GCC 12+ / NVCC 12.4+; upstream's own Perlmutter profile pairs GCC 13
+with NVCC 13.2.78), conda Open MPI 5.0.10 (CUDA-aware), CMake 3.28.4 >= 3.25,
+Ninja. openPMD (needs HDF5/ADIOS2 for useful output; AMReX plotfiles are
+produced without it), QED (PICSAR download) and Python are off for the
+bring-up; they are documented options, not modifications. Build time on
+dgx003: **1219 s (20.3 min)** at `-j32` (367 targets, AMReX built by the
+superbuild), **0 compiler warning lines**; executable
+`build/level3/warpx/cuda/bin/warpx.3d.MPI.CUDA.DP.PDP.EB` (726 MB), installed
+under `.deps/level3/warpx/install` with the fingerprint (upstream commit,
+AMReX commit, compiler, CUDA 13.2.78, MPI, CMake options).
+
+Why not the others: the Spack `warpx` recipe stops at 26.08 and takes the
+architecture only through the legacy `^amrex cuda_arch=` path; the local Spack
+checkout is 2025-05 (warpx 25.04); no Apptainer on the node and the only
+upstream container recipes are Perlmutter-specific (sm_80); site modules are
+broken. HIP: `build.sh HIP` carries `-DWarpX_COMPUTE=HIP -DAMReX_AMD_ARCH=gfx950`
+and exits with a clear message here (no ROCm). **Untested.**
+
+## Changes from upstream
+
+Class **A -- no source modification.** `run.sh` writes a derived inputs file
+into the build tree: upstream's physics/numerics lines verbatim, plus
+`warpx.numprocs` (one box per rank), the per-mode `amr.n_cell` /
+`amr.max_grid_size` / `max_step`, `warpx.random_seed = 1`, and reduced
+diagnostics instead of the plotfile/checkpoint diagnostics (I/O). For the
+validation case only the openPMD diagnostic is dropped (openPMD not built).
+
+## Execution model
+
+One MPI rank per GPU (AMReX docs: "MPI ranks == Number of GPUs"). The common
+launcher's per-rank wrapper gives each rank one visible GPU; AMReX then binds
+device 0, and the launcher audits the mapping (4/4 verified at 4 GPUs).
+`warpx.numprocs PX PY PZ` decomposes the domain into exactly one box per
+rank, so the requested rank count is the decomposition (`PX*PY*PZ == ranks`,
+each per-rank extent a multiple of the blocking factor; otherwise the run is
+refused, nothing silently changed). GPU-aware MPI: AMReX auto-detects the
+CUDA-aware Open MPI (`MPIX_Query_cuda_support`) and uses device buffers;
+`HPCPERF_WARPX_GPU_AWARE=0` forces pinned-host staging (see the measurement
+below). CPU binding: runtime default (no OpenMP threads used).
+
+## Inputs (`HPCPERF_SCALE_MODE`, case `uniform_plasma`)
+
+| Mode | Grid (cells) | Macroparticles (2/cell) | Per rank @4 GPU | Topology (numprocs) | Steps | Memory/GPU (est.) | Time/step on B200 | Validation quantity |
+|---|---|---|---|---|---|---|---|---|
+| smoke (default) | 64x32x32 (upstream) | 131,072 | 32,768 | 2x2x1 | 10 | < 0.1 GB | 0.02 s | particle number (exact) + energies recorded |
+| strong | G^3, G=`HPCPERF_WARPX_GLOBAL` (256) | 33,554,432 | 8,388,608 | 2x2x1 (128x128x256 boxes) | 20 | ~4 GB | 0.029 s (1 GPU) / 0.081 s (4 GPU, GPU-aware MPI) / 0.026 s (4 GPU, host-staged) | run completes; reduced diagnostics |
+| weak | (L*PX)x(L*PY)x(L*PZ), L=`HPCPERF_WARPX_LOCAL` (128) | 4,194,304 x N | 4,194,304 | `hpcperf_topology.py` grid | 20 | ~1 GB | 0.050 s (4 GPU) | run completes |
+
+Memory estimate ~100 B per particle plus 6 field components (double) per cell
+-- far below 180 GB at all sizes. The strong default is a correctness/bring-up
+size (33.6M particles, 20 steps in under a second); per-step times come from
+WarpX's own `Evolve time ... Avg. per step` output and exclude initialisation
+(`warpx.serialize_initial_conditions = 1`, upstream default in this deck,
+serialises particle initialisation across ranks, so `Total Time` grows with
+the rank count and must not be read as a scaling number).
+
+**Transport observation (single node, `pml ob1 / btl self,sm,smcuda`):** the
+4-GPU strong run takes 0.081 s/step with AMReX's auto-enabled GPU-aware MPI
+and 0.026 s/step with `amrex.use_gpu_aware_mpi = 0`; the 1-GPU run takes
+0.029 s/step. Device-buffer MPI through `smcuda` is therefore the bottleneck
+for WarpX's halo/particle exchange on this node (LAMMPS shows the opposite:
+`gpu/aware on` 2.38 s vs `off` 5.87 s; SPARTA is indifferent). Recorded as a
+site/transport finding; the default stays upstream's (auto-detect), correctness
+is unaffected.
+
+## Validation (`validate.sh`, upstream mechanism)
+
+Upstream's regression mechanism is checksums with baselines that upstream
+documents as architecture-dependent, so they cannot serve as a reference on a
+B200. `validate.sh` therefore runs upstream's **analytic** regression test
+`test_3d_langmuir_multi` (`Examples/Tests/langmuir/inputs_base_3d`: electron
++ positron Langmuir wave, 64^3 cells, 524,288 particles, 40 steps) and
+re-implements `analysis_3d.py`'s checks on the final plotfile (read directly;
+upstream's script needs yt/openPMD-viewer, absent here):
+
+1. `max|E_sim - E_th| / max|E_th| < 5e-2` for Ex, Ey, Ez against the
+   analytic solution `E = eps m_e c^2 k/e sin(kx) cos(ky) cos(kz) sin(wp t)`
+   (and cyclic), evaluated at cell centres exactly as upstream does;
+2. charge conservation `max|divE - rho/eps0| / max|rho/eps0| < 1e-11`
+   (upstream's tolerance for Esirkepov deposition), with WarpX's own CODATA
+   2022 constants (`Source/ablastr/constant.H`; using CODATA 2018 eps0 would
+   show a spurious 6.8e-10 offset);
+3. the `uniform_plasma` smoke run: macroparticle number constant at every
+   step (exact); the particle+field energy series is recorded for information
+   only -- the shipped 2-particles-per-cell thermal plasma with zero initial
+   fields is not an energy-conservation test for the momentum-conserving Yee
+   scheme (a few percent change in the first plasma periods is expected).
+
+Observed on dgx003 (2026-09-04): **PASS at 1, 2 and 4 GPUs** --
+`error_rel = 3.351e-02` for all three components and all three rank counts
+(the decomposition does not change the result), charge-conservation residual
+1.8e-12 / 1.9e-12 / 1.3e-12, particle number 131,072 at every step; recorded
+energy change -3.45 % / -3.56 % / -3.47 % over 10 steps (initial energies
+differ by the per-rank random sampling, as expected).
+
+## Results on dgx003 (4x B200, CUDA 13.2.78, Slurm job 9552083)
+
+| Run | Ranks x GPUs | rank->GPU | CPU binding | Topology | Problem | Time | Validation |
+|---|---|---|---|---|---|---|---|
+| langmuir | 1 x 1 | wrapper; audit 1/1 verified | runtime default | 1x1x1 | 64^3, 524k particles, 40 steps | Total 0.58 s | PASS |
+| langmuir | 2 x 2 | wrapper; 2/2 verified | runtime default | 2x1x1 | same | Total 0.70 s | PASS |
+| langmuir | 4 x 4 | wrapper; 4/4 verified | runtime default | 2x2x1 | same | Total 0.67 s | PASS |
+| smoke | 1/2/4 | wrapper; verified where sampled | runtime default | 1x1x1 / 2x1x1 / 2x2x1 | 64x32x32, 131k particles, 10 steps | Total 0.21-0.35 s | particle number exact |
+| strong | 1 x 1 | wrapper; 1/1 verified | runtime default | 1x1x1 | 256^3, 33.6M particles, 20 steps | 0.0289 s/step | completes |
+| strong | 4 x 4 | wrapper; 4/4 verified | runtime default | 2x2x1 | 256^3, 33.6M particles | 0.0811 s/step (GPU-aware) / 0.0258 s/step (host-staged) | completes |
+| weak | 4 x 4 | wrapper; 4/4 verified | runtime default | 2x2x1 | 256x256x128, 16.8M particles (4.2M/rank) | 0.0500 s/step | completes |
+
+Dry-runs (`HPCPERF_DRY_RUN=1`, hypothetical allocations) -- **DRY-RUN /
+UNVALIDATED**, nothing executed:
+
+| GPUs | Nodes x GPUs/node | Mode | Grid | Particles | Per rank | numprocs | Launch |
+|---|---|---|---|---|---|---|---|
+| 8 | 1 x 8 | strong | 256^3 | 33.6M | 4.2M | 2x2x2 (128^3 boxes) | `mpirun -np 8 --host dgx003:8 --map-by ppr:8:node ...` (single node) |
+| 40 | 5 x 8 | weak | 640x512x256 | 167.8M | 4.2M | 5x4x2 | 5 nodes x 8 -- multi-node BLOCKED on this site |
+| 80 | 10 x 8 | weak | 640x512x512 | 335.5M | 4.2M | 5x4x4 | 10 nodes x 8 -- multi-node BLOCKED on this site |
+
+## Limitations
+
+- Multi-node: BLOCKED/UNVERIFIED on this site; 40/80-GPU shapes are plans.
+- HIP: recipe present, untested (no AMD GPU); no gfx950 statement upstream.
+- openPMD/HDF5 output, QED and Python bindings are not built (documented
+  options, off for bring-up).
+- Upstream's checksum baselines are not used (platform-dependent by
+  upstream's own statement); validation is the analytic Langmuir test plus
+  charge and particle conservation.
diff --git a/level3/warpx/build.sh b/level3/warpx/build.sh
new file mode 100755
index 0000000..9c3e841
--- /dev/null
+++ b/level3/warpx/build.sh
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+# Build WarpX (3D, MPI, CUDA or HIP) with upstream's native CMake superbuild,
+# using the AMReX 26.09 checkout WarpX pins (no configure-time downloads).
+#
+#   ./build.sh [CUDA|HIP]        (default CUDA)
+#
+# Layout (Level 3 isolation): sources _upstream/level3/{WarpX,amrex}, build
+# build/level3/warpx/<cuda|hip>, install .deps/level3/warpx/install
+# (+ .hpcperf-l3-fingerprint), logs .deps/level3/warpx/logs. AMReX is built
+# by WarpX's superbuild from the local source (-DWarpX_amrex_src) -- it is
+# WarpX's private copy, nothing is shared with other Level 3 applications.
+#
+# Configuration (bring-up subset of the documented options): WarpX_COMPUTE=CUDA,
+# CMAKE_CUDA_ARCHITECTURES=100 (sm_100), WarpX_DIMS=3, WarpX_MPI=ON,
+# WarpX_OPENPMD=OFF (plotfile output only; no HDF5/ADIOS2), WarpX_QED=OFF
+# (PICSAR-QED not needed by the uniform-plasma/laser benchmarks; avoids a
+# download), WarpX_PYTHON=OFF, WarpX_FFT=OFF (no PSATD). Host compiler conda
+# GCC 13.3.0 (upstream: GCC 12+, NVCC 12.4+; upstream's Perlmutter profile
+# uses GCC 13 with NVCC 13.2.78). Modification class: A (build options only).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+l3_isolate_build_env    # Level 3 builds must not see Level 2 .deps/install prefixes
+
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+SRC="$R/_upstream/level3/WarpX"; AMREX="$R/_upstream/level3/amrex"
+[ -f "$SRC/CMakeLists.txt" ] && [ -f "$AMREX/CMakeLists.txt" ] || { echo "build.sh: sources missing -- run $HERE/fetch.sh first" >&2; exit 1; }
+SHA="$(git -C "$SRC" rev-parse HEAD)"; AMREX_SHA="$(git -C "$AMREX" rev-parse HEAD)"
+l3_paths warpx
+BUILD_DIR="$R/build/level3/warpx/$MODEL"
+JOBS="${HPCPERF_BUILD_JOBS:-32}"
+
+case "$BACKEND" in
+    CUDA)
+        ARCH="${HPCPERF_CUDA_ARCH:-$(l3_gpu_arch)}"
+        GPU_FLAGS=(-DWarpX_COMPUTE=CUDA "-DCMAKE_CUDA_ARCHITECTURES=$ARCH" "-DCMAKE_CUDA_HOST_COMPILER=$CXX")
+        ARCHNOTE="sm_$ARCH" ;;
+    HIP)
+        command -v hipcc >/dev/null 2>&1 || { echo "build.sh: HIP requested but hipcc not found -- HIP build is UNTESTED on this machine (no ROCm)" >&2; exit 1; }
+        ARCH="${HPCPERF_HIP_ARCH:-gfx950}"
+        GPU_FLAGS=(-DWarpX_COMPUTE=HIP "-DAMReX_AMD_ARCH=$ARCH" -DCMAKE_CXX_COMPILER=hipcc)
+        ARCHNOTE="$ARCH" ;;
+    *) echo "usage: $0 [CUDA|HIP]" >&2; exit 2 ;;
+esac
+
+CMAKE_OPTS="WarpX_COMPUTE=$BACKEND arch=$ARCHNOTE WarpX_DIMS=3 WarpX_MPI=ON WarpX_OPENPMD=OFF WarpX_QED=OFF WarpX_PYTHON=OFF WarpX_FFT=OFF WarpX_amrex_src=local BUILD_TESTING=OFF"
+FP="$(l3_fingerprint_text warpx "$SHA" "$MODEL" "amrex=26.09($AMREX_SHA) picsar-qed=off openpmd=off" "$CMAKE_OPTS" "runtime(amrex.use_gpu_aware_mpi auto)")"
+l3_fingerprint_check "$L3_INSTALL" "$FP" || exit 1
+
+echo "# WarpX $BACKEND: upstream $SHA, AMReX $AMREX_SHA (26.09), arch $ARCHNOTE, MPI $(mpirun --version 2>/dev/null | head -1)"
+mkdir -p "$BUILD_DIR"
+cmake -S "$SRC" -B "$BUILD_DIR" -G Ninja \
+    -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="$L3_INSTALL" \
+    -DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX" \
+    -DWarpX_DIMS=3 -DWarpX_MPI=ON -DWarpX_OPENPMD=OFF -DWarpX_QED=OFF -DWarpX_PYTHON=OFF -DWarpX_FFT=OFF \
+    -DWarpX_APP=ON -DWarpX_LIB=OFF -DWarpX_amrex_src="$AMREX" -DBUILD_TESTING=OFF \
+    "${GPU_FLAGS[@]}" > "$L3_LOGS/configure-$MODEL.log" 2>&1 \
+    || { tail -40 "$L3_LOGS/configure-$MODEL.log"; echo "build.sh: configure failed (log: $L3_LOGS/configure-$MODEL.log)" >&2; exit 1; }
+t0=$(date +%s)
+cmake --build "$BUILD_DIR" -j "$JOBS" > "$L3_LOGS/build-$MODEL.log" 2>&1 \
+    || { tail -40 "$L3_LOGS/build-$MODEL.log"; echo "build.sh: build failed (log: $L3_LOGS/build-$MODEL.log)" >&2; exit 1; }
+cmake --install "$BUILD_DIR" > "$L3_LOGS/install-$MODEL.log" 2>&1 || { echo "build.sh: install failed" >&2; exit 1; }
+l3_fingerprint_write "$L3_INSTALL" "$FP"
+EXE="$(find "$BUILD_DIR/bin" -maxdepth 1 -name 'warpx.3d*' -type f 2>/dev/null | head -1)"
+echo "# built in $(( $(date +%s)-t0 )) s: ${EXE:-<warpx.3d* not found under $BUILD_DIR/bin>} (installed under $L3_INSTALL)"
+echo "# compiler warning lines: $(grep -c 'warning' "$L3_LOGS/build-$MODEL.log" || true)"
diff --git a/level3/warpx/fetch.sh b/level3/warpx/fetch.sh
new file mode 100755
index 0000000..2838cef
--- /dev/null
+++ b/level3/warpx/fetch.sh
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Fetch WarpX and the AMReX release it pins into _upstream/level3/{WarpX,amrex}
+# (gitignored, read-only). WarpX 26.09 pins AMReX 26.09 (cmake/dependencies/
+# AMReX.cmake / dependencies.json); the AMReX checkout is used as
+# -DWarpX_amrex_src so no configure-time download is needed. Idempotent; a
+# checkout at another commit is an error, never silently reused.
+#
+#   ./fetch.sh
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+# Official repository (ECP-WarpX/WarpX redirects here since the project moved).
+WARPX_URL="https://github.com/BLAST-WarpX/warpx.git"; WARPX_TAG="26.09"; WARPX_SHA="0c62c75e53a9ad08241535444bd7e53fd1deba88"
+AMREX_URL="https://github.com/AMReX-Codes/amrex.git";  AMREX_TAG="26.09"; AMREX_SHA="a52ca73324ac2c7b65ec04f131e6df99eec9c576"
+
+fetch_one() { # url tag sha dest
+    local url=$1 tag=$2 sha=$3 dest=$4 have
+    if [ -d "$dest/.git" ]; then
+        have="$(git -C "$dest" rev-parse HEAD)"
+        [ "$have" = "$sha" ] && { echo "fetch.sh: $dest already at $tag ($sha)"; return 0; }
+        echo "fetch.sh: $dest is at $have, not the recorded $sha ($tag); remove it to re-fetch" >&2; return 1
+    fi
+    mkdir -p "$(dirname "$dest")"
+    echo "fetch.sh: cloning $url @ $tag (shallow)"
+    git clone --quiet --depth 1 --branch "$tag" "$url" "$dest"
+    have="$(git -C "$dest" rev-parse HEAD)"
+    [ "$have" = "$sha" ] || { echo "fetch.sh: tag $tag resolved to $have, expected $sha" >&2; return 1; }
+    echo "fetch.sh: ok -> $dest ($sha)"
+}
+fetch_one "$WARPX_URL" "$WARPX_TAG" "$WARPX_SHA" "$R/_upstream/level3/WarpX"
+fetch_one "$AMREX_URL" "$AMREX_TAG" "$AMREX_SHA" "$R/_upstream/level3/amrex"
diff --git a/level3/warpx/run.sh b/level3/warpx/run.sh
new file mode 100755
index 0000000..0a0d0d0
--- /dev/null
+++ b/level3/warpx/run.sh
@@ -0,0 +1,134 @@
+#!/usr/bin/env bash
+# Run WarpX (full 3D electromagnetic PIC: deposition, Maxwell/Yee solve,
+# particle push, periodic halo/particle exchange) on N GPUs.
+#
+#   ./run.sh [CUDA|HIP] [extra WarpX inputs overrides...]
+#
+# Cases (HPCPERF_WARPX_CASE):
+#   uniform_plasma (default) -- Examples/Physics_applications/uniform_plasma/
+#       inputs_base_3d, upstream's "commonly used to study performance" case:
+#       thermal electron plasma, 2 particles/cell. Sized by HPCPERF_SCALE_MODE.
+#   langmuir -- Examples/Tests/langmuir/inputs_base_3d (test_3d_langmuir_multi):
+#       electron/positron Langmuir wave with an analytic solution, 64^3 cells,
+#       40 steps; the correctness case used by validate.sh (sizes fixed).
+#
+# Execution model (AMReX/WarpX docs): one MPI rank per GPU. The common
+# launcher's per-rank wrapper gives each rank exactly one visible GPU (AMReX
+# then binds device 0 = that GPU; mapping audited). `warpx.numprocs PX PY PZ`
+# assigns exactly one box per rank, so the requested rank count is what is
+# decomposed: PX*PY*PZ must equal HPCPERF_GPUS and every dimension of
+# amr.n_cell must be divisible by PX*blocking_factor -- otherwise the run is
+# refused with the nearby legal rank counts (nothing is silently changed).
+# GPU-aware MPI: AMReX auto-detects it from the (CUDA-aware) Open MPI;
+# HPCPERF_WARPX_GPU_AWARE=0 forces host-staged communication.
+#
+# Resource / size controls (uniform_plasma):
+#   HPCPERF_GPUS=N|all        ranks = GPUs (default 1)
+#   HPCPERF_SCALE_MODE        smoke | strong | weak   (default smoke)
+#     smoke  : upstream grid 64x32x32 cells (131,072 macroparticles), 10 steps
+#     strong : ONE fixed global grid G^3 (G=HPCPERF_WARPX_GLOBAL, default 256:
+#              16.8M cells, 33.6M particles), decomposed over the ranks
+#     weak   : fixed per-rank block L^3 (L=HPCPERF_WARPX_LOCAL, default 128:
+#              2.1M cells, 4.2M particles per rank); grid L*PX x L*PY x L*PZ
+#   HPCPERF_WARPX_STEPS       time steps (default 10; strong/weak 20)
+#   HPCPERF_WARPX_GPU_AWARE   1|0 (default: AMReX auto-detect)
+#
+# Derived inputs (class A, written into the build tree; upstream files
+# untouched): upstream physics/numerics lines are kept verbatim;
+# uniform_plasma drops the plotfile/checkpoint diagnostics (I/O), sets
+# amr.n_cell / amr.max_grid_size / max_step / warpx.numprocs per mode,
+# warpx.random_seed = 1 (reproducible sampling) and adds reduced diagnostics
+# (ParticleEnergy, FieldEnergy, ParticleNumber); langmuir keeps upstream's
+# diag1 plotfile (the analysis input), drops only the openPMD diagnostic
+# (openPMD is not built) and adds warpx.numprocs. stdout is copied to
+# <run_dir>/stdout.log.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; # shellcheck disable=SC1091
+source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"; [ $# -gt 0 ] && shift
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+BUILD_DIR="$R/build/level3/warpx/$MODEL"
+EXE="$(find "$BUILD_DIR/bin" -maxdepth 1 -name 'warpx.3d*' -type f 2>/dev/null | head -1)"
+[ -n "$EXE" ] && [ -x "$EXE" ] || { echo "run.sh: warpx.3d* not found under $BUILD_DIR/bin -- run ./build.sh $BACKEND first" >&2; exit 1; }
+CASE="${HPCPERF_WARPX_CASE:-uniform_plasma}"
+case "$CASE" in
+    uniform_plasma) BASE="$R/_upstream/level3/WarpX/Examples/Physics_applications/uniform_plasma/inputs_base_3d" ;;
+    langmuir)       BASE="$R/_upstream/level3/WarpX/Examples/Tests/langmuir/inputs_base_3d" ;;
+    *) echo "run.sh: HPCPERF_WARPX_CASE must be uniform_plasma or langmuir" >&2; exit 2 ;;
+esac
+[ -f "$BASE" ] || { echo "run.sh: $BASE missing (run fetch.sh)" >&2; exit 1; }
+
+N_RANKS="$(hpcperf_ranks warpx yes)" || exit 2
+hpcperf_forbid_args warpx amr.n_cell amr.max_grid_size amr.blocking_factor warpx.numprocs max_step warpx.random_seed -- "$@" || exit 2
+
+if [ "$CASE" = langmuir ]; then
+    MODE=validate; NX=64; NY=64; NZ=64; STEPS=40; BF=8      # upstream test as shipped (default blocking factor)
+    TOPO="$(hpcperf_topology warpx "$N_RANKS" --divides "$((NX / BF)),$((NY / BF)),$((NZ / BF))")" || exit 2
+else
+    MODE="$(l3_scale_mode warpx)" || exit 2
+    BF=16
+    case "$MODE" in
+        smoke)  NX=64; NY=32; NZ=32; STEPS="${HPCPERF_WARPX_STEPS:-10}" ;;
+        strong) G="${HPCPERF_WARPX_GLOBAL:-256}"; NX=$G; NY=$G; NZ=$G; STEPS="${HPCPERF_WARPX_STEPS:-20}" ;;
+        weak)   L="${HPCPERF_WARPX_LOCAL:-128}"; STEPS="${HPCPERF_WARPX_STEPS:-20}"
+                [ $((L % BF)) -eq 0 ] || { echo "run.sh: HPCPERF_WARPX_LOCAL=$L must be a multiple of the blocking factor $BF" >&2; exit 2; } ;;
+    esac
+    if [ "$MODE" = weak ]; then
+        TOPO="$(hpcperf_topology warpx "$N_RANKS")" || exit 2
+        read -r PX PY PZ <<< "$TOPO"; NX=$((L * PX)); NY=$((L * PY)); NZ=$((L * PZ))
+    else
+        TOPO="$(hpcperf_topology warpx "$N_RANKS" --divides "$((NX / BF)),$((NY / BF)),$((NZ / BF))")" || {
+            echo "run.sh: $N_RANKS ranks cannot tile the ${NX}x${NY}x${NZ} grid into one ${BF}-aligned box per rank (see feasible counts above)" >&2; exit 2; }
+    fi
+fi
+read -r PX PY PZ <<< "$TOPO"
+BX=$((NX / PX)); BY=$((NY / PY)); BZ=$((NZ / PZ))
+MGS=$BX; [ "$BY" -gt "$MGS" ] && MGS=$BY; [ "$BZ" -gt "$MGS" ] && MGS=$BZ
+CELLS=$((NX * NY * NZ))
+if [ "$CASE" = langmuir ]; then PARTS=$((2 * CELLS)); else PARTS=$((2 * CELLS)); fi
+
+# l3_rundir: real runs get a fresh dir; a dry-run gets a throwaway .dryrun/ dir so it
+# can never delete or overwrite a real result directory (the old code rm -rf'd the real
+# dir before the launcher's dry-run check ever ran).
+RUN_DIR="$(l3_rundir "$BUILD_DIR/$L3_RUN_SUBDIR/$CASE.$MODE.np$N_RANKS")" || exit 2
+IN="$RUN_DIR/inputs"
+{
+    echo "# derived from upstream $(realpath --relative-to="$R/_upstream/level3/WarpX" "$BASE") (HPC-Performance-AI level3/warpx/run.sh)"
+    if [ "$CASE" = langmuir ]; then
+        grep -vE '^\s*(amr\.max_grid_size|diagnostics\.diags_names|openpmd\.)' "$BASE"
+        echo "diagnostics.diags_names = diag1"
+        echo "amr.max_grid_size = $MGS"
+    else
+        grep -vE '^\s*(max_step|amr\.n_cell|amr\.max_grid_size|amr\.blocking_factor|diagnostics\.|diag1\.|chk\.)' "$BASE"
+        echo "max_step = $STEPS"
+        echo "amr.n_cell = $NX $NY $NZ"
+        echo "amr.max_grid_size = $MGS"
+        echo "amr.blocking_factor = $BF"
+        echo "warpx.random_seed = 1"
+        # no plotfile/checkpoint diagnostics (the upstream diag1/chk lines were dropped above)
+        echo "warpx.reduced_diags_names = EP EF NP"
+        echo "EP.type = ParticleEnergy"; echo "EP.intervals = 1"
+        echo "EF.type = FieldEnergy";    echo "EF.intervals = 1"
+        echo "NP.type = ParticleNumber"; echo "NP.intervals = 1"
+    fi
+    echo "warpx.numprocs = $PX $PY $PZ"
+    [ -n "${HPCPERF_WARPX_GPU_AWARE:-}" ] && echo "amrex.use_gpu_aware_mpi = $HPCPERF_WARPX_GPU_AWARE"
+} > "$IN"
+
+echo "# WarpX $BACKEND: case=$CASE mode=$MODE ranks=$N_RANKS grid=${NX}x${NY}x${NZ} ($CELLS cells, $PARTS particles, $((PARTS / N_RANKS))/rank) numprocs=${PX}x${PY}x${PZ} box=${BX}x${BY}x${BZ} steps=$STEPS run_dir=$RUN_DIR"
+cd "$RUN_DIR"
+RUN_ID="$(l3_run_id)"
+"$L3_LAUNCHER" --gpus "$N_RANKS" --bind wrapper -- "$EXE" "$IN" "$@" 2>&1 | tee "$RUN_DIR/stdout.log"
+rc=${PIPESTATUS[0]}
+if [ -z "${HPCPERF_DRY_RUN:-}" ]; then
+    l3_manifest "$RUN_DIR" "run_id=$RUN_ID" "app=warpx" "backend=$BACKEND" "case=$CASE" "mode=$MODE" \
+        "ranks=$N_RANKS" "grid=${NX}x${NY}x${NZ}" "numprocs=${PX}x${PY}x${PZ}" "particles=$PARTS" "steps=$STEPS" \
+        "exit_code=$rc" "binary=$EXE" "binary_sha256=$(l3_sha_file "$EXE")" "input=$IN" "input_sha256=$(l3_sha_file "$IN")" \
+        "stdout=$RUN_DIR/stdout.log" "utc=$(date -u +%FT%TZ)"
+fi
+exit "$rc"
diff --git a/level3/warpx/validate.sh b/level3/warpx/validate.sh
new file mode 100755
index 0000000..a9710b7
--- /dev/null
+++ b/level3/warpx/validate.sh
@@ -0,0 +1,137 @@
+#!/usr/bin/env bash
+# Correctness check for WarpX on N GPUs, using upstream's analytic Langmuir-wave
+# regression test plus an invariant of the performance case.
+#
+#   ./validate.sh [CUDA|HIP]        HPCPERF_GPUS=N (default 1)
+#
+# [1] test_3d_langmuir_multi: an e-/e+ plasma wave with an analytic field
+#     solution. Adapted subset of upstream analysis_3d.py (which needs
+#     yt/openPMD-viewer, not installed): the final plotfile's Ex/Ey/Ez are
+#     compared with the exact solution -- require max|E_sim-E_th|/max|E_th|<5e-2
+#     each -- and (Esirkepov) charge conservation max|divE-rho/eps0|/
+#     max|rho/eps0|<1e-11, with WarpX's own CODATA-2022 constants. Reader is
+#     hardened: it FAILs on a missing/short plotfile, a box set that does not
+#     cover the domain (truncation), or any non-finite field value.
+# [2] uniform_plasma smoke: macroparticle count constant AND finite at every
+#     step, and the run must reach the final step; the energy series is recorded
+#     for information only (not a pass/fail criterion, see header of run.sh).
+# Reproducibility: the run's real exit code is captured (nonzero/timeout ->
+# FAIL); run.sh writes a fresh per-run directory (dry-run never touches it).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+R="$(cd "$HERE/../.." && pwd)"
+set +u; source "$R/hpcperf_env.sh" 2>/dev/null || true; set -u
+# shellcheck disable=SC1091
+source "$R/level3/tools/l3_common.sh"
+BACKEND="$(echo "${1:-CUDA}" | tr '[:lower:]' '[:upper:]')"
+MODEL="$(echo "$BACKEND" | tr '[:upper:]' '[:lower:]')"
+N="${HPCPERF_GPUS:-1}"
+RUNS="$R/build/level3/warpx/$MODEL/$L3_RUN_SUBDIR"
+TIMEOUT="${HPCPERF_VALIDATE_TIMEOUT:-1800}"
+python3 -c 'import numpy' 2>/dev/null || { echo "validate.sh: python3 with numpy required for the plotfile analysis" >&2; exit 1; }
+export HPCPERF_GPUS="$N"
+ok=1
+
+run_case() { local case=$1 mode=$2 out=$3 rc=0; HPCPERF_WARPX_CASE="$case" HPCPERF_SCALE_MODE="$mode" timeout "$TIMEOUT" "$HERE/run.sh" "$BACKEND" > "$out" 2>&1 || rc=$?; return $rc; }
+
+echo "validate.sh: [1] WarpX $BACKEND langmuir_multi (64^3, 40 steps, analytic solution) on $N GPU(s)"
+mkdir -p "$RUNS"; L1="$RUNS/validate.langmuir.np$N.stdout"
+rc=0; run_case langmuir validate "$L1" || rc=$?
+grep -aE '^#|hpcperf-launch: audit summary|Total Time|ERROR|abort' "$L1" || true
+if [ "$rc" -eq 124 ]; then echo "validate.sh: FAIL -- langmuir run timed out after ${TIMEOUT}s"; exit 1; fi
+[ "$rc" -eq 0 ] || { echo "validate.sh: FAIL -- langmuir run.sh exited $rc (see $L1)"; exit 1; }
+PLT="$RUNS/langmuir.validate.np$N/diags/diag1000040"
+[ -f "$PLT/Header" ] || { echo "validate.sh: FAIL -- plotfile $PLT not produced (run did not reach step 40)"; exit 1; }
+python3 - "$PLT" <<'PY' || ok=0
+import sys, re, os
+import numpy as np
+sys.path.insert(0, os.environ["L3_TOOLS"])
+from l3_check import require_finite, ValidationError
+c, e, epsilon_0, m_e = 299792458.0, 1.602176634e-19, 8.8541878188e-12, 9.1093837139e-31
+plt = sys.argv[1]
+try:
+    hdr = open(f"{plt}/Header").read().split("\n")
+    ncomp = int(hdr[1]); names = hdr[2:2 + ncomp]; i = 2 + ncomp
+    time = float(hdr[i + 1]); i += 3
+    lo = [float(v) for v in hdr[i].split()]; hi = [float(v) for v in hdr[i + 1].split()]
+    dom = re.search(r"\(\((\d+),(\d+),(\d+)\) \((\d+),(\d+),(\d+)\)", hdr[i + 3])
+    n = [int(dom.group(k + 4)) - int(dom.group(k + 1)) + 1 for k in range(3)]
+    for fld in ("Ex", "Ey", "Ez", "rho", "divE"):
+        if fld not in names: raise ValidationError(f"plotfile missing field '{fld}' (have {names})")
+    ch = open(f"{plt}/Level_0/Cell_H").read().split("\n")
+    boxes = [tuple(int(v) for v in m.groups()) for m in re.finditer(r"\(\((-?\d+),(-?\d+),(-?\d+)\) \((-?\d+),(-?\d+),(-?\d+)\) \(", "\n".join(ch))]
+    fabs = [(m.group(1), int(m.group(2))) for m in re.finditer(r"FabOnDisk: (\S+) (\d+)", "\n".join(ch))]
+    if not (len(boxes) == len(fabs) > 0): raise ValidationError(f"plotfile box/fab mismatch ({len(boxes)} boxes, {len(fabs)} fabs)")
+    data = np.full((ncomp, n[0], n[1], n[2]), np.nan)   # nan-init: any uncovered cell trips the finite check
+    covered = np.zeros((n[0], n[1], n[2]), bool)
+    for (lx, ly, lz, hx, hy, hz), (fname, off) in zip(boxes, fabs):
+        with open(f"{plt}/Level_0/{fname}", "rb") as f:
+            f.seek(off); line = b""
+            while not line.endswith(b"\n"): line += f.read(1)
+            h = line.decode()
+            order = re.search(r"\(\d+, \((\d)(?: \d){7}\)\)\)", h).group(1)
+            dt = "<f8" if order == "8" else ">f8"
+            nc = int(h.strip().split()[-1])
+            shape = (hx - lx + 1, hy - ly + 1, hz - lz + 1)
+            raw = np.frombuffer(f.read(8 * nc * int(np.prod(shape))), dtype=dt)
+            if raw.size != nc * int(np.prod(shape)): raise ValidationError(f"FAB {fname} truncated: {raw.size} of {nc*int(np.prod(shape))} reals")
+            arr = raw.reshape((nc, shape[2], shape[1], shape[0])).transpose(0, 3, 2, 1)
+            data[:, lx:hx + 1, ly:hy + 1, lz:hz + 1] = arr
+            covered[lx:hx + 1, ly:hy + 1, lz:hz + 1] = True
+    if not covered.all(): raise ValidationError(f"plotfile boxes cover only {covered.mean()*100:.1f}% of the {n} domain (missing boxes)")
+    comp = {nm: data[k] for k, nm in enumerate(names)}
+    epsilon, nden = 0.01, 4.0e24
+    kx, ky, kz = [2.0 * np.pi * 2 / (hi[d] - lo[d]) for d in range(3)]
+    wp = np.sqrt(nden * e**2 / (m_e * epsilon_0))
+    kmap = {"Ex": kx, "Ey": ky, "Ez": kz}; cosf = {"Ex": (0, 1, 1), "Ey": (1, 0, 1), "Ez": (1, 1, 0)}
+    def contrib(is_cos, kk, d):
+        du = (hi[d] - lo[d]) / n[d]; u = lo[d] + du * (0.5 + np.arange(n[d]))
+        return np.cos(kk * u) if is_cos else np.sin(kk * u)
+    def theory(field, t):
+        amp = epsilon * (m_e * c**2 * kmap[field]) / e * np.sin(wp * t); cf = cosf[field]
+        return amp * contrib(cf[0], kx, 0)[:, None, None] * contrib(cf[1], ky, 1)[None, :, None] * contrib(cf[2], kz, 2)[None, None, :]
+    require_finite("plotfile time", time)
+    print(f"    plotfile time t = {time:.6e} s (wp t = {wp*time:.4f}), grid {n}, {len(boxes)} box(es), full coverage")
+    err = 0.0
+    for fld in ("Ex", "Ey", "Ez"):
+        th = theory(fld, time)
+        m = require_finite(f"max|{fld}_sim-{fld}_th|/max|{fld}_th|", abs(comp[fld] - th).max() / abs(th).max())
+        err = max(err, m); print(f"    {fld}: max|E_sim-E_th|/max|E_th| = {m:.3e}")
+    print(f"    error_rel = {err:.3e} (upstream tolerance_rel 5e-2) {'ok' if err < 5e-2 else 'BAD'}")
+    ce = require_finite("charge-conservation residual", np.amax(np.abs(comp["divE"] - comp["rho"]/epsilon_0)) / np.amax(np.abs(comp["rho"]/epsilon_0)))
+    print(f"    charge conservation max|divE-rho/eps0|/max|rho/eps0| = {ce:.3e} (upstream tolerance 1e-11) {'ok' if ce < 1e-11 else 'BAD'}")
+    sys.exit(0 if (err < 5e-2 and ce < 1e-11) else 1)
+except ValidationError as ex:
+    print(f"  VALIDATION ERROR: {ex}"); sys.exit(1)
+PY
+
+echo "validate.sh: [2] WarpX $BACKEND uniform_plasma smoke (64x32x32, 131,072 particles) on $N GPU(s)"
+L2="$RUNS/validate.uniform_plasma.np$N.stdout"
+rc=0; run_case uniform_plasma smoke "$L2" || rc=$?
+grep -aE '^#|hpcperf-launch: audit summary|Total Time|ERROR|abort' "$L2" || true
+if [ "$rc" -eq 124 ]; then echo "validate.sh: FAIL -- uniform_plasma run timed out"; exit 1; fi
+[ "$rc" -eq 0 ] || { echo "validate.sh: FAIL -- uniform_plasma run.sh exited $rc (see $L2)"; ok=0; }
+D="$RUNS/uniform_plasma.smoke.np$N/diags/reducedfiles"
+[ -f "$D/NP.txt" ] || { echo "validate.sh: FAIL -- reduced diagnostics not produced under $D"; exit 1; }
+python3 - "$D" <<'PY' || ok=0
+import sys, os
+sys.path.insert(0, os.environ["L3_TOOLS"])
+from l3_check import require_finite, ValidationError
+d = sys.argv[1]
+def load(p): return [[float(x) for x in l.split()] for l in open(p) if l.strip() and not l.startswith('#')]
+try:
+    npart, ep, ef = load(f"{d}/NP.txt"), load(f"{d}/EP.txt"), load(f"{d}/EF.txt")
+    if not npart: raise ValidationError("NP.txt has no rows")
+    steps = [int(r[0]) for r in npart]
+    if steps[-1] != 10: raise ValidationError(f"uniform_plasma reached step {steps[-1]}, expected final step 10 (incomplete run)")
+    vals = sorted(set(require_finite(f"Np@{int(r[0])}", r[2]) for r in npart))
+    print(f"    ParticleNumber over steps {steps[0]}..{steps[-1]}: {vals} -> {'ok (exact, finite)' if len(vals) == 1 else 'BAD'}")
+    e0 = require_finite("E0", ep[0][2] + ef[0][2]); e1 = require_finite("E1", ep[-1][2] + ef[-1][2])
+    print(f"    for the record: E_particles+E_fields = {e0:.6e} J at step {int(ep[0][0])}, {e1:.6e} J at step {int(ep[-1][0])} (rel change {(e1-e0)/e0:+.3e}; informational, see header)")
+    sys.exit(0 if len(vals) == 1 else 1)
+except ValidationError as ex:
+    print(f"  VALIDATION ERROR: {ex}"); sys.exit(1)
+PY
+
+if [ "$ok" -eq 1 ]; then echo "WarpX $BACKEND validation ($N GPU, langmuir_multi analytic + charge conservation, particle conservation): PASS"; exit 0; fi
+echo "WarpX $BACKEND validation ($N GPU): FAIL"; exit 1
diff --git a/tools/runtime/README.md b/tools/runtime/README.md
new file mode 100644
index 0000000..65ded28
--- /dev/null
+++ b/tools/runtime/README.md
@@ -0,0 +1,34 @@
+# Shared runtime tools -- commonization proposal (not yet moved)
+
+The GPU-count-aware launcher and its helpers currently live under
+`level2/tools/` and are validated there:
+
+| Tool | Role |
+|---|---|
+| `level2/tools/hpcperf_mpi_launch.sh` | allocation parser, `HPCPERF_GPUS` selection, CPU binding, site profiles, GPU binding audit |
+| `level2/tools/mpi_gpu_bind.sh` | scheduler-safe one-GPU-per-rank wrapper |
+| `level2/tools/hpcperf_topology.py` | process-grid helper with constraint checking |
+| `level2/tools/hpcperf_launch_common.sh` | `run.sh` helpers (rank resolution, argument guards) |
+| `level2/tools/site/*.sh` | site profiles (transport, launcher choice) |
+| `level2/tools/tests/` | regression tests |
+
+Level 3 uses the same semantics (`HPCPERF_GPUS=N|all`, `HPCPERF_NODES`,
+`HPCPERF_GPUS_PER_NODE`, `HPCPERF_CPUS_PER_RANK`, `HPCPERF_SCALE_MODE`,
+`HPCPERF_SITE_PROFILE`, `HPCPERF_DRY_RUN=1`) and MUST not fork them.
+
+## Minimal commonization plan
+
+1. **Now (this round):** Level 3 scripts reference the tools through one
+   variable, `HPCPERF_RUNTIME_DIR` (default `level2/tools`), set in
+   `level3/tools/l3_common.sh`. No file is moved or copied; Level 2 is not
+   disturbed.
+2. **Next (separate PR):** `git mv level2/tools/{hpcperf_mpi_launch.sh,
+   mpi_gpu_bind.sh,hpcperf_topology.py,hpcperf_launch_common.sh,site,tests}
+   tools/runtime/`, leave thin forwarding shims at the old paths for one
+   release (`exec "$(dirname "$0")/../../tools/runtime/<name>" "$@"`), switch
+   the `HPCPERF_RUNTIME_DIR` default to `tools/runtime`, re-run
+   `tools/runtime/tests/run_all.sh` plus the Level 2 4-GPU smoke set before
+   merging.
+3. Only then remove the shims.
+
+Until step 2 lands, `tools/runtime/` holds this note only.