Add Level 3 multi-GPU applications and validation safeguards - #4
Draft
bowencui123 wants to merge 25 commits into
Draft
Add Level 3 multi-GPU applications and validation safeguards#4bowencui123 wants to merge 25 commits into
bowencui123 wants to merge 25 commits into
Conversation
added 25 commits
September 4, 2026 21:28
…EM3D/nekRS bring-up STEP 1-2: level3/APPLICATION_AUDIT.md covers all ten candidates (repository, release, commit, license, LOC, build system, GPU model, CUDA/HIP/MPI support, rank->GPU mapping, inputs, correctness mechanism, scaling inputs, build/disk estimates, B200+CUDA 13.2 and MI355X risk, containers, Spack, priority, blockers). level3/BUILD_STRATEGY.md compares Native / Spack / Apptainer / site-native per application and records BUILD_RECOMMENDATION (NATIVE for eight, NATIVE+SPACK_DEPS for CP2K and QMCPACK), the Spack and container policies, per-application dependency isolation and the modification classes used. STEP 3-6: per-application fetch/build/run/validate scripts and README under level3/<app>/, private trees under .deps/level3/<app>/ with schema l3-1 fingerprints (level3/tools/l3_common.sh), upstream sources as read-only shallow clones under _upstream/level3/. All five first-batch applications build natively for CUDA 13.2 / sm_100 and pass their upstream correctness mechanism at 1, 2 and 4 GPUs on dgx003 through the common launcher (one rank per GPU, per-rank GPU wrapper, mapping audited): LAMMPS stable_22Jul2025_update6 thermo vs shipped reference log (identical) SPARTA 27Aug2026 statistical stats vs shipped reference log WarpX 26.09 + AMReX 26.09 analytic Langmuir test + charge conservation SPECFEM3D v4.1.1 reference seismograms via upstream script nekRS v26.0 upstream --cimode CI on the analytic Ethier case Strong and weak inputs are defined and run at 1/4 GPUs; 8/40/80-GPU shapes are launcher dry-runs only (DRY-RUN / UNVALIDATED); multi-node is BLOCKED/ UNVERIFIED on this site; HIP build branches exist and are untested. Compatibility changes (no class E): SPECFEM3D two upstream-devel back-ports (CUDA 13 deviceOverlap guard, Blackwell device block) plus make-time GENCODE and SCOTCH-without-zlib; nekRS HYPRE sm_100 list and CUDA 13 / Thrust 3.2 compatibility (thrust::pair, explicit reverse_iterator/pair headers, not1 -> not_fn), Makefiles generator, OMPI_FC/-fno-lto/-fPIC/unset AR, osc ^ucx and ulimit -s at run time. Level 2 files are untouched. tools/runtime/README.md records the plan to move the shared launcher tools out of level2/tools without breaking Level 2 (Level 3 references them through HPCPERF_RUNTIME_DIR).
…se-solver decision Validators capture the real run exit code (timeout/nonzero/missing output -> FAIL), reject NaN/Inf (level3/tools/l3_check.py), require the complete step/field/trace/check sets, and write a per-run manifest. Dry-runs are routed to a .dryrun/ scratch directory and can no longer clobber real results. Fingerprint schema l3-2 records ordered patch-content hashes; the nekRS source-cache key is SHA + patch series hash. Level 3 builds strip the Level 2 .deps/install prefixes (LAMMPS/SPARTA rebuilt isolated, identical results). CPU-only negative tests: level3/tools/tests/run_all.sh (13/13). nekRS: the Ethier workload runs its HYPRE coarse solve on the CPU by default (cimode 2); the three HYPRE/Thrust patches are needed only for GPU HYPRE, which is now verified separately with cimode 3 (DEVICE coarse, 9/9 at 1/4 GPU, hypregpu variant). A patch-free cpucoarse variant (ENABLE_HYPRE_GPU=OFF) passes cimode 2 at 1/2/4 GPU and rejects a DEVICE request explicitly. Both variants are kept, isolated per variant. See level3/CORRECTNESS_FIXES.md and level3/nekrs/COMPATIBILITY.md.
…the second batch
l3_paths_profile <app> <profile> gives every second-batch configuration its own
.deps/level3/<app>/<profile>/{src,build,install,logs,cache} tree and build
directory (profiles never share a mutable source tree or install);
l3_version_mm derives the profile name components. l3_binary_backend_check now
accepts a CUDA binary that links cudart statically (CMake's default for
AMReX-based applications) when cuobjdump finds embedded device code, instead
of reporting it as non-CUDA. Fingerprint schema unchanged (l3-2).
…/4 GPUs Native CMake build of Nyx 26.09 against a private AMReX 26.09 install (the AMReX commit Nyx pins cannot emit sm_100 through CMake: its convert_cuda_archs drops SM >= 10.0 and autodetects 8.6+PTX on this node; 26.09 is a strict descendant and resolves sm_100 correctly). Profiles cuda132-gcc133-adiabatic and a cpu-gcc133-adiabatic reference (with AMReX plotfile tools and particle_compare); double-precision particles as upstream's regression builds. Cases are upstream's decks: MiniSB (nightly GPU regression test, inputs.32 + ppm_type=0), LyA-adiabatic (inputs.rt.garuda), the 64^3 LyA science deck as a named adiabatic derivative for strong scaling, and the Scaling deck (RandomPerCell, labelled synthetic) for strong/weak. Fixed BoxArray across rank counts; ranks > boxes refused. validate.sh: completeness/finiteness, upstream's fcompare tolerance (2e-10) against a same-configuration rerun (1 GPU) or the 1-GPU run (2/4 GPUs), a CPU-backend reference at a pre-fixed 1e-8, baryon-mass conservation and exact DM counts. AMReX's particle_compare cannot compare across rank counts (header equality incl. next_id; exit code 0 even on "FAIL"), so nyx_particle_compare.py matches particles through their exact t=0 positions (checkpoints at step 0 and the final step) and applies the same norms. Results: VALIDATED_PASS at 1/2/4 GPUs for both official decks (max rel err 1.4e-10 vs 1-GPU, <= 1.8e-10 vs CPU, particles <= 2e-15, mass exact); strong/weak runs completed; 8/40/80 dry-runs planned or refused as designed.
…s; real grep; robust static-cudart check l3_clean_conda_build_env drops the conda env's CFLAGS/CXXFLAGS/LDFLAGS/AR/... (they pin -march=nocona and conda binutils, which broke OpenBLAS's AVX512 kernels and would apply to every system-GCC dependency build). l3_common.sh now unsets an exported `grep` shell function (a ugrep wrapper inherited from interactive sessions changed grep's exit codes inside the scripts) and l3_binary_backend_check captures cuobjdump's output before grepping so the callers' pipefail no longer turns cuobjdump's SIGPIPE on large binaries into a false 'not a CUDA binary'.
… GPUs; synthetic strong deck SUNDIALS probe first (its own CUDA example regression tests: 5/6 pass, the cuSolverSp batched-QR example differs only in integrator statistics and is not on Nyx's CVDiag path), then the staged heatcool profile (SUNDIALS with CVODE + ARKODE -- AMReX 26.09 requires the arkode component -- AMReX with SUNDIALS, Nyx HEATCOOL) plus a CPU heatcool reference. lya_heatcool runs Exec/LyA/inputs.rt as shipped. validate.sh applies per-case tolerances: the adiabatic decks keep 2e-10/1e-8; the heat/cool deck uses upstream's own nightly LyA tolerance 5e-5 and excludes the CVODE rate diagnostic I_R (O(1) noise even between identical runs) while every state variable agrees to <=1.6e-13 (rerun, 2/4 GPU, CPU). run.sh adds a strong mode for the synthetic RandomPerCell deck (fixed 256^3).
… validated at 1/2/4 GPUs Upstream toolchain (install_cp2k_toolchain.sh) with the B200 back-port of upstream commit 378b2fab (--gpu-ver=B200 -> ARCH_NUM 100; DBCSR H100 kernel parameters reused for B200), system GCC 14.2.1 for C/C++/Fortran, conda Open MPI 5.0.10, CUDA 13.2.78, CMAKE_CUDA_ARCHITECTURES=100. DBCSR 2.10.0 verified first (its own ctest suite, 4 ranks x 4 threads on the 4 B200s: 19/19). CP2K validation: adapted subset of upstream's regression tests (regtest-gpw-1 Ar/H2O-geoopt/pyridine, regtest-dm-ls-scf-1 H2-big-1/5) within upstream tolerances at 1, 2 and 4 GPUs; benchmarks/QS/H2O-64.inp GPW-DFT NVE MD (10 steps): all MD-step SCF cycles converged, GRID/DBCSR/PW GPU evidence from CP2K's own output, MD-step FORCE_EVAL energies within 8.7e-12 Ha of the 1-GPU run (pre-fixed tolerance 1e-8 Ha). The deck's own IGNORE_CONVERGENCE_FAILURE (initial ATOMIC-guess SCF, MAX_SCF=50) is reported explicitly and would fail without that declaration.
…on checkers test_l3_validators.sh exercises cp2k_md_summary.py, qmc_check.py and dftfe_check.py on genuine outputs (when present under build/level3/) and on mutated copies: truncated run, NaN, non-converged MD-step SCF, non-converged initial SCF without IGNORE_CONVERGENCE_FAILURE, zero accelerator devices, missing offload/CUDA banners, GRID tasks on CPU only, nonzero exit code, truncated/shifted scalar files, 'QMCPACK ERROR'. Each mutation must be rejected; the genuine output must be accepted.
…ith a private LLVM 23.1.0 toolchain Upstream's recommended NVIDIA configuration QMC_GPU="openmp;cuda" needs a Clang with NVPTX offload: LLVM 23.1.0 is built from source on local scratch (clang/lld + openmp/offload runtimes, GPU runtimes target nvptx64-nvidia-cuda -> libompdevice/libomptarget-nvptx.bc; the official binary release ships no device runtime), then probed before any application build (OMP_TARGET_OFFLOAD=MANDATORY, target region off the initial device, numerics vs host, MPI 2/4 ranks through the common launcher). HDF5 1.14.5 parallel, Boost 1.90 headers and OpenBLAS 0.3.30 are private per profile; the node's zlib-ng CMake package (missing libz.a) is bypassed for HDF5 via CMAKE_IGNORE_PATH. run.sh drives upstream's diamondC_2x1x1_pp batched VMC+DMC deck (verbatim for smoke; strong/weak change only the walker population); validate.sh runs upstream's unit and deterministic ctests on the built tree (64/64 and 526/526 passed) and applies upstream's own check_scalars.py criterion to the N-GPU run plus a cross-rank statistical consistency check; qmc_check.py gates on completion, offload/CUDA banners, reported device code paths and device memory allocated through the offload runtime.
… paths for system-toolchain builds C_INCLUDE_PATH/CPLUS_INCLUDE_PATH/CPATH/LIBRARY_PATH on this node point at other conda environments (empty include dirs, but linker search paths); they are now cleared together with the conda build variables in l3_clean_conda_build_env.
…II 9.6.2 / ELPA 2026.02.001 stack Recipe transcribed from upstream install_DFTFE (frontierDevelop) to this node: OpenBLAS -> ScaLAPACK 2.2.2 -> libxc 7.0.0 -> spglib -> ALGLIB -> p4est 2.8.7 (dftfe's Cray-oriented p4est-setup.sh given the Open MPI wrappers, LIBS=-lm and the 2.8.7 header location) -> Kokkos 4.6.00 Serial -> deal.II 9.6.2 -> ELPA with NVIDIA sm_100 kernels (-march=native for its AVX-512 probe; ScaLAPACK paths in LDFLAGS for its cublas check) -> dftfe_real (CMAKE_CUDA_ARCHITECTURES=100). deal.II 9.7.1 (the version the current recipe pairs with dftfe develop) was tried first and rejected: 9.7 removed Utilities::MPI::create_group, Triangulation::load(name, autopartition) and VtkFlags::ZlibCompressionLevel that release 1.2.0 still uses. One 2-line source patch (std::isnan for two unqualified isnan calls, GCC 14). run.sh drives upstream's GPU regression deck Input_MD_0.prm (32-atom Al BOMD) verbatim, a synthetic Al-supercell weak series and the 192-atom LLZO ground state; validate.sh compares with upstream's own GPU reference output through dftfe_check.py (pre-fixed tolerances) and requires the independent ELPA GPU probe (elpa_probe.sh: ELPA's residual/orthogonality tests on 1/2/4 GPUs). At 1/2/4 GPUs the Al BOMD energies, temperatures and forces reproduce upstream's reference at printed precision.
…eads one); BLAS-resolution guard
The first build inherited conda's LDFLAGS (-Wl,--disable-new-dtags) and the MPI
wrapper's rpath, so cp2k.psmp/libcp2k.so resolved libopenblas.so.0 to the conda
pthreads OpenBLAS at run time ("OpenBLAS Warning : Detect OpenMP Loop"); the
validations had passed with it, but the configuration was not the recorded one
and H2O-128 ran 1.7x slower. build.sh now clears the conda build variables,
passes the toolchain libopenblas.a as CP2K's CUSTOM BLAS/LAPACK and puts the
toolchain library directories first in the RPATH; run.sh refuses to run unless
ldd resolves BLAS under the toolchain (or it is linked statically) and records
blas_resolved= in the manifest. Re-validated at 1/2/4 GPUs with the relinked
binary; attempts 1-2 archived under install/ATTEMPT-*.
LyA heat/cool 64^3 on 1/2/4 GPUs: 2.85 / 2.62 / 2.65 s (too small to scale, fixed-cost dominated); dry-runs 8 planned, 40 IMBALANCED, 80 refused as designed. nyx_particle_compare.py made executable.
run.sh refuses more than HPCPERF_QMCPACK_MAX_WALKERS_PER_GPU (300) walkers per GPU unless HPCPERF_QMCPACK_FORCE_POPULATION is set: this build exhausts device memory at ~320 MB per walker (cuSOLVER INTERNAL_ERROR at 1024 walkers/GPU; open issue documented in the README, not worked around). Strong = verbatim 256-walker deck over N GPUs (534/332/211 s on 1/2/4), weak = 256 walkers per GPU (534/632/656 s); every run within upstream's check_scalars window. Manifest records walkers_per_gpu.
…ng validation 1/2/4 GPUs Build: upstream thirdPartyLibs superbuild (config-build.py -n) with three build-system patches (superlu_dist 65-char hash, RAJA_ENABLE_VECTORIZATION overridable -> OFF for nvcc 13.2 + GCC 14/x86-64-v3, hdf5 step generator and build commands), ENABLE_HYPREDRV=OFF (hypredrive step lacks Umpire's include; GEOS-documented option, unused by the beam workflow); GEOS with a private host-config (hypre on device, sm_100), TPL METIS/ParMETIS include dirs ahead of the conda MPI include dir (unrelated 32-bit metis.h), and the back-port of LLNL/blt 38b46203 for BLT's CUDA runtime smoke test (memoryClockRate removed in CUDA 13); tests kept at upstream's default ON. 1718 s at -j32. Validation re-implements geos-ats's checks for beamBending (curve_check.py metric ||u-u_ref||_2/N <= 2e-4 vs the analytic Euler-Bernoulli script, restart_check.py rules incl. LvArray permutation handling and the default exclusions, atol 1e-3 / rtol 1e-7) and was verified against upstream's published baseline first. Results: hypre GMRES+AMG on the device at 1/2/4 GPUs, curve metric 1.376e-4 (= upstream's baseline value), cross-rank rel L-inf <= 1.1e-8; shipped direct-solver deck vs the public restart baseline beamBending_smoke_01: 0 of 1191 arrays disagree (worst rel 3.6e-12); the three GPU-build LinearSolverParameters defaults are reported, not gating. Strong/weak/dry-run records: the official beam decks are too small to scale.
… 1/2/4 PASS with the probe elpa_probe.sh runs ELPA's own validate_*_gpu_analytic programs (the real binaries in build/elpa/.libs/; the top-level names are libtool relink wrappers) on 1/2/4 GPUs and applies ELPA's limits from test_analytic_template.F90 (max eigenvalue error 5e-14, eigenvector error 6e-10), exit 0, GPU timers present and a launcher audit with 0 mismatch. The two earlier versions looked for wrapper-script names and then for the %Error Residual/Orthogonality lines of the random-matrix programs (which the analytic ones never print; the empty grep aborted the script under pipefail) -- both recorded in the script header and README. Probe: PASS on all six runs (errors <= 7.3e-15 / 1.0e-11). Validation 1/2/4 re-run with the probe: PASS (al_md identical to upstream's GPU reference). README: results table, the std::isnan patch described as the one source change.
…ts fail on this GPU build)
ctest on GPU 0 with upstream's default ENABLE_TESTS=ON: 254 of 261 passed.
testMath (float asinhf vs host within 1 ulp) and testErrorHandling (abort
on purpose under prterun) are test/harness artefacts; testCompMultiphaseFlow,
testCompMultiphaseFlowHybrid, testThermalEstimator{Prod,Inj}Well and
testReservoirThermalSinglePhaseMSWells_RateInj fail grossly (derivative and
Jacobian checks off by O(1), ExternalError in the well solvers) -- cause not
determined; the compositional-flow and well modules of this build are marked
UNVERIFIED. The validated solid-mechanics beam workflow is unaffected.
SECOND_BATCH_STATUS.md (new): status table for Nyx, CP2K, QMCPACK, DFT-FE and GEOS (version/SHA, build strategy, compiler/Toolkit, dependency probe, CUDA build, 1/2/4-GPU validation, strong, weak/size sweep, 40/80 dry-run, multi-node, HIP, source changes, blocker), per-application sections with the measured numbers and the open items, cross-cutting notes, raw-material locations and the local commit list. APPLICATION_AUDIT.md / BUILD_STRATEGY.md: second-batch corrections (CP2K B200 back-port and BLAS relink, Nyx AMReX pin, QMCPACK toolchain, DFT-FE deal.II 9.6.2 + std::isnan patch, GEOS develop SHA, RAJA vectorization, BLT back-port, metis include order, unit-test findings), the 'what was actually built and how' table, Spack policy realised as NATIVE.
…epository) Environment facts of the validated node (CUDA 13.2.78, conda vs system GCC, Open MPI transport, no ROCm, Slurm slots), the git rules (no attribution trailers, local commits only, never push/PR/merge unprompted, never add the environment/dependency trees), the Level 2/3 per-application conventions (fetch/build/run/validate/checker/patches/README, per-profile isolation, fingerprints, manifests, common launcher interface and dry-run semantics), the validation principles and change policy, the node/toolchain pitfalls collected over Levels 1-3, the current state of the branches and the open decisions, and how reports are expected.
…BDIR for regression campaigns l3_clean_env.sh / l3_clean_env_exec: re-execute a command under env -i with an explicit allow-list (session/locale, Slurm, MPI/PMIx/UCX, CUDA, compilers and build flags, conda/python, OpenMP/BLAS knobs, HPCPERF_*). Used around every tool that records its process environment; CP2K's toolchain installer (which writes a declare -x dump into toolchain.env) now runs through it. Compatibility checked on dgx003: nvcc/mpirun/cmake/gcc/python and the launcher dry-run work under the wrapper with hpcperf_env.sh sourced, no secret-named variable is visible inside. HPCPERF_L3_RUN_SUBDIR (default run): every application's run.sh/validate.sh now builds its run directories under build/level3/<app>/<profile>/$L3_RUN_SUBDIR, so a regression campaign can write into a fresh sibling tree (run.regress-<sha>) without touching historical results. Mechanical, line-local edits in the 10 applications; level3/tools/tests/run_all.sh now also runs test_l3_validators.sh and test_nyx_validator.sh.
…ovenance corrected, negative tests compare() trusted an awk filter over fcompare's table: it dropped inf/nan and message rows before the finiteness check, treated any rc=1 as an over-tolerance case, and 'nv>=5' passed as a completeness check -- a reviewer's CPU injection (Ne abs=1 rel=inf on a zero reference, a missing variable with fcompare's warning, a NaN row) was accepted. nyx_fcompare_check.py now parses both plotfile Headers (variable set, dim, levels, time, domain, cell sizes, box arrays), runs fextrema on both files for raw finiteness and zero-reference detection, requires exactly one parsed row per variable and level (no message rows, no duplicates, no truncation), judges zero-reference fields by an absolute rule (exact by default), never drops a non-finite error, keeps diagnostic fields reported but ungated, and refuses parser/tool disagreement. validate.sh calls it, gains OFFLINE/report-dir/tools-dir hooks (offline re-checks never write into historical run directories), and downgrades a case with a diagnostic field to STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING (exit 3) instead of PASS. Provenance, corrected in the header and README: MiniSB 2e-10 is upstream's; LyA-adiabatic 2e-10 is a project choice 25x stricter than upstream's 5e-9; the heat/cool 5e-5 is upstream's plt00354 tolerance adopted after the first heat/cool validation at 2e-10 FAILED; the I_R exclusion is project-defined and was introduced at that time (upstream's report does compare I_R). README section 'I_R' records what the field is (SDC reaction integral, state-carried, a ~1e-3 residual of cancelling O(a^2 rho e/dt) terms in this deck), its O(1) run-to-run variation that leaves no imprint on any state field, why no state-derived absolute criterion accepts it, and what is still unidentified. Offline re-check of the saved 2026-09-06 plotfiles with the new comparator: MiniSB/LyA-adiabatic 1/2/4 GPUs PASS at 2e-10 / 1e-8 (worst Temp 1.4e-10 on MiniSB np4, 6.7e-14 on LyA); heat/cool 1/2/4 GPUs STATE_AND_PARTICLES_PASS, I_R_CHECK_PENDING. Tests: level3/tools/tests/test_nyx_validator.sh (26 checks: 16 comparator cases incl. the three reviewer injections, zero/zero, zero-vs- nonzero, duplicate, truncated, header mismatch, non-finite raw, diagnostic; 10 through the real validate.sh chain in OFFLINE mode).
…-listed environment wrapper
…tial deny-rule - nyx_fcompare_check.py: the BoxArray of every level is checked geometrically (physical extents -> cell boxes via prob_lo/dx): boxes inside the level domain, no overlap, level 0 covers the domain exactly; a missing box / overlap / coverage deficit / different covered cell sets is STRUCTURAL (exit 2), while a legal re-blocking (same cells, different partition) is UNSUPPORTED_LAYOUT (exit 4) -- reported explicitly, never a science FAIL, never skipped silently (fcompare would need --allow_diff_grids, not enabled). Same boxes in another order are compared normally. - nyx/validate.sh: comparator exit 4 -> case verdict UNSUPPORTED_LAYOUT, script exit 4; precedence FAIL(1) > UNSUPPORTED_LAYOUT(4) > I_R_CHECK_PENDING(3) > PASS(0); header states that I_R is consumed by the next step's predictor and that short-horizon state agreement does not prove it has no later effect. - l3_verdict.py (new): classes PASS / PENDING / UNSUPPORTED_LAYOUT / FAIL / MISSING from (exit code, log); exit 3/4 are never PASS; an exit code that contradicts the log is FAIL; campaign summary with launcher-audit columns. l3_common.sh: l3_run_recorded records a queue step's exit code and continues. - l3_clean_env.sh: credential deny-rule applied after the allow-list (a *_TOKEN/*_KEY/... name never survives through an allow-listed prefix); --show names denied variables (names only). Compatibility on dgx003 unchanged: nvcc, mpirun, gcc, cmake, python/numpy, nvidia-smi, SLURM_JOB_ID, CUDA_VISIBLE_DEVICES, OMPI_MCA_opal_cuda_support visible inside. - Tests: test_nyx_validator A17-A21 (re-blocking -> exit 4; missing box, overlap, out-of-domain -> STRUCTURAL; reordered boxes -> compared) and B11-B12 (real validate.sh chain: exit 4 UNSUPPORTED_LAYOUT, missing box -> FAIL); test_l3_infra 6a-6f (clean env) and 7a-7b (queue continues past exit 3); test_l3_verdict (13 checks). run_all.sh: 88 checks, 0 failures. - Offline re-judgement of the 2026-09-07 regression plotfiles with this comparator: identical verdicts (adiabatic 1/2/4 PASS, heat/cool 1/2/4 exit 3), 18/18 comparisons "identical box array".
…, second-batch table
- level3/README.md: status table now carries the five second-batch applications
with their real state (Nyx adiabatic decks PASS re-run 2026-09-07; heat/cool
STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING -- not a pass; CP2K/QMCPACK/DFT-FE/
GEOS historical 2026-09-05/06 results, not re-run on GPU); the 2026-09-07
joint-HEAD regression (27 validate.sh calls: 24 PASS, 3 PENDING, 0 FAIL; four
logs with unverified launcher ranks listed as a binding-evidence gap); verdict
classes; strong/weak, 40/80-GPU, multi-node and HIP stated as they are; the
Nyx heat/cool tolerance history recorded as the one exception to
"no tolerance changed after a run".
- SECOND_BATCH_STATUS.md: Nyx row/section split into adiabatic PASS and heat/cool
PENDING ("Nothing remaining" removed; the I_R follow-up listed); joint-HEAD
regression section with what was and was not re-run; CPU test groups and
script versions; commit list updated (branch submitted as a Draft PR).
- nyx/README.md: heat/cool case no longer "not built"; box-layout policy
(UNSUPPORTED_LAYOUT vs STRUCTURAL); 2026-09-07 regression table; I_R section
reworded -- I_R feeds the next predictor, 10-step state agreement is not proof
of no later effect; the 1.9e-2 figure is a diagnostic estimate with its
formula, inputs (tol 5e-5, max|a^2 rho e|/(dt a_half) = 3.7e2, dt = 2.99e5 code
time units, a = 0.01094), units and assumptions, not a verified bound;
heat/cool excluded from the correctness-pass count and performance summaries.
- CLAUDE.md: validate.sh exit classes, queue/verdict helpers, the unwrapped
run/profiler path as a follow-up, current state (2026-09-08) and open items.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Level 3 of HPC-Performance-AI: ten full production applications brought up on one node (dgx003: 4x NVIDIA B200 / sm_100, CUDA 13.2.78, Open MPI 5.0.10, Slurm job 9552083), each with
fetch.sh/build.sh/run.sh/validate.sh, upstream decks, pre-fixed correctness criteria, per-run manifests and fingerprints, plus the shared Level 3 helpers (level3/tools/). The branch contains the first batch (level3/full-apps-bringup: LAMMPS, SPARTA, WarpX, SPECFEM3D, nekRS), the second batch (Nyx, CP2K, QMCPACK, DFT-FE, GEOS), the validator rework and the joint-HEAD regression of 2026-09-07/08. Base:main(= PR #3, Level 3 preparation).Status documents:
level3/README.md(status table),level3/SECOND_BATCH_STATUS.md,level3/CORRECTNESS_FIXES.md,level3/APPLICATION_AUDIT.md,level3/BUILD_STRATEGY.md, per-applicationREADME.md.This PR does not claim a uniform re-validation of every case of all ten applications, and it is not "all tests PASS". What was run, when, and at which code state is listed below; PENDING items are counted separately.
Application status (per case / variant, 1/2/4 GPUs, one MPI rank per GPU)
bench/in.ljsmoke vs shipped reference log (thermo, bit-identical)fc4d2a1)bench/in.collidesmoke vs shipped reference (count exact, temp 2 %, Natt 15 %)langmuir_multianalytic (5e-2) + charge conservation (1e-11) +uniform_plasmasmokehomogeneous_halfspacevs reference seismograms (upstream comparison script)ethier --cimode: hypregpu cimode 2, hypregpu cimode 3 (device coarse), cpucoarse cimode 2 (9/9 CI checks each)nekrs/COMPATIBILITY.mdinputs.rtheat/cool, state fields at 5e-5 (upstream's value, adopted after a first run at 2e-10 FAILED), particles, mass;I_Rreported, not accepteddiamondC_2x1x1_ppVMC/DMC verbatim, upstreamcheck_scalars.pywindow, 1-GPU consistency; unit ctests 64/64, deterministic 526/526run.shrefuses > 300 walkers/GPU; cause openInput_MD_0.prm32-atom Al BOMD vs upstream GPU reference: identical at printed precision; ELPA GPU probe 1/2/4b7a0f13305+ thirdPartyLibs9b55672beamBending80x8x4, hypre GMRES+AMG on device: geos-ats curve metric 1.376e-4 <= 2e-4 (= upstream baseline), 1-GPU consistency; direct-solver deck vs public restart baseline 0 disagree (worst 3.6e-12)testMathfloat ulp,testErrorHandlinglauncher interactionEvery validation captures the real exit code, fails on timeout / missing / non-finite output, requires the launcher audit, and writes
run_manifest.txt(run id, hashes, GPU evidence). No kernel, algorithm, precision or solver-placement change in any upstream code; patches are classified (A none, B build-system, C node adaptation, D minimal back-port) and documented per application.Joint-HEAD regression (2026-09-07)
Code state
fc4d2a156d4e8aefe8a7f4e4c3ec88613b3a352c; the following docs-only commit74c1d4d00babd830c9513c01f152665d3c611bb0differs from it inCLAUDE.mdalone (verified withgit diff --stat). After the shared-helper change (HPCPERF_L3_RUN_SUBDIR: regression run treesrun.regress-<sha>that never overwrite historicalrun/directories) and the Nyx validator rework, 27validate.shcalls were made on dgx003 with new run ids:level3/tools/tests/test_l3_validators.sh. A GPU regression of these four under the new directory logic is a follow-up.level3/tools/tests/run_all.sh-- four groups (infra helpers 21, second-batch checkers 21, Nyx strict comparator +validate.shchain 33, verdict classes 13 = 88 checks, 0 failures);level2/tools/tests/run_all.sh-- topology self-test, deps markers (9), env dep profiles (11), launcher dry-run (12), run.sh guards (23), all passed; Level 2 test scripts last changed in7f79d9c(2026-09-04). Passing suites say nothing about the open items listed per application.-b2worktree):build/level3/regress-firstbatch-fc4d2a1/(validate_full.<app>.np<N>.log,queue_firstbatch.log,SUMMARY_TABLE.md,level2_tests_run_all.log,level3_tests_run_all*.log),build/level3/nyx/regress-fc4d2a1/(validate_full.hc{NO,YES}.np{1,2,4}.log,queue_nyx.log),build/level3/nyx/offline_recheck-2026-09-07/andoffline_recheck-2026-09-08-layoutfix/(read-only re-judgement of saved plotfiles). Run trees:build/level3/<app>/<profile>/run.regress-fc4d2a1/(36 run ids); historicalrun/trees untouched.Nyx heat/cool: why it is PENDING, not PASS
I_Ris the SDC reaction-integral state component (ode_eos_finalize_struct), written to plotfiles and consumed by the next step's hydro predictor. In the 10-step official deck it is a ~1e-3 residual of nearly cancelling terms; between any two runs (same-config rerun, 1-vs-2/4 GPUs, GPU-vs-CPU) it differs by O(1) relative (0.28-0.33 absolute) while every state field agrees to 1e-13 and particles to 1e-15. Upstream's nightly comparison of this field happens at plt00354 in a regime where the reaction term is physical, so its 4.2e-8 cannot be transferred. No criterion derived from the state tolerance accepts the observed variation (a one-step linear coupling estimate gives ~1.9e-2 as the largest tolerable change -- recorded as a diagnostic estimate with its formula, inputs and assumptions, not a proven bound), and the mechanism (CVODE tolerance noise, batched buffers, a race or an uninitialised read) is not identified. The validator therefore parses and reportsI_R(it must exist and be finite), keeps the state/particle/mass checks at 5e-5, and exits 3. Short-horizon state agreement is not taken as proof that the variation has no later effect. Heat/cool is excluded from the correctness-pass count and from any performance summary.Validation safeguards added in this branch
level3/nyx/nyx_fcompare_check.py: strict wrapper around AMReXfcompare-- Header structure (variable set, dimension, levels, time, domain, cell sizes), geometric BoxArray check (inside domain, no overlap, level-0 coverage; a legal re-blocking is reported asUNSUPPORTED_LAYOUT, exit 4, never a science FAIL and never skipped silently), raw finiteness of both plotfiles viafextrema, one parsed row per variable and level (message rows, duplicates,WARNING/ERRORare STRUCTURAL), zero-reference fields judged by an absolute rule, parser/tool exit-status consistency. Replaces an awk filter that acceptedinf/nan/missing-variable rows.validate.shexit-code contract: 0 PASS, 1 FAIL, 3 PENDING, 4 UNSUPPORTED_LAYOUT (precedence FAIL > UNSUPPORTED_LAYOUT > PENDING > PASS).level3/tools/l3_verdict.pyclassifies (PASS / PENDING / UNSUPPORTED_LAYOUT / FAIL / MISSING; an exit code that contradicts the log is FAIL) and summarises campaigns;l3_run_recordedlets queues record every step's exit code and continue.level3/tools/l3_clean_env.sh/l3_clean_env_exec: allow-listedenv -ire-execution with a credential deny-rule, used around tools that record their process environment (CP2K's toolchain installer). The common launcher /run.sh/ profiler path is not wrapped yet (follow-up).HPCPERF_L3_RUN_SUBDIR: regression campaigns get their own run trees; historical results are never overwritten.level3/tools/tests/), plus the earlier Level 3 hardening (CORRECTNESS_FIXES.md: real exit codes, NaN/Inf rejection, dry-run sentinel, fingerprints).Security note (no values): a build-time environment dump written by an upstream toolchain installer under an ignored path was found to contain credentials from the login shell. It was sanitized locally, the installer now runs through the allow-listed wrapper, and nothing in the pushed history contains credential values or names beyond the wrapper's comment (scanned by variable name and by token pattern before pushing). Credential rotation is handled outside this PR.
Boundaries (stated as they are)
HPCPERF_DRY_RUN=1, HYPOTHETICAL node counts); no 40/80-GPU run has ever happened. Rank counts the decomposition cannot support are refused (e.g. Nyx MiniSB at 80 ranks, GEOS at 3 ranks).pml ob1 / btl self,sm,smcudatransport; UCX hangs on device buffers).build.sh, untested (no ROCm on the node).Follow-ups (not done in this PR)
run.sh/ profiler path in the clean environment; moving the shared runtime (tools/runtime/README.mdplan) is not part of this PR.Final HEAD of this PR:
0dd99a170ede656d3f3d6442c59c673e0e8961d6.