From 1e4264b4bf6282a6e9b72f42d86a019f2da345ff Mon Sep 17 00:00:00 2001
From: Paulius Velesko <pvelesko@pglc.io>
Date: Wed, 14 Oct 2026 04:31:47 +0300
Subject: [PATCH 01/31] ci: add libs-apps.sh, every library and app on one
 Intel GPU

No CI lane builds the full install_chipstar.py --all --with-tests set or
runs the tests/apps applications, so a device header change such as
making warpSize non-constexpr (PR 1934) can break rocPRIM and everything
built on it without any lane noticing.

The script builds chipStar plus every library with its tests, runs each
library's ctest suite with the same .ci/exclude.txt and
.ci/ctest-labels.txt convention as libraries-presubmit.yml and the
manual Aurora libs lane, then builds zeroRK and OpenSn and runs the two
zeroRK hydrogen CPU-vs-GPU correctness checks (both
ZERORK_REACTOR_USE_LU modes) and the OpenSn strong scaling input, once
per backend (Level Zero and OpenCL by default). The zeroRK box reactor
check is not run (summary.txt records the SKIP):
zerork_correctness_box_reactor.sh reads PBS_NODEFILE
and launches 12 ranks with Intel MPI options (-ppn, -d, -env) through
Aurora's gpu_tile_compact.sh, none of which exist on a workstation
runner. The hydrogen outputs are copied into logs/ because the next
backend's run deletes them.

The GPU may be shared with other work on the runner, so:
- before each GPU step it waits until fuser shows no process holding the
  GPU's render node (resolved from GPU_PCI via sysfs);
- a journal cursor is taken before the first GPU step. Any reset,
  wedge, hang or timed-out job kernel line from the GPU's driver (xe or
  i915) for its PCI address after it
  (matched case-insensitively: xe logs both guc_exec_queue_timedout_job
  and "Timedout job ... in <process>"),
  seen during a step (polled every second), between steps, or while
  waiting for another holder to let go, kills the running step and
  skips all remaining GPU work, including the OpenCL platform probe
  (clinfo also runs as a GPU step), since a wedged GPU needs a reboot
  and every later test would only hang on it. A cursor rather than a
  line count, because journal rotation can shrink the count and hide a
  new line. The script refuses to start GPU work if it cannot read the
  kernel journal, and a later journal read error ("Failed to seek to
  cursor", "No journal files", or a nonzero journalctl exit) counts as
  a fault. The first observation
  is kept in logs/fault, which makes it sticky (a transient read error
  cannot be forgotten by the next query) and keeps the first 100
  matching lines followed by the first 900 driver lines since the cursor,
  so the line naming the process whose job hung is kept. The match uses a here-string, not a pipe: under pipefail,
  `echo "$j" | grep -q` loses a large reset storm when grep exits early
  and echo dies of SIGPIPE. The STOP line names the ctest test that was running;
- each GPU step runs under setsid and the fault path kills the step's
  pid (in case setsid has not run yet) and then its whole session
  (pkill -s). Killing only the process group is not enough:
  OpenMPI's mpiexec starts its rank in a separate process group of the
  same session, which survived a group kill. ctest's test processes stay
  in ctest's group and session, so they die either way;
- GPU work is refused if the boot's kernel log already declares this
  GPU wedged (a cursor taken later would not see it) or cannot be read,
  and when fuser is missing;
- SIGINT/SIGTERM to the script kills the running step's session, which
  setsid has taken out of the terminal's process group;
- ctest runs serially (-j1) with a per-test timeout (CTEST_TIMEOUT,
  default 900 s).

The GPU is the first Intel display-class PCI device off bus 00, where
integrated GPUs sit, unless GPU_PCI is set: cupcake's Arc B570 (xe) and
meatloaf's Arc A770 (i915), both at 0000:03:00.0. Its driver name comes
from sysfs and selects the kernel log lines watched above.

Device selection uses NEO's FilterBdfPath debug key
(NEOReadDebugKeys=1) so neither Level Zero nor the Intel OpenCL
platform enumerates the integrated GPU next to the discrete one; NEO otherwise opens
every Intel GPU in each process. Verified on cupcake: with the filter,
hipInfo reports only the Arc B570 under both CHIP_BE=level0 and
CHIP_BE=opencl. The OpenCL platform index is taken from clinfo.
CHIP_DEVICE_TYPE is unset with the other selectors because chipStar
rejects it next to CHIP_PLATFORM.

zeroRK is built with the flags of its scripts/build_aurora.sh, minus
the module loads and the unbounded `make -j`, and with the system BLAS
(ZERORK_HAVE_MKL=OFF): its cmake/blas.cmake turns MKL on whenever
ONEAPI_ROOT is set and then links $ONEAPI_ROOT/mkl/latest/lib/intel64
static archives, a layout cupcake's flat oneAPI install does not have. CMAKE_POLICY_VERSION_MINIMUM
is exported because both apps pull in projects (yaml-cpp, SuperLU) whose
cmake_minimum_required is below 3.5, which CMake 4 rejects; a -D flag
does not reach their nested ExternalProject configures, the environment
variable does. OpenSn's dependencies (PETSc, VTK, HDF5, Caliper, Boost)
do not involve chipStar, so they are built once per runner under
DEPS_CACHE keyed by the OpenSn commit; the cache is valid once the deps
build's final set_opensn_env.sh exists and is non-empty, a rebuild starts from an empty
prefix, and a lock file serializes concurrent invocations sharing the
cache. A hard machine freeze during a deps build on cupcake left
hundreds of zero-byte installed files; the next build into the same
prefix kept them as "Up-to-date" and OpenSn then failed on an empty
caliper/cali-manager.h. Boost is forced into that build
because the system Boost 1.83 satisfies the deps script but not OpenSn's
>= 1.86 requirement. The outer deps build runs with -j1 because each
dependency already runs make -j$JOBS and nested makes do not share a
jobserver.

The build phase starts from an empty install prefix and library
staging dir, so a reused work dir cannot test against files or test
binaries a previous revision left behind (a failed `git pull` in a
reused clone would otherwise leave the old build's tests runnable). The prefix
is date-stamped by install_chipstar.py; a later libs/apps-only run uses
the newest one.

Arc A-series GPUs (meatloaf's A770) have no FP64 hardware, and zeroRK,
OpenSn and most library tests use double: on the A770 every double
kernel fails to build with "Double type is not supported on this
platform". Before the first backend the script reads clinfo --raw for
the selected GPU and, when its Intel OpenCL Graphics device reports no
CL_DEVICE_DOUBLE_FP_CONFIG, exports OverrideDefaultFP64Settings=1 and
IGC_EnableDPEmulation=1, the two settings compute-runtime's
opencl/doc/FAQ.md names for FP64 emulation, and records a NOTE line in
summary.txt. GPUs with FP64 hardware (B570, PVC) keep running double
natively. Emulation is slow: one zeroRK hydrogen run takes about 25
minutes on the A770, and it matches the CPU history exactly.

OpenSn passes -DCMAKE_HIP_PLATFORM=spirv, which CMake accepts only from
4.3 (4.2.3's CMakeDetermineHIPCompiler.cmake has no spirv), and Ubuntu
24.04 ships 3.28. Rather than replace a CI runner's system CMake under
every other job, build_opensn downloads the CMake 4.3.2 release into
DEPS_CACHE, checks its SHA-256 against Kitware's published sum, and uses
it for the OpenSn configure and build only.

The zeroRK check script and OpenSn inputs are taken from the tree under
test, so a PR that changes them is tested with its own version.

tests/apps/run_apps.sh is not reused directly: it hard-codes Aurora
modules, /lus paths and gpu_tile_compact.sh, and its performance
baselines are PVC numbers.
---
 scripts/ci/libs-apps.sh | 179 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 179 insertions(+)
 create mode 100755 scripts/ci/libs-apps.sh

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
new file mode 100755
index 000000000..54e1759ef
--- /dev/null
+++ b/scripts/ci/libs-apps.sh
@@ -0,0 +1,179 @@
+#!/bin/bash
+# Build chipStar plus every downstream library with its tests, then run each
+# library's ctest suite and the zeroRK and OpenSn checks on one Intel GPU per
+# backend. Waits while another process holds the GPU and stops all GPU work at
+# the first reset or wedge the kernel logs for it. Results go to summary.txt.
+#
+#   scripts/ci/libs-apps.sh -w WORK_DIR [-s CHIPSTAR_SRC] [-j JOBS] [-b "level0 opencl"] [-p build,libs,apps]
+set -o pipefail
+
+HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
+SRC=$HERE WRK="" JOBS="" BACKENDS="level0 opencl" PHASES=build,libs,apps
+# First Intel display device off bus 00, where the integrated GPU sits.
+GPU_PCI=${GPU_PCI:-$(grep -l 0x8086 /sys/bus/pci/devices/*/vendor | xargs -n1 dirname |
+  while read -r d; do grep -q '^0x03' "$d/class" && echo "${d##*/}"; done | grep -v '^0000:00:' | head -1)}
+DEPS_CACHE=${DEPS_CACHE:-$HOME/.cache/chipstar-libs-apps}
+OPENSN_COMMIT=d0644cd9c633c6ae5e7110d3f8721641e5ef2982
+while getopts "s:w:j:b:p:" o; do
+  case $o in
+    s) SRC=$(cd "$OPTARG" && pwd) || exit 2 ;; w) WRK=$OPTARG ;; j) JOBS=$OPTARG ;;
+    b) BACKENDS=$OPTARG ;; p) PHASES=$OPTARG ;; *) sed -n '2,7p' "$0"; exit 2 ;;
+  esac
+done
+[ -n "$WRK" ] && mkdir -p "$WRK/logs" && WRK=$(cd "$WRK" && pwd) || { sed -n '2,7p' "$0"; exit 2; }
+LOGS=$WRK/logs SUMMARY=$WRK/summary.txt FAILED=0 GPU_DEAD=0
+# idle cores minus one, at most one job per 3 GB of available RAM
+: "${JOBS:=$(awk -v n="$(nproc)" -v m="$(free -g | awk '/^Mem:/ {print $7}')" \
+  '{ j = int(n - $1 - 1); if (m / 3 < j) j = int(m / 3); print (j < 1 ? 1 : j) }' /proc/loadavg)}"
+RENDER=/dev/dri/$(basename "$(ls -d /sys/bus/pci/devices/"$GPU_PCI"/drm/renderD*)")
+DRV=$(basename "$(readlink /sys/bus/pci/devices/"$GPU_PCI"/driver)")
+source "$HERE/scripts/module-env.sh"
+module load oneapi/2025.0.4 llvm/23.0 || exit 1
+# zeroRK and OpenSn bundle projects that declare a CMake minimum below 3.5.
+export CMAKE_POLICY_VERSION_MINIMUM=3.5
+
+trap '[ -n "$STEP" ] && { kill -KILL "$STEP"; pkill -KILL -s "$STEP"; } 2>/dev/null; exit 143' INT TERM
+wants() { case ,$PHASES, in *,$1,*) return 0 ;; esac; return 1; }
+note() { echo "$*" | tee -a "$SUMMARY"; }
+result() { [ "$2" -eq 0 ] && note "PASS  $1" || { note "FAIL  $1 (rc=$2, $3)"; FAILED=$((FAILED + 1)); }; }
+# A journal read error counts as a fault: the GPU cannot be watched. Sticky, so a transient error is not forgotten.
+gpu_faults() {
+  local j p="$DRV $GPU_PCI.*(reset|wedged|timedout|hang)|^Failed|^No journal"
+  [ -e "$LOGS/fault" ] && { echo 1; return; }
+  j=$(journalctl -k -b -q --no-pager --after-cursor="$CURSOR" 2>&1) || j="Failed: journalctl exit $?
+$j"
+  j=$(grep -E "$DRV $GPU_PCI|^Failed|^No journal" <<< "$j")
+  grep -qiE "$p" <<< "$j" || { echo 0; return; }
+  { grep -iE "$p" <<< "$j" | head -100; head -900 <<< "$j"; } > "$LOGS/fault"; echo 1
+}
+
+# gpu_step NAME LOG CMD...; CMD gets its own session so a fault kills all of it.
+gpu_step() {
+  local name=$1 log=$2 pid="" f; shift 2
+  [ $GPU_DEAD -eq 0 ] || { note "SKIP  $name"; return; }
+  while f=$(gpu_faults) && [ "$f" -eq 0 ] && fuser "$RENDER" > /dev/null 2>&1; do echo "[gpu] $RENDER busy, waiting"; sleep 60; done
+  if [ "$f" -eq 0 ]; then
+    setsid "$@" > "$log" 2>&1 < /dev/null &
+    pid=$! STEP=$!
+    while kill -0 $pid 2>/dev/null && f=$(gpu_faults) && [ "$f" -eq 0 ]; do sleep 1; done
+    [ "$f" -gt 0 ] || f=$(gpu_faults)
+  fi
+  if [ "$f" -gt 0 ]; then
+    [ -n "$pid" ] && { kill -KILL $pid; pkill -KILL -s $pid; } 2>/dev/null; GPU_DEAD=1
+    note "STOP  GPU reset, wedge or journal error at $name $(grep -so 'Start *[0-9]*: [^ ]*' "$log" | tail -1), see logs/fault"
+    result "$name" 137 "$log"; STEP=""; return
+  fi
+  wait $pid
+  result "$name" $? "$log"; STEP=""
+}
+
+backend_env() {
+  export CHIP_BE=$1 CHIP_LOGLEVEL=err NEOReadDebugKeys=1 FilterBdfPath=$GPU_PCI
+  unset CHIP_PLATFORM CHIP_DEVICE CHIP_DEVICE_TYPE
+  [ "$1" = opencl ] || return 0
+  # FilterBdfPath leaves only this GPU on the Intel graphics platform.
+  gpu_step "opencl platform probe" "$LOGS/opencl/clinfo.txt" clinfo -l
+  CHIP_PLATFORM=$(grep -s '^Platform' "$LOGS/opencl/clinfo.txt" | grep -n 'Intel(R) OpenCL Graphics' | cut -d: -f1)
+  export CHIP_PLATFORM=$((CHIP_PLATFORM - 1))
+}
+
+# Same .ci/exclude.txt and .ci/ctest-labels.txt convention as libraries-presubmit.yml.
+run_libs() {
+  local lib s ex lb
+  for s in "$WRK"/staging/*/; do
+    lib=$(basename "$s")
+    [ -f "$s/build/CTestTestfile.cmake" ] || continue
+    ex=$(grep -Evhs '^\s*(#|$)' "$s/.ci/exclude.txt" | paste -sd '|' -)
+    lb=$(grep -Evhs '^\s*(#|$)' "$s/.ci/ctest-labels.txt" | paste -sd '|' -)
+    rm -f "$LOGS/$1/$lib.log"
+    gpu_step "$1 $lib" "$LOGS/$1/$lib.log" ctest --test-dir "$s/build" -j1 --output-on-failure \
+      --timeout "${CTEST_TIMEOUT:-900}" ${lb:+-L "$lb"} ${ex:+-E "$ex"}
+    [ $GPU_DEAD -eq 0 ] && grep -hs 'tests passed' "$LOGS/$1/$lib.log" | sed 's/^/      /' | tee -a "$SUMMARY"
+  done
+}
+
+# zeroRK's scripts/build_aurora.sh without its module loads and unbounded make -j.
+build_zerork() {
+  git clone -q --depth 1 -b hip-chipStar git@github.com:CHIP-SPV/zero-rk.git zero-rk &&
+    mkdir zero-rk/build_aurora && cd zero-rk/build_aurora &&
+    cmake -DCMAKE_EXE_LINKER_FLAGS=-lintlc -DENABLE_GPU=ON -DENABLE_MAGMA=OFF -DCMAKE_CXX_COMPILER=hipcc \
+      -DCMAKE_C_COMPILER=hipcc -DZERORK_TESTS=OFF -DZERORK_HAVE_MKL=OFF .. && make -j"$JOBS" && make install
+}
+
+# OpenSn's dependencies do not use chipStar, so each runner builds them once.
+build_opensn() {
+  local deps=$DEPS_CACHE/opensn-deps-$OPENSN_COMMIT
+  mkdir -p "$DEPS_CACHE" && exec 9> "$DEPS_CACHE/lock" && flock 9 || return 1
+  git clone -q https://github.com/Open-Sn/opensn.git && cd opensn && git checkout -q $OPENSN_COMMIT || return 1
+  [ -s "$deps/bin/set_opensn_env.sh" ] || {
+    rm -rf "$deps" && cmake -S tools/dependencies -B build_deps -DCMAKE_INSTALL_PREFIX="$deps" -DJOBS="$JOBS" \
+      -DCMAKE_DISABLE_FIND_PACKAGE_Boost=ON &&
+      cmake --build build_deps -j1; } || { rm -rf "$deps"; return 1; }
+  # CMAKE_HIP_PLATFORM=spirv needs CMake 4.3.
+  local cm=$DEPS_CACHE/cmake-4.3.2-linux-x86_64 tgz=$DEPS_CACHE/cmake-4.3.2.tar.gz
+  [ -x "$cm/bin/cmake" ] || {
+    curl -sSfL -o "$tgz" https://github.com/Kitware/CMake/releases/download/v4.3.2/cmake-4.3.2-linux-x86_64.tar.gz &&
+      echo "791ae3604841ca03cb3889a3ad89165346e4b180ae3448efd4b0caa9ef46d245  $tgz" | sha256sum -c - &&
+      tar xzf "$tgz" -C "$DEPS_CACHE" && rm -f "$tgz"; } || return 1
+  source "$deps/bin/set_opensn_env.sh" &&
+    CC=hipcc CXX=hipcc "$cm/bin/cmake" -S . -B build -DOPENSN_WITH_HIP=ON -DCMAKE_HIP_ARCHITECTURES=spirv \
+      -DCMAKE_HIP_PLATFORM=spirv -DCMAKE_BUILD_TYPE=RelWithDebInfo &&
+    "$cm/bin/cmake" --build build -j"$JOBS"
+}
+
+run_apps() {
+  local lu out=$WRK/apps/zero-rk/build_aurora/inst_dir/share/zerork/examples/cfd_plugin_tester/outputs_gpu_lu
+  note "SKIP  $1 zeroRK box_reactor (zerork_correctness_box_reactor.sh needs Aurora's PBS and MPI launcher)"
+  for lu in 1 0; do
+    rm -rf "$out$lu"
+    gpu_step "$1 zeroRK hydrogen lu$lu" "$LOGS/$1/zerork-hydrogen-lu$lu.log" bash -c 'cd "$0" && bash "$1" "$2"' \
+      "$WRK/apps/zero-rk" "$SRC/tests/apps/ZeroRK/zerork_correctness_hydrogen.sh" "$lu"
+    cp -r "$out$lu" "$LOGS/$1/" 2>/dev/null
+  done
+  rm -rf "$WRK/apps/opensn-run" && mkdir "$WRK/apps/opensn-run" && cp "$SRC"/tests/apps/OpenSn/* "$WRK/apps/opensn-run"
+  gpu_step "$1 OpenSn strong_scaling" "$LOGS/$1/opensn.log" bash -c \
+    'cd "$0" && source "$1" && CHIP_LOGLEVEL=off mpiexec -n 1 "$2" -i strong_scaling.py' "$WRK/apps/opensn-run" \
+    "$DEPS_CACHE/opensn-deps-$OPENSN_COMMIT/bin/set_opensn_env.sh" "$WRK/apps/opensn/build/python/opensn"
+}
+
+note "=== $(date -u +%FT%TZ) $(hostname) GPU $GPU_PCI $DRV ($RENDER) chipStar $(git -C "$SRC" log -1 --format='%h %s') -j$JOBS"
+if wants build; then
+  rm -rf "$WRK/install" "$WRK/staging"
+  (cd "$SRC" && ./install_chipstar.py --all --with-tests -j "$JOBS" --install-dir "$WRK/install" \
+    --staging-dir "$WRK/staging" --module-dir "$WRK/modulefiles") > "$LOGS/build.log" 2>&1
+  result "build: install_chipstar.py --all --with-tests" $? "$LOGS/build.log"
+  sed 's/\x1b\[[0-9;]*m//g' "$LOGS/build.log" | grep -E '^\[(SUCCESS|FAILED)\]' | cut -c1-150 | sed 's/^/      /' | tee -a "$SUMMARY"
+  for s in "$WRK"/staging/*/; do note "      $(basename "$s") $(git -C "$s" rev-parse --short HEAD)"; done
+fi
+PREFIX=$(ls -d "$WRK"/install/chipStar/*/ | tail -1) && PREFIX=${PREFIX%/}
+[ -x "$PREFIX/bin/hipcc" ] || { note "FAIL  no chipStar install under $WRK/install"; exit 1; }
+export HIP_PATH=$PREFIX HIP_PLATFORM=spirv PATH=$PREFIX/bin:$PATH CMAKE_PREFIX_PATH=$PREFIX:$CMAKE_PREFIX_PATH
+export LD_LIBRARY_PATH=$PREFIX/lib:$LD_LIBRARY_PATH LIBRARY_PATH=$PREFIX/lib:$LIBRARY_PATH
+if wants apps; then
+  rm -rf "$WRK/apps" && mkdir "$WRK/apps"
+  (cd "$WRK/apps" && build_zerork) > "$LOGS/zerork-build.log" 2>&1; result "build zeroRK" $? "$LOGS/zerork-build.log"
+  (cd "$WRK/apps" && build_opensn) > "$LOGS/opensn-build.log" 2>&1; result "build OpenSn" $? "$LOGS/opensn-build.log"
+fi
+wants libs || wants apps || BACKENDS=""
+rm -f "$LOGS/fault"
+CURSOR=$(journalctl -k -b -n0 --show-cursor -q --no-pager | sed -n 's/^-- cursor: //p')
+[ -z "$BACKENDS" ] || [ -n "$CURSOR" ] || { note "FAIL  kernel journal unreadable, cannot watch the GPU"; exit 1; }
+[ -z "$BACKENDS" ] || { n=$(journalctl -k -b -q --no-pager | grep -ciE "$DRV $GPU_PCI.*wedged"; echo "${PIPESTATUS[0]}"); [ "$(echo $n)" = "0 0" ]; } ||
+  { note "FAIL  $GPU_PCI was declared wedged earlier in this boot, or the kernel journal is unreadable"; exit 1; }
+[ -z "$BACKENDS" ] || command -v fuser > /dev/null || { note "FAIL  fuser not found, cannot see other GPU users"; exit 1; }
+# Arc A-series GPUs have no FP64 hardware; the apps and most libraries need double, so emulate it there.
+if [ -n "$BACKENDS" ]; then
+  gpu_step "fp64 probe" "$LOGS/clinfo-raw.txt" env NEOReadDebugKeys=1 FilterBdfPath="$GPU_PCI" clinfo --raw
+  t=$(sed -n 's/^\[\([A-Z]*\)\/\*\] *CL_PLATFORM_NAME *Intel(R) OpenCL Graphics$/\1/p' "$LOGS/clinfo-raw.txt" | head -1)
+  grep -q "^\[$t/0\] *CL_DEVICE_DOUBLE_FP_CONFIG" "$LOGS/clinfo-raw.txt" || {
+    export OverrideDefaultFP64Settings=1 IGC_EnableDPEmulation=1
+    note "NOTE  $GPU_PCI has no FP64 hardware, double runs under NEO's FP64 emulation"
+  }
+fi
+for be in $BACKENDS; do
+  mkdir -p "$LOGS/$be" && backend_env "$be"
+  wants libs && run_libs "$be"
+  wants apps && run_apps "$be"
+done
+note "=== $FAILED failed"
+[ "$FAILED" -eq 0 ] && [ "$GPU_DEAD" -eq 0 ]

From abf550b0654a9a4aea1410832f24954d5f243858 Mon Sep 17 00:00:00 2001
From: Paulius Velesko <pvelesko@pglc.io>
Date: Wed, 14 Oct 2026 04:31:59 +0300
Subject: [PATCH 02/31] ci: add libs-apps workflow, manual or /check-apps on a
 PR

Runs scripts/ci/libs-apps.sh on a runner labelled [self-hosted,
intel-dgpu]; any Intel discrete GPU will do, and meatloaf (Arc A770)
carries the label. The label is dedicated so that a runner added only
for this lane does not also pick up unit-tests-x86-intel.yml jobs through
the default Linux/X64 labels.

Two triggers. workflow_dispatch from the Actions tab takes a pull
request number or a ref. A /check-apps comment on a pull request, from
the same allow list as /run-aurora-ci in trigger-gitlab-ci.yml, tests
that pull request's head commit and reacts with a rocket. Whenever a
pull request is involved the job posts a check-apps commit status on its
head commit, pending at the start and success or failure at the end,
linking to the run. A run started from the Actions tab is otherwise
attached to the dispatched branch's commit, not to the pull request, so
the status is what makes the result show in the pull request's checks.

The job passes -j to the script: one job per 3 GB of RAM, at most
nproc - 1. The script's own default subtracts the 1-minute load
average, which on a runner still includes the previous job's build: a
hand run on meatloaf started while a cross-compile job was finishing
picked -j1 for its whole multi-hour build.

The header lists what the runner needs installed; meatloaf lacked the
GoogleTest, GoogleMock, BLAS, LAPACK, Boost and pybind11 development
packages, which rocSPARSE's clients, zeroRK and OpenSn find with
find_package.

Both triggers only work once this file is on the default branch, which
is where GitHub reads workflow_dispatch and issue_comment workflows from.

The script is taken from a checkout of the default or dispatched branch
and the tree under test from a second checkout, so a pull request opened
before the script existed can still be tested. The concurrency group is
on the job, not the workflow: every issue comment in the repository
starts a run of this workflow whose job is skipped, and a workflow-level
group would let those skipped runs replace a queued /check-apps run. One
run at a time because the job owns the GPU. The summary and all logs are
uploaded whatever the outcome.
---
 .github/workflows/libs-apps.yml | 92 +++++++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)
 create mode 100644 .github/workflows/libs-apps.yml

diff --git a/.github/workflows/libs-apps.yml b/.github/workflows/libs-apps.yml
new file mode 100644
index 000000000..8cc861030
--- /dev/null
+++ b/.github/workflows/libs-apps.yml
@@ -0,0 +1,92 @@
+name: Libraries and apps (Intel dGPU)
+
+# chipStar plus every downstream library with its tests, and the zeroRK and
+# OpenSn checks, on an Intel discrete GPU under Level Zero and OpenCL. Run it
+# from the Actions tab, or comment /check-apps on a pull request to get a
+# check-apps status on its head commit.
+#
+# Runner needs: modules oneapi/2025.0.4 and llvm/23.0, kernel journal access
+# (adm group), clinfo, fuser, OpenMPI, gfortran, and on Ubuntu libgtest-dev
+# libgmock-dev libblas-dev liblapack-dev libboost-dev pybind11-dev.
+on:
+  workflow_dispatch:
+    inputs:
+      pr:
+        description: Pull request number to test (overrides ref)
+        required: false
+      ref:
+        description: Branch, tag or sha to test, by default the dispatched branch
+        required: false
+  issue_comment:
+    types: [created]
+
+permissions:
+  contents: read
+  statuses: write
+  pull-requests: write
+
+jobs:
+  libs-apps:
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      (github.event.issue.pull_request && startsWith(github.event.comment.body, '/check-apps') &&
+       contains(fromJSON('["colleeneb", "pvelesko"]'), github.event.comment.user.login))
+    runs-on: [self-hosted, intel-dgpu]
+    timeout-minutes: 1440
+    concurrency:
+      group: libs-apps
+    env:
+      PR: ${{ github.event.issue.number || inputs.pr }}
+      REF: ${{ inputs.ref }}
+    steps:
+      - name: Resolve the commit to test
+        id: target
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const pr = process.env.PR;
+            if (!pr) return core.setOutput('sha', process.env.REF || context.sha);
+            const { data } = await github.rest.pulls.get({ ...context.repo, pull_number: Number(pr) });
+            core.setOutput('sha', data.head.sha);
+            await github.rest.repos.createCommitStatus({
+              ...context.repo, sha: data.head.sha, state: 'pending', context: 'check-apps',
+              description: 'libraries and apps running',
+              target_url: `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`,
+            });
+            if (context.eventName === 'issue_comment')
+              await github.rest.reactions.createForIssueComment({
+                ...context.repo, comment_id: context.payload.comment.id, content: 'rocket',
+              });
+      # The script comes from the default or dispatched branch, so it can test PRs older than it.
+      - uses: actions/checkout@v4
+        with:
+          path: ci
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ steps.target.outputs.sha }}
+          path: src
+          fetch-depth: 0
+          submodules: recursive
+      # The job owns the machine; the script's load-based default would shrink by the previous job's load.
+      - name: Build and test
+        run: |
+          jobs=$(awk -v n="$(nproc)" '/^MemTotal/ { j = int($2 / 3145728); if (n - 1 < j) j = n - 1; print (j < 1 ? 1 : j) }' /proc/meminfo)
+          ci/scripts/ci/libs-apps.sh -s src -w "${{ runner.temp }}/libs-apps" -j "$jobs"
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: libs-apps-logs
+          path: |
+            ${{ runner.temp }}/libs-apps/summary.txt
+            ${{ runner.temp }}/libs-apps/logs
+      - name: Report check-apps
+        if: always() && env.PR && steps.target.outputs.sha
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const ok = '${{ job.status }}' === 'success';
+            await github.rest.repos.createCommitStatus({
+              ...context.repo, sha: '${{ steps.target.outputs.sha }}', state: ok ? 'success' : 'failure',
+              context: 'check-apps', description: ok ? 'libraries and apps pass' : 'see summary.txt in the run artifacts',
+              target_url: `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`,
+            });

From 44aa151c8420f042304839c3f962286614a75665 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sat, 17 Oct 2026 21:03:13 +0300
Subject: [PATCH 03/31] ci: libs-apps checkouts do not persist the GITHUB_TOKEN

actions/checkout writes the token into .git/config by default. The job
then builds and runs the tested tree's own code, which could read it.
Neither checkout pushes, so persist-credentials: false costs nothing.
---
 .github/workflows/libs-apps.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.github/workflows/libs-apps.yml b/.github/workflows/libs-apps.yml
index 8cc861030..7863b3f58 100644
--- a/.github/workflows/libs-apps.yml
+++ b/.github/workflows/libs-apps.yml
@@ -61,10 +61,12 @@ jobs:
       - uses: actions/checkout@v4
         with:
           path: ci
+          persist-credentials: false
       - uses: actions/checkout@v4
         with:
           ref: ${{ steps.target.outputs.sha }}
           path: src
+          persist-credentials: false
           fetch-depth: 0
           submodules: recursive
       # The job owns the machine; the script's load-based default would shrink by the previous job's load.

From 2e969de3631ec95e9a3ccdfdd40bc267e5fde9a1 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sat, 17 Oct 2026 21:03:25 +0300
Subject: [PATCH 04/31] ci: libs-apps selects the GPU without NEO debug keys,
 always emulates FP64

NEOReadDebugKeys=1 puts the Intel driver into debug mode for every
process of every test, which makes high-scratch kernels fail with
CL_OUT_OF_RESOURCES (see the comment above chipUnrestrictedAllocSize in
src/CHIPDriver.cc) and may change performance. FilterBdfPath, the only
reason for it, is itself a debug key.

Level Zero and the Intel OpenCL platform both honour ZE_AFFINITY_MASK,
and ZE_ENABLE_PCI_ID_DEVICE_ORDER=1 makes its index follow PCI order, so
the index is the position of GPU_PCI among the Intel display devices
sorted by address. On meatloaf (iGPU 0000:00:02.0, Arc A770 0000:03:00.0)
index 1 leaves a single device named Arc A770 under both CHIP_BE=level0
and CHIP_BE=opencl, and clinfo reports PCI-E 0000:03:00.0 for it. With
only one OpenCL platform left, the clinfo -l platform lookup and
CHIP_PLATFORM are no longer needed and go.

FP64 emulation (IGC_EnableDPEmulation, OverrideDefaultFP64Settings) is
exported unconditionally, as the maintainers' shells do, which removes
the clinfo --raw probe and its NOTE line.
---
 scripts/ci/libs-apps.sh | 33 +++++++++++----------------------
 1 file changed, 11 insertions(+), 22 deletions(-)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index 54e1759ef..6b0a2eec7 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -9,9 +9,10 @@ set -o pipefail
 
 HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
 SRC=$HERE WRK="" JOBS="" BACKENDS="level0 opencl" PHASES=build,libs,apps
+INTEL_GPUS=$(grep -l 0x8086 /sys/bus/pci/devices/*/vendor | xargs -n1 dirname |
+  while read -r d; do grep -q '^0x03' "$d/class" && echo "${d##*/}"; done | sort)
 # First Intel display device off bus 00, where the integrated GPU sits.
-GPU_PCI=${GPU_PCI:-$(grep -l 0x8086 /sys/bus/pci/devices/*/vendor | xargs -n1 dirname |
-  while read -r d; do grep -q '^0x03' "$d/class" && echo "${d##*/}"; done | grep -v '^0000:00:' | head -1)}
+GPU_PCI=${GPU_PCI:-$(grep -v '^0000:00:' <<< "$INTEL_GPUS" | head -1)}
 DEPS_CACHE=${DEPS_CACHE:-$HOME/.cache/chipstar-libs-apps}
 OPENSN_COMMIT=d0644cd9c633c6ae5e7110d3f8721641e5ef2982
 while getopts "s:w:j:b:p:" o; do
@@ -27,6 +28,13 @@ LOGS=$WRK/logs SUMMARY=$WRK/summary.txt FAILED=0 GPU_DEAD=0
   '{ j = int(n - $1 - 1); if (m / 3 < j) j = int(m / 3); print (j < 1 ? 1 : j) }' /proc/loadavg)}"
 RENDER=/dev/dri/$(basename "$(ls -d /sys/bus/pci/devices/"$GPU_PCI"/drm/renderD*)")
 DRV=$(basename "$(readlink /sys/bus/pci/devices/"$GPU_PCI"/driver)")
+# Level Zero and the Intel OpenCL platform both hide every GPU but this one, indexed in PCI order.
+ZE_IDX=$(($(grep -nx "$GPU_PCI" <<< "$INTEL_GPUS" | cut -d: -f1) - 1))
+[ "$ZE_IDX" -ge 0 ] || { echo "$GPU_PCI is not an Intel display device"; exit 2; }
+export ZE_ENABLE_PCI_ID_DEVICE_ORDER=1 ZE_AFFINITY_MASK=$ZE_IDX CHIP_LOGLEVEL=err
+unset CHIP_PLATFORM CHIP_DEVICE CHIP_DEVICE_TYPE
+# Arc A-series GPUs have no FP64 hardware; the apps and most libraries need double.
+export OverrideDefaultFP64Settings=1 IGC_EnableDPEmulation=1
 source "$HERE/scripts/module-env.sh"
 module load oneapi/2025.0.4 llvm/23.0 || exit 1
 # zeroRK and OpenSn bundle projects that declare a CMake minimum below 3.5.
@@ -67,16 +75,6 @@ gpu_step() {
   result "$name" $? "$log"; STEP=""
 }
 
-backend_env() {
-  export CHIP_BE=$1 CHIP_LOGLEVEL=err NEOReadDebugKeys=1 FilterBdfPath=$GPU_PCI
-  unset CHIP_PLATFORM CHIP_DEVICE CHIP_DEVICE_TYPE
-  [ "$1" = opencl ] || return 0
-  # FilterBdfPath leaves only this GPU on the Intel graphics platform.
-  gpu_step "opencl platform probe" "$LOGS/opencl/clinfo.txt" clinfo -l
-  CHIP_PLATFORM=$(grep -s '^Platform' "$LOGS/opencl/clinfo.txt" | grep -n 'Intel(R) OpenCL Graphics' | cut -d: -f1)
-  export CHIP_PLATFORM=$((CHIP_PLATFORM - 1))
-}
-
 # Same .ci/exclude.txt and .ci/ctest-labels.txt convention as libraries-presubmit.yml.
 run_libs() {
   local lib s ex lb
@@ -161,17 +159,8 @@ CURSOR=$(journalctl -k -b -n0 --show-cursor -q --no-pager | sed -n 's/^-- cursor
 [ -z "$BACKENDS" ] || { n=$(journalctl -k -b -q --no-pager | grep -ciE "$DRV $GPU_PCI.*wedged"; echo "${PIPESTATUS[0]}"); [ "$(echo $n)" = "0 0" ]; } ||
   { note "FAIL  $GPU_PCI was declared wedged earlier in this boot, or the kernel journal is unreadable"; exit 1; }
 [ -z "$BACKENDS" ] || command -v fuser > /dev/null || { note "FAIL  fuser not found, cannot see other GPU users"; exit 1; }
-# Arc A-series GPUs have no FP64 hardware; the apps and most libraries need double, so emulate it there.
-if [ -n "$BACKENDS" ]; then
-  gpu_step "fp64 probe" "$LOGS/clinfo-raw.txt" env NEOReadDebugKeys=1 FilterBdfPath="$GPU_PCI" clinfo --raw
-  t=$(sed -n 's/^\[\([A-Z]*\)\/\*\] *CL_PLATFORM_NAME *Intel(R) OpenCL Graphics$/\1/p' "$LOGS/clinfo-raw.txt" | head -1)
-  grep -q "^\[$t/0\] *CL_DEVICE_DOUBLE_FP_CONFIG" "$LOGS/clinfo-raw.txt" || {
-    export OverrideDefaultFP64Settings=1 IGC_EnableDPEmulation=1
-    note "NOTE  $GPU_PCI has no FP64 hardware, double runs under NEO's FP64 emulation"
-  }
-fi
 for be in $BACKENDS; do
-  mkdir -p "$LOGS/$be" && backend_env "$be"
+  mkdir -p "$LOGS/$be" && export CHIP_BE=$be
   wants libs && run_libs "$be"
   wants apps && run_apps "$be"
 done

From 9e7ddc1bb60c14465db6703b980e0633c066ec6b Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sat, 17 Oct 2026 21:03:26 +0300
Subject: [PATCH 05/31] ci: libs-apps caches the zeroRK and OpenSn builds per
 runner

Both apps were rebuilt on every run. They are now built under DEPS_CACHE,
keyed on the app source commit (zero-rk hip-chipStar head from git
ls-remote, OPENSN_COMMIT), marked complete by a .built file, and taken
under the same flock as the OpenSn dependencies. The zeroRK key also
carries a checksum of the installed rocprim and thrust headers, since
zeroRK compiles against them.

zeroRK's src/zerork/CMakeLists.txt hardcodes the include directories of
one machine (/space/pvelesko/install/HIP/rocThrust/main/include and
rocPRIM/2025.04.22/include). Where those exist, an old rocPRIM that
treats warpSize as constexpr is picked and fails to compile against
current chipStar. Passing -I$PREFIX/include in CMAKE_CXX_FLAGS puts this
install's headers first on the command line, ahead of the target's own
-I. zero-rk itself is not edited.

The cached binaries carry a RUNPATH, not an RPATH, so the LD_LIBRARY_PATH
the script exports (this run's $PREFIX/lib) takes precedence: ldd on
meatloaf resolves libCHIP, libhipblas, libhipsolver, libhipsparse and
librocsparse to the install under test.
---
 scripts/ci/libs-apps.sh | 33 ++++++++++++++++++++++-----------
 1 file changed, 22 insertions(+), 11 deletions(-)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index 6b0a2eec7..26c06ca86 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -90,19 +90,28 @@ run_libs() {
   done
 }
 
-# zeroRK's scripts/build_aurora.sh without its module loads and unbounded make -j.
+# Built trees are cached per runner by app commit; the libraries they load come from this run's PREFIX.
+lock() { mkdir -p "$DEPS_CACHE" && exec 9> "$DEPS_CACHE/lock" && flock 9; }
+
+# zeroRK's scripts/build_aurora.sh without its module loads and unbounded make -j. Its CMakeLists hardcodes
+# rocPRIM and rocThrust include dirs of one machine; the earlier -I makes this install's headers win.
 build_zerork() {
-  git clone -q --depth 1 -b hip-chipStar git@github.com:CHIP-SPV/zero-rk.git zero-rk &&
-    mkdir zero-rk/build_aurora && cd zero-rk/build_aurora &&
-    cmake -DCMAKE_EXE_LINKER_FLAGS=-lintlc -DENABLE_GPU=ON -DENABLE_MAGMA=OFF -DCMAKE_CXX_COMPILER=hipcc \
-      -DCMAKE_C_COMPILER=hipcc -DZERORK_TESTS=OFF -DZERORK_HAVE_MKL=OFF .. && make -j"$JOBS" && make install
+  lock || return 1
+  [ -e "$ZRK/.built" ] && { echo "reusing $ZRK"; return 0; }
+  rm -rf "$ZRK" && git clone -q --depth 1 -b hip-chipStar git@github.com:CHIP-SPV/zero-rk.git "$ZRK" &&
+    mkdir "$ZRK/build_aurora" && cd "$ZRK/build_aurora" &&
+    cmake -DCMAKE_CXX_FLAGS="-I$PREFIX/include" -DCMAKE_EXE_LINKER_FLAGS=-lintlc -DENABLE_GPU=ON -DENABLE_MAGMA=OFF -DCMAKE_CXX_COMPILER=hipcc \
+      -DCMAKE_C_COMPILER=hipcc -DZERORK_TESTS=OFF -DZERORK_HAVE_MKL=OFF .. && make -j"$JOBS" && make install &&
+    touch "$ZRK/.built"
 }
 
 # OpenSn's dependencies do not use chipStar, so each runner builds them once.
 build_opensn() {
   local deps=$DEPS_CACHE/opensn-deps-$OPENSN_COMMIT
-  mkdir -p "$DEPS_CACHE" && exec 9> "$DEPS_CACHE/lock" && flock 9 || return 1
-  git clone -q https://github.com/Open-Sn/opensn.git && cd opensn && git checkout -q $OPENSN_COMMIT || return 1
+  lock || return 1
+  [ -e "$OSN/.built" ] && { echo "reusing $OSN"; return 0; }
+  rm -rf "$OSN" && git clone -q https://github.com/Open-Sn/opensn.git "$OSN" && cd "$OSN" &&
+    git checkout -q $OPENSN_COMMIT || return 1
   [ -s "$deps/bin/set_opensn_env.sh" ] || {
     rm -rf "$deps" && cmake -S tools/dependencies -B build_deps -DCMAKE_INSTALL_PREFIX="$deps" -DJOBS="$JOBS" \
       -DCMAKE_DISABLE_FIND_PACKAGE_Boost=ON &&
@@ -116,22 +125,22 @@ build_opensn() {
   source "$deps/bin/set_opensn_env.sh" &&
     CC=hipcc CXX=hipcc "$cm/bin/cmake" -S . -B build -DOPENSN_WITH_HIP=ON -DCMAKE_HIP_ARCHITECTURES=spirv \
       -DCMAKE_HIP_PLATFORM=spirv -DCMAKE_BUILD_TYPE=RelWithDebInfo &&
-    "$cm/bin/cmake" --build build -j"$JOBS"
+    "$cm/bin/cmake" --build build -j"$JOBS" && touch "$OSN/.built"
 }
 
 run_apps() {
-  local lu out=$WRK/apps/zero-rk/build_aurora/inst_dir/share/zerork/examples/cfd_plugin_tester/outputs_gpu_lu
+  local lu out=$ZRK/build_aurora/inst_dir/share/zerork/examples/cfd_plugin_tester/outputs_gpu_lu
   note "SKIP  $1 zeroRK box_reactor (zerork_correctness_box_reactor.sh needs Aurora's PBS and MPI launcher)"
   for lu in 1 0; do
     rm -rf "$out$lu"
     gpu_step "$1 zeroRK hydrogen lu$lu" "$LOGS/$1/zerork-hydrogen-lu$lu.log" bash -c 'cd "$0" && bash "$1" "$2"' \
-      "$WRK/apps/zero-rk" "$SRC/tests/apps/ZeroRK/zerork_correctness_hydrogen.sh" "$lu"
+      "$ZRK" "$SRC/tests/apps/ZeroRK/zerork_correctness_hydrogen.sh" "$lu"
     cp -r "$out$lu" "$LOGS/$1/" 2>/dev/null
   done
   rm -rf "$WRK/apps/opensn-run" && mkdir "$WRK/apps/opensn-run" && cp "$SRC"/tests/apps/OpenSn/* "$WRK/apps/opensn-run"
   gpu_step "$1 OpenSn strong_scaling" "$LOGS/$1/opensn.log" bash -c \
     'cd "$0" && source "$1" && CHIP_LOGLEVEL=off mpiexec -n 1 "$2" -i strong_scaling.py' "$WRK/apps/opensn-run" \
-    "$DEPS_CACHE/opensn-deps-$OPENSN_COMMIT/bin/set_opensn_env.sh" "$WRK/apps/opensn/build/python/opensn"
+    "$DEPS_CACHE/opensn-deps-$OPENSN_COMMIT/bin/set_opensn_env.sh" "$OSN/build/python/opensn"
 }
 
 note "=== $(date -u +%FT%TZ) $(hostname) GPU $GPU_PCI $DRV ($RENDER) chipStar $(git -C "$SRC" log -1 --format='%h %s') -j$JOBS"
@@ -149,6 +158,8 @@ export HIP_PATH=$PREFIX HIP_PLATFORM=spirv PATH=$PREFIX/bin:$PATH CMAKE_PREFIX_P
 export LD_LIBRARY_PATH=$PREFIX/lib:$LD_LIBRARY_PATH LIBRARY_PATH=$PREFIX/lib:$LIBRARY_PATH
 if wants apps; then
   rm -rf "$WRK/apps" && mkdir "$WRK/apps"
+  ZRK=$DEPS_CACHE/zerork-$(git ls-remote git@github.com:CHIP-SPV/zero-rk.git hip-chipStar | cut -c1-40)-$(find "$PREFIX"/include/rocprim "$PREFIX"/include/thrust -type f | sort | xargs cat | cksum | cut -d' ' -f1)
+  OSN=$DEPS_CACHE/opensn-$OPENSN_COMMIT
   (cd "$WRK/apps" && build_zerork) > "$LOGS/zerork-build.log" 2>&1; result "build zeroRK" $? "$LOGS/zerork-build.log"
   (cd "$WRK/apps" && build_opensn) > "$LOGS/opensn-build.log" 2>&1; result "build OpenSn" $? "$LOGS/opensn-build.log"
 fi

From 97e79fb6c1ba2f1b78f9d11607165f7986ac541d Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sat, 17 Oct 2026 21:03:35 +0300
Subject: [PATCH 06/31] ci: libs-apps runs five zeroRK GPU code paths instead
 of four hydrogen runs

The four hydrogen runs (ZERORK_REACTOR_USE_LU 0 and 1, two backends)
all took the same Seulex finite-difference path. zerork_gpu_paths.sh
runs the h2 mechanism (10 species, a few seconds) through five
configurations of zerork_cfd_plugin_tester_gpu.x, each against a CPU
reference, with 2048 reactors (it must stay at or above n_reactors_min,
256, or zeroRK silently falls back to the CPU):

 cvode_analytic_cv  integrator 0, iterative, analytic Jacobian: SUNDIALS
                    nvector_hip reductions, SPGMR, analytic kernels
 seulex_fd_cv       integrator 1, finite-difference Jacobian kernels
 radau_analytic_cv  integrator 3: complex hipBLAS getrf/getri/getrs
 cvode_analytic_cp  constant pressure reactor kernels
 seulex_fd_notemp_batch  always_solve_temperature 0, n_reactors_max 1024:
                    the no-temperature Jacobian and multi-group batching

ZERORK_REACTOR_USE_LU alternates 1,0,1,0,1 so both the getrf+getrs and
the getrf+getri+bdmv factorisations are covered. Only the dense GPU path
exists in this build (MAGMA and hipSolverRf off), dense 0 on the GPU hits
assert(false), and CVODE with iterative 0 on the GPU falls back to a host
LAPACK solver, so none of those is used. The constant volume and
constant pressure CPU references are computed once and reused by both
backends. Each case prints one PASS or FAIL line, which libs-apps.sh
copies into summary.txt.

The base ymls lack constant_volume, always_solve_temperature and
n_reactors_max, so the new script appends keys that are missing.
zerork_correctness_hydrogen.sh keeps its <0|1> interface for
tests/apps/run_apps.sh, which still calls it.
---
 scripts/ci/libs-apps.sh               | 12 ++--
 tests/apps/ZeroRK/zerork_gpu_paths.sh | 80 +++++++++++++++++++++++++++
 2 files changed, 85 insertions(+), 7 deletions(-)
 create mode 100755 tests/apps/ZeroRK/zerork_gpu_paths.sh

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index 26c06ca86..0a8a806ce 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -129,14 +129,12 @@ build_opensn() {
 }
 
 run_apps() {
-  local lu out=$ZRK/build_aurora/inst_dir/share/zerork/examples/cfd_plugin_tester/outputs_gpu_lu
   note "SKIP  $1 zeroRK box_reactor (zerork_correctness_box_reactor.sh needs Aurora's PBS and MPI launcher)"
-  for lu in 1 0; do
-    rm -rf "$out$lu"
-    gpu_step "$1 zeroRK hydrogen lu$lu" "$LOGS/$1/zerork-hydrogen-lu$lu.log" bash -c 'cd "$0" && bash "$1" "$2"' \
-      "$ZRK" "$SRC/tests/apps/ZeroRK/zerork_correctness_hydrogen.sh" "$lu"
-    cp -r "$out$lu" "$LOGS/$1/" 2>/dev/null
-  done
+  rm -rf "$LOGS/$1/zerork-out"
+  gpu_step "$1 zeroRK GPU paths" "$LOGS/$1/zerork-paths.log" bash -c 'cd "$0" && bash "$1" "$2"' \
+    "$ZRK" "$SRC/tests/apps/ZeroRK/zerork_gpu_paths.sh" "$WRK/apps/zerork-out"
+  [ $GPU_DEAD -eq 0 ] && grep -hsE '^(PASS|FAIL) ' "$LOGS/$1/zerork-paths.log" | sed "s/^/      $1 /" | tee -a "$SUMMARY"
+  cp -r "$WRK/apps/zerork-out" "$LOGS/$1/" 2>/dev/null
   rm -rf "$WRK/apps/opensn-run" && mkdir "$WRK/apps/opensn-run" && cp "$SRC"/tests/apps/OpenSn/* "$WRK/apps/opensn-run"
   gpu_step "$1 OpenSn strong_scaling" "$LOGS/$1/opensn.log" bash -c \
     'cd "$0" && source "$1" && CHIP_LOGLEVEL=off mpiexec -n 1 "$2" -i strong_scaling.py' "$WRK/apps/opensn-run" \
diff --git a/tests/apps/ZeroRK/zerork_gpu_paths.sh b/tests/apps/ZeroRK/zerork_gpu_paths.sh
new file mode 100755
index 000000000..1c7e00899
--- /dev/null
+++ b/tests/apps/ZeroRK/zerork_gpu_paths.sh
@@ -0,0 +1,80 @@
+#!/bin/bash
+# ZeroRK GPU code path check: run the h2 mechanism on the GPU through five
+# integrator and reactor configurations and compare each with a CPU reference.
+#
+# Usage: zerork_gpu_paths.sh <outputs_dir>
+#   CPU references already in <outputs_dir> are reused, so the CPU runs happen
+#   once for any number of invocations (one per backend).
+#
+# Must be invoked from the zero-rk source root with build_aurora/ built and installed.
+
+[ $# -eq 1 ] || { echo "Usage: $0 <outputs_dir>"; exit 1; }
+out=$(mkdir -p "$1" && cd "$1" && pwd)
+export CHIP_JIT_FLAGS_OVERRIDE="-ze-opt-enable-auto-large-GRF-mode" CHIP_LOGLEVEL=off
+TOLERANCE=1e-4   # max relative difference in temperature and pressure
+
+cd build_aurora/inst_dir || exit 1
+exe=$PWD/bin/zerork_cfd_plugin_tester_gpu.x
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD/lib:$PWD/lib64
+cd share/zerork/examples/cfd_plugin_tester || exit 1
+
+# Replace KEY in FILE, or append it: the base ymls lack some keys.
+setYML() {
+  grep -q "^ *$2 *:" "$1" && sed -i "s|^ *$2 *:.*$|$2: $3|" "$1" || echo "$2: $3" >> "$1"
+}
+
+# run NAME GPU CV INTEGRATOR ANALYTIC ALWAYS_SOLVE_T N_REACTORS N_REACTORS_MAX LU
+run() {
+  local name=$1 in=$out/$1_input.yml zrk=$out/$1_zerork.yml
+  local fuel oxid
+  fuel=$(grep -v '^#' inputs/h2_fracs.log | grep -Eiv '^(O2|N2)\b' | sed 's/$/,/' | tr '\n' ' ')
+  oxid=$(grep -v '^#' inputs/h2_fracs.log | grep -Ei '^(O2|N2)\b' | sed 's/$/,/' | tr '\n' ' ')
+  cp inputs/base_tester.yml "$in"; cp inputs/base_plugin.yml "$zrk"
+  setYML "$in" mechanism_file "$PWD/../../mechanisms/hydrogen/h2_v1b_mech.txt"
+  setYML "$in" thermo_file "$PWD/../../mechanisms/hydrogen/h2_v1a_therm.txt"
+  setYML "$in" solution_time "$(echo "8.2565998*10^-7 * 2" | bc -l)"
+  setYML "$in" fuel_composition "{ $fuel }"
+  setYML "$in" oxidizer_composition "{ $oxid }"
+  setYML "$in" zerork_cfd_plugin_input "$zrk"
+  setYML "$in" n_reactors "$7"
+  setYML "$in" constant_volume "$3"
+  setYML "$in" always_solve_temperature "$6"
+  setYML "$in" reactor_history_file_prefix "$out/$name"
+  setYML "$zrk" mechanism_parsing_log "$out/$name.cklog"
+  setYML "$zrk" reactor_timing_log "$out/$name.log"
+  setYML "$zrk" preconditioner_threshold 3.20e-5
+  setYML "$zrk" gpu "$2"
+  setYML "$zrk" dense 1
+  setYML "$zrk" iterative 1
+  setYML "$zrk" integrator "$4"
+  setYML "$zrk" analytic "$5"
+  setYML "$zrk" n_reactors_max "$8"
+  ZERORK_REACTOR_USE_LU=$9 "$exe" "$in" > "$out/$name.stdout" 2>&1
+}
+
+# Temperature (col 3) and pressure (col 4) row by row against the CPU reference.
+compare() {
+  [ -s "$1" ] && [ -s "$2" ] && [ "$(wc -l < "$1")" -eq "$(wc -l < "$2")" ] || return 1
+  paste "$1" "$2" | awk -v tol=$TOLERANCE '/^#/ { next } {
+    h = NF / 2; dt = ($(h+3) - $3) / $3; dp = ($(h+4) - $4) / $4
+    if (dt < 0) dt = -dt; if (dp < 0) dp = -dp
+    if (dt > tol || dp > tol) { printf "step %s: T %g vs %g, P %g vs %g\n", $1, $3, $(h+3), $4, $(h+4); bad = 1 } }
+    END { exit bad }'
+}
+
+# n_reactors must stay >= the plugin's n_reactors_min (256) or the GPU is skipped silently.
+for cv in 1 0; do [ -s "$out/ref_cv${cv}_000.hist" ] || run ref_cv$cv 0 $cv 1 1 1 32 2048 0; done
+rc=0
+# name  constant_volume integrator analytic always_solve_temperature n_reactors_max ZERORK_REACTOR_USE_LU
+while read -r name cv integ analytic ast nmax lu; do
+  run "$name" 1 "$cv" "$integ" "$analytic" "$ast" 2048 "$nmax" "$lu"
+  if diff=$(compare "$out/ref_cv${cv}_000.hist" "$out/${name}_000.hist"); then echo "PASS  zeroRK $name"
+  else echo "FAIL  zeroRK $name ($out/$name.stdout)"; echo "$diff" | head -5; rc=1; fi
+done << 'CASES'
+cvode_analytic_cv      1 0 1 1 2048 1
+seulex_fd_cv           1 1 0 1 2048 0
+radau_analytic_cv      1 3 1 1 2048 1
+cvode_analytic_cp      0 0 1 1 2048 0
+seulex_fd_notemp_batch 1 1 0 0 1024 1
+CASES
+exit $rc

From d040cc55cc18d828e11e4a36f5ee68e5ab438796 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sat, 17 Oct 2026 21:03:36 +0300
Subject: [PATCH 07/31] ci: libs-apps bounds the zeroRK and OpenSn steps with
 timeout

A hang that logs nothing to the kernel journal is invisible to the
fault watcher, so the step would run until the 24 hour job limit.
1800 seconds is well above either app's runtime.
---
 scripts/ci/libs-apps.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index 0a8a806ce..2c4d89e0c 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -131,12 +131,12 @@ build_opensn() {
 run_apps() {
   note "SKIP  $1 zeroRK box_reactor (zerork_correctness_box_reactor.sh needs Aurora's PBS and MPI launcher)"
   rm -rf "$LOGS/$1/zerork-out"
-  gpu_step "$1 zeroRK GPU paths" "$LOGS/$1/zerork-paths.log" bash -c 'cd "$0" && bash "$1" "$2"' \
+  gpu_step "$1 zeroRK GPU paths" "$LOGS/$1/zerork-paths.log" timeout 1800 bash -c 'cd "$0" && bash "$1" "$2"' \
     "$ZRK" "$SRC/tests/apps/ZeroRK/zerork_gpu_paths.sh" "$WRK/apps/zerork-out"
   [ $GPU_DEAD -eq 0 ] && grep -hsE '^(PASS|FAIL) ' "$LOGS/$1/zerork-paths.log" | sed "s/^/      $1 /" | tee -a "$SUMMARY"
   cp -r "$WRK/apps/zerork-out" "$LOGS/$1/" 2>/dev/null
   rm -rf "$WRK/apps/opensn-run" && mkdir "$WRK/apps/opensn-run" && cp "$SRC"/tests/apps/OpenSn/* "$WRK/apps/opensn-run"
-  gpu_step "$1 OpenSn strong_scaling" "$LOGS/$1/opensn.log" bash -c \
+  gpu_step "$1 OpenSn strong_scaling" "$LOGS/$1/opensn.log" timeout 1800 bash -c \
     'cd "$0" && source "$1" && CHIP_LOGLEVEL=off mpiexec -n 1 "$2" -i strong_scaling.py' "$WRK/apps/opensn-run" \
     "$DEPS_CACHE/opensn-deps-$OPENSN_COMMIT/bin/set_opensn_env.sh" "$OSN/build/python/opensn"
 }

From e8c7ab078779a17192f52cc8a29874793bf9bac6 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sat, 17 Oct 2026 22:38:34 +0300
Subject: [PATCH 08/31] ci: libs-apps runner needs list drops clinfo, adds what
 the script uses

The script no longer calls clinfo. It does call bc (zeroRK case
setup), curl and sha256sum (CMake download for OpenSn), and git
clone over ssh from github.com (zero-rk).
---
 .github/workflows/libs-apps.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/libs-apps.yml b/.github/workflows/libs-apps.yml
index 7863b3f58..e93332a78 100644
--- a/.github/workflows/libs-apps.yml
+++ b/.github/workflows/libs-apps.yml
@@ -6,7 +6,7 @@ name: Libraries and apps (Intel dGPU)
 # check-apps status on its head commit.
 #
 # Runner needs: modules oneapi/2025.0.4 and llvm/23.0, kernel journal access
-# (adm group), clinfo, fuser, OpenMPI, gfortran, and on Ubuntu libgtest-dev
+# (adm group), fuser, bc, curl, sha256sum, git over ssh to github.com, OpenMPI, gfortran, and on Ubuntu libgtest-dev
 # libgmock-dev libblas-dev liblapack-dev libboost-dev pybind11-dev.
 on:
   workflow_dispatch:

From 57ed9b6c38586d494fff4a4979532355553822aa Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sat, 17 Oct 2026 22:38:35 +0300
Subject: [PATCH 09/31] ci: libs-apps splits resolve, run and report so the
 self-hosted job holds no write token

The self-hosted job runs the tested tree's code as the runner user. That
user can rewrite the downloaded action files under _work/_actions, and
the later github-script step and the checkout post step would then run
them with the job token, which had statuses:write and pull-requests:write.

resolve (ubuntu-latest) keeps the allow list gate, resolves the commit,
sets the pending status and reacts to the comment, and exposes the sha.
run (self-hosted, concurrency libs-apps) needs resolve and has
contents: read only; it checks out, builds, tests and uploads logs.
report (ubuntu-latest) needs both, runs always when a PR is set, and
posts the final status from needs.run.result. Top level permissions are
now empty so each job states what it needs. resolve skipped by the gate
skips the other two with it.
---
 .github/workflows/libs-apps.yml | 49 +++++++++++++++++++++++----------
 1 file changed, 34 insertions(+), 15 deletions(-)

diff --git a/.github/workflows/libs-apps.yml b/.github/workflows/libs-apps.yml
index e93332a78..f042abef3 100644
--- a/.github/workflows/libs-apps.yml
+++ b/.github/workflows/libs-apps.yml
@@ -20,21 +20,21 @@ on:
   issue_comment:
     types: [created]
 
-permissions:
-  contents: read
-  statuses: write
-  pull-requests: write
+permissions: {}
 
 jobs:
-  libs-apps:
+  # Runs on GitHub's hosted runners: the job below executes the tested tree's code, so it never holds the write token.
+  resolve:
     if: >-
       github.event_name == 'workflow_dispatch' ||
       (github.event.issue.pull_request && startsWith(github.event.comment.body, '/check-apps') &&
        contains(fromJSON('["colleeneb", "pvelesko"]'), github.event.comment.user.login))
-    runs-on: [self-hosted, intel-dgpu]
-    timeout-minutes: 1440
-    concurrency:
-      group: libs-apps
+    runs-on: ubuntu-latest
+    permissions:
+      statuses: write
+      pull-requests: write
+    outputs:
+      sha: ${{ steps.target.outputs.sha }}
     env:
       PR: ${{ github.event.issue.number || inputs.pr }}
       REF: ${{ inputs.ref }}
@@ -57,6 +57,16 @@ jobs:
               await github.rest.reactions.createForIssueComment({
                 ...context.repo, comment_id: context.payload.comment.id, content: 'rocket',
               });
+
+  run:
+    needs: resolve
+    runs-on: [self-hosted, intel-dgpu]
+    timeout-minutes: 1440
+    permissions:
+      contents: read
+    concurrency:
+      group: libs-apps
+    steps:
       # The script comes from the default or dispatched branch, so it can test PRs older than it.
       - uses: actions/checkout@v4
         with:
@@ -64,7 +74,7 @@ jobs:
           persist-credentials: false
       - uses: actions/checkout@v4
         with:
-          ref: ${{ steps.target.outputs.sha }}
+          ref: ${{ needs.resolve.outputs.sha }}
           path: src
           persist-credentials: false
           fetch-depth: 0
@@ -81,14 +91,23 @@ jobs:
           path: |
             ${{ runner.temp }}/libs-apps/summary.txt
             ${{ runner.temp }}/libs-apps/logs
-      - name: Report check-apps
-        if: always() && env.PR && steps.target.outputs.sha
-        uses: actions/github-script@v7
+
+  report:
+    needs: [resolve, run]
+    if: always() && needs.resolve.outputs.sha != '' && (github.event.issue.number || inputs.pr)
+    runs-on: ubuntu-latest
+    permissions:
+      statuses: write
+    steps:
+      - uses: actions/github-script@v7
+        env:
+          RESULT: ${{ needs.run.result }}
+          SHA: ${{ needs.resolve.outputs.sha }}
         with:
           script: |
-            const ok = '${{ job.status }}' === 'success';
+            const ok = process.env.RESULT === 'success';
             await github.rest.repos.createCommitStatus({
-              ...context.repo, sha: '${{ steps.target.outputs.sha }}', state: ok ? 'success' : 'failure',
+              ...context.repo, sha: process.env.SHA, state: ok ? 'success' : 'failure',
               context: 'check-apps', description: ok ? 'libraries and apps pass' : 'see summary.txt in the run artifacts',
               target_url: `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`,
             });

From 80fdee15a7bb3377e1687ba4e11c907a562954d7 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sat, 17 Oct 2026 22:38:58 +0300
Subject: [PATCH 10/31] ci: libs-apps kills the step's session after wait and
 escalates timeout to KILL

gpu_step only killed the session on the fault path. On the normal path
an mpiexec rank that survived the SIGTERM from timeout could outlive the
step and keep the GPU busy. pkill -s after wait removes any survivor, and
timeout -k 60 sends SIGKILL if the command ignores SIGTERM.
---
 scripts/ci/libs-apps.sh | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index 2c4d89e0c..3d9c35330 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -57,7 +57,7 @@ $j"
 
 # gpu_step NAME LOG CMD...; CMD gets its own session so a fault kills all of it.
 gpu_step() {
-  local name=$1 log=$2 pid="" f; shift 2
+  local name=$1 log=$2 pid="" f rc; shift 2
   [ $GPU_DEAD -eq 0 ] || { note "SKIP  $name"; return; }
   while f=$(gpu_faults) && [ "$f" -eq 0 ] && fuser "$RENDER" > /dev/null 2>&1; do echo "[gpu] $RENDER busy, waiting"; sleep 60; done
   if [ "$f" -eq 0 ]; then
@@ -71,8 +71,9 @@ gpu_step() {
     note "STOP  GPU reset, wedge or journal error at $name $(grep -so 'Start *[0-9]*: [^ ]*' "$log" | tail -1), see logs/fault"
     result "$name" 137 "$log"; STEP=""; return
   fi
-  wait $pid
-  result "$name" $? "$log"; STEP=""
+  wait $pid; rc=$?
+  pkill -KILL -s $pid 2>/dev/null
+  result "$name" $rc "$log"; STEP=""
 }
 
 # Same .ci/exclude.txt and .ci/ctest-labels.txt convention as libraries-presubmit.yml.
@@ -131,12 +132,12 @@ build_opensn() {
 run_apps() {
   note "SKIP  $1 zeroRK box_reactor (zerork_correctness_box_reactor.sh needs Aurora's PBS and MPI launcher)"
   rm -rf "$LOGS/$1/zerork-out"
-  gpu_step "$1 zeroRK GPU paths" "$LOGS/$1/zerork-paths.log" timeout 1800 bash -c 'cd "$0" && bash "$1" "$2"' \
+  gpu_step "$1 zeroRK GPU paths" "$LOGS/$1/zerork-paths.log" timeout -k 60 1800 bash -c 'cd "$0" && bash "$1" "$2"' \
     "$ZRK" "$SRC/tests/apps/ZeroRK/zerork_gpu_paths.sh" "$WRK/apps/zerork-out"
   [ $GPU_DEAD -eq 0 ] && grep -hsE '^(PASS|FAIL) ' "$LOGS/$1/zerork-paths.log" | sed "s/^/      $1 /" | tee -a "$SUMMARY"
   cp -r "$WRK/apps/zerork-out" "$LOGS/$1/" 2>/dev/null
   rm -rf "$WRK/apps/opensn-run" && mkdir "$WRK/apps/opensn-run" && cp "$SRC"/tests/apps/OpenSn/* "$WRK/apps/opensn-run"
-  gpu_step "$1 OpenSn strong_scaling" "$LOGS/$1/opensn.log" timeout 1800 bash -c \
+  gpu_step "$1 OpenSn strong_scaling" "$LOGS/$1/opensn.log" timeout -k 60 1800 bash -c \
     'cd "$0" && source "$1" && CHIP_LOGLEVEL=off mpiexec -n 1 "$2" -i strong_scaling.py' "$WRK/apps/opensn-run" \
     "$DEPS_CACHE/opensn-deps-$OPENSN_COMMIT/bin/set_opensn_env.sh" "$OSN/build/python/opensn"
 }

From b98d13a4307eb3f8590a8f379a4ed5180fb8757a Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sat, 17 Oct 2026 22:39:06 +0300
Subject: [PATCH 11/31] ci: libs-apps runs zerork_gpu_paths.sh from the control
 checkout

The script is taken from the default or dispatched branch (HERE, where
libs-apps.sh itself lives) so testing a ref older than the helper does
not fail with rc 127. Only the tested tree's chipStar sources come from
SRC.
---
 scripts/ci/libs-apps.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index 3d9c35330..6840d825d 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -133,7 +133,7 @@ run_apps() {
   note "SKIP  $1 zeroRK box_reactor (zerork_correctness_box_reactor.sh needs Aurora's PBS and MPI launcher)"
   rm -rf "$LOGS/$1/zerork-out"
   gpu_step "$1 zeroRK GPU paths" "$LOGS/$1/zerork-paths.log" timeout -k 60 1800 bash -c 'cd "$0" && bash "$1" "$2"' \
-    "$ZRK" "$SRC/tests/apps/ZeroRK/zerork_gpu_paths.sh" "$WRK/apps/zerork-out"
+    "$ZRK" "$HERE/tests/apps/ZeroRK/zerork_gpu_paths.sh" "$WRK/apps/zerork-out"
   [ $GPU_DEAD -eq 0 ] && grep -hsE '^(PASS|FAIL) ' "$LOGS/$1/zerork-paths.log" | sed "s/^/      $1 /" | tee -a "$SUMMARY"
   cp -r "$WRK/apps/zerork-out" "$LOGS/$1/" 2>/dev/null
   rm -rf "$WRK/apps/opensn-run" && mkdir "$WRK/apps/opensn-run" && cp "$SRC"/tests/apps/OpenSn/* "$WRK/apps/opensn-run"

From c8e3766c716ffdfa8a9f7a1f6039ef2bd5e5418c Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sat, 17 Oct 2026 22:39:06 +0300
Subject: [PATCH 12/31] ci: libs-apps corrects the comment on the zeroRK
 include path

CMake orders a target's compile line DEFINES, INCLUDES, FLAGS, so the
CMAKE_CXX_FLAGS -I$PREFIX/include comes after zero-rk's hardcoded
/space/pvelesko/install/HIP/rocThrust and rocPRIM include dirs, which
win where they exist. The earlier comment claimed the opposite. Tracked
in https://github.com/CHIP-SPV/chipStar/issues/2006, fixed in zero-rk.
---
 scripts/ci/libs-apps.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index 6840d825d..2fe9ab276 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -94,8 +94,8 @@ run_libs() {
 # Built trees are cached per runner by app commit; the libraries they load come from this run's PREFIX.
 lock() { mkdir -p "$DEPS_CACHE" && exec 9> "$DEPS_CACHE/lock" && flock 9; }
 
-# zeroRK's scripts/build_aurora.sh without its module loads and unbounded make -j. Its CMakeLists hardcodes
-# rocPRIM and rocThrust include dirs of one machine; the earlier -I makes this install's headers win.
+# zeroRK's scripts/build_aurora.sh without its module loads and unbounded make -j.
+# The -I does not beat zero-rk's hardcoded rocPRIM and rocThrust include dirs, see https://github.com/CHIP-SPV/chipStar/issues/2006
 build_zerork() {
   lock || return 1
   [ -e "$ZRK/.built" ] && { echo "reusing $ZRK"; return 0; }

From 39dc3941ca3de068bfe0c456f3a716080f92d287 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sat, 17 Oct 2026 22:39:06 +0300
Subject: [PATCH 13/31] ci: libs-apps cache keys include a checksum of the
 chipStar install

hipcc embeds the install's headers, device bitcode (lib/hip-device-lib)
and LLVM pass plugins (lib/llvm/libLLVMHip*.so) into what it builds, so
a cached zeroRK or OpenSn built against another chipStar would be reused
stale. Both keys now carry a cksum over include, lib/hip-device-lib,
lib/llvm and bin of the install. The include tree contains rocprim and
thrust, so the separate rocPRIM/rocThrust checksum is subsumed. The sum
is of file contents only, so it is stable between computations.
---
 scripts/ci/libs-apps.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index 2fe9ab276..7a16f6e4c 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -157,8 +157,10 @@ export HIP_PATH=$PREFIX HIP_PLATFORM=spirv PATH=$PREFIX/bin:$PATH CMAKE_PREFIX_P
 export LD_LIBRARY_PATH=$PREFIX/lib:$LD_LIBRARY_PATH LIBRARY_PATH=$PREFIX/lib:$LIBRARY_PATH
 if wants apps; then
   rm -rf "$WRK/apps" && mkdir "$WRK/apps"
-  ZRK=$DEPS_CACHE/zerork-$(git ls-remote git@github.com:CHIP-SPV/zero-rk.git hip-chipStar | cut -c1-40)-$(find "$PREFIX"/include/rocprim "$PREFIX"/include/thrust -type f | sort | xargs cat | cksum | cut -d' ' -f1)
-  OSN=$DEPS_CACHE/opensn-$OPENSN_COMMIT
+  # hipcc embeds headers, device bitcode and pass plugins of the install into what it builds.
+  INST=$(find "$PREFIX/include" "$PREFIX/lib/hip-device-lib" "$PREFIX/lib/llvm" "$PREFIX/bin" -type f | sort | xargs cat | cksum | cut -d' ' -f1)
+  ZRK=$DEPS_CACHE/zerork-$(git ls-remote git@github.com:CHIP-SPV/zero-rk.git hip-chipStar | cut -c1-40)-$INST
+  OSN=$DEPS_CACHE/opensn-$OPENSN_COMMIT-$INST
   (cd "$WRK/apps" && build_zerork) > "$LOGS/zerork-build.log" 2>&1; result "build zeroRK" $? "$LOGS/zerork-build.log"
   (cd "$WRK/apps" && build_opensn) > "$LOGS/opensn-build.log" 2>&1; result "build OpenSn" $? "$LOGS/opensn-build.log"
 fi

From 49f3d4e42bd95e90fac39b96f67dd3e3cb49ecfd Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sat, 17 Oct 2026 22:39:30 +0300
Subject: [PATCH 14/31] ci: zerork_gpu_paths.sh sets always_solve_temperature
 in the plugin yaml

always_solve_temperature is a key of the cfd plugin input ($zrk), read
by zerork_cfd_plugin, not of the tester input ($in). Written to the
tester yaml it was ignored, so the no-temperature case ran the normal
temperature path.
---
 tests/apps/ZeroRK/zerork_gpu_paths.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/apps/ZeroRK/zerork_gpu_paths.sh b/tests/apps/ZeroRK/zerork_gpu_paths.sh
index 1c7e00899..40849a6f5 100755
--- a/tests/apps/ZeroRK/zerork_gpu_paths.sh
+++ b/tests/apps/ZeroRK/zerork_gpu_paths.sh
@@ -38,12 +38,12 @@ run() {
   setYML "$in" zerork_cfd_plugin_input "$zrk"
   setYML "$in" n_reactors "$7"
   setYML "$in" constant_volume "$3"
-  setYML "$in" always_solve_temperature "$6"
   setYML "$in" reactor_history_file_prefix "$out/$name"
   setYML "$zrk" mechanism_parsing_log "$out/$name.cklog"
   setYML "$zrk" reactor_timing_log "$out/$name.log"
   setYML "$zrk" preconditioner_threshold 3.20e-5
   setYML "$zrk" gpu "$2"
+  setYML "$zrk" always_solve_temperature "$6"
   setYML "$zrk" dense 1
   setYML "$zrk" iterative 1
   setYML "$zrk" integrator "$4"

From 3cab4ff83f411596830783ec44faebb7d96861de Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sat, 17 Oct 2026 22:39:36 +0300
Subject: [PATCH 15/31] ci: zerork_gpu_paths.sh fails a case unless it ran
 fully on the GPU with fresh output

zeroRK retries failed GPU reactors on the CPU, and runs everything on
the CPU when no device is found or a group is below n_reactors_min, with
the hist still matching the CPU reference. A case now passes only if
every row of its timing log has n_cpu == 0 and n_gpu > 0 (columns 3 and
5), and for the no-temperature case also n_gpu_notemp > 0 (column 6).

Each case removes its previous hist, log and stdout before running, so a
run that dies early cannot be judged on the other backend's leftovers,
and a nonzero exit status of the executable is a FAIL. A CPU reference
whose run fails is removed so it is not reused. Valid references are
still computed once and shared by both backends.
---
 tests/apps/ZeroRK/zerork_gpu_paths.sh | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/tests/apps/ZeroRK/zerork_gpu_paths.sh b/tests/apps/ZeroRK/zerork_gpu_paths.sh
index 40849a6f5..90bb29322 100755
--- a/tests/apps/ZeroRK/zerork_gpu_paths.sh
+++ b/tests/apps/ZeroRK/zerork_gpu_paths.sh
@@ -49,6 +49,7 @@ run() {
   setYML "$zrk" integrator "$4"
   setYML "$zrk" analytic "$5"
   setYML "$zrk" n_reactors_max "$8"
+  rm -f "$out/$name"_*.hist "$out/$name.log" "$out/$name.stdout"
   ZERORK_REACTOR_USE_LU=$9 "$exe" "$in" > "$out/$name.stdout" 2>&1
 }
 
@@ -62,13 +63,20 @@ compare() {
     END { exit bad }'
 }
 
+# Every timing log row must show all reactors on the GPU (zeroRK retries failed GPU reactors on the CPU);
+# with $2 = 1 also n_gpu_notemp > 0. Columns: 3 n_cpu, 5 n_gpu, 6 n_gpu_notemp.
+onGpu() {
+  awk -v nt="$2" '/^#/ { next } { n++; if ($3 != 0 || $5 <= 0 || (nt && $6 <= 0)) { print "row " $1 ": n_cpu " $3 ", n_gpu " $5 ", n_gpu_notemp " $6; bad = 1 } }
+    END { exit (bad || !n) }' "$1"
+}
+
 # n_reactors must stay >= the plugin's n_reactors_min (256) or the GPU is skipped silently.
-for cv in 1 0; do [ -s "$out/ref_cv${cv}_000.hist" ] || run ref_cv$cv 0 $cv 1 1 1 32 2048 0; done
+for cv in 1 0; do [ -s "$out/ref_cv${cv}_000.hist" ] || run ref_cv$cv 0 $cv 1 1 1 32 2048 0 || rm -f "$out/ref_cv${cv}_000.hist"; done
 rc=0
 # name  constant_volume integrator analytic always_solve_temperature n_reactors_max ZERORK_REACTOR_USE_LU
 while read -r name cv integ analytic ast nmax lu; do
-  run "$name" 1 "$cv" "$integ" "$analytic" "$ast" 2048 "$nmax" "$lu"
-  if diff=$(compare "$out/ref_cv${cv}_000.hist" "$out/${name}_000.hist"); then echo "PASS  zeroRK $name"
+  if run "$name" 1 "$cv" "$integ" "$analytic" "$ast" 2048 "$nmax" "$lu" &&
+    diff=$(onGpu "$out/$name.log" $((1 - ast)) && compare "$out/ref_cv${cv}_000.hist" "$out/${name}_000.hist"); then echo "PASS  zeroRK $name"
   else echo "FAIL  zeroRK $name ($out/$name.stdout)"; echo "$diff" | head -5; rc=1; fi
 done << 'CASES'
 cvode_analytic_cv      1 0 1 1 2048 1

From 4a24d1075421c5d3ddad0e1232989b0644fe98e1 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sun, 18 Oct 2026 00:02:35 +0300
Subject: [PATCH 16/31] ci: zerork_gpu_paths.sh uses 512 reactors, 4 steps and
 a matching no-temperature reference

On the A770 seulex_fd_cv took about 155 s per solve at 2048 reactors and
10 solves, roughly 26 minutes, which used the whole 1800 s step budget
so the next case was killed. The timeout stays; the cases shrink to 512
reactors (still above n_reactors_min, 256) and 4 solves, with
n_reactors_max 512 and 256 so the batch case still forms two groups.

The no-temperature case differed from the CPU reference by about 1e-2 in
T because the reference solved temperature. References are now keyed by
constant_volume and always_solve_temperature (ref_cv1_t0 for that case),
so tolerance is not loosened. n_gpu_notemp is nonzero only on some
solves (one of four in the logs), so that check requires it on at least
one row, while n_cpu == 0 and n_gpu > 0 are still required on every row.
---
 tests/apps/ZeroRK/zerork_gpu_paths.sh | 29 +++++++++++++++++----------
 1 file changed, 18 insertions(+), 11 deletions(-)

diff --git a/tests/apps/ZeroRK/zerork_gpu_paths.sh b/tests/apps/ZeroRK/zerork_gpu_paths.sh
index 90bb29322..1b0f5e0bf 100755
--- a/tests/apps/ZeroRK/zerork_gpu_paths.sh
+++ b/tests/apps/ZeroRK/zerork_gpu_paths.sh
@@ -37,6 +37,7 @@ run() {
   setYML "$in" oxidizer_composition "{ $oxid }"
   setYML "$in" zerork_cfd_plugin_input "$zrk"
   setYML "$in" n_reactors "$7"
+  setYML "$in" n_steps 4
   setYML "$in" constant_volume "$3"
   setYML "$in" reactor_history_file_prefix "$out/$name"
   setYML "$zrk" mechanism_parsing_log "$out/$name.cklog"
@@ -64,25 +65,31 @@ compare() {
 }
 
 # Every timing log row must show all reactors on the GPU (zeroRK retries failed GPU reactors on the CPU);
-# with $2 = 1 also n_gpu_notemp > 0. Columns: 3 n_cpu, 5 n_gpu, 6 n_gpu_notemp.
+# with $2 = 1 also some row with n_gpu_notemp > 0 (zeroRK counts them only on some solves).
+# Columns: 3 n_cpu, 5 n_gpu, 6 n_gpu_notemp.
 onGpu() {
-  awk -v nt="$2" '/^#/ { next } { n++; if ($3 != 0 || $5 <= 0 || (nt && $6 <= 0)) { print "row " $1 ": n_cpu " $3 ", n_gpu " $5 ", n_gpu_notemp " $6; bad = 1 } }
-    END { exit (bad || !n) }' "$1"
+  awk -v nt="$2" '/^#/ { next } { n++; if ($6 > 0) any = 1
+    if ($3 != 0 || $5 <= 0) { print "row " $1 ": n_cpu " $3 ", n_gpu " $5; bad = 1 } }
+    END { if (nt && !any) { print "n_gpu_notemp is 0 on every row"; bad = 1 } exit (bad || !n) }' "$1"
 }
 
 # n_reactors must stay >= the plugin's n_reactors_min (256) or the GPU is skipped silently.
-for cv in 1 0; do [ -s "$out/ref_cv${cv}_000.hist" ] || run ref_cv$cv 0 $cv 1 1 1 32 2048 0 || rm -f "$out/ref_cv${cv}_000.hist"; done
+N=512
+# One CPU reference per constant_volume and always_solve_temperature pair.
+for ref in cv1_t1 cv0_t1 cv1_t0; do
+  [ -s "$out/ref_${ref}_000.hist" ] || run ref_$ref 0 ${ref:2:1} 1 1 ${ref:5:1} 32 $N 0 || rm -f "$out/ref_${ref}_000.hist"
+done
 rc=0
 # name  constant_volume integrator analytic always_solve_temperature n_reactors_max ZERORK_REACTOR_USE_LU
 while read -r name cv integ analytic ast nmax lu; do
-  if run "$name" 1 "$cv" "$integ" "$analytic" "$ast" 2048 "$nmax" "$lu" &&
-    diff=$(onGpu "$out/$name.log" $((1 - ast)) && compare "$out/ref_cv${cv}_000.hist" "$out/${name}_000.hist"); then echo "PASS  zeroRK $name"
+  if run "$name" 1 "$cv" "$integ" "$analytic" "$ast" $N "$nmax" "$lu" &&
+    diff=$(onGpu "$out/$name.log" $((1 - ast)) && compare "$out/ref_cv${cv}_t${ast}_000.hist" "$out/${name}_000.hist"); then echo "PASS  zeroRK $name"
   else echo "FAIL  zeroRK $name ($out/$name.stdout)"; echo "$diff" | head -5; rc=1; fi
 done << 'CASES'
-cvode_analytic_cv      1 0 1 1 2048 1
-seulex_fd_cv           1 1 0 1 2048 0
-radau_analytic_cv      1 3 1 1 2048 1
-cvode_analytic_cp      0 0 1 1 2048 0
-seulex_fd_notemp_batch 1 1 0 0 1024 1
+cvode_analytic_cv      1 0 1 1 512 1
+seulex_fd_cv           1 1 0 1 512 0
+radau_analytic_cv      1 3 1 1 512 1
+cvode_analytic_cp      0 0 1 1 512 0
+seulex_fd_notemp_batch 1 1 0 0 256 1
 CASES
 exit $rc

From fe7211600c8b0cdbc100bb4ce89de4746205e187 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sun, 18 Oct 2026 08:55:35 +0300
Subject: [PATCH 17/31] ci: zerork_gpu_paths.sh fails a case whose temperature
 or pressure is not finite

compare() flagged a row only when dt > tol || dp > tol. Every comparison
with NaN is false in awk, so a GPU history with -nan in the temperature
or pressure column passed. The test is now !(dt <= tol && dp <= tol),
which is true for NaN as well as for a difference above the tolerance.
---
 tests/apps/ZeroRK/zerork_gpu_paths.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/apps/ZeroRK/zerork_gpu_paths.sh b/tests/apps/ZeroRK/zerork_gpu_paths.sh
index 1b0f5e0bf..90afe99a5 100755
--- a/tests/apps/ZeroRK/zerork_gpu_paths.sh
+++ b/tests/apps/ZeroRK/zerork_gpu_paths.sh
@@ -60,7 +60,7 @@ compare() {
   paste "$1" "$2" | awk -v tol=$TOLERANCE '/^#/ { next } {
     h = NF / 2; dt = ($(h+3) - $3) / $3; dp = ($(h+4) - $4) / $4
     if (dt < 0) dt = -dt; if (dp < 0) dp = -dp
-    if (dt > tol || dp > tol) { printf "step %s: T %g vs %g, P %g vs %g\n", $1, $3, $(h+3), $4, $(h+4); bad = 1 } }
+    if (!(dt <= tol && dp <= tol)) { printf "step %s: T %g vs %g, P %g vs %g\n", $1, $3, $(h+3), $4, $(h+4); bad = 1 } }
     END { exit bad }'
 }
 

From 6b8bd60bc6bc51b7cd71787463c699d75d399f91 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sun, 18 Oct 2026 08:55:44 +0300
Subject: [PATCH 18/31] ci: zerork_gpu_paths.sh builds the no-temperature
 reference the way the GPU runs

zeroRK's GPU path solves temperature on the first SolveReactors call
(solve_temperature = n_calls_ == 1 in zerork_reactor_manager.cpp) whatever
always_solve_temperature says; the CPU path does not on its first call.
A CPU reference with always_solve_temperature 0 therefore differs from
the GPU at row 1 by about 1.3e-4, above the 1e-4 tolerance, for a reason
that is not a GPU bug.

The reference now runs step 0 on the CPU with always_solve_temperature 1,
writes that row as a state file, and restarts from it with
always_solve_temperature 0 for the remaining three steps. compare() skips
GPU row 0 for that case so the restart rows line up with GPU rows 1 to 3.
A CPU run of this sequence matched the GPU path to about 1.2e-5.

run() takes the step count, solution time and state file from STEPS, SOLT
and STATE, defaulting to the 4 step run. The solution time is written as
the literal bc printed (8.2565998e-7 * 2), so bc leaves the runner needs.

The tester reads one state file per reactor, with the species mass
fractions in mechanism order and then T and P, so the two reference runs
use one reactor and run() always sets log_species to list every species
for the state file.
---
 .github/workflows/libs-apps.yml       |  2 +-
 tests/apps/ZeroRK/zerork_gpu_paths.sh | 27 ++++++++++++++++++++-------
 2 files changed, 21 insertions(+), 8 deletions(-)

diff --git a/.github/workflows/libs-apps.yml b/.github/workflows/libs-apps.yml
index f042abef3..304255432 100644
--- a/.github/workflows/libs-apps.yml
+++ b/.github/workflows/libs-apps.yml
@@ -6,7 +6,7 @@ name: Libraries and apps (Intel dGPU)
 # check-apps status on its head commit.
 #
 # Runner needs: modules oneapi/2025.0.4 and llvm/23.0, kernel journal access
-# (adm group), fuser, bc, curl, sha256sum, git over ssh to github.com, OpenMPI, gfortran, and on Ubuntu libgtest-dev
+# (adm group), fuser, curl, sha256sum, git over ssh to github.com, OpenMPI, gfortran, and on Ubuntu libgtest-dev
 # libgmock-dev libblas-dev liblapack-dev libboost-dev pybind11-dev.
 on:
   workflow_dispatch:
diff --git a/tests/apps/ZeroRK/zerork_gpu_paths.sh b/tests/apps/ZeroRK/zerork_gpu_paths.sh
index 90afe99a5..fb41e58fe 100755
--- a/tests/apps/ZeroRK/zerork_gpu_paths.sh
+++ b/tests/apps/ZeroRK/zerork_gpu_paths.sh
@@ -32,12 +32,14 @@ run() {
   cp inputs/base_tester.yml "$in"; cp inputs/base_plugin.yml "$zrk"
   setYML "$in" mechanism_file "$PWD/../../mechanisms/hydrogen/h2_v1b_mech.txt"
   setYML "$in" thermo_file "$PWD/../../mechanisms/hydrogen/h2_v1a_therm.txt"
-  setYML "$in" solution_time "$(echo "8.2565998*10^-7 * 2" | bc -l)"
+  setYML "$in" solution_time "${SOLT:-0.00000165131996}"
   setYML "$in" fuel_composition "{ $fuel }"
   setYML "$in" oxidizer_composition "{ $oxid }"
   setYML "$in" zerork_cfd_plugin_input "$zrk"
   setYML "$in" n_reactors "$7"
-  setYML "$in" n_steps 4
+  setYML "$in" log_species "[h, h2, o, o2, oh, h2o, n2, ho2, h2o2, ar]"
+  setYML "$in" n_steps "${STEPS:-4}"
+  [ -n "$STATE" ] && setYML "$in" state_files "[$STATE]"
   setYML "$in" constant_volume "$3"
   setYML "$in" reactor_history_file_prefix "$out/$name"
   setYML "$zrk" mechanism_parsing_log "$out/$name.cklog"
@@ -54,10 +56,13 @@ run() {
   ZERORK_REACTOR_USE_LU=$9 "$exe" "$in" > "$out/$name.stdout" 2>&1
 }
 
-# Temperature (col 3) and pressure (col 4) row by row against the CPU reference.
+# Temperature (col 3) and pressure (col 4) row by row against the CPU reference;
+# $3 = leading rows of the GPU history to skip.
 compare() {
-  [ -s "$1" ] && [ -s "$2" ] && [ "$(wc -l < "$1")" -eq "$(wc -l < "$2")" ] || return 1
-  paste "$1" "$2" | awk -v tol=$TOLERANCE '/^#/ { next } {
+  local gpu
+  gpu=$(awk -v s="${3:-0}" '/^#/ || ++n > s' "$2")
+  [ -s "$1" ] && [ -n "$gpu" ] && [ "$(wc -l < "$1")" -eq "$(echo "$gpu" | wc -l)" ] || return 1
+  paste "$1" <(echo "$gpu") | awk -v tol=$TOLERANCE '/^#/ { next } {
     h = NF / 2; dt = ($(h+3) - $3) / $3; dp = ($(h+4) - $4) / $4
     if (dt < 0) dt = -dt; if (dp < 0) dp = -dp
     if (!(dt <= tol && dp <= tol)) { printf "step %s: T %g vs %g, P %g vs %g\n", $1, $3, $(h+3), $4, $(h+4); bad = 1 } }
@@ -76,14 +81,22 @@ onGpu() {
 # n_reactors must stay >= the plugin's n_reactors_min (256) or the GPU is skipped silently.
 N=512
 # One CPU reference per constant_volume and always_solve_temperature pair.
-for ref in cv1_t1 cv0_t1 cv1_t0; do
+for ref in cv1_t1 cv0_t1; do
   [ -s "$out/ref_${ref}_000.hist" ] || run ref_$ref 0 ${ref:2:1} 1 1 ${ref:5:1} 32 $N 0 || rm -f "$out/ref_${ref}_000.hist"
 done
+# zeroRK solves temperature on the first GPU call whatever always_solve_temperature says, so the
+# no-temperature reference does step 0 with temperature and restarts from that state without it.
+# One reactor: the tester reads one state file per reactor (species, T, P).
+if [ ! -s "$out/ref_cv1_t0_000.hist" ]; then
+  STEPS=1 SOLT=0.00000041282999 run ref_s0 0 1 1 1 1 1 $N 0 &&
+    awk '!/^#/ { print $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $3, $4 }' "$out/ref_s0_000.hist" > "$out/ref_s0.state" &&
+    STEPS=3 SOLT=0.00000123848997 STATE=$out/ref_s0.state run ref_cv1_t0 0 1 1 1 0 1 $N 0 || rm -f "$out/ref_cv1_t0_000.hist"
+fi
 rc=0
 # name  constant_volume integrator analytic always_solve_temperature n_reactors_max ZERORK_REACTOR_USE_LU
 while read -r name cv integ analytic ast nmax lu; do
   if run "$name" 1 "$cv" "$integ" "$analytic" "$ast" $N "$nmax" "$lu" &&
-    diff=$(onGpu "$out/$name.log" $((1 - ast)) && compare "$out/ref_cv${cv}_t${ast}_000.hist" "$out/${name}_000.hist"); then echo "PASS  zeroRK $name"
+    diff=$(onGpu "$out/$name.log" $((1 - ast)) && compare "$out/ref_cv${cv}_t${ast}_000.hist" "$out/${name}_000.hist" $((1 - ast))); then echo "PASS  zeroRK $name"
   else echo "FAIL  zeroRK $name ($out/$name.stdout)"; echo "$diff" | head -5; rc=1; fi
 done << 'CASES'
 cvode_analytic_cv      1 0 1 1 512 1

From 18952cd916dc417fa67697cef83e7d24b254a0f0 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sun, 18 Oct 2026 08:55:48 +0300
Subject: [PATCH 19/31] ci: libs-apps builds zeroRK without the -I workaround
 for rocPRIM and rocThrust

zero-rk hip-chipStar now finds rocthrust through find_package
(https://github.com/CHIP-SPV/zero-rk/pull/6) instead of hardcoded include
directories, so CMAKE_PREFIX_PATH=$PREFIX already resolves rocthrust and
rocprim from this lane's install. The -DCMAKE_CXX_FLAGS=-I$PREFIX/include
that tried to override the hardcoded paths, and its comment referencing
https://github.com/CHIP-SPV/chipStar/issues/2006, are dropped. The cache
key needs no change: the zero-rk commit is in the directory name and the
headers under $PREFIX/include are already hashed.
---
 scripts/ci/libs-apps.sh | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index 7a16f6e4c..4cee8ac9b 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -95,13 +95,12 @@ run_libs() {
 lock() { mkdir -p "$DEPS_CACHE" && exec 9> "$DEPS_CACHE/lock" && flock 9; }
 
 # zeroRK's scripts/build_aurora.sh without its module loads and unbounded make -j.
-# The -I does not beat zero-rk's hardcoded rocPRIM and rocThrust include dirs, see https://github.com/CHIP-SPV/chipStar/issues/2006
 build_zerork() {
   lock || return 1
   [ -e "$ZRK/.built" ] && { echo "reusing $ZRK"; return 0; }
   rm -rf "$ZRK" && git clone -q --depth 1 -b hip-chipStar git@github.com:CHIP-SPV/zero-rk.git "$ZRK" &&
     mkdir "$ZRK/build_aurora" && cd "$ZRK/build_aurora" &&
-    cmake -DCMAKE_CXX_FLAGS="-I$PREFIX/include" -DCMAKE_EXE_LINKER_FLAGS=-lintlc -DENABLE_GPU=ON -DENABLE_MAGMA=OFF -DCMAKE_CXX_COMPILER=hipcc \
+    cmake -DCMAKE_EXE_LINKER_FLAGS=-lintlc -DENABLE_GPU=ON -DENABLE_MAGMA=OFF -DCMAKE_CXX_COMPILER=hipcc \
       -DCMAKE_C_COMPILER=hipcc -DZERORK_TESTS=OFF -DZERORK_HAVE_MKL=OFF .. && make -j"$JOBS" && make install &&
     touch "$ZRK/.built"
 }

From 9e2ff3877b3431234b71bed79584323965432fb6 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sun, 18 Oct 2026 10:14:20 +0300
Subject: [PATCH 20/31] ci: zerork_gpu_paths.sh compare() rejects nonfinite
 values under any awk and locale

The tolerance test !(dt <= tol && dp <= tol) relies on NaN comparing
false. mawk 1.3.4 (stock Ubuntu awk) reads nan and -nan as NaN but
evaluates NaN <= tol as true, so a NaN temperature or pressure from the
GPU passed. Reject any GPU temperature or pressure containing nan or inf
textually. zeroRK writes the history with %g, so the only letters in a
finite number are the exponent e.

The runner's LC_NUMERIC is lt_LT.UTF-8 (comma decimal), under which mawk
parses 1450.19 as 1450 and accepts finite mismatches. Run the comparison
under LC_ALL=C.

Checked with a synthetic row (reference 0 4.1283e-07 1450.19 2.02672e+06)
under mawk, gawk and gawk --posix with LC_NUMERIC=lt_LT.UTF-8: identical
GPU row exits 0; GPU P of nan, -nan, inf and a finite mismatch 2.02700e+06
exit 1.
---
 tests/apps/ZeroRK/zerork_gpu_paths.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/apps/ZeroRK/zerork_gpu_paths.sh b/tests/apps/ZeroRK/zerork_gpu_paths.sh
index fb41e58fe..4c5e6fc49 100755
--- a/tests/apps/ZeroRK/zerork_gpu_paths.sh
+++ b/tests/apps/ZeroRK/zerork_gpu_paths.sh
@@ -62,10 +62,10 @@ compare() {
   local gpu
   gpu=$(awk -v s="${3:-0}" '/^#/ || ++n > s' "$2")
   [ -s "$1" ] && [ -n "$gpu" ] && [ "$(wc -l < "$1")" -eq "$(echo "$gpu" | wc -l)" ] || return 1
-  paste "$1" <(echo "$gpu") | awk -v tol=$TOLERANCE '/^#/ { next } {
+  paste "$1" <(echo "$gpu") | LC_ALL=C awk -v tol=$TOLERANCE '/^#/ { next } {
     h = NF / 2; dt = ($(h+3) - $3) / $3; dp = ($(h+4) - $4) / $4
     if (dt < 0) dt = -dt; if (dp < 0) dp = -dp
-    if (!(dt <= tol && dp <= tol)) { printf "step %s: T %g vs %g, P %g vs %g\n", $1, $3, $(h+3), $4, $(h+4); bad = 1 } }
+    if (!(dt <= tol && dp <= tol) || $(h+3) $(h+4) ~ /[nN][aA][nN]|[iI][nN][fF]/) { printf "step %s: T %g vs %g, P %g vs %g\n", $1, $3, $(h+3), $4, $(h+4); bad = 1 } }
     END { exit bad }'
 }
 

From 0a56b206d72dc912d378561025ba21218919400f Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sun, 18 Oct 2026 10:14:32 +0300
Subject: [PATCH 21/31] ci: zerork_gpu_paths.sh derives the step 0 state from
 the temperature reference

The no-temperature CPU reference restarted from a state produced by a
separate one-step CPU solve (ref_s0). That solve is the same CPU
configuration as ref_cv1_t1, with the same first step dt and uniform
initial conditions, so row 1 of ref_cv1_t1_000.hist is byte identical to
ref_s0_000.hist. Take the state from the first data row of
ref_cv1_t1_000.hist instead (awk exits after that row) and drop the
duplicate solve.
---
 tests/apps/ZeroRK/zerork_gpu_paths.sh | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/tests/apps/ZeroRK/zerork_gpu_paths.sh b/tests/apps/ZeroRK/zerork_gpu_paths.sh
index 4c5e6fc49..62b5c4372 100755
--- a/tests/apps/ZeroRK/zerork_gpu_paths.sh
+++ b/tests/apps/ZeroRK/zerork_gpu_paths.sh
@@ -85,11 +85,10 @@ for ref in cv1_t1 cv0_t1; do
   [ -s "$out/ref_${ref}_000.hist" ] || run ref_$ref 0 ${ref:2:1} 1 1 ${ref:5:1} 32 $N 0 || rm -f "$out/ref_${ref}_000.hist"
 done
 # zeroRK solves temperature on the first GPU call whatever always_solve_temperature says, so the
-# no-temperature reference does step 0 with temperature and restarts from that state without it.
+# no-temperature reference restarts from step 0 of the temperature reference without it.
 # One reactor: the tester reads one state file per reactor (species, T, P).
 if [ ! -s "$out/ref_cv1_t0_000.hist" ]; then
-  STEPS=1 SOLT=0.00000041282999 run ref_s0 0 1 1 1 1 1 $N 0 &&
-    awk '!/^#/ { print $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $3, $4 }' "$out/ref_s0_000.hist" > "$out/ref_s0.state" &&
+  awk '!/^#/ { print $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $3, $4; exit }' "$out/ref_cv1_t1_000.hist" > "$out/ref_s0.state" &&
     STEPS=3 SOLT=0.00000123848997 STATE=$out/ref_s0.state run ref_cv1_t0 0 1 1 1 0 1 $N 0 || rm -f "$out/ref_cv1_t0_000.hist"
 fi
 rc=0

From 0cb16aa2dc0dc3f30b7854b1d453f2f765f28ab6 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sun, 18 Oct 2026 10:19:45 +0300
Subject: [PATCH 22/31] ci: zerork_gpu_paths.sh compare() also rejects a
 nonfinite CPU reference

The nonfinite check in compare() tested only the GPU fields, but the CPU
reference can hold a nonfinite value too: zeroRK GetState computes the
final T and P without checking that they are finite. With nan or inf in
the reference columns 3 or 4, dt or dp becomes NaN, and mawk accepts a
NaN comparison, so !(dt <= tol && dp <= tol) is false and the row passed.

Reproduced under mawk with LC_ALL=C: reference row
"0 4.1283e-07 nan 2.02672e+06" against GPU row
"0 4.1283e-07 1450.19 2.02672e+06" returned rc=0, as did a CPU P of inf.

Test the reference fields $3 $4 together with the GPU fields.
---
 tests/apps/ZeroRK/zerork_gpu_paths.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/apps/ZeroRK/zerork_gpu_paths.sh b/tests/apps/ZeroRK/zerork_gpu_paths.sh
index 62b5c4372..2938dbaed 100755
--- a/tests/apps/ZeroRK/zerork_gpu_paths.sh
+++ b/tests/apps/ZeroRK/zerork_gpu_paths.sh
@@ -65,7 +65,7 @@ compare() {
   paste "$1" <(echo "$gpu") | LC_ALL=C awk -v tol=$TOLERANCE '/^#/ { next } {
     h = NF / 2; dt = ($(h+3) - $3) / $3; dp = ($(h+4) - $4) / $4
     if (dt < 0) dt = -dt; if (dp < 0) dp = -dp
-    if (!(dt <= tol && dp <= tol) || $(h+3) $(h+4) ~ /[nN][aA][nN]|[iI][nN][fF]/) { printf "step %s: T %g vs %g, P %g vs %g\n", $1, $3, $(h+3), $4, $(h+4); bad = 1 } }
+    if (!(dt <= tol && dp <= tol) || $3 $4 $(h+3) $(h+4) ~ /[nN][aA][nN]|[iI][nN][fF]/) { printf "step %s: T %g vs %g, P %g vs %g\n", $1, $3, $(h+3), $4, $(h+4); bad = 1 } }
     END { exit bad }'
 }
 

From 169b5f495fed216c50be0ea2f7ce78a44a23a98f Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sun, 18 Oct 2026 10:33:38 +0300
Subject: [PATCH 23/31] ci: libs-apps report leaves check-apps to a newer
 request on the same commit

The report job runs on a hosted runner, outside the libs-apps concurrency
group (only the self-hosted run job is in it), and it always writes the
final check-apps status. Two requests on one head can therefore finish
out of order:

1. Request A and request B are made on the same head. B's resolve job
   writes pending with B's run URL. A's run finishes later than B's
   resolve, and A's report then overwrites B's pending with success or
   failure while B is still running.
2. A is cancelled while queued by B. Its run job is cancelled, but its
   report job still runs with RESULT=cancelled and writes failure over
   B's pending or final status.

The report now reads the latest check-apps status of the commit and skips
the write unless that status carries this run's URL, which is the exact
URL the resolve job wrote as pending. A newer request has replaced it
with its own URL, so the older report leaves the status alone.

Concurrency stays at job level (every issue comment triggers this
workflow) and the run job still holds no write token.
---
 .github/workflows/libs-apps.yml | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/.github/workflows/libs-apps.yml b/.github/workflows/libs-apps.yml
index 304255432..c16e83ebd 100644
--- a/.github/workflows/libs-apps.yml
+++ b/.github/workflows/libs-apps.yml
@@ -106,8 +106,12 @@ jobs:
         with:
           script: |
             const ok = process.env.RESULT === 'success';
+            const url = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
+            // A newer request on this commit owns the status.
+            const { data } = await github.rest.repos.getCombinedStatusForRef({ ...context.repo, ref: process.env.SHA });
+            if (data.statuses.find(s => s.context === 'check-apps')?.target_url !== url) return;
             await github.rest.repos.createCommitStatus({
               ...context.repo, sha: process.env.SHA, state: ok ? 'success' : 'failure',
               context: 'check-apps', description: ok ? 'libraries and apps pass' : 'see summary.txt in the run artifacts',
-              target_url: `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`,
+              target_url: url,
             });

From 2a91215c99d740d94a0b06659d4951057ddebde7 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sun, 18 Oct 2026 10:47:34 +0300
Subject: [PATCH 24/31] ci: libs-apps reports libraries that register no CTest
 tests

run_libs skipped any component without build/CTestTestfile.cmake and said
nothing. rocSPARSE builds rocsparse-test (BUILD_CLIENTS_TESTS=ON) but only
clients/ enables testing and no add_test exists, so its tests never run and
nothing in summary.txt said so. chipRCCL and chipStdPar are in the same
state. libraries-presubmit.yml already prints "no CTest tests" for this
case; write the equivalent SKIP line to the summary.
---
 scripts/ci/libs-apps.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index 4cee8ac9b..ab6f9b604 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -81,7 +81,7 @@ run_libs() {
   local lib s ex lb
   for s in "$WRK"/staging/*/; do
     lib=$(basename "$s")
-    [ -f "$s/build/CTestTestfile.cmake" ] || continue
+    [ -f "$s/build/CTestTestfile.cmake" ] || { note "SKIP  $1 $lib (no CTest tests)"; continue; }
     ex=$(grep -Evhs '^\s*(#|$)' "$s/.ci/exclude.txt" | paste -sd '|' -)
     lb=$(grep -Evhs '^\s*(#|$)' "$s/.ci/ctest-labels.txt" | paste -sd '|' -)
     rm -f "$LOGS/$1/$lib.log"

From 4388185d48ffa89dc95766943a79edd7f882a425 Mon Sep 17 00:00:00 2001
From: t <t@t>
Date: Sun, 18 Oct 2026 10:47:38 +0300
Subject: [PATCH 25/31] ci: libs-apps queues GPU runs instead of cancelling a
 waiting one

The run job's concurrency group libs-apps uses the default single-slot
queue: with run A in progress and B pending, a third request C cancels B.
B's report job (if: always()) then posts a failure for a run that never
executed. Setting queue: max keeps up to 100 pending runs in the group
(valid because cancel-in-progress is not set), so each request runs in
order on the single GPU.
---
 .github/workflows/libs-apps.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.github/workflows/libs-apps.yml b/.github/workflows/libs-apps.yml
index c16e83ebd..84ab9d01b 100644
--- a/.github/workflows/libs-apps.yml
+++ b/.github/workflows/libs-apps.yml
@@ -66,6 +66,7 @@ jobs:
       contents: read
     concurrency:
       group: libs-apps
+      queue: max
     steps:
       # The script comes from the default or dispatched branch, so it can test PRs older than it.
       - uses: actions/checkout@v4

From 8ecbae01032e87dd4fbbf6a67c12714645b71405 Mon Sep 17 00:00:00 2001
From: Paulius Velesko <pvelesko@pglc.io>
Date: Wed, 21 Oct 2026 07:41:14 +0300
Subject: [PATCH 26/31] ci: libs-apps trigger is /run-app-tests, matching
 /run-math-tests

Renamed the comment command from /check-apps to /run-app-tests, the
workflow name to "Run app tests (/run-app-tests)", and the commit status
context from check-apps to app-tests, to match the /run-math-tests
workflow on main.

The match now requires the exact command followed by end of comment, a
space, a newline or a carriage return, so a longer command that shares
the prefix, such as /run-app-tests-foo, does not start a GPU run.
---
 .github/workflows/libs-apps.yml | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/.github/workflows/libs-apps.yml b/.github/workflows/libs-apps.yml
index 84ab9d01b..4689a72dc 100644
--- a/.github/workflows/libs-apps.yml
+++ b/.github/workflows/libs-apps.yml
@@ -1,9 +1,9 @@
-name: Libraries and apps (Intel dGPU)
+name: Run app tests (/run-app-tests)
 
 # chipStar plus every downstream library with its tests, and the zeroRK and
 # OpenSn checks, on an Intel discrete GPU under Level Zero and OpenCL. Run it
-# from the Actions tab, or comment /check-apps on a pull request to get a
-# check-apps status on its head commit.
+# from the Actions tab, or comment /run-app-tests on a pull request to get an
+# app-tests status on its head commit.
 #
 # Runner needs: modules oneapi/2025.0.4 and llvm/23.0, kernel journal access
 # (adm group), fuser, curl, sha256sum, git over ssh to github.com, OpenMPI, gfortran, and on Ubuntu libgtest-dev
@@ -27,7 +27,11 @@ jobs:
   resolve:
     if: >-
       github.event_name == 'workflow_dispatch' ||
-      (github.event.issue.pull_request && startsWith(github.event.comment.body, '/check-apps') &&
+      (github.event.issue.pull_request &&
+       (github.event.comment.body == '/run-app-tests' ||
+        startsWith(github.event.comment.body, '/run-app-tests ') ||
+        startsWith(github.event.comment.body, fromJSON('"/run-app-tests\n"')) ||
+        startsWith(github.event.comment.body, fromJSON('"/run-app-tests\r"'))) &&
        contains(fromJSON('["colleeneb", "pvelesko"]'), github.event.comment.user.login))
     runs-on: ubuntu-latest
     permissions:
@@ -49,7 +53,7 @@ jobs:
             const { data } = await github.rest.pulls.get({ ...context.repo, pull_number: Number(pr) });
             core.setOutput('sha', data.head.sha);
             await github.rest.repos.createCommitStatus({
-              ...context.repo, sha: data.head.sha, state: 'pending', context: 'check-apps',
+              ...context.repo, sha: data.head.sha, state: 'pending', context: 'app-tests',
               description: 'libraries and apps running',
               target_url: `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`,
             });
@@ -110,9 +114,9 @@ jobs:
             const url = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
             // A newer request on this commit owns the status.
             const { data } = await github.rest.repos.getCombinedStatusForRef({ ...context.repo, ref: process.env.SHA });
-            if (data.statuses.find(s => s.context === 'check-apps')?.target_url !== url) return;
+            if (data.statuses.find(s => s.context === 'app-tests')?.target_url !== url) return;
             await github.rest.repos.createCommitStatus({
               ...context.repo, sha: process.env.SHA, state: ok ? 'success' : 'failure',
-              context: 'check-apps', description: ok ? 'libraries and apps pass' : 'see summary.txt in the run artifacts',
+              context: 'app-tests', description: ok ? 'libraries and apps pass' : 'see summary.txt in the run artifacts',
               target_url: url,
             });

From 9d5d55f24b9ade718d0cbd296fb89f23c624cdbc Mon Sep 17 00:00:00 2001
From: Paulius Velesko <pvelesko@pglc.io>
Date: Fri, 30 Oct 2026 01:20:55 +0200
Subject: [PATCH 27/31] ci: libs-apps keeps testing after a GPU reset the
 driver recovers from

i915 and xe answer a hung job by resetting the engine and banning the
offending context; the GPU keeps working afterwards. Only "wedged" means
the driver gave up and every later GPU call fails until reboot.

gpu_step killed the step on any reset, hang or timedout line, set
GPU_DEAD and skipped every later step. One recovered reset therefore threw
away the rest of the current ctest suite and every later library and app
result.

Now only a wedged line for this GPU or a journal read error is fatal and
keeps the kill, GPU_DEAD, STOP note and SKIP of later steps. A reset, hang
or timedout line lets the step run on, so ctest moves to its next test and
CTEST_TIMEOUT or the timeout wrappers bound the hung one. When the step
ends, those lines are saved to "<step log>.gpu-faults", a NOTE names the
file, and the step is reported FAIL even if its rc was 0. The journal
cursor is refreshed at each step start so a reset is attributed only to the
step it happened in. The per-step summary lines still print.
---
 scripts/ci/libs-apps.sh | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index ab6f9b604..727c21289 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -1,8 +1,8 @@
 #!/bin/bash
 # Build chipStar plus every downstream library with its tests, then run each
 # library's ctest suite and the zeroRK and OpenSn checks on one Intel GPU per
-# backend. Waits while another process holds the GPU and stops all GPU work at
-# the first reset or wedge the kernel logs for it. Results go to summary.txt.
+# backend. Waits while another process holds the GPU and stops all GPU work only
+# when the kernel declares it wedged. Results go to summary.txt.
 #
 #   scripts/ci/libs-apps.sh -w WORK_DIR [-s CHIPSTAR_SRC] [-j JOBS] [-b "level0 opencl"] [-p build,libs,apps]
 set -o pipefail
@@ -44,9 +44,10 @@ trap '[ -n "$STEP" ] && { kill -KILL "$STEP"; pkill -KILL -s "$STEP"; } 2>/dev/n
 wants() { case ,$PHASES, in *,$1,*) return 0 ;; esac; return 1; }
 note() { echo "$*" | tee -a "$SUMMARY"; }
 result() { [ "$2" -eq 0 ] && note "PASS  $1" || { note "FAIL  $1 (rc=$2, $3)"; FAILED=$((FAILED + 1)); }; }
-# A journal read error counts as a fault: the GPU cannot be watched. Sticky, so a transient error is not forgotten.
+cursor() { journalctl -k -b -n0 --show-cursor -q --no-pager | sed -n 's/^-- cursor: //p'; }
+# Fatal: the driver gave up on the GPU (wedged) or the journal is unreadable so the GPU cannot be watched. Sticky.
 gpu_faults() {
-  local j p="$DRV $GPU_PCI.*(reset|wedged|timedout|hang)|^Failed|^No journal"
+  local j p="$DRV $GPU_PCI.*wedged|^Failed|^No journal"
   [ -e "$LOGS/fault" ] && { echo 1; return; }
   j=$(journalctl -k -b -q --no-pager --after-cursor="$CURSOR" 2>&1) || j="Failed: journalctl exit $?
 $j"
@@ -57,10 +58,11 @@ $j"
 
 # gpu_step NAME LOG CMD...; CMD gets its own session so a fault kills all of it.
 gpu_step() {
-  local name=$1 log=$2 pid="" f rc; shift 2
+  local name=$1 log=$2 pid="" f rc c resets; shift 2
   [ $GPU_DEAD -eq 0 ] || { note "SKIP  $name"; return; }
   while f=$(gpu_faults) && [ "$f" -eq 0 ] && fuser "$RENDER" > /dev/null 2>&1; do echo "[gpu] $RENDER busy, waiting"; sleep 60; done
   if [ "$f" -eq 0 ]; then
+    c=$(cursor) && [ -n "$c" ] && CURSOR=$c
     setsid "$@" > "$log" 2>&1 < /dev/null &
     pid=$! STEP=$!
     while kill -0 $pid 2>/dev/null && f=$(gpu_faults) && [ "$f" -eq 0 ]; do sleep 1; done
@@ -68,11 +70,14 @@ gpu_step() {
   fi
   if [ "$f" -gt 0 ]; then
     [ -n "$pid" ] && { kill -KILL $pid; pkill -KILL -s $pid; } 2>/dev/null; GPU_DEAD=1
-    note "STOP  GPU reset, wedge or journal error at $name $(grep -so 'Start *[0-9]*: [^ ]*' "$log" | tail -1), see logs/fault"
+    note "STOP  GPU wedged or journal error at $name $(grep -so 'Start *[0-9]*: [^ ]*' "$log" | tail -1), see logs/fault"
     result "$name" 137 "$log"; STEP=""; return
   fi
   wait $pid; rc=$?
   pkill -KILL -s $pid 2>/dev/null
+  # A reset the driver recovered from: the hung test fails by itself, the rest keep running.
+  resets=$(journalctl -k -b -q --no-pager --after-cursor="$CURSOR" 2>&1 | grep -iE "$DRV $GPU_PCI.*(reset|timedout|hang)")
+  [ -z "$resets" ] || { echo "$resets" > "$log.gpu-faults"; note "NOTE  GPU reset during $name, see $log.gpu-faults"; [ $rc -ne 0 ] || rc=1; }
   result "$name" $rc "$log"; STEP=""
 }
 
@@ -165,7 +170,7 @@ if wants apps; then
 fi
 wants libs || wants apps || BACKENDS=""
 rm -f "$LOGS/fault"
-CURSOR=$(journalctl -k -b -n0 --show-cursor -q --no-pager | sed -n 's/^-- cursor: //p')
+CURSOR=$(cursor)
 [ -z "$BACKENDS" ] || [ -n "$CURSOR" ] || { note "FAIL  kernel journal unreadable, cannot watch the GPU"; exit 1; }
 [ -z "$BACKENDS" ] || { n=$(journalctl -k -b -q --no-pager | grep -ciE "$DRV $GPU_PCI.*wedged"; echo "${PIPESTATUS[0]}"); [ "$(echo $n)" = "0 0" ]; } ||
   { note "FAIL  $GPU_PCI was declared wedged earlier in this boot, or the kernel journal is unreadable"; exit 1; }

From 2657937384273fd0eabe96977428d9853f0847d0 Mon Sep 17 00:00:00 2001
From: Paulius Velesko <pvelesko@pglc.io>
Date: Fri, 30 Oct 2026 01:32:56 +0200
Subject: [PATCH 28/31] ci: libs-apps treats i915 reset failures as fatal and
 never skips journal lines

Three defects in the GPU fault handling of the previous commit.

i915 can wedge the GPU without printing "wedged". In Linux v7.0
drivers/gpu/drm/i915/gt/intel_reset.c, intel_gt_reset() logs "GPU reset
not supported", "Failed to reset chip" or "Failed to initialise HW
following reset (%d)" and then calls __intel_gt_set_wedged(), which logs
nothing with "wedged"; intel_wedge_me() logs "<name> timed out,
cancelling all in-flight rendering." and wedges as well. The fatal
pattern is now one variable, FATAL_MSG, matching wedged or those
messages, used by gpu_faults and the boot precheck. The recoverable scan
excludes FATAL_MSG, so "Resetting chip" and "GPU HANG" stay recoverable
while "Failed to reset chip" is fatal.

gpu_step advanced the fatal monitor's CURSOR before each launch, so a
fatal line logged between the last gpu_faults read and that cursor()
call was skipped for good. CURSOR is now never advanced; a separate
per-step cursor is used only to attribute recoverable resets to a step.

The post-step journalctl read ignored failure, and a failed cursor() was
ignored too, so a step could pass with the journal unreadable. A failed
read or an empty per-step cursor now marks the step FAIL with a note.
---
 scripts/ci/libs-apps.sh | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index 727c21289..f31dcce92 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -45,9 +45,11 @@ wants() { case ,$PHASES, in *,$1,*) return 0 ;; esac; return 1; }
 note() { echo "$*" | tee -a "$SUMMARY"; }
 result() { [ "$2" -eq 0 ] && note "PASS  $1" || { note "FAIL  $1 (rc=$2, $3)"; FAILED=$((FAILED + 1)); }; }
 cursor() { journalctl -k -b -n0 --show-cursor -q --no-pager | sed -n 's/^-- cursor: //p'; }
-# Fatal: the driver gave up on the GPU (wedged) or the journal is unreadable so the GPU cannot be watched. Sticky.
+# i915 wedges without saying so after these (intel_reset.c); the word reset also appears in recoverable lines.
+FATAL_MSG="wedged|GPU reset not supported|Failed to reset chip|Failed to initialise HW following reset|timed out, cancelling all in-flight rendering"
+# Fatal: the driver gave up on the GPU (FATAL_MSG) or the journal is unreadable so the GPU cannot be watched. Sticky.
 gpu_faults() {
-  local j p="$DRV $GPU_PCI.*wedged|^Failed|^No journal"
+  local j p="$DRV $GPU_PCI.*($FATAL_MSG)|^Failed|^No journal"
   [ -e "$LOGS/fault" ] && { echo 1; return; }
   j=$(journalctl -k -b -q --no-pager --after-cursor="$CURSOR" 2>&1) || j="Failed: journalctl exit $?
 $j"
@@ -58,11 +60,11 @@ $j"
 
 # gpu_step NAME LOG CMD...; CMD gets its own session so a fault kills all of it.
 gpu_step() {
-  local name=$1 log=$2 pid="" f rc c resets; shift 2
+  local name=$1 log=$2 pid="" f rc sc resets; shift 2
   [ $GPU_DEAD -eq 0 ] || { note "SKIP  $name"; return; }
   while f=$(gpu_faults) && [ "$f" -eq 0 ] && fuser "$RENDER" > /dev/null 2>&1; do echo "[gpu] $RENDER busy, waiting"; sleep 60; done
   if [ "$f" -eq 0 ]; then
-    c=$(cursor) && [ -n "$c" ] && CURSOR=$c
+    sc=$(cursor)
     setsid "$@" > "$log" 2>&1 < /dev/null &
     pid=$! STEP=$!
     while kill -0 $pid 2>/dev/null && f=$(gpu_faults) && [ "$f" -eq 0 ]; do sleep 1; done
@@ -76,7 +78,10 @@ gpu_step() {
   wait $pid; rc=$?
   pkill -KILL -s $pid 2>/dev/null
   # A reset the driver recovered from: the hung test fails by itself, the rest keep running.
-  resets=$(journalctl -k -b -q --no-pager --after-cursor="$CURSOR" 2>&1 | grep -iE "$DRV $GPU_PCI.*(reset|timedout|hang)")
+  if [ -z "$sc" ] || ! resets=$(journalctl -k -b -q --no-pager --after-cursor="$sc" 2>&1); then
+    note "NOTE  journal unreadable after $name, cannot rule out a GPU reset"; resets=""; [ $rc -ne 0 ] || rc=1
+  fi
+  resets=$(grep -iE "$DRV $GPU_PCI.*(reset|timedout|hang)" <<< "$resets" | grep -viE "$DRV $GPU_PCI.*($FATAL_MSG)")
   [ -z "$resets" ] || { echo "$resets" > "$log.gpu-faults"; note "NOTE  GPU reset during $name, see $log.gpu-faults"; [ $rc -ne 0 ] || rc=1; }
   result "$name" $rc "$log"; STEP=""
 }
@@ -172,8 +177,8 @@ wants libs || wants apps || BACKENDS=""
 rm -f "$LOGS/fault"
 CURSOR=$(cursor)
 [ -z "$BACKENDS" ] || [ -n "$CURSOR" ] || { note "FAIL  kernel journal unreadable, cannot watch the GPU"; exit 1; }
-[ -z "$BACKENDS" ] || { n=$(journalctl -k -b -q --no-pager | grep -ciE "$DRV $GPU_PCI.*wedged"; echo "${PIPESTATUS[0]}"); [ "$(echo $n)" = "0 0" ]; } ||
-  { note "FAIL  $GPU_PCI was declared wedged earlier in this boot, or the kernel journal is unreadable"; exit 1; }
+[ -z "$BACKENDS" ] || { n=$(journalctl -k -b -q --no-pager | grep -ciE "$DRV $GPU_PCI.*($FATAL_MSG)"; echo "${PIPESTATUS[0]}"); [ "$(echo $n)" = "0 0" ]; } ||
+  { note "FAIL  $GPU_PCI was declared wedged or failed to reset earlier in this boot, or the kernel journal is unreadable"; exit 1; }
 [ -z "$BACKENDS" ] || command -v fuser > /dev/null || { note "FAIL  fuser not found, cannot see other GPU users"; exit 1; }
 for be in $BACKENDS; do
   mkdir -p "$LOGS/$be" && export CHIP_BE=$be

From 5422017a35688e11d5e76188a603c81c10bbb95d Mon Sep 17 00:00:00 2001
From: Paulius Velesko <pvelesko@pglc.io>
Date: Fri, 30 Oct 2026 01:45:31 +0200
Subject: [PATCH 29/31] ci: libs-apps acts on a fatal GPU line seen only by the
 final journal read

gpu_step polled gpu_faults() while the step ran and once at exit, but the
per-step journal read that collects recoverable resets came after the last
poll and filtered FATAL_MSG lines out with grep -v without acting on them.
A fatal line logged in that window was dropped and the step could PASS, and
a journal read error there was not sticky.

Move the wait and reset scan before the fatal block and call gpu_faults()
once more after it. CURSOR is continuous, so that read is later than the
snapshot, sees any fatal line, and also covers journal errors and the
sticky $LOGS/fault. A fault takes the existing fatal path (GPU_DEAD=1,
STOP note, result 137).
---
 scripts/ci/libs-apps.sh | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index f31dcce92..9fe85e790 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -69,20 +69,24 @@ gpu_step() {
     pid=$! STEP=$!
     while kill -0 $pid 2>/dev/null && f=$(gpu_faults) && [ "$f" -eq 0 ]; do sleep 1; done
     [ "$f" -gt 0 ] || f=$(gpu_faults)
+    [ "$f" -gt 0 ] || {
+      wait $pid; rc=$?
+      pkill -KILL -s $pid 2>/dev/null
+      # A reset the driver recovered from: the hung test fails by itself, the rest keep running.
+      if [ -z "$sc" ] || ! resets=$(journalctl -k -b -q --no-pager --after-cursor="$sc" 2>&1); then
+        note "NOTE  journal unreadable after $name, cannot rule out a GPU reset"; resets=""; [ $rc -ne 0 ] || rc=1
+      fi
+      resets=$(grep -iE "$DRV $GPU_PCI.*(reset|timedout|hang)" <<< "$resets" | grep -viE "$DRV $GPU_PCI.*($FATAL_MSG)")
+      [ -z "$resets" ] || { echo "$resets" > "$log.gpu-faults"; note "NOTE  GPU reset during $name, see $log.gpu-faults"; [ $rc -ne 0 ] || rc=1; }
+      # A fatal line logged after the last poll is still fatal.
+      f=$(gpu_faults)
+    }
   fi
   if [ "$f" -gt 0 ]; then
     [ -n "$pid" ] && { kill -KILL $pid; pkill -KILL -s $pid; } 2>/dev/null; GPU_DEAD=1
     note "STOP  GPU wedged or journal error at $name $(grep -so 'Start *[0-9]*: [^ ]*' "$log" | tail -1), see logs/fault"
     result "$name" 137 "$log"; STEP=""; return
   fi
-  wait $pid; rc=$?
-  pkill -KILL -s $pid 2>/dev/null
-  # A reset the driver recovered from: the hung test fails by itself, the rest keep running.
-  if [ -z "$sc" ] || ! resets=$(journalctl -k -b -q --no-pager --after-cursor="$sc" 2>&1); then
-    note "NOTE  journal unreadable after $name, cannot rule out a GPU reset"; resets=""; [ $rc -ne 0 ] || rc=1
-  fi
-  resets=$(grep -iE "$DRV $GPU_PCI.*(reset|timedout|hang)" <<< "$resets" | grep -viE "$DRV $GPU_PCI.*($FATAL_MSG)")
-  [ -z "$resets" ] || { echo "$resets" > "$log.gpu-faults"; note "NOTE  GPU reset during $name, see $log.gpu-faults"; [ $rc -ne 0 ] || rc=1; }
   result "$name" $rc "$log"; STEP=""
 }
 

From 44dd638abd0927c5d99fd42aa620f32447e79143 Mon Sep 17 00:00:00 2001
From: Paulius Velesko <pvelesko@pglc.io>
Date: Fri, 30 Oct 2026 01:45:41 +0200
Subject: [PATCH 30/31] ci: zerork_gpu_paths.sh bounds each run so a hung case
 does not hide the rest

libs-apps.sh wraps the whole script in one timeout and no single tester run
was bounded, so one hung case killed the loop and the later cases never
reported. Wrap each tester run in timeout -k 30 1800 so the hung case is
reported FAIL and the loop continues.

Measured on meatloaf (~/libs-apps-1945/work/logs/level0/zerork-out/*.log and
file mtimes): the slowest GPU case is seulex_fd_cv at about 320s summed over
its four solves (about 380s by mtime); radau_analytic_cv about 107s,
cvode_analytic_cp about 78s, seulex_fd_notemp_batch about 91s,
cvode_analytic_cv about 26s; CPU references under 1s each. 1800s is about
5x the slowest.

Raise the outer budget from 1800 to 22000 so it covers the worst case of
11 runs (6 CPU references and 5 GPU cases) at 1830s each.
---
 scripts/ci/libs-apps.sh               | 2 +-
 tests/apps/ZeroRK/zerork_gpu_paths.sh | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index 9fe85e790..e838ec166 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -145,7 +145,7 @@ build_opensn() {
 run_apps() {
   note "SKIP  $1 zeroRK box_reactor (zerork_correctness_box_reactor.sh needs Aurora's PBS and MPI launcher)"
   rm -rf "$LOGS/$1/zerork-out"
-  gpu_step "$1 zeroRK GPU paths" "$LOGS/$1/zerork-paths.log" timeout -k 60 1800 bash -c 'cd "$0" && bash "$1" "$2"' \
+  gpu_step "$1 zeroRK GPU paths" "$LOGS/$1/zerork-paths.log" timeout -k 60 22000 bash -c 'cd "$0" && bash "$1" "$2"' \
     "$ZRK" "$HERE/tests/apps/ZeroRK/zerork_gpu_paths.sh" "$WRK/apps/zerork-out"
   [ $GPU_DEAD -eq 0 ] && grep -hsE '^(PASS|FAIL) ' "$LOGS/$1/zerork-paths.log" | sed "s/^/      $1 /" | tee -a "$SUMMARY"
   cp -r "$WRK/apps/zerork-out" "$LOGS/$1/" 2>/dev/null
diff --git a/tests/apps/ZeroRK/zerork_gpu_paths.sh b/tests/apps/ZeroRK/zerork_gpu_paths.sh
index 2938dbaed..fb17362dc 100755
--- a/tests/apps/ZeroRK/zerork_gpu_paths.sh
+++ b/tests/apps/ZeroRK/zerork_gpu_paths.sh
@@ -53,7 +53,7 @@ run() {
   setYML "$zrk" analytic "$5"
   setYML "$zrk" n_reactors_max "$8"
   rm -f "$out/$name"_*.hist "$out/$name.log" "$out/$name.stdout"
-  ZERORK_REACTOR_USE_LU=$9 "$exe" "$in" > "$out/$name.stdout" 2>&1
+  ZERORK_REACTOR_USE_LU=$9 timeout -k 30 1800 "$exe" "$in" > "$out/$name.stdout" 2>&1
 }
 
 # Temperature (col 3) and pressure (col 4) row by row against the CPU reference;

From 1b648c4f38f227b2440403f8af933d8333d39947 Mon Sep 17 00:00:00 2001
From: Paulius Velesko <pvelesko@pglc.io>
Date: Fri, 30 Oct 2026 01:58:13 +0200
Subject: [PATCH 31/31] ci: libs-apps lets journald catch up before scanning a
 step for GPU resets

journald reads /dev/kmsg asynchronously (systemd src/journal/journald-kmsg.c),
so a recovered reset logged as the step exits could appear in the journal
after the scan. The next step's per-step cursor would then exclude it, and
the step would PASS unreported. 2 s covers the ingestion lag at a cost of
2 s per GPU step.
---
 scripts/ci/libs-apps.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/ci/libs-apps.sh b/scripts/ci/libs-apps.sh
index e838ec166..4f05e88a4 100755
--- a/scripts/ci/libs-apps.sh
+++ b/scripts/ci/libs-apps.sh
@@ -72,6 +72,8 @@ gpu_step() {
     [ "$f" -gt 0 ] || {
       wait $pid; rc=$?
       pkill -KILL -s $pid 2>/dev/null
+      # journald ingests kernel messages asynchronously: let a reset logged at the end of the step reach the journal.
+      sleep 2
       # A reset the driver recovered from: the hung test fails by itself, the rest keep running.
       if [ -z "$sc" ] || ! resets=$(journalctl -k -b -q --no-pager --after-cursor="$sc" 2>&1); then
         note "NOTE  journal unreadable after $name, cannot rule out a GPU reset"; resets=""; [ $rc -ne 0 ] || rc=1