Repository navigation
Conversation
pvelesko
force-pushed
the
2026-10-13-github-1935-libs-apps-dispatch
branch
from
October 16, 2026 09:40
0ac0351 to
969338a
Compare
pvelesko
changed the title
pvelesko
force-pushed
the
2026-10-13-github-1935-libs-apps-dispatch
branch
from
October 16, 2026 15:42
969338a to
f2e8aa6
Compare
This was referenced Oct 17, 2026
Open
pvelesko
marked this pull request as ready for review
October 18, 2026 15:19
pvelesko
changed the title
pvelesko
added
the
category:ci
No CI lane builds the full install_chipstar.py --all --with-tests set or
runs the tests/apps applications, so a device header change such as
making warpSize non-constexpr (PR 1934) can break rocPRIM and everything
built on it without any lane noticing.
The script builds chipStar plus every library with its tests, runs each
library's ctest suite with the same .ci/exclude.txt and
.ci/ctest-labels.txt convention as libraries-presubmit.yml and the
manual Aurora libs lane, then builds zeroRK and OpenSn and runs the two
zeroRK hydrogen CPU-vs-GPU correctness checks (both
ZERORK_REACTOR_USE_LU modes) and the OpenSn strong scaling input, once
per backend (Level Zero and OpenCL by default). The zeroRK box reactor
check is not run (summary.txt records the SKIP):
zerork_correctness_box_reactor.sh reads PBS_NODEFILE
and launches 12 ranks with Intel MPI options (-ppn, -d, -env) through
Aurora's gpu_tile_compact.sh, none of which exist on a workstation
runner. The hydrogen outputs are copied into logs/ because the next
backend's run deletes them.
The GPU may be shared with other work on the runner, so:
- before each GPU step it waits until fuser shows no process holding the
GPU's render node (resolved from GPU_PCI via sysfs);
- a journal cursor is taken before the first GPU step. Any reset,
wedge, hang or timed-out job kernel line from the GPU's driver (xe or
i915) for its PCI address after it
(matched case-insensitively: xe logs both guc_exec_queue_timedout_job
and "Timedout job ... in <process>"),
seen during a step (polled every second), between steps, or while
waiting for another holder to let go, kills the running step and
skips all remaining GPU work, including the OpenCL platform probe
(clinfo also runs as a GPU step), since a wedged GPU needs a reboot
and every later test would only hang on it. A cursor rather than a
line count, because journal rotation can shrink the count and hide a
new line. The script refuses to start GPU work if it cannot read the
kernel journal, and a later journal read error ("Failed to seek to
cursor", "No journal files", or a nonzero journalctl exit) counts as
a fault. The first observation
is kept in logs/fault, which makes it sticky (a transient read error
cannot be forgotten by the next query) and keeps the first 100
matching lines followed by the first 900 driver lines since the cursor,
so the line naming the process whose job hung is kept. The match uses a here-string, not a pipe: under pipefail,
`echo "$j" | grep -q` loses a large reset storm when grep exits early
and echo dies of SIGPIPE. The STOP line names the ctest test that was running;
- each GPU step runs under setsid and the fault path kills the step's
pid (in case setsid has not run yet) and then its whole session
(pkill -s). Killing only the process group is not enough:
OpenMPI's mpiexec starts its rank in a separate process group of the
same session, which survived a group kill. ctest's test processes stay
in ctest's group and session, so they die either way;
- GPU work is refused if the boot's kernel log already declares this
GPU wedged (a cursor taken later would not see it) or cannot be read,
and when fuser is missing;
- SIGINT/SIGTERM to the script kills the running step's session, which
setsid has taken out of the terminal's process group;
- ctest runs serially (-j1) with a per-test timeout (CTEST_TIMEOUT,
default 900 s).
The GPU is the first Intel display-class PCI device off bus 00, where
integrated GPUs sit, unless GPU_PCI is set: cupcake's Arc B570 (xe) and
meatloaf's Arc A770 (i915), both at 0000:03:00.0. Its driver name comes
from sysfs and selects the kernel log lines watched above.
Device selection uses NEO's FilterBdfPath debug key
(NEOReadDebugKeys=1) so neither Level Zero nor the Intel OpenCL
platform enumerates the integrated GPU next to the discrete one; NEO otherwise opens
every Intel GPU in each process. Verified on cupcake: with the filter,
hipInfo reports only the Arc B570 under both CHIP_BE=level0 and
CHIP_BE=opencl. The OpenCL platform index is taken from clinfo.
CHIP_DEVICE_TYPE is unset with the other selectors because chipStar
rejects it next to CHIP_PLATFORM.
zeroRK is built with the flags of its scripts/build_aurora.sh, minus
the module loads and the unbounded `make -j`, and with the system BLAS
(ZERORK_HAVE_MKL=OFF): its cmake/blas.cmake turns MKL on whenever
ONEAPI_ROOT is set and then links $ONEAPI_ROOT/mkl/latest/lib/intel64
static archives, a layout cupcake's flat oneAPI install does not have. CMAKE_POLICY_VERSION_MINIMUM
is exported because both apps pull in projects (yaml-cpp, SuperLU) whose
cmake_minimum_required is below 3.5, which CMake 4 rejects; a -D flag
does not reach their nested ExternalProject configures, the environment
variable does. OpenSn's dependencies (PETSc, VTK, HDF5, Caliper, Boost)
do not involve chipStar, so they are built once per runner under
DEPS_CACHE keyed by the OpenSn commit; the cache is valid once the deps
build's final set_opensn_env.sh exists and is non-empty, a rebuild starts from an empty
prefix, and a lock file serializes concurrent invocations sharing the
cache. A hard machine freeze during a deps build on cupcake left
hundreds of zero-byte installed files; the next build into the same
prefix kept them as "Up-to-date" and OpenSn then failed on an empty
caliper/cali-manager.h. Boost is forced into that build
because the system Boost 1.83 satisfies the deps script but not OpenSn's
>= 1.86 requirement. The outer deps build runs with -j1 because each
dependency already runs make -j$JOBS and nested makes do not share a
jobserver.
The build phase starts from an empty install prefix and library
staging dir, so a reused work dir cannot test against files or test
binaries a previous revision left behind (a failed `git pull` in a
reused clone would otherwise leave the old build's tests runnable). The prefix
is date-stamped by install_chipstar.py; a later libs/apps-only run uses
the newest one.
Arc A-series GPUs (meatloaf's A770) have no FP64 hardware, and zeroRK,
OpenSn and most library tests use double: on the A770 every double
kernel fails to build with "Double type is not supported on this
platform". Before the first backend the script reads clinfo --raw for
the selected GPU and, when its Intel OpenCL Graphics device reports no
CL_DEVICE_DOUBLE_FP_CONFIG, exports OverrideDefaultFP64Settings=1 and
IGC_EnableDPEmulation=1, the two settings compute-runtime's
opencl/doc/FAQ.md names for FP64 emulation, and records a NOTE line in
summary.txt. GPUs with FP64 hardware (B570, PVC) keep running double
natively. Emulation is slow: one zeroRK hydrogen run takes about 25
minutes on the A770, and it matches the CPU history exactly.
OpenSn passes -DCMAKE_HIP_PLATFORM=spirv, which CMake accepts only from
4.3 (4.2.3's CMakeDetermineHIPCompiler.cmake has no spirv), and Ubuntu
24.04 ships 3.28. Rather than replace a CI runner's system CMake under
every other job, build_opensn downloads the CMake 4.3.2 release into
DEPS_CACHE, checks its SHA-256 against Kitware's published sum, and uses
it for the OpenSn configure and build only.
The zeroRK check script and OpenSn inputs are taken from the tree under
test, so a PR that changes them is tested with its own version.
tests/apps/run_apps.sh is not reused directly: it hard-codes Aurora
modules, /lus paths and gpu_tile_compact.sh, and its performance
baselines are PVC numbers.
Runs scripts/ci/libs-apps.sh on a runner labelled [self-hosted, intel-dgpu]; any Intel discrete GPU will do, and meatloaf (Arc A770) carries the label. The label is dedicated so that a runner added only for this lane does not also pick up unit-tests-x86-intel.yml jobs through the default Linux/X64 labels. Two triggers. workflow_dispatch from the Actions tab takes a pull request number or a ref. A /check-apps comment on a pull request, from the same allow list as /run-aurora-ci in trigger-gitlab-ci.yml, tests that pull request's head commit and reacts with a rocket. Whenever a pull request is involved the job posts a check-apps commit status on its head commit, pending at the start and success or failure at the end, linking to the run. A run started from the Actions tab is otherwise attached to the dispatched branch's commit, not to the pull request, so the status is what makes the result show in the pull request's checks. The job passes -j to the script: one job per 3 GB of RAM, at most nproc - 1. The script's own default subtracts the 1-minute load average, which on a runner still includes the previous job's build: a hand run on meatloaf started while a cross-compile job was finishing picked -j1 for its whole multi-hour build. The header lists what the runner needs installed; meatloaf lacked the GoogleTest, GoogleMock, BLAS, LAPACK, Boost and pybind11 development packages, which rocSPARSE's clients, zeroRK and OpenSn find with find_package. Both triggers only work once this file is on the default branch, which is where GitHub reads workflow_dispatch and issue_comment workflows from. The script is taken from a checkout of the default or dispatched branch and the tree under test from a second checkout, so a pull request opened before the script existed can still be tested. The concurrency group is on the job, not the workflow: every issue comment in the repository starts a run of this workflow whose job is skipped, and a workflow-level group would let those skipped runs replace a queued /check-apps run. One run at a time because the job owns the GPU. The summary and all logs are uploaded whatever the outcome.
actions/checkout writes the token into .git/config by default. The job then builds and runs the tested tree's own code, which could read it. Neither checkout pushes, so persist-credentials: false costs nothing.
… FP64 NEOReadDebugKeys=1 puts the Intel driver into debug mode for every process of every test, which makes high-scratch kernels fail with CL_OUT_OF_RESOURCES (see the comment above chipUnrestrictedAllocSize in src/CHIPDriver.cc) and may change performance. FilterBdfPath, the only reason for it, is itself a debug key. Level Zero and the Intel OpenCL platform both honour ZE_AFFINITY_MASK, and ZE_ENABLE_PCI_ID_DEVICE_ORDER=1 makes its index follow PCI order, so the index is the position of GPU_PCI among the Intel display devices sorted by address. On meatloaf (iGPU 0000:00:02.0, Arc A770 0000:03:00.0) index 1 leaves a single device named Arc A770 under both CHIP_BE=level0 and CHIP_BE=opencl, and clinfo reports PCI-E 0000:03:00.0 for it. With only one OpenCL platform left, the clinfo -l platform lookup and CHIP_PLATFORM are no longer needed and go. FP64 emulation (IGC_EnableDPEmulation, OverrideDefaultFP64Settings) is exported unconditionally, as the maintainers' shells do, which removes the clinfo --raw probe and its NOTE line.
Both apps were rebuilt on every run. They are now built under DEPS_CACHE, keyed on the app source commit (zero-rk hip-chipStar head from git ls-remote, OPENSN_COMMIT), marked complete by a .built file, and taken under the same flock as the OpenSn dependencies. The zeroRK key also carries a checksum of the installed rocprim and thrust headers, since zeroRK compiles against them. zeroRK's src/zerork/CMakeLists.txt hardcodes the include directories of one machine (/space/pvelesko/install/HIP/rocThrust/main/include and rocPRIM/2025.04.22/include). Where those exist, an old rocPRIM that treats warpSize as constexpr is picked and fails to compile against current chipStar. Passing -I$PREFIX/include in CMAKE_CXX_FLAGS puts this install's headers first on the command line, ahead of the target's own -I. zero-rk itself is not edited. The cached binaries carry a RUNPATH, not an RPATH, so the LD_LIBRARY_PATH the script exports (this run's $PREFIX/lib) takes precedence: ldd on meatloaf resolves libCHIP, libhipblas, libhipsolver, libhipsparse and librocsparse to the install under test.
…n runs
The four hydrogen runs (ZERORK_REACTOR_USE_LU 0 and 1, two backends)
all took the same Seulex finite-difference path. zerork_gpu_paths.sh
runs the h2 mechanism (10 species, a few seconds) through five
configurations of zerork_cfd_plugin_tester_gpu.x, each against a CPU
reference, with 2048 reactors (it must stay at or above n_reactors_min,
256, or zeroRK silently falls back to the CPU):
cvode_analytic_cv integrator 0, iterative, analytic Jacobian: SUNDIALS
nvector_hip reductions, SPGMR, analytic kernels
seulex_fd_cv integrator 1, finite-difference Jacobian kernels
radau_analytic_cv integrator 3: complex hipBLAS getrf/getri/getrs
cvode_analytic_cp constant pressure reactor kernels
seulex_fd_notemp_batch always_solve_temperature 0, n_reactors_max 1024:
the no-temperature Jacobian and multi-group batching
ZERORK_REACTOR_USE_LU alternates 1,0,1,0,1 so both the getrf+getrs and
the getrf+getri+bdmv factorisations are covered. Only the dense GPU path
exists in this build (MAGMA and hipSolverRf off), dense 0 on the GPU hits
assert(false), and CVODE with iterative 0 on the GPU falls back to a host
LAPACK solver, so none of those is used. The constant volume and
constant pressure CPU references are computed once and reused by both
backends. Each case prints one PASS or FAIL line, which libs-apps.sh
copies into summary.txt.
The base ymls lack constant_volume, always_solve_temperature and
n_reactors_max, so the new script appends keys that are missing.
zerork_correctness_hydrogen.sh keeps its <0|1> interface for
tests/apps/run_apps.sh, which still calls it.
A hang that logs nothing to the kernel journal is invisible to the fault watcher, so the step would run until the 24 hour job limit. 1800 seconds is well above either app's runtime.
The script no longer calls clinfo. It does call bc (zeroRK case setup), curl and sha256sum (CMake download for OpenSn), and git clone over ssh from github.com (zero-rk).
…olds no write token The self-hosted job runs the tested tree's code as the runner user. That user can rewrite the downloaded action files under _work/_actions, and the later github-script step and the checkout post step would then run them with the job token, which had statuses:write and pull-requests:write. resolve (ubuntu-latest) keeps the allow list gate, resolves the commit, sets the pending status and reacts to the comment, and exposes the sha. run (self-hosted, concurrency libs-apps) needs resolve and has contents: read only; it checks out, builds, tests and uploads logs. report (ubuntu-latest) needs both, runs always when a PR is set, and posts the final status from needs.run.result. Top level permissions are now empty so each job states what it needs. resolve skipped by the gate skips the other two with it.
…ut to KILL gpu_step only killed the session on the fault path. On the normal path an mpiexec rank that survived the SIGTERM from timeout could outlive the step and keep the GPU busy. pkill -s after wait removes any survivor, and timeout -k 60 sends SIGKILL if the command ignores SIGTERM.
The script is taken from the default or dispatched branch (HERE, where libs-apps.sh itself lives) so testing a ref older than the helper does not fail with rc 127. Only the tested tree's chipStar sources come from SRC.
CMake orders a target's compile line DEFINES, INCLUDES, FLAGS, so the CMAKE_CXX_FLAGS -I$PREFIX/include comes after zero-rk's hardcoded /space/pvelesko/install/HIP/rocThrust and rocPRIM include dirs, which win where they exist. The earlier comment claimed the opposite. Tracked in #2006, fixed in zero-rk.
hipcc embeds the install's headers, device bitcode (lib/hip-device-lib) and LLVM pass plugins (lib/llvm/libLLVMHip*.so) into what it builds, so a cached zeroRK or OpenSn built against another chipStar would be reused stale. Both keys now carry a cksum over include, lib/hip-device-lib, lib/llvm and bin of the install. The include tree contains rocprim and thrust, so the separate rocPRIM/rocThrust checksum is subsumed. The sum is of file contents only, so it is stable between computations.
always_solve_temperature is a key of the cfd plugin input ($zrk), read by zerork_cfd_plugin, not of the tester input ($in). Written to the tester yaml it was ignored, so the no-temperature case ran the normal temperature path.
…ith fresh output zeroRK retries failed GPU reactors on the CPU, and runs everything on the CPU when no device is found or a group is below n_reactors_min, with the hist still matching the CPU reference. A case now passes only if every row of its timing log has n_cpu == 0 and n_gpu > 0 (columns 3 and 5), and for the no-temperature case also n_gpu_notemp > 0 (column 6). Each case removes its previous hist, log and stdout before running, so a run that dies early cannot be judged on the other backend's leftovers, and a nonzero exit status of the executable is a FAIL. A CPU reference whose run fails is removed so it is not reused. Valid references are still computed once and shared by both backends.
…temperature reference On the A770 seulex_fd_cv took about 155 s per solve at 2048 reactors and 10 solves, roughly 26 minutes, which used the whole 1800 s step budget so the next case was killed. The timeout stays; the cases shrink to 512 reactors (still above n_reactors_min, 256) and 4 solves, with n_reactors_max 512 and 256 so the batch case still forms two groups. The no-temperature case differed from the CPU reference by about 1e-2 in T because the reference solved temperature. References are now keyed by constant_volume and always_solve_temperature (ref_cv1_t0 for that case), so tolerance is not loosened. n_gpu_notemp is nonzero only on some solves (one of four in the logs), so that check requires it on at least one row, while n_cpu == 0 and n_gpu > 0 are still required on every row.
… not finite compare() flagged a row only when dt > tol || dp > tol. Every comparison with NaN is false in awk, so a GPU history with -nan in the temperature or pressure column passed. The test is now !(dt <= tol && dp <= tol), which is true for NaN as well as for a difference above the tolerance.
…he GPU runs zeroRK's GPU path solves temperature on the first SolveReactors call (solve_temperature = n_calls_ == 1 in zerork_reactor_manager.cpp) whatever always_solve_temperature says; the CPU path does not on its first call. A CPU reference with always_solve_temperature 0 therefore differs from the GPU at row 1 by about 1.3e-4, above the 1e-4 tolerance, for a reason that is not a GPU bug. The reference now runs step 0 on the CPU with always_solve_temperature 1, writes that row as a state file, and restarts from it with always_solve_temperature 0 for the remaining three steps. compare() skips GPU row 0 for that case so the restart rows line up with GPU rows 1 to 3. A CPU run of this sequence matched the GPU path to about 1.2e-5. run() takes the step count, solution time and state file from STEPS, SOLT and STATE, defaulting to the 4 step run. The solution time is written as the literal bc printed (8.2565998e-7 * 2), so bc leaves the runner needs. The tester reads one state file per reactor, with the species mass fractions in mechanism order and then T and P, so the two reference runs use one reactor and run() always sets log_species to list every species for the state file.
… rocThrust zero-rk hip-chipStar now finds rocthrust through find_package (CHIP-SPV/zero-rk#6) instead of hardcoded include directories, so CMAKE_PREFIX_PATH=$PREFIX already resolves rocthrust and rocprim from this lane's install. The -DCMAKE_CXX_FLAGS=-I$PREFIX/include that tried to override the hardcoded paths, and its comment referencing #2006, are dropped. The cache key needs no change: the zero-rk commit is in the directory name and the headers under $PREFIX/include are already hashed.
…awk and locale The tolerance test !(dt <= tol && dp <= tol) relies on NaN comparing false. mawk 1.3.4 (stock Ubuntu awk) reads nan and -nan as NaN but evaluates NaN <= tol as true, so a NaN temperature or pressure from the GPU passed. Reject any GPU temperature or pressure containing nan or inf textually. zeroRK writes the history with %g, so the only letters in a finite number are the exponent e. The runner's LC_NUMERIC is lt_LT.UTF-8 (comma decimal), under which mawk parses 1450.19 as 1450 and accepts finite mismatches. Run the comparison under LC_ALL=C. Checked with a synthetic row (reference 0 4.1283e-07 1450.19 2.02672e+06) under mawk, gawk and gawk --posix with LC_NUMERIC=lt_LT.UTF-8: identical GPU row exits 0; GPU P of nan, -nan, inf and a finite mismatch 2.02700e+06 exit 1.
… reference The no-temperature CPU reference restarted from a state produced by a separate one-step CPU solve (ref_s0). That solve is the same CPU configuration as ref_cv1_t1, with the same first step dt and uniform initial conditions, so row 1 of ref_cv1_t1_000.hist is byte identical to ref_s0_000.hist. Take the state from the first data row of ref_cv1_t1_000.hist instead (awk exits after that row) and drop the duplicate solve.
The nonfinite check in compare() tested only the GPU fields, but the CPU reference can hold a nonfinite value too: zeroRK GetState computes the final T and P without checking that they are finite. With nan or inf in the reference columns 3 or 4, dt or dp becomes NaN, and mawk accepts a NaN comparison, so !(dt <= tol && dp <= tol) is false and the row passed. Reproduced under mawk with LC_ALL=C: reference row "0 4.1283e-07 nan 2.02672e+06" against GPU row "0 4.1283e-07 1450.19 2.02672e+06" returned rc=0, as did a CPU P of inf. Test the reference fields $3 $4 together with the GPU fields.
… commit The report job runs on a hosted runner, outside the libs-apps concurrency group (only the self-hosted run job is in it), and it always writes the final check-apps status. Two requests on one head can therefore finish out of order: 1. Request A and request B are made on the same head. B's resolve job writes pending with B's run URL. A's run finishes later than B's resolve, and A's report then overwrites B's pending with success or failure while B is still running. 2. A is cancelled while queued by B. Its run job is cancelled, but its report job still runs with RESULT=cancelled and writes failure over B's pending or final status. The report now reads the latest check-apps status of the commit and skips the write unless that status carries this run's URL, which is the exact URL the resolve job wrote as pending. A newer request has replaced it with its own URL, so the older report leaves the status alone. Concurrency stays at job level (every issue comment triggers this workflow) and the run job still holds no write token.
run_libs skipped any component without build/CTestTestfile.cmake and said nothing. rocSPARSE builds rocsparse-test (BUILD_CLIENTS_TESTS=ON) but only clients/ enables testing and no add_test exists, so its tests never run and nothing in summary.txt said so. chipRCCL and chipStdPar are in the same state. libraries-presubmit.yml already prints "no CTest tests" for this case; write the equivalent SKIP line to the summary.
The run job's concurrency group libs-apps uses the default single-slot queue: with run A in progress and B pending, a third request C cancels B. B's report job (if: always()) then posts a failure for a run that never executed. Setting queue: max keeps up to 100 pending runs in the group (valid because cancel-in-progress is not set), so each request runs in order on the single GPU.
Renamed the comment command from /check-apps to /run-app-tests, the workflow name to "Run app tests (/run-app-tests)", and the commit status context from check-apps to app-tests, to match the /run-math-tests workflow on main. The match now requires the exact command followed by end of comment, a space, a newline or a carriage return, so a longer command that shares the prefix, such as /run-app-tests-foo, does not start a GPU run.
i915 and xe answer a hung job by resetting the engine and banning the offending context; the GPU keeps working afterwards. Only "wedged" means the driver gave up and every later GPU call fails until reboot. gpu_step killed the step on any reset, hang or timedout line, set GPU_DEAD and skipped every later step. One recovered reset therefore threw away the rest of the current ctest suite and every later library and app result. Now only a wedged line for this GPU or a journal read error is fatal and keeps the kill, GPU_DEAD, STOP note and SKIP of later steps. A reset, hang or timedout line lets the step run on, so ctest moves to its next test and CTEST_TIMEOUT or the timeout wrappers bound the hung one. When the step ends, those lines are saved to "<step log>.gpu-faults", a NOTE names the file, and the step is reported FAIL even if its rc was 0. The journal cursor is refreshed at each step start so a reset is attributed only to the step it happened in. The per-step summary lines still print.
…rnal lines Three defects in the GPU fault handling of the previous commit. i915 can wedge the GPU without printing "wedged". In Linux v7.0 drivers/gpu/drm/i915/gt/intel_reset.c, intel_gt_reset() logs "GPU reset not supported", "Failed to reset chip" or "Failed to initialise HW following reset (%d)" and then calls __intel_gt_set_wedged(), which logs nothing with "wedged"; intel_wedge_me() logs "<name> timed out, cancelling all in-flight rendering." and wedges as well. The fatal pattern is now one variable, FATAL_MSG, matching wedged or those messages, used by gpu_faults and the boot precheck. The recoverable scan excludes FATAL_MSG, so "Resetting chip" and "GPU HANG" stay recoverable while "Failed to reset chip" is fatal. gpu_step advanced the fatal monitor's CURSOR before each launch, so a fatal line logged between the last gpu_faults read and that cursor() call was skipped for good. CURSOR is now never advanced; a separate per-step cursor is used only to attribute recoverable resets to a step. The post-step journalctl read ignored failure, and a failed cursor() was ignored too, so a step could pass with the journal unreadable. A failed read or an empty per-step cursor now marks the step FAIL with a note.
… read gpu_step polled gpu_faults() while the step ran and once at exit, but the per-step journal read that collects recoverable resets came after the last poll and filtered FATAL_MSG lines out with grep -v without acting on them. A fatal line logged in that window was dropped and the step could PASS, and a journal read error there was not sticky. Move the wait and reset scan before the fatal block and call gpu_faults() once more after it. CURSOR is continuous, so that read is later than the snapshot, sees any fatal line, and also covers journal errors and the sticky $LOGS/fault. A fault takes the existing fatal path (GPU_DEAD=1, STOP note, result 137).
…the rest libs-apps.sh wraps the whole script in one timeout and no single tester run was bounded, so one hung case killed the loop and the later cases never reported. Wrap each tester run in timeout -k 30 1800 so the hung case is reported FAIL and the loop continues. Measured on meatloaf (~/libs-apps-1945/work/logs/level0/zerork-out/*.log and file mtimes): the slowest GPU case is seulex_fd_cv at about 320s summed over its four solves (about 380s by mtime); radau_analytic_cv about 107s, cvode_analytic_cp about 78s, seulex_fd_notemp_batch about 91s, cvode_analytic_cv about 26s; CPU references under 1s each. 1800s is about 5x the slowest. Raise the outer budget from 1800 to 22000 so it covers the worst case of 11 runs (6 CPU references and 5 GPU cases) at 1830s each.
…esets journald reads /dev/kmsg asynchronously (systemd src/journal/journald-kmsg.c), so a recovered reset logged as the step exits could appear in the journal after the scan. The next step's per-step cursor would then exclude it, and the step would PASS unreported. 2 s covers the ingestion lag at a cost of 2 s per GPU step.
pvelesko
force-pushed
the
2026-10-13-github-1935-libs-apps-dispatch
branch
from
October 30, 2026 00:40
c533d03 to
1b648c4
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds libs-apps.yml, which runs scripts/ci/libs-apps.sh on a runner labelled [self-hosted, intel-dgpu]. The script builds chipStar plus every library from install_chipstar.py --all --with-tests, runs each library's ctest suite with the same .ci/exclude.txt and .ci/ctest-labels.txt convention as libraries-presubmit.yml, then builds zeroRK and OpenSn and runs five zeroRK GPU code path checks against CPU references and the OpenSn strong scaling input, on the first Intel discrete GPU under Level Zero and OpenCL. Logs and a summary are uploaded.
It can be started from the Actions tab with a PR number or ref, or by commenting /run-app-tests on a PR (same allow list as /run-aurora-ci). With a PR it posts an app-tests commit status on the PR head, so the result shows in the PR's checks. Both triggers work only once the workflow is on main.
GPU steps run one at a time and wait while another process holds the render node. A step during which the GPU's driver logs a reset, hang or timed-out job is reported as failed and the run continues; all GPU work stops only when the driver declares the GPU wedged. The zeroRK box reactor check is not run because its launcher needs Aurora's PBS and MPI setup. FP64 emulation is always enabled, so double also runs on GPUs without FP64 hardware (Arc A-series). The zeroRK and OpenSn builds are cached per runner.
Fixes #1935
This PR was generated using AI