SRE-4015 ci: run full PR verification only on the stack tip

[ryon-jensen]

Note

The red build on this PR is not caused by this PR. Build 3 FAILED at
125 min with exactly one failing stage, Test Library — the downstream
daos release/2.6 axis. That axis fails in NLT on EL 8.8 (UNSTABLE),
a pre-existing breakage on 2.6 that was already observed and documented on
the same axis during SRE-4019 (#534), before any of this branch's code
existed. It needs its own ticket.

Every one of this library's own 27 executed stages passed, including
JUnit Tests, Self Unit Test and Commit Pragma tests. The other five
downstream axes — master, release/2.8, weekly-testing,
weekly-2.6-testing, weekly-2.8-testing — are all SUCCESS.

This PR will also need an admin merge regardless, per SRE-3162: master
requires five contexts whose stages are hard-disabled with
expression { false }, so no pipeline-lib PR can go green on its own.

GitHub stacked PRs open one pull request per layer, and today every
layer runs the complete DAOS verification pipeline. For a stack N deep
that is N full runs of work to validate a single change set, even
though only the tip contains the cumulative result that will land.

Add the primitives needed to make the pipeline stack aware:

prStack() fetches and caches the GitHub REST 'stack' object for
the current PR, failing open to an empty map so a
GitHub outage cannot break unrelated builds
isStackTip() true for non-stacked PRs and for the top layer of a
stack, honouring the CI_STACK_TIP_ONLY parameter and a
Skip-stack-optimization commit pragma as escape hatches
targetBranch() resolves the branch the change will actually land on

targetBranch() also fixes a latent correctness bug. For a mid-stack PR
env.CHANGE_TARGET is the parent layer's branch, not the landing branch,
so repository selection, package versions and stage skipping were all
computed against the wrong branch. The stack's base.ref is authoritative
and is now preferred when present.

skipStage() and skipFunctionalTestStage() skip everything on a mid-stack
PR except a small allow list of cheap checks plus a compile-only build,
so a broken layer still fails fast without paying for hardware testing.

githubAccess() is promoted out of setupDownstreamTesting so prStack can
share the same credential binding.

Signed-off-by: Ryon Jensen ryon.jensen@hpe.com

Validation

Exercised against a real 3-deep stack in daos-stack/daos (stack 19005, PRs #19002-#19004, now closed and their branches deleted):

Layer	prStack()	Stages run	Result	Duration
1 of 3	layer 1 of 3 onto master	5	SUCCESS	6m34s
2 of 3	layer 2 of 3 onto master	5	SUCCESS	10m01s
3 of 3 (tip)	layer 3 of 3 onto master	22	full pipeline	aborted once gating was confirmed, to free test hardware

The mid-stack layers ran only Cancel Previous Builds, Pre-build, Python Bandit check, Build and Build on EL 9.

targetBranch() was confirmed to fix the landing-branch bug: on PR #19003, whose CHANGE_TARGET was ryon-jensen/test_stacked_prs_1, the build ran ci/gen_commit_pragmas.py --target origin/master.

Two things were found and fixed during that validation, both of which the fail-open path absorbed without breaking a build:

1. prStack() is called while the Jenkinsfile is still loading, before any node exists, so sh() threw MissingContextVariableException.
2. The library is sandboxed, so a raw HttpURLConnection was rejected for setRequestProperty.

Both are why the lookup now uses only the httpRequest and readJSON steps.

Follow-up commit: letting the skipped layers go green

Skipping a stage is not free on a protected branch. Every Jenkins GitHub
status context is published from inside its stage body — sconsBuild,
runTest, buildRpm and provisionNodes all call scmNotify/stepResult
with a context:. A stage that never executes never publishes anything, so
GitHub leaves the context as "Expected — waiting for status to be reported"
indefinitely. There is no end-of-build backfill anywhere in this library.

daos-stack/daos master requires 20 contexts, so without this the mid-stack
layers of a stack could never be merged. Measured on the original test stack
19005: mid-stack PR #19002 published 4 of the 20 and sat at
mergeable_state: blocked.

A trap worth writing down: GET /commits/{sha}/status returns success
for these PRs, because it only aggregates the contexts that were
published. Branch protection evaluates the required contexts, which are
absent. Never judge this from the combined status endpoint.

vars/notifySkippedRequiredChecks.groovy reads the required status-check
contexts of the branch the change will actually land on (via targetBranch()),
maps each test/- or build/-prefixed context back to its stage name, and
publishes a SUCCESS status for each one whose stage was deliberately
skipped. The description names the stack and the layer, so the reason is
visible on the PR rather than buried in a build log.

It no-ops unless CHANGE_ID and DAOS_JENKINS_NOTIFY_STATUS are both set,
and fails open on any exception — the same posture as prStack().

Ordering is load-bearing. The synthetic statuses are published before
the stages run. Commit statuses are last-write-wins per context, so any stage
that does execute overwrites its synthetic value with the real result,
including a real failure. Publishing afterwards would mask genuine failures.

Contexts without a test/ or build/ prefix are left alone: they are not
published by this pipeline. On daos master that means the six GitHub Actions
checks, which continue to run in full on every layer.

Which stages a mid-stack layer still runs

stack_mid_stages() now includes Build on Leap 15 alongside
Build on EL 9. A mid-stack layer is only ever proven to compile, so it
should compile on both toolchains the project ships; building only EL 9 left
a Leap 15 break invisible until the tip ran, which is exactly the feedback
the layer build exists to give. It is a compile-only stage of the same cost
class, and it added 3.8 min.

The daos Jenkinsfile carries the matching entry in its own mid-stack allow
list, since master and 2.8 gate stages through shouldStageRun() rather than
this library's skipStage().

This also means build/Build on Leap 15 is no longer synthesized: the stage
runs, so it publishes its own real result.

Validation

Against a fresh 2-deep stack (stack 19075, daos PRs #19073 and #19074):

prStack: PR 19073 is layer 1 of 2 in stack 19075 onto master
updateRunStage: Detected mid-stack PR, only building
notifySkippedRequiredChecks: publishing success for 'test/Functional Hardware Medium MD on SSD': Runs on the tip of stack 19075, this is layer 1 of 2

Required contexts on the mid-stack layer	Before	After
Jenkins-owned	4 of 14	14 of 14, green
GitHub Actions (run in full regardless)	6 of 6	6 of 6

Ten contexts were synthesized, all of them the required test/* ones.
build/Build on EL 9 and build/Build on Leap 15 stayed real, because both
stages are on the mid-stack allow list. The layer finished in 6.6 min.

src/test/groovy/testNotifySkippedRequiredChecks.groovy adds 10 tests; 40
pass in total, spotlessCheck clean.

Note

daos-stack/daos needs the companion change, currently daos PR #19073 on
branch ryon-jensen/SRE-4015, which calls isStackTip(), targetBranch()
and notifySkippedRequiredChecks(). This must land first or that
Jenkinsfile will not load.

[ryon-jensen]

Rebased on master; #534 has landed

pipeline-lib#534 (SRE-4019) merged as 072a1f2, so the downstream matrix is now build-only by default. This branch has been rebased onto it (3341507 → 028d3f6) and rebuilt.

The rebase was clean — the two PRs touch no files in common. Locally: 30 tests pass, spotlessCheck clean.

This is the cheap verification run referred to earlier: with build-only downstream testing on master, this PR's matrix should complete in minutes-to-about-an-hour per axis rather than the hours-to-days that forced build 1 to be aborted.

Note for reviewers on the required checks

This PR will still show as blocked, and it is not caused by this change. Branch protection requires five checks that cannot report, because the stages producing them are hard-disabled in the Jenkinsfile with expression { false } pending SRE-3162 (open since June 2025):

• test/provisionNodes on EL 8 with slurm
• test/provisionNodes on EL 9 with master Repo
• test/provisionNodes on Leap 15 with slurm
• test/publishToRepository RPM tests
• test/publishToRepository DEB tests

They are disabled on master too and untouched by this branch. This affects every PR in the repo, so landing requires an admin merge, as #534 did.

Follow-up

Once this lands, the matching daos PR will be raised. It cannot be proposed sooner because its Jenkinsfile calls isStackTip(), which does not exist until this merges.