SRE-4019 ci: make downstream testing build-only and fix axis list

[ryon-jensen]

Note

The red release/2.6 axis is not caused by this PR. Build RPM on Leap 15.5 fails in %build with Checking for C library fabric... no / BuildRequired: ofi needs to be built, and it failed identically on build 1 of this PR — before any of this branch's library code was loaded. It is a broken Leap 15.5 mock environment for 2.6 and needs its own ticket. NLT on EL 8.8 is UNSTABLE for the same pre-existing reason. Details under Pre-existing failures surfaced below.

Problem

The DAOS Build and Test matrix creates a real branch in daos-stack/daos per axis with the @Library line pointed at the PR, then triggers the full DAOS pipeline against it — functional, fault injection and hardware stages included.

Measured over the last 9 pipeline-lib master builds:

Metric	Value
Every stage except Test Library	~4 min total
Test Library axis duration	4 min – 14,094 min
Axis pass rate	11 of 36 (31%)
Only fully green build (48)	26.6 h
Worst build (45)	17,611 min (~12 days)

Two defects:

1. Wrong depth. The question the matrix answers — "does this library load and drive this branch's pipeline correctly" — is settled by the build and unit test stages. Everything past that re-tests DAOS and inherits DAOS CI's flakiness and hardware contention. At a 31% pass rate the result is not usable as a gate, so the cost buys almost no assurance.
2. Wrong branches. release/2.8 and weekly-2.8-testing got zero coverage, while release/2.6 consumed two of the four axes and most of the elapsed time.

Changes

vars/buildDaosJob.groovy — the machinery existed but was inverted: a full test run was the default and skipping was opt-in. Now build-only is the default and a full downstream run is opt-in via Full-downstream-test: true. Skip-downstream-test: true still works and takes precedence over Full-downstream-test.

The subtlety is that Skip-test: true does not cover everything. A build-only run emits three pragmas:

Pragma	Why it is needed
Skip-test: true	Test parent stage: functional and RPM tests on all branches
Skip-test-hardware: true	daos master and release/2.8 gate hardware stages on their own parent pragma in getStageNameSkipPragmas(); skip-test is never consulted for them
Skip-fault-injection-test: true	on release/2.6 the fault injection stage is gated by this library's skipStage(), which checks only skip-fault-injection-test / skip-func-test

The last two were each confirmed empirically, not by reading code alone — see below.

Jenkinsfile — axis list is now master, release/2.6, release/2.8, weekly-testing, weekly-2.6-testing, weekly-2.8-testing.

Axis-list decision

release/2.6 keeps one axis and loses the second. It is the only remaining branch whose Jenkinsfile is driven by this library's own skipStage() (27 call sites; the other five branches all use their in-repo getStageNameSkipPragmas() and zero skipStage() calls). That makes it the highest-value integration target we have, and it is exactly the kind of breakage the mocked JUnit tests cannot catch — it is also the branch that exposed the fault injection gap above.

All three weekly branches are kept. An earlier revision of this PR dropped weekly-2.6-testing on the grounds that it is structurally the same generation of Jenkinsfile as weekly-testing and weekly-2.8-testing and largely duplicated their coverage. That de-duplication is not worth making: once downstream runs are build-only the weekly axes finish in under a minute each, so the marginal cost of keeping all three is effectively nil, and having every weekly branch covered is worth more than saving a minute.

Net: 4 axes to 6, with release/2.8 and weekly-2.8-testing newly covered. Widening only makes sense because the first change bounds the per-axis cost, which is why both are in one commit.

How this was verified

A pipeline-lib PR does not self-test vars/ changes: the PR job logs Loading library pipeline-lib@master, because this repo's own @Library line is commented out. Only the downstream daos branches get @Library(value='pipeline-lib@<pr-branch>'), and the pragmas are produced upstream by buildDaosJob. So build 1 of this PR ran entirely on master's buildDaosJob and emitted no pragmas at all — Functional Hardware Medium VMD provisioned five real nodes and ran for 25.6 minutes.

To get real evidence, the @Library line was temporarily pointed at this branch (the mechanism documented in the comment above it). That commit is reverted before merge.

Downstream results with the change actually loaded:

Axis	Duration	Result	Functional / fault injection / hardware time
weekly-2.8-testing	0.8 min	SUCCESS	none
weekly-testing	0.8 min	SUCCESS	none
weekly-2.6-testing	not re-measured; same build-only path as the other two weeklies	—	none
release/2.8	58.9 min	SUCCESS	none
master	75.1 min	SUCCESS	none
release/2.6	77.1 min	FAILED (pre-existing, see below)	none

On release/2.6, Fault injection testing on EL 8.8 and Test Hardware both report NOT_EXECUTED. Before the third pragma was added, fault injection ran for 19.7 min.

Downstream commit message confirmed on the created branch:

Update pipeline-lib branch to self

Skip-test: true
Skip-test-hardware: true
Skip-fault-injection-test: true

The remaining time on master, release/2.8 and release/2.6 is builds and unit tests, which are deliberately kept — unitTest(), provisionNodes() and runTest() are this library's code and are worth exercising.

Pre-existing failures surfaced

The release/2.6 axis is red for reasons unrelated to this change, and was red identically in build 1 before this library was loaded:

• Build RPM on Leap 15.5 fails in %build with Checking for C library fabric... no / BuildRequired: ofi needs to be built — a broken Leap 15.5 mock environment for 2.6.
• NLT on EL 8.8 is UNSTABLE.

Arguably that is the point: a build-only axis surfaces a broken 2.6 RPM build in about 20 minutes instead of burying it behind a multi-day functional run.

Tests

New src/test/groovy/testBuildDaosJob.groovy covers the pragma logic: build-only default, the three skip pragmas always emitted together, Full-downstream-test opt-in, Skip-downstream-test precedence and no-op-on-its-own behaviour, Test-skip-build, and that the joined pragma block has no stray blank lines (the old string concatenation could emit a leading newline).

./gradle test spotlessCheck   # BUILD SUCCESSFUL

The temporary @Library commit has now been reverted, so the @Library line is commented out again as it should be on master. Because that revert means this PR's own final CI run falls back to master's buildDaosJob — i.e. the old full-test behaviour this PR is removing — the revert commit carries Skip-daos-build-and-test: true.

Out of scope

Moving a full downstream run to a nightly/weekly trigger on pipeline-lib master — deferred to a separate ticket.

[ryon-jensen]

Ready to land, and best landed before #533

This PR is ready to merge as it stands. Verification is complete and recorded in the PR body: all five axes were exercised with this branch's buildDaosJob actually loaded, and functional, fault injection and hardware stages consumed zero time on every axis. Build 4 was SUCCESS in 6.6 min.

Why the matrix shows NOT_EXECUTED on the latest build. The HEAD commit carries Skip-daos-build-and-test: true deliberately. After the temporary @Library self-load was reverted, this PR's own matrix would run under master's buildDaosJob — i.e. the old full-test behaviour this PR exists to remove. Running it would have cost hours and proved nothing about the change. The evidence that matters was gathered while the library was actually loaded, and is in the table in the description.

Sequencing

This is worth landing ahead of #533 (SRE-4015). The two PRs touch no files in common, so the order is free to choose — but #533 still needs a green build to land, and until this change is on master that build runs the full four-axis matrix at hours-to-days cost. With this merged first, #533's verification becomes build-only at roughly 77 minutes worst case while still exercising the library.

CI is currently overloaded, which is the reason for caring about the order at all.

/cc SRE-4019