SRE-4037 test: Fix doca-ofed install on Cluster Box stages by phender · Pull Request #19069 · daos-stack/daos

phender · 2026-09-15T14:33:02Z

The doca-ofed install on cluster box nodes failed due to a script error when attempting to
obtain the Mellanox adapter type. Once this issue was fixed it was discovered that the
doca-ofed package ended up being removed when installing daos packages due to a
conflict between dependencies requiring libmpi.so.40()(64bit) and
libmpi.so.40()(64bit)(openmpi-x86_64). This had been previously resolved by the
daos-mofed-shim package, but after moving to fpm it was no longer being built. The
building of the daos-mofed-shim package has been reintroduced and now the daos
packages can be installed w/o removing doca-ofed.

In addition to the fix the following changes have also been included to support testing
the changes:

the harness/setup.py functional test has been updated to support testing on all
hardware and cluster box stages
any functional hardware or cluster box stage that supports running with a different
provider (via params or pragmas) now installs both mercury-libfabric and mercury-ucx.
Any stage running a fixed provider will install the appropriate mercury package.

Test-provider: ucx+dc_x
Quick-functional: true
Skip-test: true
Skip-Functional-Hardware-Medium-Verbs-Provider-MD-on-SSD: true
Skip-Functional-Hardware-Large-MD-on-SSD: true
Test-tag: test_setup

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

Fix the post_provision_config_nodes Mellanox adapter type detection to account for virtual devices. This will fix installing doca-ofed on cluster box nodes. Test-provider: ucx+dc_x Quick-functional: true Skip-test: true Skip-Functional-Hardware-Medium-Verbs-Provider-MD-on-SSD: true Skip-Functional-Hardware-Large-MD-on-SSD: true Test-tag: test_setup Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

github-actions · 2026-09-15T14:33:19Z

Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/SRE-4037

Test-provider: ucx+dc_x Quick-functional: true Skip-test: true Skip-Functional-Hardware-Medium-Verbs-Provider-MD-on-SSD: true Skip-Functional-Hardware-Large-MD-on-SSD: true Test-tag: test_setup Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2026-09-15T16:42:05Z

Test stage Functional Cluster Box Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19069/3/execution/node/743/log

daosbuild3 · 2026-09-15T22:08:24Z

Test stage Functional Cluster Box Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19069/4/execution/node/865/log

Test-provider: ucx+dc_x Quick-functional: true Skip-test: true Skip-Functional-Hardware-Medium-Verbs-Provider-MD-on-SSD: true Skip-Functional-Hardware-Large-MD-on-SSD: true Test-tag: test_setup Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2026-09-16T13:56:55Z

Test stage Functional Cluster Box Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19069/5/execution/node/743/log

Test-provider: ucx+dc_x Quick-functional: true Skip-test: true Test-tag: test_setup Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2026-09-16T14:42:03Z

Test stage Functional Cluster Box Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19069/6/execution/node/744/log

Test-provider: ucx+dc_x Quick-functional: true Skip-test: true Test-tag: test_setup Skip-Functional-Hardware-Medium: false Skip-Functional-Hardware-Medium-VMD: false Skip-Functional-Hardware-Medium-Verbs-Provider: false Skip-Functional-Hardware-Medium-UCX-Provider: false Skip-Functional-Hardware-Large: false Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daltonbohning · 2026-09-16T16:08:09Z

+        :avocado: tags=harness,test_setup
+        :avocado: tags=HarnessSetupTest,test_setup_hw


Lint is failing due to

(required) 4 tests with invalid test_ tag test_setup_hw test_setup_hw_provider test_setup_cb test_setup_cb_provider

because all of these variants are tagged with test_setup

Right, I was hoping to still be able to use the old tag - test_setup - to run any of the tests that matched the stages. This would also be useful if the test_setup tag was the default for any downstream testing. We can just use the class tag for this behavior.

Test-provider: ucx+dc_x Quick-functional: true Skip-test: true Test-tag: test_setup Skip-Functional-Hardware-Medium: false Skip-Functional-Hardware-Medium-VMD: false Skip-Functional-Hardware-Medium-Verbs-Provider: false Skip-Functional-Hardware-Medium-UCX-Provider: false Skip-Functional-Hardware-Large: false Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Test-provider: ucx+dc_x Quick-functional: true Skip-test: true Test-tag: HarnessSetupTest Skip-Functional-Hardware-Medium: false Skip-Functional-Hardware-Medium-VMD: false Skip-Functional-Hardware-Medium-Verbs-Provider: false Skip-Functional-Hardware-Medium-UCX-Provider: false Skip-Functional-Hardware-Large: false Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2026-09-16T16:42:53Z

Test stage Functional Hardware Medium VMD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19069/9/execution/node/775/log

phender · 2026-09-16T18:28:10Z

Test stage Functional Hardware Medium VMD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19069/9/execution/node/775/log

The ERROR: scan fabric: no fabric interfaces found with provider ucx+dc_x failure occurs because the opa-[100,102-105] cluster does not support the ucx+dc_x provider with its hardware. When the provider is specified for the build with a pragma or build param, launch.py does not verify that the provider is supported - this only occurs when auto-detecting the provider. So this error is expected in this case, but ultimately verifies the src/tests/ftest/harness/setup.py by proving the test tags will allow the test to run in this stage - where normally it would not run with the ucx+dc_x provider.

Test-provider: ucx+dc_x Quick-functional: true Test-tag: server_setup Skip-Functional-on-Leap-15: false Skip-Functional-on-SLES-15: false Skip-Functional-Hardware-Medium: false Skip-Functional-Hardware-Medium-VMD: false Skip-Functional-Hardware-Medium-Verbs-Provider: false Skip-Functional-Hardware-Medium-UCX-Provider: false Skip-Functional-Hardware-Large: false Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2026-09-16T19:24:29Z

Test stage Functional on Leap 15 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19069/10/execution/node/892/log

daosbuild3 · 2026-09-16T19:25:47Z

Test stage Functional on SLES 15 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19069/10/execution/node/912/log

daosbuild3 · 2026-09-16T19:28:26Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19069/10/execution/node/907/log

daosbuild3 · 2026-09-16T19:41:32Z

Test stage Functional Hardware Medium VMD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19069/10/execution/node/1265/log

phender added 2 commits September 15, 2026 10:40

Add building daos-mofed-shim RPM.

c6ef4f9

Test-provider: ucx+dc_x Quick-functional: true Skip-test: true Skip-Functional-Hardware-Medium-Verbs-Provider-MD-on-SSD: true Skip-Functional-Hardware-Large-MD-on-SSD: true Test-tag: test_setup Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Updates

edb2f77

Test-provider: ucx+dc_x Quick-functional: true Skip-test: true Test-tag: test_setup Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

This was referenced Sep 16, 2026

SRE-4037 test: Update openmpi dependency to use generic libmpi.so daos-stack/hdf5-vol-daos#41

Closed

SRE-4037 test: Update openmpi dependency to use generic libmpi.so daos-stack/hdf5#42

Closed

phender marked this pull request as ready for review September 16, 2026 16:03

phender requested review from a team as code owners September 16, 2026 16:03

daltonbohning reviewed Sep 16, 2026

View reviewed changes

phender added 2 commits September 16, 2026 12:11

phender requested review from daltonbohning and ryon-jensen September 16, 2026 16:17

daltonbohning previously approved these changes Sep 16, 2026

View reviewed changes

Comment thread src/tests/ftest/harness/setup.py Outdated

phender dismissed daltonbohning’s stale review via c7836ad September 16, 2026 18:45

daltonbohning approved these changes Sep 16, 2026

View reviewed changes

Navigation Menu

Uh oh!

SRE-4037 test: Fix doca-ofed install on Cluster Box stages - #19069

SRE-4037 test: Fix doca-ofed install on Cluster Box stages#19069
phender wants to merge 10 commits into
masterfrom
hendersp/SRE-4037

phender commented Sep 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 15, 2026

Uh oh!

daosbuild3 commented Sep 15, 2026

Uh oh!

daosbuild3 commented Sep 15, 2026

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

daltonbohning Sep 16, 2026

Uh oh!

phender Sep 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

phender commented Sep 16, 2026 •

edited

Loading

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

		:avocado: tags=harness,test_setup
		:avocado: tags=HarnessSetupTest,test_setup_hw

Uh oh!

Conversation

phender commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Sep 15, 2026

Uh oh!

daosbuild3 commented Sep 15, 2026

Uh oh!

daosbuild3 commented Sep 15, 2026

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

daltonbohning Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

phender Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

phender commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

phender commented Sep 15, 2026 •

edited

Loading

phender Sep 16, 2026 •

edited

Loading

phender commented Sep 16, 2026 •

edited

Loading