Skip to content

Set MPILIB from the case in standalone Omega builds - #600

Merged
sbrus89 merged 3 commits into
E3SM-Project:developfrom
xylar:omega/fix-pm-mpi-flags
Oct 14, 2026
Merged

sbrus89 merged 3 commits into
E3SM-Project:developfrom
xylar:omega/fix-pm-mpi-flags

Conversation

@xylar

@xylar xylar commented Oct 11, 2026 •

Copy link
Copy Markdown

Standalone Omega builds include the machine's Macros.cmake without setting MPILIB, which those macros use to choose compilers. Since #566, pm-cpu and pm-gpu use the Cray CC wrapper only when MPILIB is mpich, so standalone builds there have been linking with mpicxx; on pm-gpu, omega.exe then lacks Cray's GPU transport layer and aborts at startup. This sets MPILIB from the case, as CIME builds do with -DMPILIB, and drops MPILIB_NAME.

Among supported machines, only the pm-cpu (gnu) and pm-gpu (gnugpu) macros branch on MPILIB, so other builds are unchanged apart from OmegaBuild.cmake's own mpi-serial checks, which now work.

This also updates the PR template's testing rows: Aurora's and Chrysalis's compilers are now intel rather than oneapi-ifx, as in E3SM's machine configs since #566, and Aurora's GPU build (intelgpu) gets its own row.

Checklist

  • Linting

  • Building

    • CMake build does not produce any new warnings from changes in this PR
  • Testing

    aurora, intel, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    aurora, intelgpu, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    chrysalis, intel, openmpi

    • CTests Pass
    • Polaris omega_pr Pass

    frontier, craygnu, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    frontier, craygnu-mphipcc, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    pm-cpu, gnu, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    pm-gpu, gnugpu, mpich

    • CTests Pass
    • Polaris omega_pr Pass
  • Provide relevant details in a comment to the PR titled Testing

Fixes #599


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

The machine Macros.cmake selects compilers and flags by MPILIB; on
pm-gpu and pm-cpu, mpich uses the Cray CC wrapper (which links GTL on
pm-gpu) and anything else uses mpicxx. CIME builds pass -DMPILIB, but
read_cime_config stored the case's value only as MPILIB_NAME, so
MPILIB was empty when Macros.cmake was included and standalone pm-gpu
builds linked omega.exe without libmpi_gtl_cuda.

Query MPILIB directly so it is set before Macros.cmake is included,
which also makes the existing mpi-serial checks work, and replace the
remaining MPILIB_NAME uses (including a $ENV{MPILIB_NAME} that was
never set).

Fixes E3SM-Project#599

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar

xylar commented Oct 11, 2026

Copy link
Copy Markdown
Author

Testing

I ran Polaris' omega_pr suite with this branch on pm-cpu (gnu, mpich) and pm-gpu (gnugpu, mpich), once from Polaris main (mache 3.12.0) and once from E3SM-Project/polaris#835 (mache 5.0.0rc2), with no baseline. All 29 tasks passed in all four runs, and every build now links through CC, with libmpi_gtl_cuda on pm-gpu:

/pscratch/sd/x/xylar/polaris_1.1/omega_fix_pm_mpi_flags/main_pm-cpu_gnu
/pscratch/sd/x/xylar/polaris_1.1/omega_fix_pm_mpi_flags/main_pm-gpu_gnugpu
/pscratch/sd/x/xylar/polaris_1.1/omega_fix_pm_mpi_flags/mache5_pm-cpu_gnu
/pscratch/sd/x/xylar/polaris_1.1/omega_fix_pm_mpi_flags/mache5_pm-gpu_gnugpu

So the fix works with Polaris main and does not need mache 5.0.0. Without it, every pm-gpu task fails (#599).


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

Since E3SM-Project#566, Aurora's and Chrysalis's Intel compilers are called intel
rather than oneapi-ifx.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar xylar self-assigned this Oct 11, 2026
@xylar xylar added the CMake CMake-related issues label Oct 11, 2026
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar

xylar commented Oct 11, 2026 •

Copy link
Copy Markdown
Author

Testing: chrysalis, intel, openmpi

CTests: 58 of 58 passed. omega_pr: 29 of 29 tasks passed.

Tested the merge of #600 (e40578ac7b) into develop (55c3de8c15) as 35ca64cc33, against develop (55c3de8c15) rather than the Polaris submodule, because Polaris pins Omega #524, and answer-changing PRs have merged into develop since; this PR only changes the build, so rows should match develop.

Polaris omega_pr suite

  • Baseline workdir: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/baselines/chrysalis_intel_polaris-386dfb1_omega-55c3de8/omega_pr
  • Baseline build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/baselines/chrysalis_intel_polaris-386dfb1_omega-55c3de8/build
  • PR build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr600-e40578a/chrysalis_intel/build
  • PR workdir: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr600-e40578a/chrysalis_intel/omega_pr
  • Polaris: 386dfb17820e6f9f87ee0b9459650e179adc66c5 (1.0.0-961-g386dfb178)
  • Baseline Omega: 55c3de8c156eb0659dc52e5ffa6ae1090a39b16e (bp_maint3.2-3406-g55c3de8c15)
  • PR Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Machine: chrysalis
  • Partition: debug
  • Compiler: intel
  • Build type: Release
  • Log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr600-e40578a/chrysalis_intel/omega_pr/polaris_omega_pr.o1306317
  • Result: All tests passed
Recent commits

Polaris:

386dfb178 Merge Polaris main for Omega PR 600 testing
8fe479ec3 Leave a test branch alone when its manifest pins the same commits
1070a8b8a Count a repeated warning as new only in files the PR changes
f7fdd1bba Make one agent testing over ssh the default for Omega PR testing
b591226fc Read the machines an Omega PR tester reaches over ssh from the config

Baseline Omega:

55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc
ca1387035e Merge pull request #548 from xylar/omega/fix-analysis-units
31393ac215 Merge pull request #591 from brian-oneill/omega/fix-mpi-sumdd-and-tend-test
dd2b61e166 Merge pull request #593 from xylar/omega/fix-kokkos-memory-error

PR Omega:

35ca64cc33 Test merge of E3SM-Project/Omega#600 (e40578ac7b) into develop (55c3de8c15)
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc
ca1387035e Merge pull request #548 from xylar/omega/fix-analysis-units
31393ac215 Merge pull request #591 from brian-oneill/omega/fix-mpi-sumdd-and-tend-test

CTest unit tests:

  • Machine: chrysalis
  • Compiler: intel
  • Build type: Release
  • Build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr600-e40578a/chrysalis_intel/build
  • Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Result: All tests passed
  • Log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr600-e40578a/chrysalis_intel/build/ctests.log

Build warnings

No new warnings (37 in the PR build, 37 in the baseline build).

  • Baseline build log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/baselines/chrysalis_intel_polaris-386dfb1_omega-55c3de8/build/build_omega.log
  • PR build log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr600-e40578a/chrysalis_intel/build/build_omega.log

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@xylar

xylar commented Oct 11, 2026 •

Copy link
Copy Markdown
Author

Testing: aurora, intel, mpich

CTests: 58 of 58 passed. omega_pr: 29 of 29 tasks passed.

Tested the merge of #600 (e40578ac7b) into develop (55c3de8c15) as 35ca64cc33, against develop (55c3de8c15) rather than the Polaris submodule, because Polaris pins Omega #524, and answer-changing PRs have merged into develop since; this PR only changes the build, so rows should match develop.

Polaris omega_pr suite

  • Baseline workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intel_polaris-ecf9fc9_omega-55c3de8/omega_pr
  • Baseline build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intel_polaris-ecf9fc9_omega-55c3de8/build
  • PR build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr600-e40578a/aurora_intel/build
  • PR workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr600-e40578a/aurora_intel/omega_pr
  • Polaris: ecf9fc90e503197fd72d07fc187cc819b60e19fe (1.0.0-977-gecf9fc90e)
  • Baseline Omega: 55c3de8c156eb0659dc52e5ffa6ae1090a39b16e (bp_maint3.2-3406-g55c3de8c15)
  • PR Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Machine: aurora
  • Compiler: intel
  • Build type: Release
  • Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr600-e40578a/aurora_intel/omega_pr/polaris_omega_pr.o8920656
  • Result: All tests passed
Recent commits

Polaris:

ecf9fc90e Merge Polaris main for Omega PR 600 testing
696e100a8 Merge #835 (update-to-mache-5.0.0) at 5294fc48e for testing Omega PRs on Aurora
76900cd8d Describe a row without a baseline as such
38f7b1379 Let setup run a row's PR suite without a baseline
b4ee617e2 Let a baseline run from Polaris before a compiler rename

Baseline Omega:

55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc
ca1387035e Merge pull request #548 from xylar/omega/fix-analysis-units
31393ac215 Merge pull request #591 from brian-oneill/omega/fix-mpi-sumdd-and-tend-test
dd2b61e166 Merge pull request #593 from xylar/omega/fix-kokkos-memory-error

PR Omega:

35ca64cc33 Test merge of E3SM-Project/Omega#600 (e40578ac7b) into develop (55c3de8c15)
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc
ca1387035e Merge pull request #548 from xylar/omega/fix-analysis-units
31393ac215 Merge pull request #591 from brian-oneill/omega/fix-mpi-sumdd-and-tend-test

CTest unit tests:

  • Machine: aurora
  • Compiler: intel
  • Build type: Release
  • Build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr600-e40578a/aurora_intel/build
  • Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Result: All tests passed
  • Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr600-e40578a/aurora_intel/build/ctests.log

Build warnings

No new warnings (14 in the PR build, 14 in the baseline build).

  • Baseline build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intel_polaris-ecf9fc9_omega-55c3de8/build/build_omega.log
  • PR build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr600-e40578a/aurora_intel/build/build_omega.log

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@xylar

xylar commented Oct 11, 2026

Copy link
Copy Markdown
Author

Testing: aurora, intelgpu, mpich

CTests: 58 of 58 passed. omega_pr: 29 of 29 tasks passed.

Tested the merge of #600 (e40578ac7b) into develop (55c3de8c15) as 35ca64cc33, against develop (55c3de8c15) rather than the Polaris submodule, because Polaris pins Omega #524, and answer-changing PRs have merged into develop since; this PR only changes the build, so rows should match develop.

Polaris omega_pr suite

  • Baseline workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intelgpu_polaris-ecf9fc9_omega-55c3de8/omega_pr
  • Baseline build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intelgpu_polaris-ecf9fc9_omega-55c3de8/build
  • PR build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr600-e40578a/aurora_intelgpu/build
  • PR workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr600-e40578a/aurora_intelgpu/omega_pr
  • Polaris: ecf9fc90e503197fd72d07fc187cc819b60e19fe (1.0.0-977-gecf9fc90e)
  • Baseline Omega: 55c3de8c156eb0659dc52e5ffa6ae1090a39b16e (bp_maint3.2-3406-g55c3de8c15)
  • PR Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Machine: aurora
  • Compiler: intelgpu
  • Build type: Release
  • Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr600-e40578a/aurora_intelgpu/omega_pr/polaris_omega_pr.o8920675
  • Result: All tests passed
Recent commits

Polaris:

ecf9fc90e Merge Polaris main for Omega PR 600 testing
696e100a8 Merge #835 (update-to-mache-5.0.0) at 5294fc48e for testing Omega PRs on Aurora
76900cd8d Describe a row without a baseline as such
38f7b1379 Let setup run a row's PR suite without a baseline
b4ee617e2 Let a baseline run from Polaris before a compiler rename

Baseline Omega:

55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc
ca1387035e Merge pull request #548 from xylar/omega/fix-analysis-units
31393ac215 Merge pull request #591 from brian-oneill/omega/fix-mpi-sumdd-and-tend-test
dd2b61e166 Merge pull request #593 from xylar/omega/fix-kokkos-memory-error

PR Omega:

35ca64cc33 Test merge of E3SM-Project/Omega#600 (e40578ac7b) into develop (55c3de8c15)
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc
ca1387035e Merge pull request #548 from xylar/omega/fix-analysis-units
31393ac215 Merge pull request #591 from brian-oneill/omega/fix-mpi-sumdd-and-tend-test

CTest unit tests:

  • Machine: aurora
  • Compiler: intelgpu
  • Build type: Release
  • Build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr600-e40578a/aurora_intelgpu/build
  • Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Result: All tests passed
  • Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr600-e40578a/aurora_intelgpu/build/ctests.log

Build warnings

2 new warnings, 0 in files this PR changes (243 in the PR build, 235 in the baseline build). Those in files the PR changes are marked *.

New warnings
  externals/scorpio/src/clib/core/pio_darray_int.cpp: variable length arrays in C++ are a Clang extension [-Wvla-cxx-extension][ 41%] Building CXX object external/scorpio/src/clib/core/CMakeFiles/spio_core.dir/pioc_sc.cpp.o
  externals/scorpio/src/clib/core/pioc_support.cpp: [-Wformat-security] format string is not a string literal (potentially insecure) (0 -> 2)
  • Baseline build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intelgpu_polaris-ecf9fc9_omega-55c3de8/build/build_omega.log
  • PR build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr600-e40578a/aurora_intelgpu/build/build_omega.log

Notes

The 2 new warnings are not new. Both build logs have the same SCORPIO -Wformat-security and -Wvla-cxx-extension warnings, the same number of times, but in the baseline log they are interleaved with other compile jobs' output, so they were not matched. This PR does not touch SCORPIO.


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

xylar added a commit to xylar/polaris that referenced this pull request Oct 11, 2026
Since E3SM-Project/Omega#566, Omega calls Aurora's and Chrysalis's
compilers intel and intelgpu, and E3SM-Project/Omega#600 updates the
template to match, adding a row for Aurora's GPU build.  Polaris still
calls Aurora's compilers oneapi-ifx and oneapi-ifxgpu, so those rows
stay mapped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xylar added a commit to xylar/polaris that referenced this pull request Oct 11, 2026
Since E3SM-Project/Omega#566, Omega calls Aurora's and Chrysalis's
compilers intel and intelgpu, and E3SM-Project/Omega#600 updates the
template to match, adding a row for Aurora's GPU build.  With E3SM-Project#835,
Polaris uses the same names, so no row is mapped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xylar added a commit to xylar/polaris that referenced this pull request Oct 11, 2026
Since E3SM-Project/Omega#566, Omega calls Aurora's and Chrysalis's
compilers intel and intelgpu, and E3SM-Project/Omega#600 updates the
template to match, adding a row for Aurora's GPU build.  Polaris still
calls Aurora's compilers oneapi-ifx and oneapi-ifxgpu, so those rows
stay mapped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar

xylar commented Oct 11, 2026

Copy link
Copy Markdown
Author

Testing: pm-cpu, gnu, mpich

CTests: 58 of 58 passed. omega_pr: 29 of 29 tasks passed.

Tested the merge of #600 (e40578ac7b) into develop (55c3de8c15) as 35ca64cc33, against develop (55c3de8c15) rather than the Polaris submodule, because Polaris pins Omega #524, and answer-changing PRs have merged into develop since; this PR only changes the build, so rows should match develop.

Polaris omega_pr suite

  • Baseline workdir: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-cpu_gnu_polaris-069f378_omega-55c3de8/omega_pr
  • Baseline build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-cpu_gnu_polaris-069f378_omega-55c3de8/build
  • PR build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr600-e40578a/pm-cpu_gnu/build
  • PR workdir: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr600-e40578a/pm-cpu_gnu/omega_pr
  • Polaris: 069f378c3680000c89eb5976574ed6fd371d7142 (1.0.0-965-g069f378c3)
  • Baseline Omega: 55c3de8c156eb0659dc52e5ffa6ae1090a39b16e (bp_maint3.2-3406-g55c3de8c15)
  • PR Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Machine: pm-cpu
  • Compiler: gnu
  • Build type: Release
  • Log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr600-e40578a/pm-cpu_gnu/omega_pr/polaris_omega_pr.o59673551
  • Result: All tests passed
Recent commits

Polaris:

069f378c3 Merge Polaris main for Omega PR 600 testing
5957382d6 Describe a row without a baseline as such
c8af5d4e5 Let setup run a row's PR suite without a baseline
6fbc42249 Let a baseline run from Polaris before a compiler rename
3f5f8e86a Let init test a merge the requester made

Baseline Omega:

55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc
ca1387035e Merge pull request #548 from xylar/omega/fix-analysis-units
31393ac215 Merge pull request #591 from brian-oneill/omega/fix-mpi-sumdd-and-tend-test
dd2b61e166 Merge pull request #593 from xylar/omega/fix-kokkos-memory-error

PR Omega:

35ca64cc33 Test merge of E3SM-Project/Omega#600 (e40578ac7b) into develop (55c3de8c15)
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc
ca1387035e Merge pull request #548 from xylar/omega/fix-analysis-units
31393ac215 Merge pull request #591 from brian-oneill/omega/fix-mpi-sumdd-and-tend-test

CTest unit tests:

  • Machine: pm-cpu
  • Compiler: gnu
  • Build type: Release
  • Build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr600-e40578a/pm-cpu_gnu/build
  • Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Result: All tests passed
  • Log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr600-e40578a/pm-cpu_gnu/build/ctests.log

Build warnings

No new warnings (4 in the PR build, 4 in the baseline build).

  • Baseline build log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-cpu_gnu_polaris-069f378_omega-55c3de8/build/build_omega.log
  • PR build log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr600-e40578a/pm-cpu_gnu/build/build_omega.log

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@xylar

xylar commented Oct 11, 2026

Copy link
Copy Markdown
Author

Testing: pm-gpu, gnugpu, mpich

CTests: 58 of 58 passed. omega_pr: 29 of 29 tasks passed.

Tested the merge of #600 (e40578ac7b) into develop (55c3de8c15) as 35ca64cc33, without a baseline, because Omega develop cannot run on pm-gpu without this PR (#599).

Polaris omega_pr suite

  • PR build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr600-e40578a/pm-gpu_gnugpu/build
  • PR workdir: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr600-e40578a/pm-gpu_gnugpu/omega_pr
  • Polaris: 069f378c3680000c89eb5976574ed6fd371d7142 (1.0.0-965-g069f378c3)
  • PR Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Machine: pm-gpu
  • Compiler: gnugpu
  • Build type: Release
  • Log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr600-e40578a/pm-gpu_gnugpu/omega_pr/polaris_omega_pr.o59673749
  • Result: All tests passed
Recent commits

Polaris:

069f378c3 Merge Polaris main for Omega PR 600 testing
5957382d6 Describe a row without a baseline as such
c8af5d4e5 Let setup run a row's PR suite without a baseline
6fbc42249 Let a baseline run from Polaris before a compiler rename
3f5f8e86a Let init test a merge the requester made

PR Omega:

35ca64cc33 Test merge of E3SM-Project/Omega#600 (e40578ac7b) into develop (55c3de8c15)
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc
ca1387035e Merge pull request #548 from xylar/omega/fix-analysis-units
31393ac215 Merge pull request #591 from brian-oneill/omega/fix-mpi-sumdd-and-tend-test

CTest unit tests:

  • Machine: pm-gpu
  • Compiler: gnugpu
  • Build type: Release
  • Build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr600-e40578a/pm-gpu_gnugpu/build
  • Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Result: All tests passed
  • Log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr600-e40578a/pm-gpu_gnugpu/build/ctests.log

Build warnings

Not compared: this row ran without a baseline.


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@xylar

xylar commented Oct 11, 2026 •

Copy link
Copy Markdown
Author

Testing: frontier, craygnu, mpich

CTests: 58 of 58 passed. omega_pr: 28 of 29 tasks passed, 1 with baseline differences.

Tested the merge of #600 (e40578ac7b) into develop (55c3de8c15) as 35ca64cc33, against develop (55c3de8c15) rather than the Polaris submodule, because Polaris pins Omega #524, and answer-changing PRs have merged into develop since; this PR only changes the build, so rows should match develop.

Polaris omega_pr suite

  • Baseline workdir: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu_polaris-fffaaa6_omega-55c3de8/omega_pr
  • Baseline build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu_polaris-fffaaa6_omega-55c3de8/build
  • PR build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr600-e40578a/frontier_craygnu/build
  • PR workdir: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr600-e40578a/frontier_craygnu/omega_pr
  • Polaris: fffaaa6db064351fbb71ed8ddc775213d6cc331a (1.0.0-962-gfffaaa6db)
  • Baseline Omega: 55c3de8c156eb0659dc52e5ffa6ae1090a39b16e (bp_maint3.2-3406-g55c3de8c15)
  • PR Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Machine: frontier
  • Partition: batch
  • Compiler: craygnu
  • Build type: Release
  • Log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr600-e40578a/frontier_craygnu/omega_pr/polaris_omega_pr.o5651683
  • Result:
    • Diffs (1 of 29):
    - ocean/planar/manufactured_solution/convergence_both/default
      - no norms (missing file or variable)
Recent commits

Polaris:

fffaaa6db Follow Omega's intel compiler names in the PR template rows
d6af3fd42 Let init test a merge the requester made
6e84baff0 Leave a test branch alone when its manifest pins the same commits
c93412d65 Count a repeated warning as new only in files the PR changes
8cb406b55 Make one agent testing over ssh the default for Omega PR testing

Baseline Omega:

55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc
ca1387035e Merge pull request #548 from xylar/omega/fix-analysis-units
31393ac215 Merge pull request #591 from brian-oneill/omega/fix-mpi-sumdd-and-tend-test
dd2b61e166 Merge pull request #593 from xylar/omega/fix-kokkos-memory-error

PR Omega:

35ca64cc33 Test merge of E3SM-Project/Omega#600 (e40578ac7b) into develop (55c3de8c15)
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc
ca1387035e Merge pull request #548 from xylar/omega/fix-analysis-units
31393ac215 Merge pull request #591 from brian-oneill/omega/fix-mpi-sumdd-and-tend-test

CTest unit tests:

  • Machine: frontier
  • Compiler: craygnu
  • Build type: Release
  • Build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr600-e40578a/frontier_craygnu/build
  • Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Result: All tests passed
  • Log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr600-e40578a/frontier_craygnu/build/ctests.log

Build warnings

No new warnings (3 in the PR build, 3 in the baseline build).

  • Baseline build log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu_polaris-fffaaa6_omega-55c3de8/build/build_omega.log
  • PR build log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr600-e40578a/frontier_craygnu/build/build_omega.log

Notes

The convergence_both/default difference is not caused by this PR. In the baseline run, the forward_25km_19s step never ran Omega: srun failed with Unable to confirm allocation for job 5651682: Socket timed out on send/recv operation, a Slurm controller timeout on Frontier. With no baseline output, the comparison reports "no norms". In the PR run, the same step ran and passed its property checks.

We suspect the Slurm controller was briefly overloaded. Two of the three timeouts on Frontier came 11 seconds apart (11:03:15 and 11:03:26 EDT), in separate jobs on different nodes. That was just after a 2,048-node job hit its time limit (11:01:55) and as a 5,199-node job started (11:03:45). The third (11:20, PR vmix_unstable on craygnu-mphipcc) matches no large job starting or ending. The srun command lines are identical to earlier runs, and no other Frontier suite run since October 1 hit this error, so we do not think a change on develop is responsible. Rerunning the affected tasks would confirm this.


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@xylar

xylar commented Oct 11, 2026 •

Copy link
Copy Markdown
Author

Testing: frontier, craygnu-mphipcc, mpich

CTests: 58 of 58 passed. omega_pr: 28 of 29 tasks passed.

Tested the merge of #600 (e40578ac7b) into develop (55c3de8c15) as 35ca64cc33, against develop (55c3de8c15) rather than the Polaris submodule, because Polaris pins Omega #524, and answer-changing PRs have merged into develop since; this PR only changes the build, so rows should match develop.

Polaris omega_pr suite

  • Baseline workdir: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-fffaaa6_omega-55c3de8/omega_pr
  • Baseline build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-fffaaa6_omega-55c3de8/build
  • PR build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr600-e40578a/frontier_craygnu-mphipcc/build
  • PR workdir: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr600-e40578a/frontier_craygnu-mphipcc/omega_pr
  • Polaris: fffaaa6db064351fbb71ed8ddc775213d6cc331a (1.0.0-962-gfffaaa6db)
  • Baseline Omega: 55c3de8c156eb0659dc52e5ffa6ae1090a39b16e (bp_maint3.2-3406-g55c3de8c15)
  • PR Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Machine: frontier
  • Partition: batch
  • Compiler: craygnu-mphipcc
  • Build type: Release
  • Log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr600-e40578a/frontier_craygnu-mphipcc/omega_pr/polaris_omega_pr.o5651686
  • Result:
    • Failures (1 of 29):
      • ocean/column/vmix_unstable
Recent commits

Polaris:

fffaaa6db Follow Omega's intel compiler names in the PR template rows
d6af3fd42 Let init test a merge the requester made
6e84baff0 Leave a test branch alone when its manifest pins the same commits
c93412d65 Count a repeated warning as new only in files the PR changes
8cb406b55 Make one agent testing over ssh the default for Omega PR testing

Baseline Omega:

55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc
ca1387035e Merge pull request #548 from xylar/omega/fix-analysis-units
31393ac215 Merge pull request #591 from brian-oneill/omega/fix-mpi-sumdd-and-tend-test
dd2b61e166 Merge pull request #593 from xylar/omega/fix-kokkos-memory-error

PR Omega:

35ca64cc33 Test merge of E3SM-Project/Omega#600 (e40578ac7b) into develop (55c3de8c15)
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc
ca1387035e Merge pull request #548 from xylar/omega/fix-analysis-units
31393ac215 Merge pull request #591 from brian-oneill/omega/fix-mpi-sumdd-and-tend-test

CTest unit tests:

  • Machine: frontier
  • Compiler: craygnu-mphipcc
  • Build type: Release
  • Build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr600-e40578a/frontier_craygnu-mphipcc/build
  • Omega: 35ca64cc33c47ec5c36fd103d9651bcdf379f061 (bp_maint3.2-3410-g35ca64cc33)
  • Result: All tests passed
  • Log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr600-e40578a/frontier_craygnu-mphipcc/build/ctests.log

Build warnings

No new warnings (6 in the PR build, 6 in the baseline build).

  • Baseline build log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-fffaaa6_omega-55c3de8/build/build_omega.log
  • PR build log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr600-e40578a/frontier_craygnu-mphipcc/build/build_omega.log

Notes

The ocean/column/vmix_unstable failure is not caused by this PR. Both the baseline and the PR runs of this task hit the same Slurm controller timeout on Frontier, in different steps: srun: error: Unable to confirm allocation for job ...: Socket timed out on send/recv operation. Omega never started in the step that errored: forward_no_hadv_restoring in the baseline, forward_no_vadv_no_hadv in the PR. The PR's forward_no_hadv_restoring ran, but its baseline comparison failed because the baseline step has no output. Its salt-conservation property check fails with the same error (5.362e-05) as in both the baseline and PR runs on frontier/craygnu, where salt restoring is expected to change total salt.

We suspect the Slurm controller was briefly overloaded. Two of the three timeouts on Frontier came 11 seconds apart (11:03:15 and 11:03:26 EDT), in separate jobs on different nodes. That was just after a 2,048-node job hit its time limit (11:01:55) and as a 5,199-node job started (11:03:45). The third (11:20, PR vmix_unstable on craygnu-mphipcc) matches no large job starting or ending. The srun command lines are identical to earlier runs, and no other Frontier suite run since October 1 hit this error, so we do not think a change on develop is responsible. Rerunning the affected tasks would confirm this.


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@xylar
xylar requested a review from grnydawn October 11, 2026 18:45
@xylar

xylar commented Oct 11, 2026

Copy link
Copy Markdown
Author

@grnydawn, could you take a look at this? I think it needs to be fixed for things to work properly on Perlmutter following #566.

xylar added a commit to xylar/E3SM that referenced this pull request Oct 12, 2026
xylar added a commit to xylar/E3SM that referenced this pull request Oct 12, 2026
xylar added a commit to xylar/polaris that referenced this pull request Oct 12, 2026
Since E3SM-Project/Omega#566, Omega calls Aurora's and Chrysalis's
compilers intel and intelgpu, and E3SM-Project/Omega#600 updates the
template to match, adding a row for Aurora's GPU build.  Polaris still
calls Aurora's compilers oneapi-ifx and oneapi-ifxgpu, so those rows
stay mapped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xylar added a commit to xylar/polaris that referenced this pull request Oct 14, 2026
Since E3SM-Project/Omega#566, Omega calls Aurora's and Chrysalis's
compilers intel and intelgpu, and E3SM-Project/Omega#600 updates the
template to match, adding a row for Aurora's GPU build.  Polaris still
calls Aurora's compilers oneapi-ifx and oneapi-ifxgpu, so those rows
stay mapped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar xylar mentioned this pull request Oct 14, 2026
38 of 51 tasks

@grnydawn grnydawn left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR fixes a bug where the MPILIB CMake variable was not being set, preventing the proper compiler flags from being loaded in the CMake files. All CTests are passed on pm-gpu. Approved.

@sbrus89

sbrus89 commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the testing @xylar and for the review @grnydawn!

@sbrus89
sbrus89 merged commit cdb935b into E3SM-Project:develop Oct 14, 2026
1 check passed
xylar added a commit to xylar/polaris that referenced this pull request Oct 14, 2026
Omega develop is now cdb935bb5a, which merges E3SM-Project/Omega#600 so
that standalone Omega builds set MPILIB from the case, which pm-gpu needs.
It has the same tree as the Omega E3SM-Project#835's testing built: b90c3dae8 with
E3SM-Project#600 merged in.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xylar added a commit to E3SM-Project/polaris that referenced this pull request Oct 22, 2026
Since E3SM-Project/Omega#566, Omega calls Aurora's and Chrysalis's
compilers intel and intelgpu, and E3SM-Project/Omega#600 updates the
template to match, adding a row for Aurora's GPU build.  Polaris still
calls Aurora's compilers oneapi-ifx and oneapi-ifxgpu, so those rows
stay mapped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CMake CMake-related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standalone builds never set MPILIB, so pm-gpu links without GTL

3 participants