Skip to content

Minor refactoring of KPP - #621

Merged
sbrus89 merged 3 commits into
E3SM-Project:developfrom
mwarusz:omega/kpp-fixes
Oct 27, 2026
Merged

sbrus89 merged 3 commits into
E3SM-Project:developfrom
mwarusz:omega/kpp-fixes

Conversation

@mwarusz

@mwarusz mwarusz commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

This PR does two things:

  • Adds _Real suffix to all floating-point literals in KPPConstants.h that were missing it.
  • Refactors how convective mixing works with KPP. Before this PR, VertMix.h defined the ConvectiveMix class and
    VertMix had an instance of it, but this member variable was not used. Instead, convective mixing was implemented inline in computeVertMix. This PR refactors the code so that ConvectiveMix is KPP-aware and can be used as intended. In the process of refactoring this code I also removed the need to allocate KPPBoundaryLayerIndex filled with -1 when KPP is disabled.

Checklist

  • Linting

  • Building

    • CMake build does not produce any new warnings from changes in this PR
  • Testing

    aurora, intel, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    aurora, intelgpu, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    chrysalis, intel, openmpi

    • CTests Pass
    • Polaris omega_pr Pass
    • CIME Tests:
      • PEM_D.T62_oQU240.GOMEGA-IAF
      • ERS_Ld3.ne4pg2_oQU240.WCYCL1850NS-OMEGA

    frontier, craygnu, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    frontier, craygnu-mphipcc, mpich

    • CTests Pass
    • Polaris omega_pr Pass
    • CIME Tests:
    • ERS_Ld5.T62_oQU240.GOMEGA-IAF
    • ERS_Ld3.ne4pg2_oQU240.WCYCL1850NS-OMEGA

    pm-cpu, gnu, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    pm-gpu, gnugpu, mpich

    • CTests Pass
    • Polaris omega_pr Pass
  • Provide relevant details in a comment to the PR titled Testing with the following:

    • Which machines CTest unit tests
      have been run on and indicate that are all passing.
    • The Polaris omega_pr test suite
      has passed, using the Polaris e3sm_submodules/Omega baseline
    • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
    • Indicate "All tests passed" or document failing tests
    • Document testing used to verify the changes including any tests that are added/modified/impacted.

@mwarusz

mwarusz commented Oct 20, 2026

Copy link
Copy Markdown
Member Author

Testing: CTests

All tests passed on:

  • frontier (craygnu)
  • frontier (craygnu-mphipcc)
  • aurora (intel)

One test (SUBMESOEDDIES_TEST) failed on aurora with intelgpu, which is an unrelated issue #618 .

Testing: omega_pr suite

frontier (craygnu)

  • Baseline workdir: /lustre/orion/cli115/scratch/mwaruszewski/omega-pr-testing/kpp-fixes/../math/baseline-craygnu
  • Baseline build: /lustre/orion/cli115/scratch/mwaruszewski/omega-pr-testing/math/baseline-craygnu/ocean_build
  • PR build: /lustre/orion/cli115/scratch/mwaruszewski/omega-pr-testing/kpp-fixes/build-craygnu
  • PR workdir: /lustre/orion/cli115/scratch/mwaruszewski/omega-pr-testing/kpp-fixes/pr-craygnu
  • Baseline Omega: cdb935bb5a8d995b0878616e001a2701a2627519 (Omega-v0.1.0-alpha.1-6142-gcdb935bb5a)
  • PR Omega: f5288d1991079a0f7d65763f14b87a3f00b33795 (omega-v0.0.0-alpha0.1-26103-gf5288d1991) (at setup, not build)
  • Machine: frontier
  • Partition: batch
  • Compiler: craygnu
  • Build type: Release
  • Log: /lustre/orion/cli115/scratch/mwaruszewski/omega-pr-testing/kpp-fixes/pr-craygnu/polaris_omega_pr.o5712385
  • Result: All tests passed
Recent commits

Baseline Omega:

cdb935bb5a Merge pull request #600 from xylar/omega/fix-pm-mpi-flags
b90c3dae8d Merge pull request #601 from andrewdnolan/omega/cxx20-for-coupled
9641c6cf9e Merge pull request #546 from xylar/omega/fix-cf-units
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc

PR Omega:

f5288d1991 Fix precision of literals in KPPConstants.h
c98a9f11fc Refactor convective mixing with KPP
260d05971a Work
9656466bb6 Merge PR #570 into develop, resolving Tendencies.cpp initializer order
d1d3ea6506 Merge pull request #466 from alicebarthel/omega/add-frazil-basic-rebased

frontier (craygnu-mphipcc)

  • Baseline workdir: /lustre/orion/cli115/scratch/mwaruszewski/omega-pr-testing/kpp-fixes/../math/baseline-craygnu-mphipcc/
  • Baseline build: /lustre/orion/cli115/scratch/mwaruszewski/omega-pr-testing/math/baseline-craygnu-mphipcc/ocean_build
  • PR build: /lustre/orion/cli115/scratch/mwaruszewski/omega-pr-testing/kpp-fixes/build-craygnu-mphipcc
  • PR workdir: /lustre/orion/cli115/scratch/mwaruszewski/omega-pr-testing/kpp-fixes/pr-craygnu-mphipcc
  • Baseline Omega: cdb935bb5a8d995b0878616e001a2701a2627519 (Omega-v0.1.0-alpha.1-6142-gcdb935bb5a)
  • PR Omega: f5288d1991079a0f7d65763f14b87a3f00b33795 (omega-v0.0.0-alpha0.1-26103-gf5288d1991) (at setup, not build)
  • Machine: frontier
  • Partition: batch
  • Compiler: craygnu-mphipcc
  • Build type: Release
  • Log: /lustre/orion/cli115/scratch/mwaruszewski/omega-pr-testing/kpp-fixes/pr-craygnu-mphipcc/polaris_omega_pr.o5719553
  • Result: All tests passed
Recent commits

Baseline Omega:

cdb935bb5a Merge pull request #600 from xylar/omega/fix-pm-mpi-flags
b90c3dae8d Merge pull request #601 from andrewdnolan/omega/cxx20-for-coupled
9641c6cf9e Merge pull request #546 from xylar/omega/fix-cf-units
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc

PR Omega:

f5288d1991 Fix precision of literals in KPPConstants.h
c98a9f11fc Refactor convective mixing with KPP
260d05971a Work
9656466bb6 Merge PR #570 into develop, resolving Tendencies.cpp initializer order
d1d3ea6506 Merge pull request #466 from alicebarthel/omega/add-frazil-basic-rebased

aurora (intel)

  • Baseline workdir: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/kpp-fixes/../math/baseline-intel
  • Baseline build: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/math/baseline-intel/ocean_build
  • PR build: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/kpp-fixes/build-intel
  • PR workdir: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/kpp-fixes/pr-intel
  • Baseline Omega: cdb935bb5a8d995b0878616e001a2701a2627519 (Omega-v0.1.0-alpha.1-6142-gcdb935bb5a)
  • PR Omega: bd5bf880ba6b60f7862608f8b749aa2c75e88049 (acme-split-2019-02-28-51458-gbd5bf880ba) (at setup, not build)
  • Machine: aurora
  • Compiler: intel
  • Build type: Release
  • Log: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/kpp-fixes/pr-intel/polaris_omega_pr.o8948331
  • Result: All tests passed
Recent commits

Baseline Omega:

cdb935bb5a Merge pull request #600 from xylar/omega/fix-pm-mpi-flags
b90c3dae8d Merge pull request #601 from andrewdnolan/omega/cxx20-for-coupled
9641c6cf9e Merge pull request #546 from xylar/omega/fix-cf-units
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc

PR Omega:

bd5bf880ba Fix precision of literals in KPPConstants.h
4d9b089fb2 Refactor convective mixing with KPP
044014eb4d Work
135755f734 Merge pull request #438 from mwarusz/omega/submesoscale-eddies
a341e65245 Merge pull request #562 from andrewdnolan/omega/guard-division-by-zero

aurora (intelgpu)

  • Baseline workdir: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/kpp-fixes/../math/baseline-intelgpu/
  • Baseline build: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/math/baseline-intelgpu/ocean_build
  • PR build: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/kpp-fixes/build-intelgpu
  • PR workdir: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/kpp-fixes/pr-intelgpu
  • Baseline Omega: cdb935bb5a8d995b0878616e001a2701a2627519 (Omega-v0.1.0-alpha.1-6142-gcdb935bb5a)
  • PR Omega: bd5bf880ba6b60f7862608f8b749aa2c75e88049 (acme-split-2019-02-28-51458-gbd5bf880ba) (at setup, not build)
  • Machine: aurora
  • Compiler: intelgpu
  • Build type: Release
  • Log: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/kpp-fixes/pr-intelgpu/polaris_omega_pr.o8948342
  • Result: All tests passed
Recent commits

Baseline Omega:

cdb935bb5a Merge pull request #600 from xylar/omega/fix-pm-mpi-flags
b90c3dae8d Merge pull request #601 from andrewdnolan/omega/cxx20-for-coupled
9641c6cf9e Merge pull request #546 from xylar/omega/fix-cf-units
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc

PR Omega:

bd5bf880ba Fix precision of literals in KPPConstants.h
4d9b089fb2 Refactor convective mixing with KPP
044014eb4d Work
135755f734 Merge pull request #438 from mwarusz/omega/submesoscale-eddies
a341e65245 Merge pull request #562 from andrewdnolan/omega/guard-division-by-zero

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Only a minor documentation nit remains, with no blocking issues.

1 open issue
What changed in this PR

This PR refactors KPP-aware convective mixing and standardizes floating-point literals.

Changes:

  • Adds _Real suffixes to KPP constants.
  • Makes convective mixing KPP-aware.
  • Removes unnecessary disabled-KPP index allocation.
  • Updates the related test.
File Summary
components/​omega/​test/​ocn/​VertMixTest.cpp Updates convective-mixing test arguments.
components/​omega/​src/​ocn/​VertMix.h Adds KPP-aware mixing logic; includes a minor grammatical nit.
components/​omega/​src/​ocn/​VertMix.cpp Integrates the refactored convective mixer.
components/​omega/​src/​ocn/​KPPConstants.h Standardizes floating-point literal types.

Review effort: Lite

Comment thread components/omega/src/ocn/VertMix.h Outdated
@mwarusz
mwarusz requested a review from vanroekel October 20, 2026 21:40
@mwarusz

mwarusz commented Oct 20, 2026

Copy link
Copy Markdown
Member Author

@vanroekel Would you mind having a look ?

@vanroekel

Copy link
Copy Markdown
Collaborator

Sure, I can review

@vanroekel vanroekel left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving by visual inspection. I also ran the kpp suite and looked through the results and they appear consistent with develop

@sbrus89 sbrus89 added Omega non-BFB PR makes roundoff changes to answers. labels Oct 26, 2026
@sbrus89

sbrus89 commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

Testing on chrysalis

Tested the merge of #621 (ef102d3dce) into develop (f368d66830) as 5f6c2c5369, against develop (f368d66830) rather than the Polaris submodule, because the Omega submodule in Polaris main has not been updated since Omega PR #616 merged into develop.

Row CTests omega_pr New build warnings
chrysalis, intel, openmpi 61 of 61 passed 34 of 34 passed 0
chrysalis, intel, openmpi

Polaris omega_pr suite

  • Baseline workdir: /lcrc/group/e3sm/sbrus/omega_pr_test/baselines/chrysalis_intel_polaris-2a5289a_omega-f368d66/omega_pr
  • Baseline build: /lcrc/group/e3sm/sbrus/omega_pr_test/baselines/chrysalis_intel_polaris-2a5289a_omega-f368d66/build
  • PR build: /lcrc/group/e3sm/sbrus/omega_pr_test/pr621-ef102d3/chrysalis_intel/build
  • PR workdir: /lcrc/group/e3sm/sbrus/omega_pr_test/pr621-ef102d3/chrysalis_intel/omega_pr
  • Polaris: 2a5289ab41d8e9bd9be8150b1aace06163c19847 (1.1.0-458-g2a5289ab41-dirty)
  • Baseline Omega: f368d668307f86cce95b0d98835bc02310b83f1e (Omega-v0.1.0-alpha.1-6289-gf368d66830)
  • PR Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Machine: chrysalis
  • Partition: debug
  • Compiler: intel
  • Build type: Release
  • Log: /lcrc/group/e3sm/sbrus/omega_pr_test/pr621-ef102d3/chrysalis_intel/omega_pr/polaris_omega_pr.o1316085
  • Result: All tests passed
Recent commits

Polaris:

2a5289ab41 Merge pull request #906 from xylar/add-pr-test-cleanup
15850cedb5 Merge pull request #904 from xylar/fix-omega-pr-handoff-early
335cc496de Merge pull request #643 from xylar/add-realistic-global-ocean-general-forward
f17d3695a3 Merge pull request #903 from xylar/fix-global-stats-anomaly-label
c3fc893b7a Merge pull request #730 from xylar/add-omega-analysis-climatology-maps

Baseline Omega:

f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp

PR Omega:

5f6c2c5369 Test merge of E3SM-Project/Omega#621 (ef102d3dce) into develop (f368d66830)
f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect

CTest unit tests:

  • Machine: chrysalis
  • Compiler: intel
  • Build type: Release
  • Build: /lcrc/group/e3sm/sbrus/omega_pr_test/pr621-ef102d3/chrysalis_intel/build
  • Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Result: All tests passed
  • Log: /lcrc/group/e3sm/sbrus/omega_pr_test/pr621-ef102d3/chrysalis_intel/build/ctests.log

Build warnings

No new warnings (37 in the PR build, 37 in the baseline build).

  • Baseline build log: /lcrc/group/e3sm/sbrus/omega_pr_test/baselines/chrysalis_intel_polaris-2a5289a_omega-f368d66/build/build_omega.log
  • PR build log: /lcrc/group/e3sm/sbrus/omega_pr_test/pr621-ef102d3/chrysalis_intel/build/build_omega.log

@sbrus89 sbrus89 removed the non-BFB PR makes roundoff changes to answers. label Oct 26, 2026
@sbrus89

sbrus89 commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Testing on frontier

Tested the merge of #621 (ef102d3dce) into develop (f368d66830) as 5f6c2c5369, against develop (f368d66830) rather than the Polaris submodule, because the Omega submodule in Polaris main has not been updated since Omega PR #616 merged into develop.

Row CTests omega_pr New build warnings
frontier, craygnu, mpich 61 of 61 passed 34 of 34 passed 0
frontier, craygnu-mphipcc, mpich 61 of 61 passed 34 of 34 passed 0
frontier, craygnu, mpich

Polaris omega_pr suite

  • Baseline workdir: /ccs/home/brus/run/omega_pr_test/baselines/frontier_craygnu_polaris-2a5289a_omega-f368d66/omega_pr
  • Baseline build: /ccs/home/brus/run/omega_pr_test/baselines/frontier_craygnu_polaris-2a5289a_omega-f368d66/build
  • PR build: /ccs/home/brus/run/omega_pr_test/pr621-ef102d3/frontier_craygnu/build
  • PR workdir: /ccs/home/brus/run/omega_pr_test/pr621-ef102d3/frontier_craygnu/omega_pr
  • Polaris: 2a5289ab41d8e9bd9be8150b1aace06163c19847 (1.1.0-458-g2a5289ab41)
  • Baseline Omega: f368d668307f86cce95b0d98835bc02310b83f1e (Omega-v0.1.0-alpha.1-6289-gf368d66830)
  • PR Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Machine: frontier
  • Partition: batch
  • Compiler: craygnu
  • Build type: Release
  • Log: /ccs/home/brus/run/omega_pr_test/pr621-ef102d3/frontier_craygnu/omega_pr/polaris_omega_pr.o5769269
  • Result: All tests passed
Recent commits

Polaris:

2a5289ab41 Merge pull request #906 from xylar/add-pr-test-cleanup
15850cedb5 Merge pull request #904 from xylar/fix-omega-pr-handoff-early
335cc496de Merge pull request #643 from xylar/add-realistic-global-ocean-general-forward
f17d3695a3 Merge pull request #903 from xylar/fix-global-stats-anomaly-label
c3fc893b7a Merge pull request #730 from xylar/add-omega-analysis-climatology-maps

Baseline Omega:

f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp

PR Omega:

5f6c2c5369 Test merge of E3SM-Project/Omega#621 (ef102d3dce) into develop (f368d66830)
f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect

CTest unit tests:

  • Machine: frontier
  • Compiler: craygnu
  • Build type: Release
  • Build: /ccs/home/brus/run/omega_pr_test/pr621-ef102d3/frontier_craygnu/build
  • Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Result: All tests passed
  • Log: /ccs/home/brus/run/omega_pr_test/pr621-ef102d3/frontier_craygnu/build/ctests.log

Build warnings

No new warnings (3 in the PR build, 3 in the baseline build).

  • Baseline build log: /ccs/home/brus/run/omega_pr_test/baselines/frontier_craygnu_polaris-2a5289a_omega-f368d66/build/build_omega.log
  • PR build log: /ccs/home/brus/run/omega_pr_test/pr621-ef102d3/frontier_craygnu/build/build_omega.log
frontier, craygnu-mphipcc, mpich

Polaris omega_pr suite

  • Baseline workdir: /ccs/home/brus/run/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-2a5289a_omega-f368d66/omega_pr
  • Baseline build: /ccs/home/brus/run/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-2a5289a_omega-f368d66/build
  • PR build: /ccs/home/brus/run/omega_pr_test/pr621-ef102d3/frontier_craygnu-mphipcc/build
  • PR workdir: /ccs/home/brus/run/omega_pr_test/pr621-ef102d3/frontier_craygnu-mphipcc/omega_pr
  • Polaris: 2a5289ab41d8e9bd9be8150b1aace06163c19847 (1.1.0-458-g2a5289ab41)
  • Baseline Omega: f368d668307f86cce95b0d98835bc02310b83f1e (Omega-v0.1.0-alpha.1-6289-gf368d66830)
  • PR Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Machine: frontier
  • Partition: batch
  • Compiler: craygnu-mphipcc
  • Build type: Release
  • Log: /ccs/home/brus/run/omega_pr_test/pr621-ef102d3/frontier_craygnu-mphipcc/omega_pr/polaris_omega_pr.o5769358
  • Result: All tests passed
Recent commits

Polaris:

2a5289ab41 Merge pull request #906 from xylar/add-pr-test-cleanup
15850cedb5 Merge pull request #904 from xylar/fix-omega-pr-handoff-early
335cc496de Merge pull request #643 from xylar/add-realistic-global-ocean-general-forward
f17d3695a3 Merge pull request #903 from xylar/fix-global-stats-anomaly-label
c3fc893b7a Merge pull request #730 from xylar/add-omega-analysis-climatology-maps

Baseline Omega:

f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp

PR Omega:

5f6c2c5369 Test merge of E3SM-Project/Omega#621 (ef102d3dce) into develop (f368d66830)
f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect

CTest unit tests:

  • Machine: frontier
  • Compiler: craygnu-mphipcc
  • Build type: Release
  • Build: /ccs/home/brus/run/omega_pr_test/pr621-ef102d3/frontier_craygnu-mphipcc/build
  • Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Result: All tests passed
  • Log: /ccs/home/brus/run/omega_pr_test/pr621-ef102d3/frontier_craygnu-mphipcc/build/ctests.log

Build warnings

No new warnings (6 in the PR build, 6 in the baseline build).

  • Baseline build log: /ccs/home/brus/run/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-2a5289a_omega-f368d66/build/build_omega.log
  • PR build log: /ccs/home/brus/run/omega_pr_test/pr621-ef102d3/frontier_craygnu-mphipcc/build/build_omega.log

@sbrus89

sbrus89 commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Testing on pm-cpu and pm-gpu

Tested the merge of #621 (ef102d3dce) into develop (f368d66830) as 5f6c2c5369, against develop (f368d66830) rather than the Polaris submodule, because the Omega submodule in Polaris main has not been updated since Omega PR #616 merged into develop.

Row CTests omega_pr New build warnings
pm-cpu, gnu, mpich 61 of 61 passed 34 of 34 passed 0
pm-gpu, gnugpu, mpich 61 of 61 passed 32 of 34 passed, 2 with baseline differences 0
pm-cpu, gnu, mpich

Polaris omega_pr suite

  • Baseline workdir: /global/homes/s/sbrus/scratch/omega_pr_test/baselines/pm-cpu_gnu_polaris-2a5289a_omega-f368d66/omega_pr
  • Baseline build: /global/homes/s/sbrus/scratch/omega_pr_test/baselines/pm-cpu_gnu_polaris-2a5289a_omega-f368d66/build
  • PR build: /global/homes/s/sbrus/scratch/omega_pr_test/pr621-ef102d3/pm-cpu_gnu/build
  • PR workdir: /global/homes/s/sbrus/scratch/omega_pr_test/pr621-ef102d3/pm-cpu_gnu/omega_pr
  • Polaris: 2a5289ab41d8e9bd9be8150b1aace06163c19847 (1.0.0-1379-g2a5289ab41-dirty)
  • Baseline Omega: f368d668307f86cce95b0d98835bc02310b83f1e (Omega-v0.1.0-alpha.1-6289-gf368d66830)
  • PR Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Machine: pm-cpu
  • Compiler: gnu
  • Build type: Release
  • Log: /global/homes/s/sbrus/scratch/omega_pr_test/pr621-ef102d3/pm-cpu_gnu/omega_pr/polaris_omega_pr.o60333312
  • Result: All tests passed
Recent commits

Polaris:

2a5289ab41 Merge pull request #906 from xylar/add-pr-test-cleanup
15850cedb5 Merge pull request #904 from xylar/fix-omega-pr-handoff-early
335cc496de Merge pull request #643 from xylar/add-realistic-global-ocean-general-forward
f17d3695a3 Merge pull request #903 from xylar/fix-global-stats-anomaly-label
c3fc893b7a Merge pull request #730 from xylar/add-omega-analysis-climatology-maps

Baseline Omega:

f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp

PR Omega:

5f6c2c5369 Test merge of E3SM-Project/Omega#621 (ef102d3dce) into develop (f368d66830)
f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect

CTest unit tests:

  • Machine: pm-cpu
  • Compiler: gnu
  • Build type: Release
  • Build: /global/homes/s/sbrus/scratch/omega_pr_test/pr621-ef102d3/pm-cpu_gnu/build
  • Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Result: All tests passed
  • Log: /global/homes/s/sbrus/scratch/omega_pr_test/pr621-ef102d3/pm-cpu_gnu/build/ctests.log

Build warnings

No new warnings (4 in the PR build, 4 in the baseline build).

  • Baseline build log: /global/homes/s/sbrus/scratch/omega_pr_test/baselines/pm-cpu_gnu_polaris-2a5289a_omega-f368d66/build/build_omega.log
  • PR build log: /global/homes/s/sbrus/scratch/omega_pr_test/pr621-ef102d3/pm-cpu_gnu/build/build_omega.log
pm-gpu, gnugpu, mpich

Polaris omega_pr suite

  • Baseline workdir: /global/homes/s/sbrus/scratch/omega_pr_test/baselines/pm-gpu_gnugpu_polaris-2a5289a_omega-f368d66/omega_pr
  • Baseline build: /global/homes/s/sbrus/scratch/omega_pr_test/baselines/pm-gpu_gnugpu_polaris-2a5289a_omega-f368d66/build
  • PR build: /global/homes/s/sbrus/scratch/omega_pr_test/pr621-ef102d3/pm-gpu_gnugpu/build
  • PR workdir: /global/homes/s/sbrus/scratch/omega_pr_test/pr621-ef102d3/pm-gpu_gnugpu/omega_pr
  • Polaris: 2a5289ab41d8e9bd9be8150b1aace06163c19847 (1.0.0-1379-g2a5289ab41-dirty)
  • Baseline Omega: f368d668307f86cce95b0d98835bc02310b83f1e (Omega-v0.1.0-alpha.1-6289-gf368d66830)
  • PR Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Machine: pm-gpu
  • Compiler: gnugpu
  • Build type: Release
  • Log: /global/homes/s/sbrus/scratch/omega_pr_test/pr621-ef102d3/pm-gpu_gnugpu/omega_pr/polaris_omega_pr.o60334620
  • Result:
    • Diffs (2 of 34):
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/split_explicit
      - normalVelocity: l1=7.796e-04  l2=3.499e-05  linf=1.067e-05
      - temperature:    l1=1.016e-04  l2=5.772e-06  linf=3.117e-06
      - salinity:       l1=2.442e-05  l2=7.990e-07  linf=3.918e-07
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/rk4
      - normalVelocity: l1=2.934e-04  l2=6.575e-06  linf=2.593e-06
      - temperature:    l1=1.023e-05  l2=2.204e-07  linf=4.634e-08
      - salinity:       l1=1.375e-06  l2=4.787e-08  linf=1.068e-08
Recent commits

Polaris:

2a5289ab41 Merge pull request #906 from xylar/add-pr-test-cleanup
15850cedb5 Merge pull request #904 from xylar/fix-omega-pr-handoff-early
335cc496de Merge pull request #643 from xylar/add-realistic-global-ocean-general-forward
f17d3695a3 Merge pull request #903 from xylar/fix-global-stats-anomaly-label
c3fc893b7a Merge pull request #730 from xylar/add-omega-analysis-climatology-maps

Baseline Omega:

f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp

PR Omega:

5f6c2c5369 Test merge of E3SM-Project/Omega#621 (ef102d3dce) into develop (f368d66830)
f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect

CTest unit tests:

  • Machine: pm-gpu
  • Compiler: gnugpu
  • Build type: Release
  • Build: /global/homes/s/sbrus/scratch/omega_pr_test/pr621-ef102d3/pm-gpu_gnugpu/build
  • Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Result: All tests passed
  • Log: /global/homes/s/sbrus/scratch/omega_pr_test/pr621-ef102d3/pm-gpu_gnugpu/build/ctests.log

Build warnings

No new warnings (852 in the PR build, 852 in the baseline build).

  • Baseline build log: /global/homes/s/sbrus/scratch/omega_pr_test/baselines/pm-gpu_gnugpu_polaris-2a5289a_omega-f368d66/build/build_omega.log
  • PR build log: /global/homes/s/sbrus/scratch/omega_pr_test/pr621-ef102d3/pm-gpu_gnugpu/build/build_omega.log

Notes

The pm-gpu baseline differences in the two e3sm-qu240 cached_forward tasks look like run-to-run nondeterminism on the GPU rather than a change from this PR. Rerunning each task with the unchanged baseline build, on the same single GPU, also differed from the baseline by the same order of magnitude (split_explicit: normalVelocity linf up to 4e-6 and temperature 2e-6, against 1e-5 and 3e-6 for the PR; rk4: normalVelocity 1e-7 and temperature 1e-6). Both tasks match the baseline bit for bit on pm-cpu. The split_explicit norms are nearly the same on pm-gpu and Aurora's GPU, which the rerun does not explain, so a small GPU-only change from the PR cannot be ruled out underneath the noise.

@sbrus89

sbrus89 commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Testing on aurora

Tested the merge of #621 (ef102d3dce) into develop (f368d66830) as 5f6c2c5369, against develop (f368d66830) rather than the Polaris submodule, because the Omega submodule in Polaris main has not been updated since Omega PR #616 merged into develop.

Row CTests omega_pr New build warnings
aurora, intel, mpich 61 of 61 passed 34 of 34 passed 0
aurora, intelgpu, mpich 61 of 61 passed 32 of 34 passed, 2 with baseline differences 1
aurora, intel, mpich

Polaris omega_pr suite

  • Baseline workdir: /home/brus/run/omega_pr_test/baselines/aurora_intel_polaris-2a5289a_omega-f368d66/omega_pr
  • Baseline build: /home/brus/run/omega_pr_test/baselines/aurora_intel_polaris-2a5289a_omega-f368d66/build
  • PR build: /home/brus/run/omega_pr_test/pr621-ef102d3/aurora_intel/build
  • PR workdir: /home/brus/run/omega_pr_test/pr621-ef102d3/aurora_intel/omega_pr
  • Polaris: 2a5289ab41d8e9bd9be8150b1aace06163c19847 (1.1.0-458-g2a5289ab4-dirty)
  • Baseline Omega: f368d668307f86cce95b0d98835bc02310b83f1e (Omega-v0.1.0-alpha.1-6289-gf368d66830)
  • PR Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Machine: aurora
  • Compiler: intel
  • Build type: Release
  • Log: /home/brus/run/omega_pr_test/pr621-ef102d3/aurora_intel/omega_pr/polaris_omega_pr.o8969915
  • Result: All tests passed
Recent commits

Polaris:

2a5289ab4 Merge pull request #906 from xylar/add-pr-test-cleanup
15850cedb Merge pull request #904 from xylar/fix-omega-pr-handoff-early
335cc496d Merge pull request #643 from xylar/add-realistic-global-ocean-general-forward
f17d3695a Merge pull request #903 from xylar/fix-global-stats-anomaly-label
c3fc893b7 Merge pull request #730 from xylar/add-omega-analysis-climatology-maps

Baseline Omega:

f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp

PR Omega:

5f6c2c5369 Test merge of E3SM-Project/Omega#621 (ef102d3dce) into develop (f368d66830)
f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect

CTest unit tests:

  • Machine: aurora
  • Compiler: intel
  • Build type: Release
  • Build: /home/brus/run/omega_pr_test/pr621-ef102d3/aurora_intel/build
  • Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Result: All tests passed
  • Log: /home/brus/run/omega_pr_test/pr621-ef102d3/aurora_intel/build/ctests.log

Build warnings

No new warnings (14 in the PR build, 14 in the baseline build).

  • Baseline build log: /home/brus/run/omega_pr_test/baselines/aurora_intel_polaris-2a5289a_omega-f368d66/build/build_omega.log
  • PR build log: /home/brus/run/omega_pr_test/pr621-ef102d3/aurora_intel/build/build_omega.log
aurora, intelgpu, mpich

Polaris omega_pr suite

  • Baseline workdir: /home/brus/run/omega_pr_test/baselines/aurora_intelgpu_polaris-2a5289a_omega-f368d66/omega_pr
  • Baseline build: /home/brus/run/omega_pr_test/baselines/aurora_intelgpu_polaris-2a5289a_omega-f368d66/build
  • PR build: /home/brus/run/omega_pr_test/pr621-ef102d3/aurora_intelgpu/build
  • PR workdir: /home/brus/run/omega_pr_test/pr621-ef102d3/aurora_intelgpu/omega_pr
  • Polaris: 2a5289ab41d8e9bd9be8150b1aace06163c19847 (1.1.0-458-g2a5289ab4-dirty)
  • Baseline Omega: f368d668307f86cce95b0d98835bc02310b83f1e (Omega-v0.1.0-alpha.1-6289-gf368d66830)
  • PR Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Machine: aurora
  • Compiler: intelgpu
  • Build type: Release
  • Log: /home/brus/run/omega_pr_test/pr621-ef102d3/aurora_intelgpu/omega_pr/polaris_omega_pr.o8969986
  • Result:
    • Diffs (2 of 34):
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/split_explicit
      - normalVelocity: l1=7.855e-04  l2=3.499e-05  linf=1.067e-05
      - temperature:    l1=1.021e-04  l2=5.772e-06  linf=3.117e-06
      - salinity:       l1=2.449e-05  l2=7.989e-07  linf=3.918e-07
    - ocean/spherical/realistic_global/e3sm-qu240/cached_forward/rk4
      - normalVelocity: l1=4.100e-05  l2=7.605e-07  linf=2.758e-07
      - temperature:    l1=3.405e-06  l2=2.356e-07  linf=8.643e-08
      - salinity:       l1=5.204e-07  l2=1.920e-08  linf=6.288e-09
Recent commits

Polaris:

2a5289ab4 Merge pull request #906 from xylar/add-pr-test-cleanup
15850cedb Merge pull request #904 from xylar/fix-omega-pr-handoff-early
335cc496d Merge pull request #643 from xylar/add-realistic-global-ocean-general-forward
f17d3695a Merge pull request #903 from xylar/fix-global-stats-anomaly-label
c3fc893b7 Merge pull request #730 from xylar/add-omega-analysis-climatology-maps

Baseline Omega:

f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp

PR Omega:

5f6c2c5369 Test merge of E3SM-Project/Omega#621 (ef102d3dce) into develop (f368d66830)
f368d66830 Merge pull request #616 from mwarusz/omega/vmix-pressure-fix
695f548841 Merge pull request #622 from sbrus89/omega/metis-root-update
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect

CTest unit tests:

  • Machine: aurora
  • Compiler: intelgpu
  • Build type: Release
  • Build: /home/brus/run/omega_pr_test/pr621-ef102d3/aurora_intelgpu/build
  • Omega: 5f6c2c5369a460daa77a0d556f1dbe064a633b8a (Omega-v0.1.0-alpha.1-6293-g5f6c2c5369)
  • Result: All tests passed
  • Log: /home/brus/run/omega_pr_test/pr621-ef102d3/aurora_intelgpu/build/ctests.log

Build warnings

1 new warnings, 0 in files this PR changes (242 in the PR build, 242 in the baseline build). Those in files the PR changes are marked *.

New warnings
  externals/scorpio/src/gptl/gptl.c: [  6%] Building C object _deps/zstd-build/lib/CMakeFiles/libzstd_static.dir/__/__/zstd-src/lib/common/xxhash.c.o
  • Baseline build log: /home/brus/run/omega_pr_test/baselines/aurora_intelgpu_polaris-2a5289a_omega-f368d66/build/build_omega.log
  • PR build log: /home/brus/run/omega_pr_test/pr621-ef102d3/aurora_intelgpu/build/build_omega.log

Notes

The intelgpu baseline differences in the two e3sm-qu240 cached_forward tasks look like run-to-run nondeterminism on the GPU rather than a change from this PR. Rerunning each task with the unchanged baseline build, on the same single GPU, also differed from the baseline by as much or more (split_explicit: normalVelocity linf up to 8e-6 and temperature 2e-6; rk4: normalVelocity 4e-5 and temperature 2e-5). The same happens on pm-gpu, and both tasks match the baseline bit for bit with intel.

The one new intelgpu build warning is an artifact of interleaved build output, a zstd build line attributed to scorpio's gptl.c, in code this PR does not touch; the PR and baseline builds both have 242 warnings.

@sbrus89

sbrus89 commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

The fix for the reproducibility issue noted in the realistic global cases is in #646

@andrewdnolan

Copy link
Copy Markdown
Collaborator

Testing on chrysalis with intel compiler

Tested with a local merge of this branch into develop at 06e81db693

  • PEM_D.T62_oQU240.GOMEGA-IAF.chrysalis_intel.20261027_130822: PASS
  • ERS_Ld3.ne4pg2_oQU240.WCYCL1850NS-OMEGA.chrysalis_gnu.20261027_130951: PASS

@andrewdnolan

Copy link
Copy Markdown
Collaborator

Testing on frontier with craygnu-mphipcc compiler

Tested with a local merge of this branch into develop at 6af2d90296.

  • ERS_Ld5.T62_oQU240.GOMEGA-IAF.frontier_craygnu-mphipcc.20261027_151315: PASS
  • ERS_Ld3.ne4pg2_oQU240.WCYCL1850NS-OMEGA.frontier_craygnu-mphipcc.20261027_151327_523684: PASS

@sbrus89
sbrus89 merged commit e09fcf8 into E3SM-Project:develop Oct 27, 2026
1 check passed
@xylar xylar mentioned this pull request Oct 27, 2026
4 of 8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants