Skip to content

Add submesoscale eddy parametrization - #438

Merged
sbrus89 merged 14 commits into
E3SM-Project:developfrom
mwarusz:omega/submesoscale-eddies
Oct 20, 2026
Merged

sbrus89 merged 14 commits into
E3SM-Project:developfrom
mwarusz:omega/submesoscale-eddies

Conversation

@mwarusz

@mwarusz mwarusz commented Jun 20, 2026 •

Copy link
Copy Markdown
Member

This PR adds the submesoscale eddy parametrization in the form presented in Fox-Kemper et al. 2011. The parametrization class provides three main methods that

  • determine mixed layer depth
  • compute buoyancy gradient
  • compute submesoscale induced velocity

respectively. A new auxiliary variable named NormalTransportVelocity has been added. This velocity variable is used to advect pseudo-thickness and tracers, and can optionally include the submesocale induced velocity.

Checklist

  • Documentation:
  • Linting
  • Building
    • CMake build does not produce any new warnings from changes in this PR
  • Testing
    • Add a comment to the PR titled Testing with the following:
      • Which machines CTest unit tests
        have been run on and indicate that are all passing.
      • The Polaris omega_pr test suite
        has passed, using the Polaris e3sm_submodules/Omega baseline
      • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
      • Indicate "All tests passed" or document failing tests
      • Document testing used to verify the changes including any tests that are added/modified/impacted.
    • New tests:
      • CTest unit tests for new features have been added per the approved design.

@mwarusz

mwarusz commented Jun 20, 2026 •

Copy link
Copy Markdown
Member Author

The parametrization unit tests are passing. However, after turning it on in the baroclinic channel polaris test with the default parameters, I see NaNs after about 3 hours of simulation time, so there is still some debugging to be done. Moreover, user and developer documentation needs to be added.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new submesoscale eddy parameterization module (Fox-Kemper
et al. 2011) into Omega’s ocean model, wires it into initialization/finalization
and tendency/auxiliary-state workflows, and adds a dedicated unit test plus
broad test-harness updates to support new analytic-field helper signatures.

Changes:

  • Add SubmesoEddies module (config-driven) and integrate it into ocean module
    init/finalize and auxiliary-state computation.
  • Introduce TransportAuxVars::NormalTransportVelocity and use it in thickness
    and tracer tendency computations as the transport velocity (optionally
    including submesoscale-induced velocity).
  • Update unit test utilities and many tests to accommodate updated analytic
    field setters; add SubmesoEddiesTest.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
components/omega/test/timeStepping/TimeStepperTest.cpp Initialize/destroy SubmesoEddies in the time stepper test harness.
components/omega/test/ocn/TendencyTermsTest.cpp Update analytic-field lambdas to match updated setScalar/setVectorEdge signatures.
components/omega/test/ocn/TendenciesTest.cpp Initialize/destroy SubmesoEddies; update analytic-field lambdas and setScalar/setVectorEdge call signatures.
components/omega/test/ocn/SubmesoEddiesTest.cpp New unit test covering mixed-layer depth, buoyancy gradient, and eddy velocity computations.
components/omega/test/ocn/OceanTestCommon.h Extend setScalar/setVectorEdge helpers to pass element indices and optional Z coordinates.
components/omega/test/ocn/HorzOperatorsTest.cpp Update analytic-field lambdas to match updated helper signatures.
components/omega/test/ocn/AuxiliaryVarsTest.cpp Update analytic-field lambdas to match updated helper signatures.
components/omega/test/ocn/AuxiliaryStateTest.cpp Initialize/destroy SubmesoEddies; update analytic-field lambdas and helper call signatures.
components/omega/test/CMakeLists.txt Register new SUBMESOEDDIES_TEST.
components/omega/src/timeStepping/ForwardBackwardStepper.cpp Pass tracer array into thickness tendency computation.
components/omega/src/ocn/Tendencies.h Extend thickness tendency API to accept a tracer array (for aux-state computations).
components/omega/src/ocn/Tendencies.cpp Route thickness/tracer tendencies through new aux-state compute routines and use NormalTransportVelocity.
components/omega/src/ocn/SubmesoEddies.h New SubmesoEddies public interface.
components/omega/src/ocn/SubmesoEddies.cpp New implementation and IO field registration for SubmesoEddies.
components/omega/src/ocn/OceanInit.cpp Initialize SubmesoEddies during ocean module init.
components/omega/src/ocn/OceanFinal.cpp Destroy SubmesoEddies during ocean finalize.
components/omega/src/ocn/auxiliaryVars/TransportAuxVars.h New aux-var container for NormalTransportVelocity.
components/omega/src/ocn/auxiliaryVars/TransportAuxVars.cpp Register/unregister IO field for NormalTransportVelocity.
components/omega/src/ocn/AuxiliaryState.h Add TransportAuxVars member and new aux compute entry points.
components/omega/src/ocn/AuxiliaryState.cpp Compute Brunt–Väisälä frequency, compute transport velocity (optionally + submeso), and use it for vertical pseudo-velocity.
components/omega/configs/Default.yml Add Omega.Submeso configuration group and defaults.

Comment thread components/omega/src/ocn/AuxiliaryState.cpp Outdated
Comment on lines +173 to +182
KOKKOS_LAMBDA(int IEdge, const TeamMember &Team) {
const int KMin = MinLayerEdgeBot(IEdge);
const int KMax = MaxLayerEdgeTop(IEdge);
const int KRange = vertRangeChunked(KMin, KMax);

parallelForInner(
Team, KRange, INNER_LAMBDA(int KChunk) {
LocPseudoThicknessAux.computeVarsOnEdge(
IEdge, KChunk, PseudoThick, NormalVelEdge);
});
Comment on lines +18 to +22
/// Destroy instance of SubmesoEddies
void SubmesoEddies::destroyInstance() {
delete Instance;
Instance = nullptr;
}
Comment thread components/omega/src/ocn/SubmesoEddies.cpp
Comment thread components/omega/src/ocn/SubmesoEddies.cpp
Comment thread components/omega/test/ocn/SubmesoEddiesTest.cpp
@mwarusz
mwarusz force-pushed the omega/submesoscale-eddies branch 2 times, most recently from 1d3f283 to 9825eda Compare July 14, 2026 21:14
@mwarusz

mwarusz commented Jul 15, 2026

Copy link
Copy Markdown
Member Author

Testing

CTest unit tests

  • Machine: pm-cpu / pm-gpu / frontier
  • Compiler: gnu / gnugpu / craygnu-mphipcc
  • Build type: Release
  • Result: All tests passed (on all machines)

Polaris omega_pr suite

  • Baseline workdir: /pscratch/sd/m/mwarusz/omega-pr-testing/submeso/baseline-gnu
  • Baseline build: /pscratch/sd/m/mwarusz/omega-pr-testing/submeso/baseline-gnu/build
  • PR build: /pscratch/sd/m/mwarusz/omega-pr-testing/submeso/build-gnu
  • PR workdir: /pscratch/sd/m/mwarusz/omega-pr-testing/submeso/pr-gnu
  • Machine: pm-cpu
  • Compiler: gnu
  • Build type: Release
  • Log: /pscratch/sd/m/mwarusz/omega-pr-testing/submeso/pr-gnu/polaris_omega_pr.o55955301
  • Result: All tests passed

@mwarusz

mwarusz commented Jul 15, 2026

Copy link
Copy Markdown
Member Author

The baroclinic channel test is now running stably with the submesoscale eddy parametrization turned on. This PR is ready for review. However, I haven't checked that the parametrization has the intended effect, and I'm not sure that I'm the best person to do that. @vanroekel could you help with this kind of testing ?

@mwarusz
mwarusz requested review from katsmith133 and vanroekel July 15, 2026 23:29
@vanroekel

Copy link
Copy Markdown
Collaborator

@mwarusz I ran a couple tests with baroclinic channel and it's too hard to tell what the effects must be. We need to add a mixed layer. Let me look and see how straight forward that might be.

@vanroekel

Copy link
Copy Markdown
Collaborator

update - I have a mixed layer variant of baroclinic channel I have a test run in the queue on frontier. Hopefully will have results by Monday

PseudoThickML, GradBuoyML, BVFreqML);

GradBuoyML /= PseudoThickML;
BVFreqML /= PseudoThickML;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mwarusz for small ML this could be be problematic, it's not clear above in the MLD code if there is a floor. maybe it's not an issue though

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that we should probably add a tiny value here. MPAS-O has it.

Team, Range{MinLyrEdgeBot, IndexMLEdge},
INNER_LAMBDA(int K, Real &AccumThick, Real &AccumGradBuoy,
Real &AccumBVFreq) {
const Real PseudoThickKm1 =

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If your function starts at MinLyrEdgeBot can this lead to values lower than min layer edge?

@mwarusz mwarusz Aug 11, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this variable is set to

((K - 1) >= MinLyrEdgeBot ? MeanPseudoThickEdge(IEdge, K - 1) : 0)

which is 0 for K = MinLyrEdgeBot. In the end an average of PseudoThickKm1 and PseudoThickK is used. This means that for K = MinLyrEdgeBot you get

PseudoThickAvg = 0.5 * (0 + MeanPseudoThickEdge(IEdge, MinLyrEdgeBot));

This is a kinda hacky treatment of the first half layer. I will add a comment to explain this.

@vanroekel

Copy link
Copy Markdown
Collaborator

@mwarusz in testing I'm getting NaN for EddyVelocity quickly. CoPilot added some guards on the ML average and zero initializing and it fixed the NaN issue. I can point you at what the changes made were if you'd like

@mwarusz

mwarusz commented Aug 11, 2026

Copy link
Copy Markdown
Member Author

@vanroekel

@mwarusz in testing I'm getting NaN for EddyVelocity quickly. CoPilot added some guards on the ML average and zero initializing and it fixed the NaN issue. I can point you at what the changes made were if you'd like

Please do.

@vanroekel

Copy link
Copy Markdown
Collaborator

here it is - https://github.com/vanroekel/E3SM/tree/temp-submeso-branch

There is some extra stuff that is adding more diagnostics as well.

@mwarusz

mwarusz commented Aug 11, 2026

Copy link
Copy Markdown
Member Author

Hmm, Copilot added this check

if (PseudoThickML > Tiny && MLDepthEdge > Tiny)

but afterwards it is also explicitly checking for NaNs and skipping calculations if it finds any. It would good to see if the first check is enough.

@mwarusz
mwarusz force-pushed the omega/submesoscale-eddies branch 2 times, most recently from 3660296 to a7bd36d Compare August 24, 2026 22:33

// Compute mixed layer index and depth based on the density difference
// criterion
void computeDenMixLayerDepth(const Array2DReal &SpecVol);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mwarusz, you were going to move this to an aux variable, correct?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I haven't worked on this PR in a while. I was planning to get back to it after the split-explicit is merged, since the transport velocity refactoring that I did there affects this PR in non-trivial ways.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Thanks!

@mwarusz
mwarusz force-pushed the omega/submesoscale-eddies branch from a7bd36d to 0bf23ad Compare September 18, 2026 23:10
@mwarusz

mwarusz commented Sep 21, 2026

Copy link
Copy Markdown
Member Author

To test this branch after rebasing on develop and fixing a couple of bugs I ran ocean/spherical/realistic_global/EC30to60E2r2/analysis_members_test for about one year using Omega and MPAS-O with the following changes to both models:

  • enable submesoscale eddy parametrization
  • use split-explicit time stepping (RK2 with 45 min time step for Omega, AB2 with 30 min time step for MPAS-O)
  • use monotonic third-order horizontal advection

Here are some plots comparing temperature, mixed layer depth, and eddy velocity kinetic energy at the final time. @vanroekel Do you think these look reasonable ?

Omega

temperature_360days DenMixLayerDepth_360days EddyKineticEnergy_360days

MPAS-O

temperature_360days dThreshMLD_360days EddyKineticEnergy_360days

Comment thread components/omega/doc/userGuide/SubmesoEddies.md Outdated
Comment thread components/omega/doc/userGuide/SubmesoEddies.md Outdated
Comment thread components/omega/doc/userGuide/SubmesoEddies.md Outdated
Comment thread components/omega/src/ocn/auxiliaryVars/MixedLayerAuxVars.h Outdated
- `Tau`: MLI timescale parameter (s).
- `Ce`: nondimensional efficiency coefficient.
- `LfMin`: minimum frontal width limiter (m).
- `DsMax`: maximum grid-length limiter used in the closure (m).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

somewhere, either here or in the SubmesoEddy class we should include reasonable ranges of these parameters.

Comment thread components/omega/src/ocn/SubmesoEddies.cpp

parallelForOuter(
LaunchConfig({Mesh->NEdgesAll},
TeamScratch<Real>(3 * NVertLayers + 3 * NVertLayersP1)),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain the 3 * here? I don't understand where they come from

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the number of scratch arrays used in this loop. Three scratch arrays for layer data and three for interface data. I added a comment and changed the code slightly to hopefully make it more clear.

const int K = MaxLyrEdgeTop + 1;
const int JCell0 = CellsOnEdge(IEdge, 0);
const int JCell1 = CellsOnEdge(IEdge, 1);
BVFSqEdge(K) = 0.5_Real * (BruntVaisalaFreqSq(JCell1, K) +

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is this part different than just above on L183?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not. In addition to BVF, the loop above interpolates fields that are located at layer mid-points (SpecVol), so it only loops over layers. This code handles the last interface interpolation for BVF. I rephrased the comment above to make it more clear.

@vanroekel vanroekel Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. thanks. Maybe say 'ocean bottom' instead of 'bottom interface' in the comment or 'bottom of the column'? But also on second read I clearly see the 'MaxLayerEdge' definition in K that I missed earlier

@mwarusz
mwarusz force-pushed the omega/submesoscale-eddies branch from 4648e59 to b97fb83 Compare October 19, 2026 15:36
@katsmith133

Copy link
Copy Markdown

I've looked over the code and it looks good to me, but Copilot found the following two issues:

  • Boundary initialization is dropped by both compatibility overloads. The bounds-taking setScalar overload accepts SetBndOpt but does not forward it, so the implementation uses SetBoundary::No; the same happens in setVectorEdge. Calls that explicitly request boundary setup—such as those in TendenciesTest.cpp:88-108—leave the special boundary element unwritten. See the forwarding calls in OceanTestCommon.h:263-273 and OceanTestCommon.h:470-482.
  • Vector functors may be invoked four times per element. The independent if constexpr branches all run when a generic or overloaded functor matches multiple supported signatures. That can repeat side effects and overwrite the vector result; the adapter should select one matching signature. See OceanTestCommon.h:290-306.

@mwarusz, if you don't think these are an issue, I am happy to approve.

@mwarusz

mwarusz commented Oct 19, 2026

Copy link
Copy Markdown
Member Author

@katsmith133

I pushed a fix for the first issue. I think the second one doesn't need a fix. I believe we never call the setX functions with generic or overloaded functors, and I don't think there is any use case for this.

@katsmith133 katsmith133 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@katsmith133

I pushed a fix for the first issue. I think the second one doesn't need a fix. I believe we never call the setX functions with generic or overloaded functors, and I don't think there is any use case for this.

Great, thanks @mwarusz! I am happy to approve this now. Approving based on visual inspection and testing done by @mwarusz.

@sbrus89

sbrus89 commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator

Testing on frontier

Tested the merge of #438 (6f4378a243) into develop (a341e65245) as 8d0c87c076, against Polaris' Omega submodule (cdb935bb5a).

Row CTests omega_pr New build warnings
frontier, craygnu, mpich 61 of 61 passed 32 of 32 passed 0
frontier, craygnu-mphipcc, mpich 61 of 61 passed 32 of 32 passed 0
frontier, craygnu, mpich

Polaris omega_pr suite

  • Baseline workdir: /ccs/home/brus/run/omega_pr_test/baselines/frontier_craygnu_polaris-46a013b_omega-cdb935b/omega_pr
  • Baseline build: /ccs/home/brus/run/omega_pr_test/baselines/frontier_craygnu_polaris-46a013b_omega-cdb935b/build
  • PR build: /ccs/home/brus/run/omega_pr_test/pr438-6f4378a/frontier_craygnu/build
  • PR workdir: /ccs/home/brus/run/omega_pr_test/pr438-6f4378a/frontier_craygnu/omega_pr
  • Polaris: 46a013b426fecdccba7c5f3a6be6830caacec322 (1.1.0-182-g46a013b42)
  • Baseline Omega: cdb935bb5a8d995b0878616e001a2701a2627519 (Omega-v0.1.0-alpha.1-6142-gcdb935bb5a)
  • PR Omega: 8d0c87c0766af598ae1b8dc9f579c0cd0f2bb6e8 (Omega-v0.1.0-alpha.1-6225-g8d0c87c076)
  • Machine: frontier
  • Partition: batch
  • Compiler: craygnu
  • Build type: Release
  • Log: /ccs/home/brus/run/omega_pr_test/pr438-6f4378a/frontier_craygnu/omega_pr/polaris_omega_pr.o5711232
  • Result: All tests passed
Recent commits

Polaris:

46a013b42 Merge remote-tracking branch 'origin/main' into HEAD
1b660aa0a Merge remote-tracking branch 'xylar/add-omega-testing-workflow' into HEAD
2a2c8ad4b Merge pull request #855 from xylar/fix-float32-diff-summary
2e0ae95f5 Merge pull request #787 from xylar/add-realistic-global-analysis-run
44c4d5601 Merge pull request #776 from xylar/check-cf-in-omega-output

Baseline Omega:

cdb935bb5a Merge pull request #600 from xylar/omega/fix-pm-mpi-flags
b90c3dae8d Merge pull request #601 from andrewdnolan/omega/cxx20-for-coupled
9641c6cf9e Merge pull request #546 from xylar/omega/fix-cf-units
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc

PR Omega:

8d0c87c076 Test merge of E3SM-Project/Omega#438 (6f4378a243) into develop (a341e65245)
a341e65245 Merge pull request #562 from andrewdnolan/omega/guard-division-by-zero
46aa54cd68 Merge pull request #603 from vanroekel/vanroekel/omega/kpp-bug-fix
9656466bb6 Merge PR #570 into develop, resolving Tendencies.cpp initializer order
d1d3ea6506 Merge pull request #466 from alicebarthel/omega/add-frazil-basic-rebased

CTest unit tests:

  • Machine: frontier
  • Compiler: craygnu
  • Build type: Release
  • Build: /ccs/home/brus/run/omega_pr_test/pr438-6f4378a/frontier_craygnu/build
  • Omega: 8d0c87c0766af598ae1b8dc9f579c0cd0f2bb6e8 (Omega-v0.1.0-alpha.1-6225-g8d0c87c076)
  • Result: All tests passed
  • Log: /ccs/home/brus/run/omega_pr_test/pr438-6f4378a/frontier_craygnu/build/ctests.log

Build warnings

No new warnings (3 in the PR build, 3 in the baseline build).

  • Baseline build log: /ccs/home/brus/run/omega_pr_test/baselines/frontier_craygnu_polaris-46a013b_omega-cdb935b/build/build_omega.log
  • PR build log: /ccs/home/brus/run/omega_pr_test/pr438-6f4378a/frontier_craygnu/build/build_omega.log
frontier, craygnu-mphipcc, mpich

Polaris omega_pr suite

  • Baseline workdir: /ccs/home/brus/run/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-46a013b_omega-cdb935b/omega_pr
  • Baseline build: /ccs/home/brus/run/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-46a013b_omega-cdb935b/build
  • PR build: /ccs/home/brus/run/omega_pr_test/pr438-6f4378a/frontier_craygnu-mphipcc/build
  • PR workdir: /ccs/home/brus/run/omega_pr_test/pr438-6f4378a/frontier_craygnu-mphipcc/omega_pr
  • Polaris: 46a013b426fecdccba7c5f3a6be6830caacec322 (1.1.0-182-g46a013b42)
  • Baseline Omega: cdb935bb5a8d995b0878616e001a2701a2627519 (Omega-v0.1.0-alpha.1-6142-gcdb935bb5a)
  • PR Omega: 8d0c87c0766af598ae1b8dc9f579c0cd0f2bb6e8 (Omega-v0.1.0-alpha.1-6225-g8d0c87c076)
  • Machine: frontier
  • Partition: batch
  • Compiler: craygnu-mphipcc
  • Build type: Release
  • Log: /ccs/home/brus/run/omega_pr_test/pr438-6f4378a/frontier_craygnu-mphipcc/omega_pr/polaris_omega_pr.o5711240
  • Result: All tests passed
Recent commits

Polaris:

46a013b42 Merge remote-tracking branch 'origin/main' into HEAD
1b660aa0a Merge remote-tracking branch 'xylar/add-omega-testing-workflow' into HEAD
2a2c8ad4b Merge pull request #855 from xylar/fix-float32-diff-summary
2e0ae95f5 Merge pull request #787 from xylar/add-realistic-global-analysis-run
44c4d5601 Merge pull request #776 from xylar/check-cf-in-omega-output

Baseline Omega:

cdb935bb5a Merge pull request #600 from xylar/omega/fix-pm-mpi-flags
b90c3dae8d Merge pull request #601 from andrewdnolan/omega/cxx20-for-coupled
9641c6cf9e Merge pull request #546 from xylar/omega/fix-cf-units
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc

PR Omega:

8d0c87c076 Test merge of E3SM-Project/Omega#438 (6f4378a243) into develop (a341e65245)
a341e65245 Merge pull request #562 from andrewdnolan/omega/guard-division-by-zero
46aa54cd68 Merge pull request #603 from vanroekel/vanroekel/omega/kpp-bug-fix
9656466bb6 Merge PR #570 into develop, resolving Tendencies.cpp initializer order
d1d3ea6506 Merge pull request #466 from alicebarthel/omega/add-frazil-basic-rebased

CTest unit tests:

  • Machine: frontier
  • Compiler: craygnu-mphipcc
  • Build type: Release
  • Build: /ccs/home/brus/run/omega_pr_test/pr438-6f4378a/frontier_craygnu-mphipcc/build
  • Omega: 8d0c87c0766af598ae1b8dc9f579c0cd0f2bb6e8 (Omega-v0.1.0-alpha.1-6225-g8d0c87c076)
  • Result: All tests passed
  • Log: /ccs/home/brus/run/omega_pr_test/pr438-6f4378a/frontier_craygnu-mphipcc/build/ctests.log

Build warnings

No new warnings (6 in the PR build, 6 in the baseline build).

  • Baseline build log: /ccs/home/brus/run/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-46a013b_omega-cdb935b/build/build_omega.log
  • PR build log: /ccs/home/brus/run/omega_pr_test/pr438-6f4378a/frontier_craygnu-mphipcc/build/build_omega.log

@xylar

xylar commented Oct 19, 2026

Copy link
Copy Markdown

While testing this branch with a new Polaris mixed-layer-eddies task, I found that DenMixLayerDepth and EddyVelocity in the same output often don't come from the same state. In SplitExplicitRK2Stepper::doStep, EddyVelocity is last computed in the Stage 3 tendencies. After that, VertMixImplicit calls computeMomVertAux on the new state (VertMix.cpp:778), which updates DenMixLayerDepth (and BruntVaisalaFreqSq) but not EddyVelocity, and the tracers are then mixed before output.

In a nearly neutral mixed layer the density-threshold depth can jump by whole layers between the two. At 8 km on day 1, the output depth was the one the eddy velocity was computed with on only about a third of the edges in the frontal zone (about 80% by day 7). Within the scheme the two are consistent, so this only affects anyone diagnosing it from output. Writing the edge mixed-layer depth the scheme used, or a note in the user guide, would help. I only looked at the split-explicit stepper.


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@sbrus89

sbrus89 commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Testing on pm-cpu and pm-gpu

Tested the merge of #438 (6f4378a243) into develop (a341e65245) as 8d0c87c076, against Polaris' Omega submodule (cdb935bb5a).

Row CTests omega_pr New build warnings
pm-cpu, gnu, mpich 58 of 58 passed 32 of 32 passed 0
pm-gpu, gnugpu, mpich 61 of 61 passed 32 of 32 passed 5
pm-cpu, gnu, mpich

Polaris omega_pr suite

  • Baseline workdir: /global/homes/s/sbrus/scratch/omega_pr_test/baselines/pm-cpu_gnu_polaris-e9e74eb_omega-cdb935b/omega_pr
  • Baseline build: /global/homes/s/sbrus/scratch/omega_pr_test/baselines/pm-cpu_gnu_polaris-e9e74eb_omega-cdb935b/build
  • PR build: /global/homes/s/sbrus/scratch/omega_pr_test/pr438-6f4378a/pm-cpu_gnu/build
  • PR workdir: /global/homes/s/sbrus/scratch/omega_pr_test/pr438-6f4378a/pm-cpu_gnu/omega_pr
  • Polaris: e9e74eb869cc2afbc38789af7fd75efd05733a95 (1.0.0-1102-ge9e74eb869)
  • Baseline Omega: cdb935bb5a8d995b0878616e001a2701a2627519 (Omega-v0.1.0-alpha.1-6142-gcdb935bb5a)
  • PR Omega: 8d0c87c0766af598ae1b8dc9f579c0cd0f2bb6e8 (Omega-v0.1.0-alpha.1-6225-g8d0c87c076)
  • Machine: pm-cpu
  • Compiler: gnu
  • Build type: Release
  • Log: /global/homes/s/sbrus/scratch/omega_pr_test/pr438-6f4378a/pm-cpu_gnu/omega_pr/polaris_omega_pr.o60005347
  • Result: All tests passed
Recent commits

Polaris:

e9e74eb869 Merge PR testing utility into main for local testing
662796b1b7 Merge pull request #864 from xylar/add-release-notes-skill
2a2c8ad4b9 Merge pull request #855 from xylar/fix-float32-diff-summary
2e0ae95f5b Merge pull request #787 from xylar/add-realistic-global-analysis-run
44c4d56012 Merge pull request #776 from xylar/check-cf-in-omega-output

Baseline Omega:

cdb935bb5a Merge pull request #600 from xylar/omega/fix-pm-mpi-flags
b90c3dae8d Merge pull request #601 from andrewdnolan/omega/cxx20-for-coupled
9641c6cf9e Merge pull request #546 from xylar/omega/fix-cf-units
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc

PR Omega:

8d0c87c076 Test merge of E3SM-Project/Omega#438 (6f4378a243) into develop (a341e65245)
a341e65245 Merge pull request #562 from andrewdnolan/omega/guard-division-by-zero
46aa54cd68 Merge pull request #603 from vanroekel/vanroekel/omega/kpp-bug-fix
9656466bb6 Merge PR #570 into develop, resolving Tendencies.cpp initializer order
d1d3ea6506 Merge pull request #466 from alicebarthel/omega/add-frazil-basic-rebased

CTest unit tests:

  • Machine: pm-cpu
  • Compiler: gnu
  • Build type: Release
  • Build: /global/homes/s/sbrus/scratch/omega_pr_test/pr438-6f4378a/pm-cpu_gnu/build
  • Omega: 8d0c87c0766af598ae1b8dc9f579c0cd0f2bb6e8 (Omega-v0.1.0-alpha.1-6225-g8d0c87c076)
  • Result: All tests passed
  • Log: /global/homes/s/sbrus/scratch/omega_pr_test/pr438-6f4378a/pm-cpu_gnu/build/ctests.log

Build warnings

No new warnings (4 in the PR build, 4 in the baseline build).

  • Baseline build log: /global/homes/s/sbrus/scratch/omega_pr_test/baselines/pm-cpu_gnu_polaris-e9e74eb_omega-cdb935b/build/build_omega.log
  • PR build log: /global/homes/s/sbrus/scratch/omega_pr_test/pr438-6f4378a/pm-cpu_gnu/build/build_omega.log
pm-gpu, gnugpu, mpich

Polaris omega_pr suite

  • Baseline workdir: /global/homes/s/sbrus/scratch/omega_pr_test/baselines/pm-gpu_gnugpu_polaris-e9e74eb_omega-cdb935b/omega_pr
  • Baseline build: /global/homes/s/sbrus/scratch/omega_pr_test/baselines/pm-gpu_gnugpu_polaris-e9e74eb_omega-cdb935b/build
  • PR build: /global/homes/s/sbrus/scratch/omega_pr_test/pr438-6f4378a/pm-gpu_gnugpu/build
  • PR workdir: /global/homes/s/sbrus/scratch/omega_pr_test/pr438-6f4378a/pm-gpu_gnugpu/omega_pr
  • Polaris: e9e74eb869cc2afbc38789af7fd75efd05733a95 (1.0.0-1102-ge9e74eb869)
  • Baseline Omega: cdb935bb5a8d995b0878616e001a2701a2627519 (Omega-v0.1.0-alpha.1-6142-gcdb935bb5a)
  • PR Omega: 8d0c87c0766af598ae1b8dc9f579c0cd0f2bb6e8 (Omega-v0.1.0-alpha.1-6225-g8d0c87c076)
  • Machine: pm-gpu
  • Compiler: gnugpu
  • Build type: Release
  • Log: /global/homes/s/sbrus/scratch/omega_pr_test/pr438-6f4378a/pm-gpu_gnugpu/omega_pr/polaris_omega_pr.o60006009
  • Result: All tests passed
Recent commits

Polaris:

e9e74eb869 Merge PR testing utility into main for local testing
662796b1b7 Merge pull request #864 from xylar/add-release-notes-skill
2a2c8ad4b9 Merge pull request #855 from xylar/fix-float32-diff-summary
2e0ae95f5b Merge pull request #787 from xylar/add-realistic-global-analysis-run
44c4d56012 Merge pull request #776 from xylar/check-cf-in-omega-output

Baseline Omega:

cdb935bb5a Merge pull request #600 from xylar/omega/fix-pm-mpi-flags
b90c3dae8d Merge pull request #601 from andrewdnolan/omega/cxx20-for-coupled
9641c6cf9e Merge pull request #546 from xylar/omega/fix-cf-units
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc

PR Omega:

8d0c87c076 Test merge of E3SM-Project/Omega#438 (6f4378a243) into develop (a341e65245)
a341e65245 Merge pull request #562 from andrewdnolan/omega/guard-division-by-zero
46aa54cd68 Merge pull request #603 from vanroekel/vanroekel/omega/kpp-bug-fix
9656466bb6 Merge PR #570 into develop, resolving Tendencies.cpp initializer order
d1d3ea6506 Merge pull request #466 from alicebarthel/omega/add-frazil-basic-rebased

CTest unit tests:

  • Machine: pm-gpu
  • Compiler: gnugpu
  • Build type: Release
  • Build: /global/homes/s/sbrus/scratch/omega_pr_test/pr438-6f4378a/pm-gpu_gnugpu/build
  • Omega: 8d0c87c0766af598ae1b8dc9f579c0cd0f2bb6e8 (Omega-v0.1.0-alpha.1-6225-g8d0c87c076)
  • Result: All tests passed
  • Log: /global/homes/s/sbrus/scratch/omega_pr_test/pr438-6f4378a/pm-gpu_gnugpu/build/ctests.log

Build warnings

5 new warnings, 2 in files this PR changes (853 in the PR build, 842 in the baseline build). Those in files the PR changes are marked *.

New warnings
* components/omega/src/ocn/AuxiliaryState.cpp: [#177-D] variable "MaxLayerEdgeTop" was declared but never referenced (10 -> 12)
* components/omega/src/ocn/AuxiliaryState.cpp: [#177-D] variable "MinLayerEdgeBot" was declared but never referenced (10 -> 12)
  components/omega/src/ocn/Frazil.cpp: [#177-D] variable "LocIceRefSal" was declared but never referenced (0 -> 2)
  components/omega/src/ocn/Frazil.cpp: [#177-D] variable "LocLatIce" was declared but never referenced (0 -> 2)
  components/omega/test/ocn/FrazilTest.cpp: [#177-D] variable "RTol" was declared but never referenced (0 -> 3)
  • Baseline build log: /global/homes/s/sbrus/scratch/omega_pr_test/baselines/pm-gpu_gnugpu_polaris-e9e74eb_omega-cdb935b/build/build_omega.log
  • PR build log: /global/homes/s/sbrus/scratch/omega_pr_test/pr438-6f4378a/pm-gpu_gnugpu/build/build_omega.log

Notes

On pm-cpu, the three KPP config tests (KPP_CONFIG_GRADIENT_TEST, KPP_CONFIG_UNSUPPORTED_TEST, KPP_CONFIG_PARABOLIC_TEST) were left out of the CTests (58 of 61 run). They are expected to abort, and today on pm-cpu the Slurm step stays running after MPI_Abort, so each test hangs until it times out and blocks the tests after it. They hang the same way with the build from #603's testing, where they passed this morning, so this is a pm-cpu problem, not this PR. They pass on pm-gpu.

Two of the new pm-gpu warnings are from this PR: in AuxiliaryState::computeTransportVelocity, the const auto &MinLayerEdgeBot and MaxLayerEdgeTop declared inside the SubEddies->Enable block are never used (they shadow the OMEGA_SCOPE copies above), so they can be removed. The three Frazil warnings come from #466, merged into develop after the baseline, and are unrelated to this PR.

@sbrus89

sbrus89 commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

@mwarusz, how would you like to address @xylar's comment above?

@mwarusz

mwarusz commented Oct 20, 2026

Copy link
Copy Markdown
Member Author

@sbrus89 @xylar

@mwarusz, how would you like to address @xylar's comment above?

I think this is #532. This is an issue that keeps coming up, but I don't think this PR is the right place to address it. We need a general solution.

@sbrus89
sbrus89 merged commit 135755f into E3SM-Project:develop Oct 20, 2026
8 checks passed
xylar added a commit to xylar/polaris that referenced this pull request Oct 20, 2026
The new develop head brings in Omega's submesoscale eddy
parameterization (E3SM-Project/Omega#438), which the mixed_layer_eddies
tasks validate, along with frazil, penetrating shortwave, a KPP sea-ice
buoyancy-flux fix and a division-by-zero guard.  Steven Brus's testing
of E3SM-Project#438 merged into this develop found omega_pr bit-for-bit with the
previous submodule.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xylar added a commit to xylar/polaris that referenced this pull request Oct 20, 2026
The new develop head brings in Omega's submesoscale eddy
parameterization (E3SM-Project/Omega#438), which the mixed_layer_eddies
tasks validate, along with frazil, penetrating shortwave, a KPP sea-ice
buoyancy-flux fix and a division-by-zero guard.  Steven Brus's testing
of E3SM-Project#438 merged into this develop found omega_pr bit-for-bit with the
previous submodule.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants