[hyungyukang]

This PR introduces a split-explicit time stepper for computationally efficient time integration in Omega.

• Added a framework for split-explicit time stepping.
  • Separated and modularized the baroclinic and barotropic time-stepping components.
• Added two new time steppers: SplitExplicitRK2 and UnsplitRK2.
  • SplitExplicitRK2: split time stepping using RK2 for the baroclinic time stepper
  • UnsplitRK2: unsplit RK2 time stepping.
• Added a barotropic time stepper: Predictor-Corrector
  • Predictor-Corrector: an explicitly subcycled predictor-corrector scheme.
• Added the related split time-stepping configuration options to config/Default.yml.

  TimeIntegration:
    CalendarType: No Leap
    TimeStepper: SplitExplicitRK2
    TimeStep: 0000_00:10:00
   ModeSplitShare:
      BtrTimeStepper: Predictor-Corrector
      BtrTimeStep: 0000_00:00:20
      NTimeStepIteration: 2
      NBclCoriolisIteration: 2
      ReinitSplitVelocity: false

• NormalBarotropicVelocity, NormalBaroclinicVelocity, BarotropicPressureAnomaly arrays were added to State.

Checklist

• Documentation:

  • User's Guide has been updated
  • Developer's Guide has been updated
  • Documentation has been built locally and changes look as expected

• Linting

  • Pre-commit run locally

• Building

  • CMake build does not produce any new warnings from changes in this PR

• Testing

  aurora, oneapi-ifx, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  chrysalis, oneapi-ifx, openmpi

  • CTests Pass
  • Polaris omega_pr Pass

  frontier, craygnu, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  frontier, craygnu-mphipcc, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  pm-cpu, gnu, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  pm-gpu, gnugpu, mpich

  • CTests Pass
  • Polaris omega_pr Pass

• Provide relevant details in a comment to the PR titled Testing with the following:

  • Which machines CTest unit tests
    have been run on and indicate that are all passing.
  • The Polaris omega_pr test suite
    has passed, using the Polaris e3sm_submodules/Omega baseline
  • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
  • Indicate "All tests passed" or document failing tests
  • Document testing used to verify the changes including any tests that are added/modified/impacted.

• New tests:

  • CTest unit tests for new features have been added per the approved design.
  • Polaris tests for new features have been added per the approved design (and included in a test suite)

[hyungyukang]

I still need to do some additional work on omega_pr in Polaris for the split time stepper. I will post the results once that is complete.

So far, I have tested this branch with the baroclinic channel, overflow, and realistic global ocean test (for initial performance measurements), and compared the results against MPAS-Ocean.

• Baroclinic channel (1km)
  • Surface temperature at day 30 for different time step sizes + total KE time series

• Overflow
  • Temperature at 6 hours

• Temporal convergence of time steppers
  • Baroclinic channel test (4 km)
  • Reference solution: $\Delta t=1 s$ for each time stepper
  • Error (l2 norm of 3D normal velocity b/t test and reference) evaluated after 10 minutes of integration

[hyungyukang]

Initial computational performance was measured on Frontier using the realistic global ocean test on the EC30to60E2r2 mesh. The model was integrated for one day using a 30 min time step and a 1 min barotropic time step. The reported runtimes are averaged over three repeated runs.

• Strong scaling

Since the Omega-GPU performance saturates relatively quickly, I compared Omega-GPU and Omega-CPU performance for the three main stages of the split time stepper: Stage 1 (BclVel), Stage 2 (BtrVel), and Stage 3 (ThickTracer).

As expected, the explicity subcycling barotropic velocity update (Stage2: BtrVel, no-patterned red histogram) is a major bottleneck of Omega-GPU performance since it is a communication heavy process.

[hyungyukang]

• Stage-wise runtime comparison between Frontier-GPU and PM-GPU

[hyungyukang]

Additional timing measurements on Frontier GPU nodes for several Stage 2 components, including the computations (*Corr, *Pred, and accum) and halo exchanges, show that receiveUnpack is the dominant contributor to the Stage 2 runtime.

receiveUnpack in Halo.h:

Omega/components/omega/src/base/Halo.h

Lines 851 to 897 in 9da1a59

	Pacer::start("Halo:receiveUnpack", 4);
	while (!AllReceived) {
	for (int INghbr = 0; INghbr < NNghbr; ++INghbr) {
	if (RecvFlags[CurElem][INghbr]) {
	if (!Neighbors[INghbr].Received) {
	MPI_Test(&Neighbors[INghbr].RReq, &Neighbors[INghbr].Received,
	MPI_STATUS_IGNORE);
	if (Neighbors[INghbr].Received) {
	++NRcvd;
	}
	}
	if (Neighbors[INghbr].Received && !Neighbors[INghbr].Unpacked) {
	// If the device buffer will be used in unpackBuffer, but the
	// exchange was done with the host buffer, a deep copy from
	// host to device is needed.
	if (UseDevBuffer && !ExchOnDev) {
	expandBuffer(Neighbors[INghbr].RecvBuffer,
	Neighbors[INghbr].RecvBufferH.size());
	// The number of elements we need to copy is different
	// than the buffer allocation size
	const I4 BufferSize =
	TotSize * Neighbors[INghbr].RecvLists[CurElem].NTot;
	auto CopyRange = Kokkos::make_pair(0, BufferSize);
	deepCopy(Kokkos::subview(Neighbors[INghbr].RecvBuffer,
	CopyRange),
	Kokkos::subview(Neighbors[INghbr].RecvBufferH,
	CopyRange));
	}
	unpackBuffer(Array, INghbr);
	Neighbors[INghbr].Unpacked = true;
	}
	}
	}
	if (NRcvd == NMessages) {
	AllReceived = true;
	}
	++IPass;
	if (IPass == MaxIter) {
	LOG_ERROR("Halo: Maximum iterations reached during halo exchange");
	IErr = -1;
	break;
	}
	}
	Pacer::stop("Halo:receiveUnpack", 4);

[hyungyukang]

Strong scaling across GPU nodes and stage-wise GPU performance, including Frontier-GPU, PM-GPU, and Aurora-GPU:
Thanks a lot, @amametjanov, for running the performance tests on Aurora!

[mark-petersen]

7245054 passes CPU but not GPU on Frontier:

Frontier CTest unit tests CPU:

• Machine: frontier
• Compiler: craygnu
• Build type: Release
• Result: All tests passed

Frontier CTest unit tests GPU

• Machine: frontier
• Compiler: craygnu-mphipcc
• Build type: Release
• Result:
  96% tests passed, 2 tests failed out of 48
  The following tests FAILED:
  32 - TIMESTEPPER_TEST (Failed)
  35 - DRIVER_TEST (Failed)

errors

32/48 Test #32: TIMESTEPPER_TEST ...........................***Failed    1.60 sec
1: [2026-08-28T12:44:31.393] error: Bind request gres/gpu:closest does not specify any devices within the allocation for task 1. Binding to the first device in the allocation instead.
0: [2026-08-28T12:44:31.393] error: Bind request gres/gpu:closest does not specify any devices within the allocation for task 0. Binding to the first device in the allocation instead.
0: MPI VERSION    : CRAY MPICH version 9.0.1.498 (ANL base 4.1.2)
0: MPI BUILD INFO : Wed Jul 16  9:39 2025 (git hash 0848216) (CH4)
0: [2026-08-28 12:44:31.761] [info] [TimeStepperTest.cpp:553] ----- Time Stepper Unit Test -----
1: [2026-08-28 12:44:31.761] [info] [TimeStepperTest.cpp:553] ----- Time Stepper Unit Test -----
1: Memory access fault by GPU node-4 (Agent handle: 0x15debd0) on address 0x2256000. Reason: Unknown.
1: Failed to allocate file: Bad file descriptor
1: GPU core dump failed
0: Memory access fault by GPU node-4 (Agent handle: 0x15df1c0) on address 0x23ab000. Reason: Unknown.
0: Failed to allocate file: Bad file descriptor
0: GPU core dump failed
srun: error: frontier04640: task 1: Aborted (core dumped)
srun: Terminating StepId=5367936.31
0: [2026-08-28T12:44:32.714] error: *** STEP 5367936.31 ON frontier04640 CANCELLED AT 2026-08-28T12:44:32 DUE TO TASK FAILURE ***
srun: error: frontier04640: task 0: Aborted (core dumped)

      Start 33: KOKKOS_FLATPAR_TEST
33/48 Test #33: KOKKOS_FLATPAR_TEST ........................   Passed    0.69 sec
      Start 34: KOKKOS_HIPAR_TEST
34/48 Test #34: KOKKOS_HIPAR_TEST ..........................   Passed    0.74 sec
      Start 35: DRIVER_TEST
35/48 Test #35: DRIVER_TEST ................................***Failed    1.74 sec
0: MPI VERSION    : CRAY MPICH version 9.0.1.498 (ANL base 4.1.2)
0: MPI BUILD INFO : Wed Jul 16  9:39 2025 (git hash 0848216) (CH4)
6: Memory access fault by GPU node-4 (Agent handle: 0x1b59a00) on address 0x3a76000. Reason: Unknown.
5: Memory access fault by GPU node-11 (Agent handle: 0x1b59a00) on address 0x3d83000. Reason: Unknown.
4: Memory access fault by GPU node-10 (Agent handle: 0x1b59a00) on address 0x3b17000. Reason: Unknown.
2: Memory access fault by GPU node-6 (Agent handle: 0x1b59a00) on address 0x3bfc000. Reason: Unknown.
7: Memory access fault by GPU node-5 (Agent handle: 0x1b59a00) on address 0x3c1e000. Reason: Unknown.
1: Memory access fault by GPU node-9 (Agent handle: 0x1b59a00) on address 0x3c7a000. Reason: Unknown.
0: Memory access fault by GPU node-8 (Agent handle: 0x1b59fb0) on address 0x2ef4000. Reason: Unknown.
3: Memory access fault by GPU node-7 (Agent handle: 0x1b59a00) on address 0x3d16000. Reason: Unknown.
6: Failed to allocate file: Bad file descriptor
7: Failed to allocate file: Bad file descriptor
7: GPU core dump failed
6: GPU core dump failed
4: Failed to allocate file: Bad file descriptor
4: GPU core dump failed
5: Failed to allocate file: Bad file descriptor
5: GPU core dump failed
0: Failed to allocate file: Bad file descriptor
0: GPU core dump failed
1: Failed to allocate file: Bad file descriptor
1: GPU core dump failed
3: Failed to allocate file: Bad file descriptor
3: GPU core dump failed
srun: error: frontier04640: task 7: Aborted
srun: Terminating StepId=5367936.34
0: [2026-08-28T12:44:35.386] error: *** STEP 5367936.34 ON frontier04640 CANCELLED AT 2026-08-28T12:44:35 DUE TO TASK FAILURE ***
srun: error: frontier04640: task 2: Terminated
srun: error: frontier04640: tasks 1,3-6: Aborted (core dumped)
srun: error: frontier04640: task 0: Aborted (core dumped)
srun: Force Terminated StepId=5367936.34

[mwarusz]

7245054 passes CPU but not GPU on Frontier

I will look into this today. I pushed a bunch of changes yesterday and rebased the branch on develop but I did not test it on any GPU machine.

[mwarusz]

The frontier GPU issue is now fixed.

[mark-petersen]

Yes, I tested Frontier craygnu-mphipcc and all tests now pass. Thanks!

[hyungyukang]

For more realistic testing on a high-resolution mesh, I locally merged this split-explicit time-stepping PR, the KPP mixing PR #512, and the mesh-scaling PR #506. I generated an initial condition for Omega on oRRS18to6v3 using oRRS18to6v3_80Layer.restartFromChrysalis.210321.nc available here.
I then ran the simulation on Frontier using 8 nodes (64 GPUs), and it ran stably for about three months. Here are the omega.yml configuration I used and a movie of the resulting kinetic energy. I will run the same test with MPAS-O as well for comparison.

omega.yml

Omega:
  Timing:
    Level: 2
    AutoFence: true
    TimingBarriers: false
    PrintAllRanks: false
  TimeIntegration:
    CalendarType: No Leap
    TimeStepper: SplitExplicitRK2
    TimeStep: 0000_00:05:00
    ModeSplitShare:
      BtrTimeStepper: Predictor-Corrector
      BtrTimeStep: 0000_00:00:10
      NTimeStepIteration: 2
      NBclCoriolisIteration: 2
      ReinitSplitVelocity: false
    StartTime: 0001-01-01_00:00:00
    StopTime: none
    RunDuration: 1000_00:30:00
  Decomp:
    HaloWidth: 3
    DecompMethod: MetisKWay
  IO:
    IOTasks: 8
    IOStride: 8
    IOBaseTask: 0
    IORearranger: box
    IODefaultFormat: pnetcdf
  State:
    NTimeLevels: 2
    PrescribeThicknessType: None
    PrescribeVelocityType: None
    Validation:
      AbortOnOOB: false
      AbortOnNan: true
  Advection:
    Coef3rdOrder: 0.25
    FluxThicknessType: Center
    HorzTracerFluxOrder: 2
    VerticalTracerFluxLimiterEnable: true
    VerticalTracerFluxOrder: 3
  SfcStress:
    InterpType: Isotropic
  SurfaceRestoring:
    TracersToRestore: [Temperature, Salinity]
    PistonVelocity: 1.585e-5
  PressureGrad:
    PressureGradType: Centered
  Tendencies:
    ThicknessFluxTendencyEnable: true
    PVTendencyEnable: true
    KETendencyEnable: true
    SSHTendencyEnable: false
    VelDiffTendencyEnable: false
    ViscDel2: 10.0
    VelHyperDiffTendencyEnable: true
    ViscDel4: 3.2e09
    DivFactor: 1.0
    SfcStressForcingTendencyEnable: true
    SurfaceTracerRestoringEnable: false
    BottomDragTendency:
      Enable: true
      Mode: Implicit
      Type: Constant
      BottomDragCoeff: 1.0e-3
    SfcThicknessForcingTendencyEnable: false
    SfcTracerForcingTendencyEnable: false
    TracerHorzAdvTendencyEnable: true
    TracerDiffTendencyEnable: false
    EddyDiff2: 10.0
    TracerHyperDiffTendencyEnable: false
    EddyDiff4: 0.0
    UseCustomTendency: false
    ManufacturedSolutionTendency: false
    ThicknessVertAdvTendencyEnable: true
    VelocityVertAdvTendencyEnable: true
    TracerVertAdvTendencyEnable: true
    PressureGradTendencyEnable: true
    VelVertMixTendencyEnable: true
    TracerVertMixTendencyEnable: true
    TracerNonLocalFluxTendencyEnable: true
    TracerNonLocalDiagnosticsEnable: true
  HorzMesh:
    MeshScaling:
      ScaleWithMesh: true
      UseRefWidth: false
      RefWidth: 6.0e3
      MaxMeshDensity: -1.0
  Tracers:
    Base: [Temperature, Salinity]
    Debug: []
  Eos:
    EosType: teos10
    Linear:
      DRhoDT: -0.2
      DRhoDS: 0.8
      RhoT0S0: 1000.0
  VertMix:
    Background:
      Diffusivity: 1.0e-4
      Viscosity: 1.0e-3
    Convective:
      Enable: true
      Diffusivity: 10.0
      TriggerBVF: 0.0
    Shear:
      Enable: true
      BaseShearValue: 0.05
      RiCrit: 0.7
      Exponent: 3.0
      RiSmoothLoops: 2
    KPP:
      Enable: true
      UseBLDSmoothing: true
      UseLangmuirCirculation: true
      CriticalBulkRichardsonNumber: 0.25
      SurfaceLayerExtent: 0.1
      # SimpleShapes or MatchBoth
      MatchTechnique: SimpleShapes
      InterpType2: LMD94
      UseEnhancedDiffusion: true
      IceFractionThresholdForLangmuir: 0.05
      IceFractionThresholdForMinimumOBL: 0.15
      MinimumOBLUnderSeaIce: 5.0
      BackgroundViscosity: 1.0e-4
      BackgroundDiffusivity: 1.0e-5
      DebugDiagnostics: false
  IOStreams:
    HorzMeshIn:
      UsePointerFile: false
      Filename: OmegaMesh.nc
      Mode: read
      Precision: double
      Freq: 1
      FreqUnits: OnStartup
      UseStartEnd: false
      Contents:
        - HorzMeshIn
    InitialVertCoord:
      UsePointerFile: false
      Filename: init.nc
      Mode: read
      Precision: double
      Freq: 1
      FreqUnits: OnStartup
      UseStartEnd: false
      Contents:
      - InitVertCoord
    # InitialState should only be used when starting from scratch.
    # For restart runs, the frequency units should be changed from
    # "OnStartup" to "never" so that the initial state file is not read.
    InitialState:
      UsePointerFile: false
      Filename: init.nc
      Mode: read
      Precision: double
      Freq: 1
      FreqUnits: OnStartup
      UseStartEnd: false
      Contents:
      - State
      - Tracers
    # Forcing is the external forcing in standalone mode.
    # It contains the surface stress, surface fluxes
    # and the surface restoring target files (if appl.).
    Forcing:
      UsePointerFile: false
      Filename: init.nc
      Mode: read
      Precision: double
      Freq: 1
      FreqUnits: OnStartup
      UseStartEnd: false
      Contents:
        - SfcStressZonal
        - SfcStressMeridional
    History:
      UsePointerFile: false
      Filename: output/ocn.hist.$Y-$M-$D_$h.$m.$s
      Mode: write
      IfExists: replace
      Precision: double
      Freq: 6
      FreqUnits: Hours
      UseStartEnd: false
      Contents:
       #- AuxiliaryState
       #- State
        - Tracers
        - KineticEnergyCell
      FileFreq: 6
      FileFreqUnits: Hours
    # Restarts are used to initialize for all job submissions after the very
    # first startup job. We use UseStartEnd with a start time just after the
    # simulation start time so that omega does not attempt to use a restart
    # for the first startup job.
    RestartRead:
      UsePointerFile: true
      PointerFilename: ocn.pointer
      Mode: read
      Precision: double
      Freq: 1
      FreqUnits: OnStartup
      UseStartEnd: true
      StartTime: 0001-06-01_00:00:00
      EndTime: 0001-12-31_00:00:00
      Contents:
        - Restart
        #- SfcStressZonal
        #- SfcStressMeridional
    RestartWrite:
      UsePointerFile: true
      PointerFilename: ocn.pointer
      Filename: restart/ocn.restart.$Y-$M-$D_$h.$m.$s
      Mode: write
      IfExists: replace
      Precision: double
      Freq: 1
      FreqUnits: Months
      UseStartEnd: false
      Contents:
        - Restart
        - SfcStressZonal
        - SfcStressMeridional
  Analysis:
    GlobalStats:
      Enable: false
      Fields: ["NormalVelocity", "PseudoThickness", "Temperature", "Salinity"]
      SpatialStats: ["Max", "Min", "Mean", "StdDev"]
      ReductionPeriod: ["1Day", "1Month"]
      SnapshotPeriod: ["6Hours"]
      Filename: global.stats.$Y
      Stream:
        FileFreq: 1
        FileFreqUnits: years

Omega_test_oRRS18to6v3_SE_KPP.mp4

[mwarusz]

@hyungyukang

I have made all the changes that I wanted in this PR. I think it is in a pretty good shape and can be taken out of draft mode. I ran CTests at commit e1f12bc on

• frontier (craygnu)
• frontier (craygnu-mphipcc)
• pm-cpu (gnu)
• pm-cpu (intel)
• pm-gpu (gnugpu)

and everything passed.

[hyungyukang]

@mwarusz , Thank you very much for working on this. I will test it and will change to the standard PR if the results look good. Thanks!

[hyungyukang]

I tested the split-explicit time stepper after the refactoring by @mwarusz . I ran two simulations: the 1 km baroclinic channel and the realistic global ocean test on the EC30to60E2r2 mesh. For comparison, I used two different commits representing the versions before (7245054f) and after (e1f12bcf1) the refactoring.

• Surface temperature at day 30 from the 1-km baroclinic channel

• Kinetic energy after 3 months of simulation from the realistic global ocean test on the EC30to60E2r2 mesh

The two commits do not necessarily need to be BFB, and the difference is only around 1.0E-6. Of course, the differences are not visually noticeable in the results. Thanks again @mwarusz for working on this. I'm converting this draft PR to a standard PR.

[sbrus89]

@mark-petersen, do you have any review suggestions, or are you happy approving based on your testing above?

[mark-petersen]

I tested SOMA with this branch and KPP together. Using TimeStepper: SplitExplicitRK2 the simulations run stably at the same time step as MPAS-Ocean:

32km/omega.yml:    TimeStep: 0000_00:24:00
32km/omega.yml:      BtrTimeStep: 0000_00:00:48
16km/omega.yml:    TimeStep: 0000_00:12:00
16km/omega.yml:      BtrTimeStep: 0000_00:00:24
8km/omega.yml:    TimeStep: 0000_00:06:00
8km/omega.yml:      BtrTimeStep: 0000_00:00:12
4km/omega.yml:    TimeStep: 0000_00:03:00
4km/omega.yml:      BtrTimeStep: 0000_00:00:06

Simulations are located at Perlmutter at

/pscratch/sd/m/mpeterse/runs/260908-mpaso-soma-noFCT
/pscratch/sd/m/mpeterse/runs/260908-omega-soma-rk4
/pscratch/sd/m/mpeterse/runs/260909b-omega-soma-Hyun-se-time-stepper-kpp

SOMA simulation results compare well with MPAS-Ocean using split-explicit and Omega using RK4. Here is the
32km SOMA at 100 days:
MPAS-O split explicit, dt=24m

Omega RK4, dt=4m

Omega TimeStepper: SplitExplicitRK2, dt=24m

32km SOMA at 200 days:
MPAS-O split explicit, dt=24m

Omega RK4, dt=4m

Omega TimeStepper: SplitExplicitRK2, dt=24m

The Omega split-explicit RK2 has a jet the points slightly northward. It is unclear if that is due to the chaotic nature of the fluid, or an actual difference in implementation. The flags are identical between the Omega RK4 and split-explicit, other than the time-stepping.

[mark-petersen]

Approving based on @hyungyukang's extensive testing above, including performance testing, and my testing with SOMA. Thank you Hyun!

[sbrus89]

@mark-petersen, did you happen to run omega_pr for this merge? I'll run on Frontier now.

[sbrus89]

Passes omega_pr on Frontier with both craygnu and craygnu-mphipcc:

Polaris omega_pr suite

• Baseline workdir: /ccs/home/brus/run/polaris_baseline_omega_pr_omega_craygnu_mpich/
• Baseline build: /ccs/home/brus/run/polaris_baseline_omega_pr_omega_craygnu_mpich/build
• PR build: /ccs/home/brus/run/polaris_split_explicit_omega_pr_omega_craygnu_mpich/build
• PR workdir: /ccs/home/brus/run/polaris_split_explicit_omega_pr_omega_craygnu_mpich
• Machine: frontier
• Partition: batch
• Compiler: craygnu
• Build type: Release
• Log: not found
• Result:
  • Diffs (2 of 22):
    • ocean/column/vmix_stable
    • ocean/column/ekman

Polaris omega_pr suite

• Baseline workdir: /ccs/home/brus/run/polaris_baseline_omega_pr_omega_craygnu-mphipcc_mpich
• Baseline build: /ccs/home/brus/run/polaris_baseline_omega_pr_omega_craygnu-mphipcc_mpich/build
• PR build: /ccs/home/brus/run/polaris_split_explicit_omega_pr_omega_craygnu-mphipcc_mpich/build
• PR workdir: /ccs/home/brus/run/polaris_split_explicit_omega_pr_omega_craygnu-mphipcc_mpich
• Machine: frontier
• Partition: batch
• Compiler: craygnu-mphipcc
• Build type: Release
• Log: not found
• Result:
  • Diffs (2 of 22):
    • ocean/column/vmix_stable
    • ocean/column/ekman

The diffs right now are expected due to baseline fails that are being fixed in polaris: E3SM-Project/polaris#752, E3SM-Project/polaris#750