Start vertical FCT from the tracer after horizontal advection by xylar · Pull Request #632 · E3SM-Project/Omega

xylar · 2026-10-24T08:59:26Z

Vertical FCT now starts from the tracer after horizontal advection, as in MPAS-Ocean. Before, each limiter bounded only its own update, so where both pushed a cell toward the same extreme, the sum went past it. One-day overflow runs with FCT now stay within their initial 10–20 °C range, except for up to 3e-5 °C with split-explicit RK2, whose final stage applies midpoint tendencies to time level n, as MPAS-Ocean's does.

This changes answers, at least at round-off, in runs where vertical FCT has vertical motion to act on. The overflow _nofct tasks, with vertical but not horizontal FCT, change most. Their unlimited horizontal advection takes them outside 10–20 °C within the first hour, with or without this PR. Vertical FCT now starts from those overshoots instead of the in-range old tracer, and at Polaris' 90 s step, where the vertical Courant number is about 1.5, the overshoots grow: the one-day minimum of linear z-star goes from −19.9 to −34.2 °C. At 15 s, the minima stay within about 1 °C of before.

Checklist

Fixes #630

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

🤖 Generated with Claude Code

xylar · 2026-10-24T15:19:31Z

Testing on chrysalis

Tested the merge of #632 (db8ebbb17c) into develop (23b13afac5) as 5ff3b755b8, against Polaris' Omega submodule (23b13afac5).

Row	CTests	`omega_pr`	New build warnings
chrysalis, intel, openmpi	61 of 61 passed	18 of 33 passed, 15 with baseline differences	0

chrysalis, intel, openmpi

Polaris `omega_pr` suite

Baseline workdir: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/baselines/chrysalis_intel_polaris-56a6d0e_omega-23b13af/omega_pr
Baseline build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/baselines/chrysalis_intel_polaris-56a6d0e_omega-23b13af/build
PR build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr632-db8ebbb/chrysalis_intel/build
PR workdir: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr632-db8ebbb/chrysalis_intel/omega_pr
Polaris: 56a6d0eb9ab81a9193ec6b35f13ce34cdd26592b (1.1.0-312-g56a6d0eb9)
Baseline Omega: 23b13afac5f4fbf9ba18d6cec416d347e4a615f7 (Omega-v0.1.0-alpha.1-6283-g23b13afac5)
PR Omega: 5ff3b755b8bc5842f5e7055dab9b6ac89586c8c4 (Omega-v0.1.0-alpha.1-6286-g5ff3b755b8)
Machine: chrysalis
Partition: debug
Compiler: intel
Build type: Release
Log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr632-db8ebbb/chrysalis_intel/omega_pr/polaris_omega_pr.o1314704
Result:
- Diffs (15 of 33):

    - ocean/planar/baroclinic_channel/split_explicit/10km/decomp
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.002e-11  l2=8.952e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/restart
      - temperature:    l1=1.253e-02  l2=3.922e-04  linf=6.592e-05
      - normalVelocity: l1=6.559e-04  l2=1.025e-05  linf=5.914e-07
      - salinity:       l1=4.724e-11  l2=6.382e-13  linf=2.842e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/threads
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.002e-11  l2=8.952e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/decomp
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.420e-10  linf=2.062e-11
      - salinity:       l1=3.855e-11  l2=7.099e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/restart
      - temperature:    l1=1.251e-06  l2=3.880e-08  linf=6.465e-09
      - normalVelocity: l1=7.158e-09  l2=1.192e-10  linf=6.847e-12
      - salinity:       l1=2.199e-11  l2=4.712e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/threads
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.420e-10  linf=2.062e-11
      - salinity:       l1=3.855e-11  l2=7.099e-13  linf=3.553e-14
    - ocean/planar/barotropic_channel/short
      - temperature:    l1=1.670e-12  l2=1.867e-13  linf=2.132e-14
    - ocean/planar/merry_go_round/default
      - tracer1:        l1=1.025e+01  l2=1.696e-01  linf=1.481e-02
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_2_del4
      - temperature:    l1=3.563e+01  l2=2.142e+00  linf=4.012e-01
      - normalVelocity: l1=2.270e+00  l2=5.595e-02  linf=2.821e-03
      - salinity:       l1=1.125e-09  l2=4.382e-12  linf=7.105e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_3
      - temperature:    l1=2.521e+01  l2=1.678e+00  linf=1.850e-01
      - normalVelocity: l1=3.647e+00  l2=1.491e-01  linf=2.064e-02
      - salinity:       l1=1.392e-09  l2=5.291e-12  linf=7.105e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_4
      - temperature:    l1=2.599e+01  l2=1.737e+00  linf=1.901e-01
      - normalVelocity: l1=7.424e+00  l2=2.974e-01  linf=3.247e-02
      - salinity:       l1=1.394e-09  l2=5.307e-12  linf=7.105e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_4_del4
      - temperature:    l1=2.820e+01  l2=1.849e+00  linf=2.113e-01
      - normalVelocity: l1=9.354e+00  l2=3.276e-01  linf=4.644e-02
      - salinity:       l1=1.341e-09  l2=4.972e-12  linf=7.105e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_3_nofct
      - temperature:    l1=3.967e+01  l2=2.160e+00  linf=3.017e-01
      - normalVelocity: l1=3.387e+00  l2=7.365e-02  linf=3.688e-03
      - salinity:       l1=1.322e-09  l2=4.916e-12  linf=7.105e-14
    - ocean/planar/seamount/linear/zstar/short
      - temperature:    l1=7.511e-03  l2=1.247e-03  linf=4.342e-04
      - normalVelocity: l1=1.180e-03  l2=5.409e-05  linf=8.667e-06
      - layerThickness: l1=2.243e-04  l2=2.066e-05  linf=5.796e-06
    - ocean/planar/seamount/nonlinear/sigma/short
      - temperature:    l1=5.722e-03  l2=2.011e-04  linf=2.436e-05
      - normalVelocity: l1=7.799e-04  l2=1.568e-05  linf=1.493e-06
      - layerThickness: l1=3.287e-05  l2=9.787e-07  linf=9.280e-08

Recent commits

Polaris:

56a6d0eb9 Merge pull request #891 from xylar/fix-dependency-pickle-growth
2fcfb1fe4 Merge pull request #890 from xylar/tear-down-polaris-parallel-on-sigterm
0144c23a8 Merge pull request #889 from xylar/fix-serial-memory-growth
3507767b2 Merge pull request #885 from xylar/document-dependency-branch-installs
56af77251 Merge pull request #867 from xylar/enable-omega-mesh-scaling

Baseline Omega:

23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags
eeef0a29dc Merge pull request #611 from brian-oneill/omega/analysis-restarts-and-samples

PR Omega:

5ff3b755b8 Test merge of E3SM-Project/Omega#632 (db8ebbb17c) into develop (23b13afac5)
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags

CTest unit tests:

Machine: chrysalis
Compiler: intel
Build type: Release
Build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr632-db8ebbb/chrysalis_intel/build
Omega: 5ff3b755b8bc5842f5e7055dab9b6ac89586c8c4 (Omega-v0.1.0-alpha.1-6286-g5ff3b755b8)
Result: All tests passed
Log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr632-db8ebbb/chrysalis_intel/build/ctests.log

Build warnings

No new warnings (37 in the PR build, 37 in the baseline build).

Baseline build log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/baselines/chrysalis_intel_polaris-56a6d0e_omega-23b13af/build/build_omega.log
PR build log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr632-db8ebbb/chrysalis_intel/build/build_omega.log

Notes

The 15 diffs include #628's 7, since this PR builds on it. The other 8 all use vertical FCT, which now starts from the tracer after horizontal advection; order_3_nofct changes most, as the description explains, and barotropic_channel only at round-off. Column and single-layer tasks have no vertical motion and are unchanged.

In one-day overflow runs, the tasks with FCT now stay within 10–20 °C to 3e-6 °C with RK4 and 3e-5 °C with SplitExplicitRK2 at 15 s (#628: 7e-3 and 2e-2). At Polaris' 90 s step, where the vertical Courant number is about 1.5, the linear tasks go up to 6e-3 °C past the range (#628: 6e-2), and nonlinear p-star reaches 1e-2 °C (#628: 1.6 °C) before failing after 7–8 hours (E3SM-Project#629):

The _nofct tasks leave the range within the first hour with either version; their minima change by about 1 °C or less at 15 s and by up to 14 °C at 90 s:

One-day overflow runs and plotting script: /lcrc/group/e3sm/ac.xylar/polaris_1.2/chrysalis/test_20261024/omega630-fct-bounds

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

Temporary. Testers build the parent of this commit.

xylar · 2026-10-24T22:20:40Z

Testing on chrysalis

Tested the merge of #632 (db8ebbb17c) into develop (23b13afac5) as 09ec281091, against Polaris' Omega submodule (23b13afac5).

Row	CTests	`omega_pr`	New build warnings
chrysalis, intel, openmpi	61 of 61 passed	18 of 33 passed, 15 with baseline differences	0

chrysalis, intel, openmpi

Polaris `omega_pr` suite

Baseline workdir: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/baselines/chrysalis_intel_polaris-fc3cf13_omega-23b13af/omega_pr
Baseline build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/baselines/chrysalis_intel_polaris-fc3cf13_omega-23b13af/build
PR build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr632-db8ebbb/chrysalis_intel/build
PR workdir: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr632-db8ebbb/chrysalis_intel/omega_pr
Polaris: fc3cf13cfcc5af58f7fcf8895d9128ae33d355ae (1.1.0-316-gfc3cf13cf-dirty)
Baseline Omega: 23b13afac5f4fbf9ba18d6cec416d347e4a615f7 (Omega-v0.1.0-alpha.1-6283-g23b13afac5)
PR Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Machine: chrysalis
Partition: debug
Compiler: intel
Build type: Release
Log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr632-db8ebbb/chrysalis_intel/omega_pr/polaris_omega_pr.o1314865
Result:
- Diffs (15 of 33):

    - ocean/planar/baroclinic_channel/split_explicit/10km/decomp
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.002e-11  l2=8.952e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/restart
      - temperature:    l1=1.253e-02  l2=3.922e-04  linf=6.592e-05
      - normalVelocity: l1=6.559e-04  l2=1.025e-05  linf=5.914e-07
      - salinity:       l1=4.724e-11  l2=6.382e-13  linf=2.842e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/threads
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.002e-11  l2=8.952e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/decomp
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.420e-10  linf=2.062e-11
      - salinity:       l1=3.855e-11  l2=7.099e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/restart
      - temperature:    l1=1.251e-06  l2=3.880e-08  linf=6.465e-09
      - normalVelocity: l1=7.158e-09  l2=1.192e-10  linf=6.847e-12
      - salinity:       l1=2.199e-11  l2=4.712e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/threads
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.420e-10  linf=2.062e-11
      - salinity:       l1=3.855e-11  l2=7.099e-13  linf=3.553e-14
    - ocean/planar/barotropic_channel/short
      - temperature:    l1=1.670e-12  l2=1.867e-13  linf=2.132e-14
    - ocean/planar/merry_go_round/default
      - tracer1:        l1=1.025e+01  l2=1.696e-01  linf=1.481e-02
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_2_del4
      - temperature:    l1=3.563e+01  l2=2.142e+00  linf=4.012e-01
      - normalVelocity: l1=2.270e+00  l2=5.595e-02  linf=2.821e-03
      - salinity:       l1=1.125e-09  l2=4.382e-12  linf=7.105e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_3
      - temperature:    l1=2.521e+01  l2=1.678e+00  linf=1.850e-01
      - normalVelocity: l1=3.647e+00  l2=1.491e-01  linf=2.064e-02
      - salinity:       l1=1.392e-09  l2=5.291e-12  linf=7.105e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_4
      - temperature:    l1=2.599e+01  l2=1.737e+00  linf=1.901e-01
      - normalVelocity: l1=7.424e+00  l2=2.974e-01  linf=3.247e-02
      - salinity:       l1=1.394e-09  l2=5.307e-12  linf=7.105e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_4_del4
      - temperature:    l1=2.820e+01  l2=1.849e+00  linf=2.113e-01
      - normalVelocity: l1=9.354e+00  l2=3.276e-01  linf=4.644e-02
      - salinity:       l1=1.341e-09  l2=4.972e-12  linf=7.105e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_3_nofct
      - temperature:    l1=3.967e+01  l2=2.160e+00  linf=3.017e-01
      - normalVelocity: l1=3.387e+00  l2=7.365e-02  linf=3.688e-03
      - salinity:       l1=1.322e-09  l2=4.916e-12  linf=7.105e-14
    - ocean/planar/seamount/linear/zstar/short
      - temperature:    l1=7.511e-03  l2=1.247e-03  linf=4.342e-04
      - normalVelocity: l1=1.180e-03  l2=5.409e-05  linf=8.667e-06
      - layerThickness: l1=2.243e-04  l2=2.066e-05  linf=5.796e-06
    - ocean/planar/seamount/nonlinear/sigma/short
      - temperature:    l1=5.722e-03  l2=2.011e-04  linf=2.436e-05
      - normalVelocity: l1=7.799e-04  l2=1.568e-05  linf=1.493e-06
      - layerThickness: l1=3.287e-05  l2=9.787e-07  linf=9.280e-08

Recent commits

Polaris:

fc3cf13cf Merge pull request #899 from xylar/restore-aurora-oneapi-2026
680e3a2c7 Merge pull request #893 from xylar/add-pr-test-parallel
56a6d0eb9 Merge pull request #891 from xylar/fix-dependency-pickle-growth
2fcfb1fe4 Merge pull request #890 from xylar/tear-down-polaris-parallel-on-sigterm
0144c23a8 Merge pull request #889 from xylar/fix-serial-memory-growth

Baseline Omega:

23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags
eeef0a29dc Merge pull request #611 from brian-oneill/omega/analysis-restarts-and-samples

PR Omega:

09ec281091 Test merge of E3SM-Project/Omega#632 (db8ebbb17c) into develop (23b13afac5)
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags

CTest unit tests:

Machine: chrysalis
Compiler: intel
Build type: Release
Build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr632-db8ebbb/chrysalis_intel/build
Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Result: All tests passed
Log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr632-db8ebbb/chrysalis_intel/build/ctests.log

Build warnings

No new warnings (37 in the PR build, 37 in the baseline build).

Baseline build log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/baselines/chrysalis_intel_polaris-fc3cf13_omega-23b13af/build/build_omega.log
PR build log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/omega_pr_test/pr632-db8ebbb/chrysalis_intel/build/build_omega.log

Notes

This reruns the earlier Chrysalis test on Polaris main with E3SM-Project#899. The same 15 tasks differ from the baseline, with the same maximum differences, so the explanation there still holds.

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

xylar · 2026-10-24T22:54:44Z

Testing on pm-cpu and pm-gpu

Tested the merge of #632 (db8ebbb17c) into develop (23b13afac5) as 09ec281091, against Polaris' Omega submodule (23b13afac5).

Row	CTests	`omega_pr`	New build warnings
pm-cpu, gnu, mpich	61 of 61 passed	18 of 33 passed, 15 with baseline differences	0
pm-gpu, gnugpu, mpich	61 of 61 passed	18 of 33 passed, 15 with baseline differences	0

pm-cpu, gnu, mpich

Polaris `omega_pr` suite

Baseline workdir: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-cpu_gnu_polaris-fc3cf13_omega-23b13af/omega_pr
Baseline build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-cpu_gnu_polaris-fc3cf13_omega-23b13af/build
PR build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr632-db8ebbb/pm-cpu_gnu/build
PR workdir: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr632-db8ebbb/pm-cpu_gnu/omega_pr
Polaris: fc3cf13cfcc5af58f7fcf8895d9128ae33d355ae (1.1.0-316-gfc3cf13cf-dirty)
Baseline Omega: 23b13afac5f4fbf9ba18d6cec416d347e4a615f7 (Omega-v0.1.0-alpha.1-6283-g23b13afac5)
PR Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Machine: pm-cpu
Compiler: gnu
Build type: Release
Log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr632-db8ebbb/pm-cpu_gnu/omega_pr/polaris_omega_pr.o60225737
Result:
- Diffs (15 of 33):

    - ocean/planar/baroclinic_channel/split_explicit/10km/decomp
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.002e-11  l2=8.952e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/restart
      - temperature:    l1=1.253e-02  l2=3.922e-04  linf=6.592e-05
      - normalVelocity: l1=6.559e-04  l2=1.025e-05  linf=5.914e-07
      - salinity:       l1=4.724e-11  l2=6.382e-13  linf=2.842e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/threads
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.002e-11  l2=8.952e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/decomp
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.420e-10  linf=2.062e-11
      - salinity:       l1=3.855e-11  l2=7.099e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/restart
      - temperature:    l1=1.251e-06  l2=3.880e-08  linf=6.465e-09
      - normalVelocity: l1=7.158e-09  l2=1.192e-10  linf=6.847e-12
      - salinity:       l1=2.199e-11  l2=4.712e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/threads
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.420e-10  linf=2.062e-11
      - salinity:       l1=3.855e-11  l2=7.099e-13  linf=3.553e-14
    - ocean/planar/barotropic_channel/short
      - temperature:    l1=1.670e-12  l2=1.867e-13  linf=2.132e-14
    - ocean/planar/merry_go_round/default
      - tracer1:        l1=1.025e+01  l2=1.696e-01  linf=1.481e-02
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_2_del4
      - temperature:    l1=3.563e+01  l2=2.142e+00  linf=4.012e-01
      - normalVelocity: l1=2.270e+00  l2=5.595e-02  linf=2.821e-03
      - salinity:       l1=1.125e-09  l2=4.382e-12  linf=7.105e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_3
      - temperature:    l1=2.521e+01  l2=1.678e+00  linf=1.850e-01
      - normalVelocity: l1=3.647e+00  l2=1.491e-01  linf=2.064e-02
      - salinity:       l1=1.392e-09  l2=5.291e-12  linf=7.105e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_4
      - temperature:    l1=2.599e+01  l2=1.737e+00  linf=1.901e-01
      - normalVelocity: l1=7.424e+00  l2=2.974e-01  linf=3.247e-02
      - salinity:       l1=1.394e-09  l2=5.307e-12  linf=7.105e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_4_del4
      - temperature:    l1=2.820e+01  l2=1.849e+00  linf=2.113e-01
      - normalVelocity: l1=9.354e+00  l2=3.276e-01  linf=4.644e-02
      - salinity:       l1=1.341e-09  l2=4.972e-12  linf=7.105e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_3_nofct
      - temperature:    l1=3.967e+01  l2=2.160e+00  linf=3.017e-01
      - normalVelocity: l1=3.387e+00  l2=7.365e-02  linf=3.688e-03
      - salinity:       l1=1.322e-09  l2=4.916e-12  linf=7.105e-14
    - ocean/planar/seamount/linear/zstar/short
      - temperature:    l1=7.511e-03  l2=1.247e-03  linf=4.342e-04
      - normalVelocity: l1=1.180e-03  l2=5.409e-05  linf=8.667e-06
      - layerThickness: l1=2.243e-04  l2=2.066e-05  linf=5.796e-06
    - ocean/planar/seamount/nonlinear/sigma/short
      - temperature:    l1=5.722e-03  l2=2.011e-04  linf=2.436e-05
      - normalVelocity: l1=7.799e-04  l2=1.568e-05  linf=1.493e-06
      - layerThickness: l1=3.287e-05  l2=9.787e-07  linf=9.280e-08

Recent commits

Polaris:

fc3cf13cf Merge pull request #899 from xylar/restore-aurora-oneapi-2026
680e3a2c7 Merge pull request #893 from xylar/add-pr-test-parallel
56a6d0eb9 Merge pull request #891 from xylar/fix-dependency-pickle-growth
2fcfb1fe4 Merge pull request #890 from xylar/tear-down-polaris-parallel-on-sigterm
0144c23a8 Merge pull request #889 from xylar/fix-serial-memory-growth

Baseline Omega:

23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags
eeef0a29dc Merge pull request #611 from brian-oneill/omega/analysis-restarts-and-samples

PR Omega:

09ec281091 Test merge of E3SM-Project/Omega#632 (db8ebbb17c) into develop (23b13afac5)
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags

CTest unit tests:

Machine: pm-cpu
Compiler: gnu
Build type: Release
Build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr632-db8ebbb/pm-cpu_gnu/build
Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Result: All tests passed
Log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr632-db8ebbb/pm-cpu_gnu/build/ctests.log

Build warnings

No new warnings (4 in the PR build, 4 in the baseline build).

Baseline build log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-cpu_gnu_polaris-fc3cf13_omega-23b13af/build/build_omega.log
PR build log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr632-db8ebbb/pm-cpu_gnu/build/build_omega.log

pm-gpu, gnugpu, mpich

Polaris `omega_pr` suite

Baseline workdir: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-gpu_gnugpu_polaris-fc3cf13_omega-23b13af/omega_pr
Baseline build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-gpu_gnugpu_polaris-fc3cf13_omega-23b13af/build
PR build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr632-db8ebbb/pm-gpu_gnugpu/build
PR workdir: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr632-db8ebbb/pm-gpu_gnugpu/omega_pr
Polaris: fc3cf13cfcc5af58f7fcf8895d9128ae33d355ae (1.1.0-316-gfc3cf13cf-dirty)
Baseline Omega: 23b13afac5f4fbf9ba18d6cec416d347e4a615f7 (Omega-v0.1.0-alpha.1-6283-g23b13afac5)
PR Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Machine: pm-gpu
Compiler: gnugpu
Build type: Release
Log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr632-db8ebbb/pm-gpu_gnugpu/omega_pr/polaris_omega_pr.o60226174
Result:
- Diffs (15 of 33):

    - ocean/planar/baroclinic_channel/split_explicit/10km/decomp
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.539e-11  l2=9.410e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/restart
      - temperature:    l1=1.253e-02  l2=3.922e-04  linf=6.592e-05
      - normalVelocity: l1=6.559e-04  l2=1.025e-05  linf=5.914e-07
      - salinity:       l1=5.500e-11  l2=7.002e-13  linf=2.842e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/threads
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.539e-11  l2=9.410e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/decomp
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.421e-10  linf=2.062e-11
      - salinity:       l1=4.151e-11  l2=7.603e-13  linf=4.974e-14
    - ocean/planar/baroclinic_channel/rk4/10km/restart
      - temperature:    l1=1.251e-06  l2=3.880e-08  linf=6.465e-09
      - normalVelocity: l1=7.160e-09  l2=1.192e-10  linf=6.858e-12
      - salinity:       l1=2.509e-11  l2=5.206e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/threads
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.421e-10  linf=2.062e-11
      - salinity:       l1=4.151e-11  l2=7.603e-13  linf=4.974e-14
    - ocean/planar/barotropic_channel/short
      - temperature:    l1=1.754e-12  l2=1.969e-13  linf=2.798e-14
    - ocean/planar/merry_go_round/default
      - tracer1:        l1=1.025e+01  l2=1.696e-01  linf=1.481e-02
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_2_del4
      - temperature:    l1=3.563e+01  l2=2.142e+00  linf=4.012e-01
      - normalVelocity: l1=2.268e+00  l2=5.594e-02  linf=2.821e-03
      - salinity:       l1=1.452e-09  l2=5.607e-12  linf=8.527e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_3
      - temperature:    l1=2.521e+01  l2=1.679e+00  linf=1.866e-01
      - normalVelocity: l1=3.660e+00  l2=1.524e-01  linf=2.071e-02
      - salinity:       l1=1.783e-09  l2=6.926e-12  linf=9.237e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_4
      - temperature:    l1=2.600e+01  l2=1.738e+00  linf=1.901e-01
      - normalVelocity: l1=7.392e+00  l2=2.970e-01  linf=3.247e-02
      - salinity:       l1=1.779e-09  l2=6.732e-12  linf=8.527e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_4_del4
      - temperature:    l1=2.827e+01  l2=1.852e+00  linf=2.132e-01
      - normalVelocity: l1=9.078e+00  l2=3.119e-01  linf=4.001e-02
      - salinity:       l1=1.713e-09  l2=6.381e-12  linf=9.237e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_3_nofct
      - temperature:    l1=3.967e+01  l2=2.160e+00  linf=3.017e-01
      - normalVelocity: l1=3.368e+00  l2=7.358e-02  linf=3.688e-03
      - salinity:       l1=1.666e-09  l2=6.209e-12  linf=9.237e-14
    - ocean/planar/seamount/linear/zstar/short
      - temperature:    l1=7.511e-03  l2=1.247e-03  linf=4.342e-04
      - normalVelocity: l1=1.180e-03  l2=5.409e-05  linf=8.667e-06
      - layerThickness: l1=2.243e-04  l2=2.066e-05  linf=5.796e-06
    - ocean/planar/seamount/nonlinear/sigma/short
      - temperature:    l1=5.722e-03  l2=2.011e-04  linf=2.436e-05
      - normalVelocity: l1=7.798e-04  l2=1.568e-05  linf=1.493e-06
      - layerThickness: l1=3.287e-05  l2=9.787e-07  linf=9.280e-08

Recent commits

Polaris:

fc3cf13cf Merge pull request #899 from xylar/restore-aurora-oneapi-2026
680e3a2c7 Merge pull request #893 from xylar/add-pr-test-parallel
56a6d0eb9 Merge pull request #891 from xylar/fix-dependency-pickle-growth
2fcfb1fe4 Merge pull request #890 from xylar/tear-down-polaris-parallel-on-sigterm
0144c23a8 Merge pull request #889 from xylar/fix-serial-memory-growth

Baseline Omega:

23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags
eeef0a29dc Merge pull request #611 from brian-oneill/omega/analysis-restarts-and-samples

PR Omega:

09ec281091 Test merge of E3SM-Project/Omega#632 (db8ebbb17c) into develop (23b13afac5)
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags

CTest unit tests:

Machine: pm-gpu
Compiler: gnugpu
Build type: Release
Build: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr632-db8ebbb/pm-gpu_gnugpu/build
Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Result: All tests passed
Log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr632-db8ebbb/pm-gpu_gnugpu/build/ctests.log

Build warnings

No new warnings (852 in the PR build, 852 in the baseline build).

Baseline build log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/baselines/pm-gpu_gnugpu_polaris-fc3cf13_omega-23b13af/build/build_omega.log
PR build log: /pscratch/sd/x/xylar/polaris_1.1/omega_pr_test/pr632-db8ebbb/pm-gpu_gnugpu/build/build_omega.log

Notes

The same 15 tasks differ from the baseline as on Chrysalis, all expected from this PR and #628, which it builds on; the earlier Chrysalis test explains them. Every task runs and passes its CF checks.

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

xylar · 2026-10-24T22:54:46Z

Testing on aurora

Tested the merge of #632 (db8ebbb17c) into develop (23b13afac5) as 09ec281091, against Polaris' Omega submodule (23b13afac5).

Row	CTests	`omega_pr`	New build warnings
aurora, intel, mpich	61 of 61 passed	18 of 33 passed, 15 with baseline differences	0
aurora, intelgpu, mpich	61 of 61 passed	18 of 33 passed, 15 with baseline differences	0

aurora, intel, mpich

Polaris `omega_pr` suite

Baseline workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intel_polaris-fc3cf13_omega-23b13af/omega_pr
Baseline build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intel_polaris-fc3cf13_omega-23b13af/build
PR build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr632-db8ebbb/aurora_intel/build
PR workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr632-db8ebbb/aurora_intel/omega_pr
Polaris: fc3cf13cfcc5af58f7fcf8895d9128ae33d355ae (1.1.0-316-gfc3cf13cf-dirty)
Baseline Omega: 23b13afac5f4fbf9ba18d6cec416d347e4a615f7 (Omega-v0.1.0-alpha.1-6283-g23b13afac5)
PR Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Machine: aurora
Compiler: intel
Build type: Release
Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr632-db8ebbb/aurora_intel/omega_pr/polaris_omega_pr.o8961368
Result:
- Diffs (15 of 33):

    - ocean/planar/baroclinic_channel/split_explicit/10km/decomp
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.002e-11  l2=8.952e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/restart
      - temperature:    l1=1.253e-02  l2=3.922e-04  linf=6.592e-05
      - normalVelocity: l1=6.559e-04  l2=1.025e-05  linf=5.914e-07
      - salinity:       l1=4.724e-11  l2=6.382e-13  linf=2.842e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/threads
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.002e-11  l2=8.952e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/decomp
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.420e-10  linf=2.062e-11
      - salinity:       l1=3.855e-11  l2=7.099e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/restart
      - temperature:    l1=1.251e-06  l2=3.880e-08  linf=6.465e-09
      - normalVelocity: l1=7.158e-09  l2=1.192e-10  linf=6.847e-12
      - salinity:       l1=2.199e-11  l2=4.712e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/threads
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.420e-10  linf=2.062e-11
      - salinity:       l1=3.855e-11  l2=7.099e-13  linf=3.553e-14
    - ocean/planar/barotropic_channel/short
      - temperature:    l1=1.670e-12  l2=1.867e-13  linf=2.132e-14
    - ocean/planar/merry_go_round/default
      - tracer1:        l1=1.025e+01  l2=1.696e-01  linf=1.481e-02
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_2_del4
      - temperature:    l1=3.563e+01  l2=2.142e+00  linf=4.012e-01
      - normalVelocity: l1=2.270e+00  l2=5.595e-02  linf=2.821e-03
      - salinity:       l1=1.125e-09  l2=4.382e-12  linf=7.105e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_3
      - temperature:    l1=2.521e+01  l2=1.678e+00  linf=1.850e-01
      - normalVelocity: l1=3.647e+00  l2=1.491e-01  linf=2.064e-02
      - salinity:       l1=1.392e-09  l2=5.291e-12  linf=7.105e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_4
      - temperature:    l1=2.599e+01  l2=1.737e+00  linf=1.901e-01
      - normalVelocity: l1=7.424e+00  l2=2.974e-01  linf=3.247e-02
      - salinity:       l1=1.394e-09  l2=5.307e-12  linf=7.105e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_4_del4
      - temperature:    l1=2.820e+01  l2=1.849e+00  linf=2.113e-01
      - normalVelocity: l1=9.354e+00  l2=3.276e-01  linf=4.644e-02
      - salinity:       l1=1.341e-09  l2=4.972e-12  linf=7.105e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_3_nofct
      - temperature:    l1=3.967e+01  l2=2.160e+00  linf=3.017e-01
      - normalVelocity: l1=3.387e+00  l2=7.365e-02  linf=3.688e-03
      - salinity:       l1=1.322e-09  l2=4.916e-12  linf=7.105e-14
    - ocean/planar/seamount/linear/zstar/short
      - temperature:    l1=7.511e-03  l2=1.247e-03  linf=4.342e-04
      - normalVelocity: l1=1.180e-03  l2=5.409e-05  linf=8.667e-06
      - layerThickness: l1=2.243e-04  l2=2.066e-05  linf=5.796e-06
    - ocean/planar/seamount/nonlinear/sigma/short
      - temperature:    l1=5.722e-03  l2=2.011e-04  linf=2.436e-05
      - normalVelocity: l1=7.799e-04  l2=1.568e-05  linf=1.493e-06
      - layerThickness: l1=3.287e-05  l2=9.787e-07  linf=9.280e-08

Recent commits

Polaris:

fc3cf13cf Merge pull request #899 from xylar/restore-aurora-oneapi-2026
680e3a2c7 Merge pull request #893 from xylar/add-pr-test-parallel
56a6d0eb9 Merge pull request #891 from xylar/fix-dependency-pickle-growth
2fcfb1fe4 Merge pull request #890 from xylar/tear-down-polaris-parallel-on-sigterm
0144c23a8 Merge pull request #889 from xylar/fix-serial-memory-growth

Baseline Omega:

23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags
eeef0a29dc Merge pull request #611 from brian-oneill/omega/analysis-restarts-and-samples

PR Omega:

09ec281091 Test merge of E3SM-Project/Omega#632 (db8ebbb17c) into develop (23b13afac5)
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags

CTest unit tests:

Machine: aurora
Compiler: intel
Build type: Release
Build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr632-db8ebbb/aurora_intel/build
Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Result: All tests passed
Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr632-db8ebbb/aurora_intel/build/ctests.log

Build warnings

No new warnings (14 in the PR build, 14 in the baseline build).

Baseline build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intel_polaris-fc3cf13_omega-23b13af/build/build_omega.log
PR build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr632-db8ebbb/aurora_intel/build/build_omega.log

aurora, intelgpu, mpich

Polaris `omega_pr` suite

Baseline workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intelgpu_polaris-fc3cf13_omega-23b13af/omega_pr
Baseline build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intelgpu_polaris-fc3cf13_omega-23b13af/build
PR build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr632-db8ebbb/aurora_intelgpu/build
PR workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr632-db8ebbb/aurora_intelgpu/omega_pr
Polaris: fc3cf13cfcc5af58f7fcf8895d9128ae33d355ae (1.1.0-316-gfc3cf13cf-dirty)
Baseline Omega: 23b13afac5f4fbf9ba18d6cec416d347e4a615f7 (Omega-v0.1.0-alpha.1-6283-g23b13afac5)
PR Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Machine: aurora
Compiler: intelgpu
Build type: Release
Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr632-db8ebbb/aurora_intelgpu/omega_pr/polaris_omega_pr.o8961441
Result:
- Diffs (15 of 33):

    - ocean/planar/baroclinic_channel/split_explicit/10km/decomp
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.599e-11  l2=9.464e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/restart
      - temperature:    l1=1.253e-02  l2=3.922e-04  linf=6.592e-05
      - normalVelocity: l1=6.559e-04  l2=1.025e-05  linf=5.914e-07
      - salinity:       l1=5.524e-11  l2=7.023e-13  linf=2.842e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/threads
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.599e-11  l2=9.464e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/decomp
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.421e-10  linf=2.062e-11
      - salinity:       l1=4.221e-11  l2=7.699e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/restart
      - temperature:    l1=1.251e-06  l2=3.880e-08  linf=6.465e-09
      - normalVelocity: l1=7.160e-09  l2=1.192e-10  linf=6.850e-12
      - salinity:       l1=2.515e-11  l2=5.250e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/threads
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.421e-10  linf=2.062e-11
      - salinity:       l1=4.221e-11  l2=7.699e-13  linf=3.553e-14
    - ocean/planar/barotropic_channel/short
      - temperature:    l1=1.757e-12  l2=1.968e-13  linf=2.642e-14
    - ocean/planar/merry_go_round/default
      - tracer1:        l1=1.025e+01  l2=1.696e-01  linf=1.481e-02
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_2_del4
      - temperature:    l1=3.563e+01  l2=2.142e+00  linf=4.012e-01
      - normalVelocity: l1=2.268e+00  l2=5.594e-02  linf=2.821e-03
      - salinity:       l1=1.438e-09  l2=5.559e-12  linf=7.816e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_3
      - temperature:    l1=2.521e+01  l2=1.679e+00  linf=1.866e-01
      - normalVelocity: l1=3.659e+00  l2=1.522e-01  linf=2.071e-02
      - salinity:       l1=1.774e-09  l2=6.903e-12  linf=9.237e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_4
      - temperature:    l1=2.600e+01  l2=1.738e+00  linf=1.901e-01
      - normalVelocity: l1=7.392e+00  l2=2.970e-01  linf=3.247e-02
      - salinity:       l1=1.786e-09  l2=6.728e-12  linf=9.237e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_4_del4
      - temperature:    l1=2.834e+01  l2=1.857e+00  linf=2.166e-01
      - normalVelocity: l1=9.556e+00  l2=3.327e-01  linf=4.535e-02
      - salinity:       l1=1.721e-09  l2=6.386e-12  linf=9.948e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_3_nofct
      - temperature:    l1=3.967e+01  l2=2.160e+00  linf=3.017e-01
      - normalVelocity: l1=3.369e+00  l2=7.368e-02  linf=3.688e-03
      - salinity:       l1=1.656e-09  l2=6.177e-12  linf=9.237e-14
    - ocean/planar/seamount/linear/zstar/short
      - temperature:    l1=7.511e-03  l2=1.247e-03  linf=4.342e-04
      - normalVelocity: l1=1.180e-03  l2=5.409e-05  linf=8.667e-06
      - layerThickness: l1=2.243e-04  l2=2.066e-05  linf=5.796e-06
    - ocean/planar/seamount/nonlinear/sigma/short
      - temperature:    l1=5.722e-03  l2=2.011e-04  linf=2.436e-05
      - normalVelocity: l1=7.798e-04  l2=1.568e-05  linf=1.493e-06
      - layerThickness: l1=3.287e-05  l2=9.787e-07  linf=9.280e-08

Recent commits

Polaris:

fc3cf13cf Merge pull request #899 from xylar/restore-aurora-oneapi-2026
680e3a2c7 Merge pull request #893 from xylar/add-pr-test-parallel
56a6d0eb9 Merge pull request #891 from xylar/fix-dependency-pickle-growth
2fcfb1fe4 Merge pull request #890 from xylar/tear-down-polaris-parallel-on-sigterm
0144c23a8 Merge pull request #889 from xylar/fix-serial-memory-growth

Baseline Omega:

23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags
eeef0a29dc Merge pull request #611 from brian-oneill/omega/analysis-restarts-and-samples

PR Omega:

09ec281091 Test merge of E3SM-Project/Omega#632 (db8ebbb17c) into develop (23b13afac5)
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags

CTest unit tests:

Machine: aurora
Compiler: intelgpu
Build type: Release
Build: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr632-db8ebbb/aurora_intelgpu/build
Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Result: All tests passed
Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr632-db8ebbb/aurora_intelgpu/build/ctests.log

Build warnings

No new warnings (236 in the PR build, 258 in the baseline build).

Baseline build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/baselines/aurora_intelgpu_polaris-fc3cf13_omega-23b13af/build/build_omega.log
PR build log: /lus/flare/projects/E3SM_Dec/xylar/polaris_1.1/aurora/omega_pr_test/pr632-db8ebbb/aurora_intelgpu/build/build_omega.log

Notes

The same 15 tasks differ from the baseline as on Chrysalis, all expected from this PR and #628, which it builds on; the earlier Chrysalis test explains them. Every task runs and passes its CF checks.

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

xylar · 2026-10-24T23:02:40Z

Testing on frontier

Tested the merge of #632 (db8ebbb17c) into develop (23b13afac5) as 09ec281091, against Polaris' Omega submodule (23b13afac5).

Row	CTests	`omega_pr`	New build warnings
frontier, craygnu, mpich	61 of 61 passed	16 of 33 passed, 17 with baseline differences	0
frontier, craygnu-mphipcc, mpich	61 of 61 passed	18 of 33 passed, 15 with baseline differences	0

frontier, craygnu, mpich

Polaris `omega_pr` suite

Baseline workdir: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu_polaris-fc3cf13_omega-23b13af/omega_pr
Baseline build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu_polaris-fc3cf13_omega-23b13af/build
PR build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr632-db8ebbb/frontier_craygnu/build
PR workdir: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr632-db8ebbb/frontier_craygnu/omega_pr
Polaris: fc3cf13cfcc5af58f7fcf8895d9128ae33d355ae (1.1.0-316-gfc3cf13cf-dirty)
Baseline Omega: 23b13afac5f4fbf9ba18d6cec416d347e4a615f7 (Omega-v0.1.0-alpha.1-6283-g23b13afac5)
PR Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Machine: frontier
Partition: batch
Compiler: craygnu
Build type: Release
Log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr632-db8ebbb/frontier_craygnu/omega_pr/polaris_omega_pr.o5756299
Result:
- Diffs (17 of 33):

    - ocean/planar/baroclinic_channel/split_explicit/10km/decomp
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=6.267e-11  l2=7.532e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/restart
      - temperature:    l1=1.253e-02  l2=3.922e-04  linf=6.592e-05
      - normalVelocity: l1=6.559e-04  l2=1.025e-05  linf=5.914e-07
      - salinity:       l1=3.678e-11  l2=5.459e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/threads
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=6.267e-11  l2=7.532e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/decomp
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.420e-10  linf=2.062e-11
      - salinity:       l1=3.772e-11  l2=7.043e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/restart
      - temperature:    l1=1.251e-06  l2=3.880e-08  linf=6.465e-09
      - normalVelocity: l1=7.159e-09  l2=1.192e-10  linf=6.844e-12
      - salinity:       l1=2.169e-11  l2=4.695e-13  linf=2.842e-14
    - ocean/planar/baroclinic_channel/rk4/10km/threads
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.420e-10  linf=2.062e-11
      - salinity:       l1=3.772e-11  l2=7.043e-13  linf=3.553e-14
    - ocean/planar/barotropic_channel/short
      - temperature:    l1=1.633e-12  l2=1.833e-13  linf=2.487e-14
    - ocean/planar/merry_go_round/default
      - tracer1:        l1=1.025e+01  l2=1.696e-01  linf=1.481e-02
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_2_del4
      - temperature:    l1=3.563e+01  l2=2.142e+00  linf=4.012e-01
      - normalVelocity: l1=2.271e+00  l2=5.595e-02  linf=2.821e-03
      - salinity:       l1=9.478e-10  l2=3.750e-12  linf=6.395e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_3
      - temperature:    l1=2.521e+01  l2=1.678e+00  linf=1.850e-01
      - normalVelocity: l1=3.647e+00  l2=1.491e-01  linf=2.064e-02
      - salinity:       l1=1.202e-09  l2=4.639e-12  linf=6.395e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_4
      - temperature:    l1=2.598e+01  l2=1.736e+00  linf=1.900e-01
      - normalVelocity: l1=7.369e+00  l2=2.966e-01  linf=3.247e-02
      - salinity:       l1=1.211e-09  l2=4.714e-12  linf=6.395e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_4_del4
      - temperature:    l1=2.805e+01  l2=1.840e+00  linf=2.200e-01
      - normalVelocity: l1=9.381e+00  l2=3.222e-01  linf=4.492e-02
      - salinity:       l1=1.152e-09  l2=4.300e-12  linf=6.395e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_3_nofct
      - temperature:    l1=3.967e+01  l2=2.160e+00  linf=3.017e-01
      - normalVelocity: l1=3.375e+00  l2=7.361e-02  linf=3.688e-03
      - salinity:       l1=1.109e-09  l2=4.163e-12  linf=7.105e-14
    - ocean/planar/seamount/linear/zstar/short
      - temperature:    l1=7.511e-03  l2=1.247e-03  linf=4.342e-04
      - normalVelocity: l1=1.180e-03  l2=5.409e-05  linf=8.667e-06
      - layerThickness: l1=2.243e-04  l2=2.066e-05  linf=5.796e-06
    - ocean/planar/seamount/nonlinear/sigma/short
      - temperature:    l1=5.722e-03  l2=2.011e-04  linf=2.436e-05
      - normalVelocity: l1=7.799e-04  l2=1.568e-05  linf=1.493e-06
      - layerThickness: l1=3.287e-05  l2=9.787e-07  linf=9.280e-08
    - ocean/spherical/icos/cosine_bell/decomp
      - tracer1:        l1=3.747e-14  l2=6.229e-15  linf=2.331e-15
    - ocean/spherical/icos/cosine_bell/restart
      - tracer1:        l1=4.838e-21  l2=3.526e-21  linf=3.388e-21

Recent commits

Polaris:

fc3cf13cf Merge pull request #899 from xylar/restore-aurora-oneapi-2026
680e3a2c7 Merge pull request #893 from xylar/add-pr-test-parallel
56a6d0eb9 Merge pull request #891 from xylar/fix-dependency-pickle-growth
2fcfb1fe4 Merge pull request #890 from xylar/tear-down-polaris-parallel-on-sigterm
0144c23a8 Merge pull request #889 from xylar/fix-serial-memory-growth

Baseline Omega:

23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags
eeef0a29dc Merge pull request #611 from brian-oneill/omega/analysis-restarts-and-samples

PR Omega:

09ec281091 Test merge of E3SM-Project/Omega#632 (db8ebbb17c) into develop (23b13afac5)
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags

CTest unit tests:

Machine: frontier
Compiler: craygnu
Build type: Release
Build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr632-db8ebbb/frontier_craygnu/build
Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Result: All tests passed
Log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr632-db8ebbb/frontier_craygnu/build/ctests.log

Build warnings

No new warnings (3 in the PR build, 3 in the baseline build).

Baseline build log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu_polaris-fc3cf13_omega-23b13af/build/build_omega.log
PR build log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr632-db8ebbb/frontier_craygnu/build/build_omega.log

frontier, craygnu-mphipcc, mpich

Polaris `omega_pr` suite

Baseline workdir: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-fc3cf13_omega-23b13af/omega_pr
Baseline build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-fc3cf13_omega-23b13af/build
PR build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr632-db8ebbb/frontier_craygnu-mphipcc/build
PR workdir: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr632-db8ebbb/frontier_craygnu-mphipcc/omega_pr
Polaris: fc3cf13cfcc5af58f7fcf8895d9128ae33d355ae (1.1.0-316-gfc3cf13cf-dirty)
Baseline Omega: 23b13afac5f4fbf9ba18d6cec416d347e4a615f7 (Omega-v0.1.0-alpha.1-6283-g23b13afac5)
PR Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Machine: frontier
Partition: batch
Compiler: craygnu-mphipcc
Build type: Release
Log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr632-db8ebbb/frontier_craygnu-mphipcc/omega_pr/polaris_omega_pr.o5756412
Result:
- Diffs (15 of 33):

    - ocean/planar/baroclinic_channel/split_explicit/10km/decomp
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.622e-11  l2=9.485e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/restart
      - temperature:    l1=1.253e-02  l2=3.922e-04  linf=6.592e-05
      - normalVelocity: l1=6.559e-04  l2=1.025e-05  linf=5.914e-07
      - salinity:       l1=5.577e-11  l2=7.050e-13  linf=2.842e-14
    - ocean/planar/baroclinic_channel/split_explicit/10km/threads
      - temperature:    l1=4.181e-02  l2=1.310e-03  linf=2.210e-04
      - normalVelocity: l1=3.086e-03  l2=4.763e-05  linf=2.606e-06
      - salinity:       l1=8.622e-11  l2=9.485e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/decomp
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.421e-10  linf=2.062e-11
      - salinity:       l1=4.157e-11  l2=7.685e-13  linf=4.263e-14
    - ocean/planar/baroclinic_channel/rk4/10km/restart
      - temperature:    l1=1.251e-06  l2=3.880e-08  linf=6.465e-09
      - normalVelocity: l1=7.159e-09  l2=1.192e-10  linf=6.849e-12
      - salinity:       l1=2.525e-11  l2=5.294e-13  linf=3.553e-14
    - ocean/planar/baroclinic_channel/rk4/10km/threads
      - temperature:    l1=2.774e-06  l2=8.889e-08  linf=1.578e-08
      - normalVelocity: l1=2.112e-08  l2=3.421e-10  linf=2.062e-11
      - salinity:       l1=4.157e-11  l2=7.685e-13  linf=4.263e-14
    - ocean/planar/barotropic_channel/short
      - temperature:    l1=1.745e-12  l2=1.958e-13  linf=2.720e-14
    - ocean/planar/merry_go_round/default
      - tracer1:        l1=1.025e+01  l2=1.696e-01  linf=1.481e-02
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_2_del4
      - temperature:    l1=3.563e+01  l2=2.142e+00  linf=4.012e-01
      - normalVelocity: l1=2.268e+00  l2=5.594e-02  linf=2.821e-03
      - salinity:       l1=1.435e-09  l2=5.560e-12  linf=8.527e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_3
      - temperature:    l1=2.521e+01  l2=1.679e+00  linf=1.866e-01
      - normalVelocity: l1=3.659e+00  l2=1.522e-01  linf=2.071e-02
      - salinity:       l1=1.781e-09  l2=6.895e-12  linf=9.948e-14
    - ocean/planar/overflow/linear/zstar/smoke_test_horiz_adv_order_4
      - temperature:    l1=2.600e+01  l2=1.738e+00  linf=1.901e-01
      - normalVelocity: l1=7.392e+00  l2=2.970e-01  linf=3.247e-02
      - salinity:       l1=1.789e-09  l2=6.766e-12  linf=8.527e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_4_del4
      - temperature:    l1=2.836e+01  l2=1.866e+00  linf=2.178e-01
      - normalVelocity: l1=1.003e+01  l2=3.467e-01  linf=5.011e-02
      - salinity:       l1=1.706e-09  l2=6.355e-12  linf=9.237e-14
    - ocean/planar/overflow/nonlinear/pstar/smoke_test_horiz_adv_order_3_nofct
      - temperature:    l1=3.967e+01  l2=2.160e+00  linf=3.017e-01
      - normalVelocity: l1=3.342e+00  l2=7.352e-02  linf=3.688e-03
      - salinity:       l1=1.666e-09  l2=6.206e-12  linf=9.948e-14
    - ocean/planar/seamount/linear/zstar/short
      - temperature:    l1=7.511e-03  l2=1.247e-03  linf=4.342e-04
      - normalVelocity: l1=1.180e-03  l2=5.409e-05  linf=8.667e-06
      - layerThickness: l1=2.243e-04  l2=2.066e-05  linf=5.796e-06
    - ocean/planar/seamount/nonlinear/sigma/short
      - temperature:    l1=5.722e-03  l2=2.011e-04  linf=2.436e-05
      - normalVelocity: l1=7.798e-04  l2=1.568e-05  linf=1.493e-06
      - layerThickness: l1=3.287e-05  l2=9.787e-07  linf=9.280e-08

Recent commits

Polaris:

fc3cf13cf Merge pull request #899 from xylar/restore-aurora-oneapi-2026
680e3a2c7 Merge pull request #893 from xylar/add-pr-test-parallel
56a6d0eb9 Merge pull request #891 from xylar/fix-dependency-pickle-growth
2fcfb1fe4 Merge pull request #890 from xylar/tear-down-polaris-parallel-on-sigterm
0144c23a8 Merge pull request #889 from xylar/fix-serial-memory-growth

Baseline Omega:

23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags
eeef0a29dc Merge pull request #611 from brian-oneill/omega/analysis-restarts-and-samples

PR Omega:

09ec281091 Test merge of E3SM-Project/Omega#632 (db8ebbb17c) into develop (23b13afac5)
23b13afac5 Merge pull request #506 from hyungyukang/omega/compute-mesh-scaling
330b297447 Merge pull request #606 from andrewdnolan/omega/optional-cout-redirect
f13879a93a Merge pull request #612 from alicebarthel/omega/add-salinity-clamp
ede8e22014 Merge pull request #625 from grnydawn/ykim/omega/standalone-cpu-arch-flags

CTest unit tests:

Machine: frontier
Compiler: craygnu-mphipcc
Build type: Release
Build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr632-db8ebbb/frontier_craygnu-mphipcc/build
Omega: 09ec281091e87e5e4d539f813303715e04f81066 (Omega-v0.1.0-alpha.1-6286-g09ec281091)
Result: All tests passed
Log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr632-db8ebbb/frontier_craygnu-mphipcc/build/ctests.log

Build warnings

No new warnings (6 in the PR build, 6 in the baseline build).

Baseline build log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-fc3cf13_omega-23b13af/build/build_omega.log
PR build log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr632-db8ebbb/frontier_craygnu-mphipcc/build/build_omega.log

Notes

Both rows run every task and pass all CTests. The test merge is this PR applied directly to the baseline commit, so the baseline differences come from this PR and from #628, which it builds on.

The GPU row differs from the baseline in the same 15 tasks as Chrysalis, pm-cpu and pm-gpu, all expected from this PR and #628; the earlier Chrysalis test explains them. The CPU row differs in those 15 plus cosine_bell/decomp and cosine_bell/restart. Their tracer1 norms are identical to those in #628's Frontier craygnu testing (linf 2.331e-15 and 3.388e-21), which traced them to gcc emitting a fused multiply-add once #628's change stopped FCTProvisionaLayerThicknesses from being inlined, not to the change itself. I did not repeat that analysis here.

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

mwarusz

It looks like horizontal FCT already computes provisional tracers after horizontal advection.

Omega/components/omega/src/ocn/TendencyTerms.h

Lines 966 to 971 in 695f548

    
           // TracerCur on LHS is provisional tracer after 
        
           //                     horizontal fluxes only. 
        
           TracerCur(ICell, K) = 
        
               (TracerCur(ICell, K) * LayerThickness(ICell, K) + 
        
                Dt * WorkTend(ICell, K)) * 
        
               HProvInv(ICell, K);

Having said that, it might require a bit of refactoring to be able to reuse that computation. A tracer dimension would need to be added to TracerCur and the standard non-FCT horizontal advection would also need to compute it. Hence, I am approving this PR as is, but we should create an issue to clean-up handling of provisional tracers and provisional thickness (TracerHorzAdvOnCell.HProv and PseudoThicknessAux.ProvPseudoThickness also appear to be the same) in tracer advection.

brian-oneill

Sorry, this looks like it was an oversight when I wrote the vertical advection. Thanks for catching it, @xylar! I agree it's best to merge these changes now and then handle the duplicate calculations. Consolidating the provisional thicknesses and tracers for both the horizontal FCT on and off cases needs a little more thought and is better left for a follow-up. Approved.

computeTracerTendenciesOnly added the horizontal and vertical FCT tendencies, but the vertical limiter took its bounds, its fluxes and its starting tracer mass (Tracers * ProvPseudoThickness) from the tracer before horizontal advection. Each limiter bounded only its own update, so where both pushed a cell toward the same extreme, the sum went past it (E3SM-Project#630). As MPAS-Ocean does, compute the tracers after horizontal advection alone, (T * h + Dt * H) / ProvPseudoThickness, while TracerTend still holds only the horizontal advection tendency H, and pass them to the vertical FCT. The array is allocated and filled only with FCT vertical advection, so the standard algorithm is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

xylar · 2026-10-27T14:21:07Z

Thanks @brian-oneill and @mwarusz. Agreed, I opened #649 about cleaning up and consolidating these calls. Please edit.

xylar · 2026-10-27T14:29:25Z

@sbrus89, this is now also ready to merge. Thank you so much!

xylar added bug Something isn't working Omega labels Oct 24, 2026

xylar requested review from brian-oneill, katsmith133 and mwarusz and removed request for katsmith133 October 24, 2026 15:15

xylar added a commit to xylar/E3SM that referenced this pull request Oct 24, 2026

Test merge of E3SM-Project#632 (db8ebbb) into develop (23b13af)

09ec281

xylar added a commit to xylar/E3SM that referenced this pull request Oct 24, 2026

Add the manifest for testing E3SM-Project#632

52d1623

Temporary. Testers build the parent of this commit.

xylar added the non-BFB PR makes roundoff changes to answers. label Oct 25, 2026

vanroekel added this to the 1st E3SM Merge milestone Oct 26, 2026

xylar force-pushed the omega/fix-fct-tracer-bounds branch from db8ebbb to 5b01d4e Compare October 26, 2026 12:52

mwarusz approved these changes Oct 26, 2026

View reviewed changes

brian-oneill approved these changes Oct 26, 2026

View reviewed changes

This was referenced Oct 27, 2026

A NaN abort from state validation leaves no record of the field or location #647

Open

Enforce velocity field masking for NormalTransportVelocity #638

Merged

xylar force-pushed the omega/fix-fct-tracer-bounds branch from 5b01d4e to 65fd64f Compare October 27, 2026 14:18

xylar marked this pull request as ready for review October 27, 2026 14:18

xylar mentioned this pull request Oct 27, 2026

Consolidate provisional thickness and tracer calculations in tracer advection #649

Open

xylar assigned sbrus89 Oct 27, 2026

xylar mentioned this pull request Oct 27, 2026

Report where state validation fails and save the state before aborting #648

Draft

3 of 5 tasks

sbrus89 merged commit 6af2d90 into E3SM-Project:develop Oct 27, 2026
1 check passed

xylar mentioned this pull request Oct 27, 2026

Update Omega submodule E3SM-Project/polaris#915

Open

4 of 8 tasks

	// TracerCur on LHS is provisional tracer after
	// horizontal fluxes only.
	TracerCur(ICell, K) =
	(TracerCur(ICell, K) * LayerThickness(ICell, K) +
	Dt * WorkTend(ICell, K)) *
	HProvInv(ICell, K);

Conversation

xylar commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xylar commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing on chrysalis

Polaris omega_pr suite

CTest unit tests:

Build warnings

Notes

Uh oh!

xylar commented Oct 24, 2026

Testing on chrysalis

Polaris omega_pr suite

CTest unit tests:

Build warnings

Notes

Uh oh!

xylar commented Oct 24, 2026

Testing on pm-cpu and pm-gpu

Polaris omega_pr suite

CTest unit tests:

Build warnings

Polaris omega_pr suite

CTest unit tests:

Build warnings

Notes

Uh oh!

xylar commented Oct 24, 2026

Testing on aurora

Polaris omega_pr suite

CTest unit tests:

Build warnings

Polaris omega_pr suite

CTest unit tests:

Build warnings

Notes

Uh oh!

xylar commented Oct 24, 2026

Testing on frontier

Polaris omega_pr suite

CTest unit tests:

Build warnings

Polaris omega_pr suite

CTest unit tests:

Build warnings

Notes

Uh oh!

mwarusz left a comment

Choose a reason for hiding this comment

Uh oh!

brian-oneill left a comment

Choose a reason for hiding this comment

Uh oh!

xylar commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xylar commented Oct 27, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

xylar commented Oct 24, 2026 •

edited

Loading

xylar commented Oct 24, 2026 •

edited

Loading

Polaris `omega_pr` suite

Polaris `omega_pr` suite

Polaris `omega_pr` suite

Polaris `omega_pr` suite

Polaris `omega_pr` suite

Polaris `omega_pr` suite

Polaris `omega_pr` suite

Polaris `omega_pr` suite

xylar commented Oct 27, 2026 •

edited

Loading