[andrewdnolan]

Disable floating point contractions. Also disabling fuse, multiply, add optimizations explicitly where needed.

Without these flags we were seeing ERS and PEM tests fail the COMPARE step with aurora_oneapi-ifxgpu and pmgpu_gnugpu. Interesting, the COMPARE failures were not for all GPU machine, frontier_craycray-mphipcc was passed (see here). I believe that is explained by the compiler defaults for hipcc versus CUDA's nvcc.

Resolves #564

Checklist

• Linting

  • Pre-commit run locally

• Building

  • CMake build does not produce any new warnings from changes in this PR

• Testing

  aurora, oneapi-ifx, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  chrysalis, oneapi-ifx, openmpi

  • CTests Pass
  • Polaris omega_pr Pass

  frontier, craygnu, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  frontier, craygnu-mphipcc, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  pm-cpu, gnu, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  pm-gpu, gnugpu, mpich

  • CTests Pass
  • Polaris omega_pr Pass

• Provide relevant details in a comment to the PR titled Testing with the following:

  • Which machines CTest unit tests
    have been run on and indicate that are all passing.
  • The Polaris omega_pr test suite
    has passed, using the Polaris e3sm_submodules/Omega baseline
  • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
  • Indicate "All tests passed" or document failing tests
  • Document testing used to verify the changes including any tests that are added/modified/impacted.

[andrewdnolan]

Testing, pmgpu with gnugpu compiler

Rerunning a case we were previously seeing fail in #564, with the changes from this PR now passes:

20261001_090719_zb3x0h: 1 test
  ERS_Ld5_Vmct.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu (Overall: PASS) details:
    PASS ERS_Ld5_Vmct.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu CREATE_NEWCASE
    PASS ERS_Ld5_Vmct.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu XML
    PASS ERS_Ld5_Vmct.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu SETUP
    PASS ERS_Ld5_Vmct.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu SHAREDLIB_BUILD time=112
    PASS ERS_Ld5_Vmct.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu MODEL_BUILD time=329
    PASS ERS_Ld5_Vmct.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu SUBMIT
    PASS ERS_Ld5_Vmct.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu RUN time=83
    PASS ERS_Ld5_Vmct.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu COMPARE_base_rest
    PASS ERS_Ld5_Vmct.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu MEMLEAK insufficient data for memleak test
    PASS ERS_Ld5_Vmct.T62_oQU240.COMEGA-IAF.pm-gpu_gnugpu SHORT_TERM_ARCHIVER

[andrewdnolan]

Unfortunately, unable to test on Aurora. Between the kokkos5 update, oneapi-ifx->intel reaname, and module changes after maintenance on 09/28 we'll need changes from #566.

I can try to test again, after rebasing onto #566, on Monday.

[xylar]

@andrewdnolan, I'll also try to point you to a Polaris branch you can use for testing, together with deployment instructions.

A lot is piling up because we're badly out of sync with E3SM!

[xylar]

@andrewdnolan, you can test on Aurora with the Polaris branch xylar/polaris:update-to-mache-5.0.0 and a Spack deployment I built for intel and intelgpu against oneapi 2026.1.0 (E3SM-Project/mache#515). Deploying this way reuses those Spack environments rather than building new ones:

git clone -b update-to-mache-5.0.0 git@github.com:xylar/polaris.git polaris-mache5
cd polaris-mache5
./deploy.py --compiler intel intelgpu \
    --mache-fork E3SM-Project/mache --mache-branch main \
    --spack-path /lus/flare/projects/E3SM_Dec/xylar/spack_test/dev_polaris_1.1.0_mache5

This writes load_polaris_aurora_intel_mpich.sh and load_polaris_aurora_intelgpu_mpich.sh.

The baseline can't come from Polaris's e3sm_submodules/Omega this time. It predates #566, so it still uses the oneapi-ifx compiler names and oneapi 2025.3.1. Build the baseline from #566 (or develop once it's merged) with --branch, and the PR build from your branch rebased onto #566:

source load_polaris_aurora_intel_mpich.sh
polaris suite -c ocean -t omega_pr --model omega \
    --branch /path/to/omega_566 -p $BASELINE/build -w $BASELINE
polaris suite -c ocean -t omega_pr --model omega \
    --branch /path/to/omega_586 -p $PR/build -w $PR -b $BASELINE

Then submit the job script in each work directory. For CTests, ./utils/omega/ctest/omega_ctest.py -o /path/to/omega_586 -s works with either load script.

---

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

[andrewdnolan]

Thanks @xylar! I'll give this a go on Monday and report back.

[andrewdnolan]

Nevermind, Aurora testing will have to wait until maintenance completes tomorrow afternoon.

[andrewdnolan]

@mwarusz I'd like hear your thoughts on how to proceed here. During an Omega developer meeting a few weeks ago, when this PR first went up, I recall you expressing reservations about the approach here and it's performance implications.

Unfortunately, I haven't had the time to get back to this. But this is definitely needed before the merge of the Omega fork back into E3SM. pm-gpu has been the workhorse machine for v4 prototyping, and we need ERS/PEM to pass on that machine if we are going to be usable component. I've enlisted @grnydawn help in making sure this is ready to go in time.

We are wondering if you are open to letting this go in as is (subject to sufficient testing)? We can open an issue to follow up on whether all floating contractions need to be disabled, or there's some subset that still allow for ERS/PEM tests to pass. Another thing to follow up on is: understanding why EAMxx can get BFB reproducibility without disabling floating point contractions.

[mwarusz]

@andrewdnolan

I am fine with merging this PR and revisiting this issue later. I am approving, but could you:

• create a GitHub issue about this
• add a CMake option OMEGA_DISABLE_FP_CONTRACT, defaulting to ON, and add these flags conditionally based on its value

[andrewdnolan]

Yes! Can do for both of those. The cmake option is a good call, these are really only needed in coupled mode.

[grnydawn]

Approve. The change is correct, it is needed for coupled runs on pm-gpu, and the flags cost little. It does not make ERS and PEM pass everywhere, though; details below.

I've added the OMEGA_DISABLE_FP_CONTRACT option Maciej asked for to this branch as 63fc21e. It defaults to ON, which gives exactly the flags of the first commit; -DOMEGA_DISABLE_FP_CONTRACT=OFF leaves contraction to the compiler.

Tested with ERS_Ld5_Vmct.T62_oQU240.COMEGA-IAF: this PR's two commits on develop (ede8e22014; also 23b13afac5 on pm-gpu), and this PR's first commit, 533e57a, on its own base on pm-gpu.

Machine (compiler)	Code	ERS without this PR	ERS with this PR	Cost of this PR
pm-gpu (gnugpu)	this PR's base 73469eb9ed	FAIL, 2 of 2	PASS, 3 of 3	–
pm-gpu (gnugpu)	develop ede8e22014, 23b13afac5	FAIL, 6 of 6	FAIL, 5 of 6	~1% OCN run time
Frontier (craycray-mphipcc)	develop ede8e22014	PASS	PASS	within run-to-run noise
Aurora (intelgpu)	develop ede8e22014	FAIL	FAIL	within run-to-run noise
Chrysalis (intel, CPU)	develop ede8e22014	PASS	PASS	none (no flags on CPU)

What works:

1. On pm-gpu the flags fix ERS at this PR's base: 0 of 2 without them, 3 of 3 with them.
2. The option works. ON, the default, gives the flags to OmegaLib, ocn and every CTest; OFF is bit for bit with develop in Polaris omega_pr on pm-gpu. CPU builds get no flags.
3. CTests pass on pm-gpu and Chrysalis, and every Polaris omega_pr task runs.

Follow-ups, not blockers for this PR:

1. pm-gpu at current develop: ERS still fails 5 of 6 with this PR. A second cause came in with the 20 merges after this PR's base (73469eb9ed..ede8e22014). It is not KPP (ERS fails with KPP switched off too), and not the METIS/ParMETIS views (#622). It needs its own issue.
2. Aurora: Omega in E3SM does not repeat itself run to run, on develop too (a rerun differs in 45 coupler fields). PEM cannot pass there, and ERS fails with or without this PR.
3. PEM fails on every machine tested, from #512 (#626), so it cannot yet show whether this PR helps.
4. pm-gpu run-to-run variation: every coupled build, with or without the flags, gives slightly different answers when rerun, though standalone Omega in Polaris repeats itself. It does not fail ERS on its own, but it is a defect.
5. Frontier: ERS passes without the flags, so the HIP flags are not shown to be needed there.

Not tested: CTests and Polaris on Frontier and Aurora.