Skip to content

Limit loop bounds to avoid division by zero - #562

Open
andrewdnolan wants to merge 4 commits into
E3SM-Project:developfrom
andrewdnolan:omega/guard-division-by-zero
Open

andrewdnolan wants to merge 4 commits into
E3SM-Project:developfrom
andrewdnolan:omega/guard-division-by-zero

Conversation

@andrewdnolan

@andrewdnolan andrewdnolan commented Sep 17, 2026 •

Copy link
Copy Markdown

Limit loop extents to avoid division by zero.

The issue was exposed using SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF, with the gnu compiler, on chrysalis. The _D modifier in the test name resulted in a debug build, which adds the -ffpe-trap=invalid,zero,overflow flags.

This has not been caught previously, because a standalone Debug build of Omega does not use the -ffpe-trap=invalid,zero,overflow flags.

Closes #556.

Checklist

  • Linting

  • Building

    • CMake build does not produce any new warnings from changes in this PR
  • Testing

    aurora, intel, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    aurora, intelgpu, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    chrysalis, intel, openmpi

    • CTests Pass
    • Polaris omega_pr Pass

    frontier, craygnu, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    frontier, craygnu-mphipcc, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    pm-cpu, gnu, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    pm-gpu, gnugpu, mpich

    • CTests Pass
    • Polaris omega_pr Pass

@andrewdnolan andrewdnolan added the bug Something isn't working label Sep 17, 2026
@andrewdnolan

Copy link
Copy Markdown
Author

Testing:

A single node, non-threaded, smoke test on chrysalis using debug build:

20260917_140012_rqz30d: 1 test
  SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu (Overall: PASS) details:
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu CREATE_NEWCASE
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu XML
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu SETUP
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu SHAREDLIB_BUILD time=276
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu MODEL_BUILD time=567
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu SUBMIT
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu RUN time=1024
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu MEMLEAK
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu SHORT_TERM_ARCHIVER
Prior to this PR, that same test failed with:

29: 
29: Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation.
29: 
29: Backtrace for this error:
...
17: #0  0x155549b1a62f in ???
22: #0  0x155549b1a62f in ???
42: #0  0x155549b1a62f in ???
52: #0  0x155549b1a62f in ???
25: #0  0x155549b1a62f in ???
30: #0  0x155549b1a62f in ???
28: #0  0x155549b1a62f in ???
41: #0  0x155549b1a62f in ???
17: #1  0xa9e6c1 in _ZZNK5OMEGA16VorticityAuxVars19computeVarsOnVertexERKN6Kokkos4Impl20HostThreadTeamMemberINS1_6SerialEEEiRKNS1_4ViewIPPdJNS1_11LayoutRightENS1_9HostSpaceEEEESF_ENKUliE2_clEi
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/ocn/auxiliaryVars/VorticityAuxVars.h:70
17: #2  0xaa28c0 in _ZN6Kokkos12parallel_forIiZNK5OMEGA16VorticityAuxVars19computeVarsOnVertexERKNS_4Impl20HostThreadTeamMemberINS_6SerialEEEiRKNS_4ViewIPPdJNS_11LayoutRightENS_9HostSpaceEEEESG_EUliE2_S6_EEvRKNS3_31TeamThreadRangeBoundariesStructIT_T1_EERKT0_PPKNSt9enable_ifIXsrNS3_26is_host_thread_team_memberISK_EE5valueEvE4typeE
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/externals/ekat/extern/kokkos/core/src/impl/Kokkos_HostThreadTeam.hpp:788
17: #3  0xaa1352 in _ZN5OMEGA16parallelForInnerIZNKS_16VorticityAuxVars19computeVarsOnVertexERKN6Kokkos4Impl20HostThreadTeamMemberINS2_6SerialEEEiRKNS2_4ViewIPPdJNS2_11LayoutRightENS2_9HostSpaceEEEESG_EUliE2_EEvS8_NS_5RangeEOT_
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/infra/OmegaKokkosHiPar.h:253
17: #4  0xa9f188 in _ZNK5OMEGA16VorticityAuxVars19computeVarsOnVertexERKN6Kokkos4Impl20HostThreadTeamMemberINS1_6SerialEEEiRKNS1_4ViewIPPdJNS1_11LayoutRightENS1_9HostSpaceEEEESF_
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/ocn/auxiliaryVars/VorticityAuxVars.h:68
17: #5  0xa89712 in operator()
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/ocn/AuxiliaryState.cpp:217
17: #6  0xa8f121 in operator()<>
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/infra/OmegaKokkos.h:241
17: #7  0xa8f121 in operator()
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/infra/OmegaKokkosHiPar.h:142
17: #8  0xa971a7 in exec<void>
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/externals/ekat/extern/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Team.hpp:229
17: #9  0xa94870 in execute
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/externals/ekat/extern/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Team.hpp:265
17: #10  0xa918af in parallel_for<Kokkos::TeamPolicy<Kokkos::Serial>, OMEGA::parallelForOuter<1, OMEGA::AuxiliaryState::computeMomAux(const OMEGA::OceanState*, const Array3DReal&, int, int, OMEGA::TimeInterval) const::<lambda(int, const TeamMember&)> >(const string&, const OMEGA::LaunchConfig<1>&, OMEGA::AuxiliaryState::computeMomAux(const OMEGA::OceanState*, const Array3DRea
l&, int, int, OMEGA::TimeInterval) const::<lambda(int, const TeamMember&)>&&)::<lambda(const TeamMember&)> >
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/externals/ekat/extern/kokkos/core/src/Kokkos_Parallel.hpp:146
17: #11  0xa8f307 in parallelForOuter<1, OMEGA::AuxiliaryState::computeMomAux(const OMEGA::OceanState*, const Array3DReal&, int, int, OMEGA::TimeInterval) const::<lambda(int, const TeamMember&)> >
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/infra/OmegaKokkosHiPar.h:139
17: #12  0xa89f3f in _ZNK5OMEGA14AuxiliaryState13computeMomAuxEPKNS_10OceanStateERKN6Kokkos4ViewIPPPdJNS4_11LayoutRightENS4_9HostSpaceEEEEiiNS_12TimeIntervalE
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/ocn/AuxiliaryState.cpp:212
17: #13  0xa8bd74 in _ZNK5OMEGA14AuxiliaryState10computeAllEPKNS_10OceanStateERKN6Kokkos4ViewIPPPdJNS4_11LayoutRightENS4_9HostSpaceEEEEiiNS_12TimeIntervalE
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/ocn/AuxiliaryState.cpp:382

Here's a full log file, if you're interested:

/lcrc/group/e3sm/ac.anolan/scratch/chrys/SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu.20260917_112659_wzv07b/run/e3sm.log.1289521.260917-114035

@andrewdnolan

Copy link
Copy Markdown
Author

Testing

CTest unit tests

  • Machine: chrysalis
  • Compiler: gnu
  • Build type: Debug
  • Result: All tests passed
    • Log: /lcrc/group/e3sm/ac.anolan/Omega_PR562/build/feature/ctest_2026-09-17_16:08.log

Tried running the omega_pr suite, but hit a filesystem error because chrysalis has run out of inodes. I can run the polaris tests once that's resolved.

@cbegeman

Copy link
Copy Markdown

Under what conditions is PseudoThickVertex zero? Is this PR indicating a horizontal or vertical index bounds issue?

@mwarusz

mwarusz commented Sep 22, 2026

Copy link
Copy Markdown
Member

@cbegeman

Under what conditions is PseudoThickVertex zero? Is this PR indicating a horizontal or vertical index bounds issue?

I think this is harmless, but can happen when computing over all vertices in the halo region. An outermost halo vertex may have only one active cell that is on a different rank.

@andrewdnolan

I think a better fix for this issue may be to limit the horizontal bounds here

parallelForOuter(
"vertexAuxState1",
LaunchConfig({Mesh->NVerticesAll},
TeamScratch<Real>(2 * VCoord->NVertLayers)),
KOKKOS_LAMBDA(int IVertex, const TeamMember &Team) {
LocVorticityAux.computeVarsOnVertex(Team, IVertex, PseudoThickCell,
NormalVelEdge);
});

to loop only over NVerticesHaloH(2). I think this should prevent division by zero and be BFB.

@andrewdnolan

Copy link
Copy Markdown
Author

@mwarusz's diagnosis is consistent with my understanding. This only occurs in the halo region when not all cells on vertex are active, so I don't think it's really accurate to call it a vertical masking issue.

The issue was exposed by SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF on chrysalis. The _D modifier in the test name resulted in a debug build, which adds the -ffpe-trap=invalid,zero,overflow flags. It's specifically the compiler flags during a debug build that cause the PseudoThickVertex to be zero in the halo region.

@mwarusz I like that suggestion! Thanks for it. I'll switch to that approach and re-test.

@andrewdnolan
andrewdnolan force-pushed the omega/guard-division-by-zero branch from 4df0539 to a328748 Compare September 23, 2026 22:13
@mwarusz mwarusz mentioned this pull request Sep 28, 2026
8 of 22 tasks
@andrewdnolan
andrewdnolan force-pushed the omega/guard-division-by-zero branch from a328748 to 6ff830b Compare October 12, 2026 22:32
@cbegeman

Copy link
Copy Markdown

@andrewdnolan do you need help with testing?

@andrewdnolan

Copy link
Copy Markdown
Author

I will be in touch depending on how my standalone (polaris) testing goes. I should be OK to handle coupled and unit tests.

@andrewdnolan
andrewdnolan force-pushed the omega/guard-division-by-zero branch from b8e36e0 to 73ad77e Compare October 13, 2026 21:07
@andrewdnolan

Copy link
Copy Markdown
Author

Rebased onto b42d508 to avoid coupled model issue currently on develop (e.g. #602). Added benefit that that commit matches the Omega submodule in polaris.

@andrewdnolan

Copy link
Copy Markdown
Author

Testing, chrysalis with gnu compiler

SMS_Vmct_D.T62_oQU240.COMEGA-IAF now passes.

After the initial test completed, I re-ran with this user_nl_omega:

Omega:
  Decomp:
    HaloWidth: 4

to confirm the dynamic halo extent in src/ocn/AuxiliaryState.cpp works as expected.

SMS_Vmct_D.T62_oQU240.COMEGA-IAF also passes, with the user_nl_omega above.

@andrewdnolan

Copy link
Copy Markdown
Author

Testing, chrysalis with gnu compiler

Polaris omega_pr suite

  • Baseline workdir: /lcrc/group/e3sm/ac.anolan/Omega_PR562/baseline
  • Baseline build: /lcrc/group/e3sm/ac.anolan/Omega_PR562/build/develop
  • PR build: /lcrc/group/e3sm/ac.anolan/Omega_PR562/build/feature
  • PR workdir: /lcrc/group/e3sm/ac.anolan/Omega_PR562/feature
  • Machine: chrysalis
  • Partition: debug
  • Compiler: gnu
  • Build type: Release
  • Log: /lcrc/group/e3sm/ac.anolan/Omega_PR562/feature/polaris_omega_pr.o1307686
  • Result: All tests passed

@andrewdnolan

Copy link
Copy Markdown
Author

@mwarusz could you take another look at this? Sorry about letting it slip through the cracks.

@andrewdnolan andrewdnolan changed the title Guard against division by zero in VorticityAuxVars.h Limit loop bounds to avoid division by zero Oct 13, 2026

@mwarusz mwarusz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have one minor suggestion.

Comment thread components/omega/src/ocn/AuxiliaryState.cpp Outdated
@andrewdnolan

Copy link
Copy Markdown
Author

I rebuilt and re-ran my SMS_Vmct_D.T62_oQU240.COMEGA-IAF test following the most recent changes, just to be safe:

PASS SMS_Vmct_D.T62_oQU240.COMEGA-IAF.chrysalis_gnu CREATE_NEWCASE
PASS SMS_Vmct_D.T62_oQU240.COMEGA-IAF.chrysalis_gnu XML
PASS SMS_Vmct_D.T62_oQU240.COMEGA-IAF.chrysalis_gnu SETUP
PASS SMS_Vmct_D.T62_oQU240.COMEGA-IAF.chrysalis_gnu SHAREDLIB_BUILD time=0
PASS SMS_Vmct_D.T62_oQU240.COMEGA-IAF.chrysalis_gnu MODEL_BUILD time=183
PASS SMS_Vmct_D.T62_oQU240.COMEGA-IAF.chrysalis_gnu SUBMIT
PASS SMS_Vmct_D.T62_oQU240.COMEGA-IAF.chrysalis_gnu RUN time=59
PASS SMS_Vmct_D.T62_oQU240.COMEGA-IAF.chrysalis_gnu MEMLEAK
PASS SMS_Vmct_D.T62_oQU240.COMEGA-IAF.chrysalis_gnu SHORT_TERM_ARCHIVER

@andrewdnolan
andrewdnolan force-pushed the omega/guard-division-by-zero branch from 010ec52 to 4801da1 Compare October 15, 2026 14:18
@xylar

xylar commented Oct 15, 2026

Copy link
Copy Markdown

I'm running the full omega_pr testing matrix. Posts should come in in the next few hours.

@xylar

xylar commented Oct 15, 2026

Copy link
Copy Markdown

Testing on frontier

Tested the merge of #562 (4801da101d) into develop (cdb935bb5a) as 5aa6097ab5, against Polaris' Omega submodule (cdb935bb5a).

Row CTests omega_pr New build warnings
frontier, craygnu, mpich 59 of 59 passed 32 of 32 passed 0
frontier, craygnu-mphipcc, mpich 59 of 59 passed 32 of 32 passed 0
frontier, craygnu, mpich

Polaris omega_pr suite

  • Baseline workdir: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu_polaris-db75895_omega-cdb935b/omega_pr
  • Baseline build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu_polaris-db75895_omega-cdb935b/build
  • PR build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr562-4801da1/frontier_craygnu/build
  • PR workdir: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr562-4801da1/frontier_craygnu/omega_pr
  • Polaris: db758959debacf1e6066741182cab2d6dfd80f9b (1.1.0-108-gdb758959d)
  • Baseline Omega: cdb935bb5a8d995b0878616e001a2701a2627519 (bp_maint3.2-3428-gcdb935bb5a)
  • PR Omega: 5aa6097ab5982410e27aa73e3afa250ef914e6aa (bp_maint3.2-3433-g5aa6097ab5)
  • Machine: frontier
  • Partition: batch
  • Compiler: craygnu
  • Build type: Release
  • Log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr562-4801da1/frontier_craygnu/omega_pr/polaris_omega_pr.o5684656
  • Result: All tests passed
Recent commits

Polaris:

db758959d Merge add-omega-testing-workflow locally for Omega PR testing
9dfb48ee5 Merge pull request #853 from xylar/add-co-drafted-signature
d6bf2af30 Merge pull request #849 from xylar/add-framework-for-pr-testing
9d6c837f9 Merge pull request #835 from xylar/update-to-mache-5.0.0
68998a2fe Merge pull request #840 from xylar/switch-mpaso-mfct-sln-back-to-rk4

Baseline Omega:

cdb935bb5a Merge pull request #600 from xylar/omega/fix-pm-mpi-flags
b90c3dae8d Merge pull request #601 from andrewdnolan/omega/cxx20-for-coupled
9641c6cf9e Merge pull request #546 from xylar/omega/fix-cf-units
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc

PR Omega:

5aa6097ab5 Test merge of E3SM-Project/Omega#562 (4801da101d) into develop (cdb935bb5a)
cdb935bb5a Merge pull request #600 from xylar/omega/fix-pm-mpi-flags
b90c3dae8d Merge pull request #601 from andrewdnolan/omega/cxx20-for-coupled
9641c6cf9e Merge pull request #546 from xylar/omega/fix-cf-units
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop

CTest unit tests:

  • Machine: frontier
  • Compiler: craygnu
  • Build type: Release
  • Build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr562-4801da1/frontier_craygnu/build
  • Omega: 5aa6097ab5982410e27aa73e3afa250ef914e6aa (bp_maint3.2-3433-g5aa6097ab5)
  • Result: All tests passed
  • Log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr562-4801da1/frontier_craygnu/build/ctests.log

Build warnings

No new warnings (3 in the PR build, 3 in the baseline build).

  • Baseline build log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu_polaris-db75895_omega-cdb935b/build/build_omega.log
  • PR build log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr562-4801da1/frontier_craygnu/build/build_omega.log
frontier, craygnu-mphipcc, mpich

Polaris omega_pr suite

  • Baseline workdir: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-db75895_omega-cdb935b/omega_pr
  • Baseline build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-db75895_omega-cdb935b/build
  • PR build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr562-4801da1/frontier_craygnu-mphipcc/build
  • PR workdir: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr562-4801da1/frontier_craygnu-mphipcc/omega_pr
  • Polaris: db758959debacf1e6066741182cab2d6dfd80f9b (1.1.0-108-gdb758959d)
  • Baseline Omega: cdb935bb5a8d995b0878616e001a2701a2627519 (bp_maint3.2-3428-gcdb935bb5a)
  • PR Omega: 5aa6097ab5982410e27aa73e3afa250ef914e6aa (bp_maint3.2-3433-g5aa6097ab5)
  • Machine: frontier
  • Partition: batch
  • Compiler: craygnu-mphipcc
  • Build type: Release
  • Log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr562-4801da1/frontier_craygnu-mphipcc/omega_pr/polaris_omega_pr.o5684683
  • Result: All tests passed
Recent commits

Polaris:

db758959d Merge add-omega-testing-workflow locally for Omega PR testing
9dfb48ee5 Merge pull request #853 from xylar/add-co-drafted-signature
d6bf2af30 Merge pull request #849 from xylar/add-framework-for-pr-testing
9d6c837f9 Merge pull request #835 from xylar/update-to-mache-5.0.0
68998a2fe Merge pull request #840 from xylar/switch-mpaso-mfct-sln-back-to-rk4

Baseline Omega:

cdb935bb5a Merge pull request #600 from xylar/omega/fix-pm-mpi-flags
b90c3dae8d Merge pull request #601 from andrewdnolan/omega/cxx20-for-coupled
9641c6cf9e Merge pull request #546 from xylar/omega/fix-cf-units
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop
9aba4a8032 Merge pull request #597 from xylar/omega/fix-real-literal-nvcc

PR Omega:

5aa6097ab5 Test merge of E3SM-Project/Omega#562 (4801da101d) into develop (cdb935bb5a)
cdb935bb5a Merge pull request #600 from xylar/omega/fix-pm-mpi-flags
b90c3dae8d Merge pull request #601 from andrewdnolan/omega/cxx20-for-coupled
9641c6cf9e Merge pull request #546 from xylar/omega/fix-cf-units
55c3de8c15 Merge pull request #566: e3sm/master into omega/develop

CTest unit tests:

  • Machine: frontier
  • Compiler: craygnu-mphipcc
  • Build type: Release
  • Build: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr562-4801da1/frontier_craygnu-mphipcc/build
  • Omega: 5aa6097ab5982410e27aa73e3afa250ef914e6aa (bp_maint3.2-3433-g5aa6097ab5)
  • Result: All tests passed
  • Log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr562-4801da1/frontier_craygnu-mphipcc/build/ctests.log

Build warnings

No new warnings (6 in the PR build, 6 in the baseline build).

  • Baseline build log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/baselines/frontier_craygnu-mphipcc_polaris-db75895_omega-cdb935b/build/build_omega.log
  • PR build log: /lustre/orion/cli115/scratch/xylar/polaris_1.1/frontier/omega_pr_test/pr562-4801da1/frontier_craygnu-mphipcc/build/build_omega.log

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

debug test fails using gnu compiler on chrysalis

5 participants