Skip to content

Fix issues #179, #181-#185, #188-#190 - #204

Merged
asalmgren merged 1 commit into
AMReX-Fluids:developmentfrom
asalmgren:fix-audit-issues-179-190
Sep 27, 2026
Merged

asalmgren merged 1 commit into
AMReX-Fluids:developmentfrom
asalmgren:fix-audit-issues-179-190

Conversation

@asalmgren

Copy link
Copy Markdown
Contributor

Fixes #179. Fixes #181. Fixes #182. Fixes #183. Fixes #184. Fixes #185. Fixes #188. Fixes #189. Fixes #190.

(#180 is not included -- it was already fixed on development by c44cbcf.)

What's in here

Issue File Fix
#179 Source/NS_derive.cpp The EB one-sided vorticity stencils in dermgvort read velocity two cells away without checking those cells are uncovered -- isConnected only certifies the immediate neighbor. Now the intermediate cell is certified before taking the quadratic form, with a two-point difference when only one cell is available and zero when both sides are disconnected (2D x/y, 3D x/y/z).
#181 Tutorials/TaylorGreen/benchmarks/ViscBench.cpp The loop over global grid indices dereferenced MultiFabs by global index, and FillVar(FArrayBox*,...) fills on one rank only, so the tool could not work under MPI. Switched to the distributed FillVar(MultiFab&,...) overload, drove the exact-solution loop from MFIter, and moved the prints to amrex::Print() so the collective norm* calls stay on every rank.
#182 Source/NS_getForce.cpp The getForceVerbose min/max loops indexed State and force by absolute state component, reading past the end of the velocity-only State FAB and the 0-indexed force FAB that scalar callers pass. Each FAB is now indexed by its own layout; the printed label keeps the absolute component.
#183 Source/main.cpp The regrid-on-restart test lacked the max_step < 0 guard the time loop applies, so with only stop_time set it regridded and wrote a spurious plotfile/checkpoint before stepping anyway.
#184 Tutorials/HIT/NS_getForce.cpp The coarse hi bound was one node short whenever ihi mod ff_factor was in [1, ff_factor-2], so the interpolation read past ff_force. Now ihi/ff_factor + 1, exactly the node the kernel touches (the three ihi == ff_ihi*ff_factor adjustments become dead and are removed). ff_force also gets an Elixir so its device memory is not freed while the asynchronous kernels still read it.
#185 Tutorials/HIT/TurbulentForcing_def.H Nothing constrained the mode indices writing into the flat stack array tmp, so large nmodes or a high-aspect domain corrupted the stack; the mode counts are now asserted to fit array_size. The GPU copy out of tmp was asynchronous with no sync before the stack frame died -- switched to the synchronous htod_memcpy. Also fixes the verbose block, which printed nxmodes in place of nymodes and nzmodes.
#188 Source/prob/prob_init.cpp meanFlowDir = +/-2 assigned vel_x = v_vort and vel_y = u_vort, giving a non-solenoidal, essentially irrotational field instead of a vortex; un-swapped so only the mean flow term moves to the y component. "No mean flow" is now an explicit case 0, with a host-side Abort for out-of-range directions that the bare default silently accepted.
#189 Source/MacProj.cpp The known-edgestate mac_sync_compute overload passed a default-constructed MultiFab as the state, which ComputeAofs indexes unconditionally -- now an alias of Sync. mac_sync_compute built the divu constraint at prev_time only, omitting the 0.5*dt*dsdt half-time correction that velocity_advection and scalar_advection apply, so the sync reconstructed edge states with a different divu than the advance used. test_umac_periodic did host-side FArrayBox work on device-resident data; it now gets pinned host copies.
#190 Source/NavierStokesBase.cpp Both limiters seeded the running maximum with numeric_limits<Real>::min(), the smallest positive normal rather than lowest(), so the upper clip bound was wrong for any all-negative stencil. With lowest() binding, covered cells are skipped so they keep the COVERED_VAL invariant. do_denminmax called ConservativeScalMinMax with Density as both the scalar and the normalizing density, clamping rho/rho == 1 against rhoo/rhoo == 1 -- a provable no-op; it now calls ConvectiveScalMinMax on the raw density.

Heads-up: benchmarks need regolding

The do_denminmax fix makes ns.do_denminmax=1 change results for the first time (it was previously a bit-exact no-op). Exec/eb_run2d/regtest.2d.hotspot and Exec/eb_run3d/regtest.3d.hotspot enable the flag, so their benchmarks will need to be regolded. On eb_run2d hotspot at step 4, tracer max goes from 1.0012879887 to 1.0000060234 and density max from 1.0011269247 to 1.0011264227 -- i.e. the limiter now does its job.

Build and test

Built clean: run2d, run3d, eb_run2d, eb_run3d, Tutorials/HIT (both USE_FAST_FORCE=TRUE and FALSE), and ViscBench with USE_MPI=TRUE.

All shipped regtests were run in 2D and 3D, EB and non-EB. eb_run2d/regtest.2d.shock_past_cylinder aborts with "MLMG failed" -- this fails identically on unmodified development, so it is pre-existing and unrelated.

Targeted verification against baseline binaries built from development:

Plotfile comparisons vs. baseline at step 4: poiseuille, traceradvect_bds, eb_double_shear_layer and eb_flow_past_cylinder are bit-identical. ns_hotspot / ns_hotspot_rz differ at 1e-12 / 1e-9 relative, from the dsdt term in #189 (tiny here because temp_cond_coef = 1e-8). eb_hotspot differs at ~0.2%, the intended #190 change described above.

Nothing here was run on GPU -- the GPU-specific parts of #184, #185 and #189 are compile-checked and reasoned, not executed on device.

🤖 Generated with Claude Code

…-Fluids#188-AMReX-Fluids#190

AMReX-Fluids#179 NS_derive: the EB one-sided vorticity stencils in dermgvort read
velocity two cells away without checking those cells are uncovered --
isConnected only certifies the immediate neighbor. Certify the
intermediate cell before taking the quadratic form, drop to a two-point
difference when only one cell is available, and leave the derivative at
zero when both sides are disconnected (2D x/y and 3D x/y/z).

AMReX-Fluids#181 ViscBench: the loop over global grid indices dereferenced MultiFabs
by global index and FillVar(FArrayBox*,...) fills on one rank only, so
the tool was broken under MPI. Use the distributed FillVar(MultiFab&,...)
overload, drive the exact-solution loop from MFIter, and print through
amrex::Print() so the collective norm calls stay on every rank.

AMReX-Fluids#182 getForce: the getForceVerbose min/max loops indexed State and force
by absolute state component, reading past the end of the velocity-only
State FAB and the 0-indexed force FAB that scalar callers pass. Index
each FAB by its own layout; the printed label keeps the absolute
component.

AMReX-Fluids#183 main: the regrid-on-restart test lacked the max_step < 0 guard that
the time loop below applies, so with only stop_time set it regridded and
wrote a spurious plotfile/checkpoint before stepping anyway.

AMReX-Fluids#184 HIT fast force: the coarse hi bound was one node short whenever
ihi mod ff_factor was in [1, ff_factor-2], so the interpolation read past
ff_force. Use ihi/ff_factor + 1, which is exactly the node the kernel
touches, and give ff_force an Elixir so its device memory is not freed
while the asynchronous kernels are still reading it.

AMReX-Fluids#185 TurbulentForcing: nothing constrained the mode indices writing into
the flat stack array tmp, so large nmodes or a high-aspect domain
corrupted the stack; assert the mode counts fit array_size. The GPU copy
out of tmp was asynchronous with no sync before the stack frame died;
use the synchronous htod_memcpy. Also fixes the verbose block, which
printed nxmodes in place of nymodes and nzmodes.

AMReX-Fluids#188 init_ConvectedVortex: cases meanFlowDir = +/-2 assigned vel_x =
v_vort and vel_y = u_vort, giving a non-solenoidal, essentially
irrotational field instead of a vortex. Un-swap them so only the mean
flow term moves to the y component. Make "no mean flow" an explicit
case 0 and abort on the host for out-of-range directions, which a bare
default silently accepted.

AMReX-Fluids#189 MacProj: the known-edgestate mac_sync_compute overload passed a
default-constructed MultiFab as the state, which ComputeAofs indexes
unconditionally; pass an alias of Sync instead. mac_sync_compute built
the divu constraint at prev_time only, omitting the 0.5*dt*dsdt
half-time correction that velocity_advection and scalar_advection apply,
so the sync reconstructed edge states with a different divu than the
advance used. test_umac_periodic did host-side FArrayBox work on
device-resident data; give it pinned host copies.

AMReX-Fluids#190 ScalMinMax: both limiters seeded the running maximum with
numeric_limits<Real>::min(), the smallest positive normal rather than
lowest(), so the upper clip bound was wrong for any all-negative
stencil. With lowest() binding, skip covered cells so they keep the
COVERED_VAL invariant. do_denminmax called ConservativeScalMinMax with
Density as both the scalar and the normalizing density, clamping
rho/rho == 1 against rhoo/rhoo == 1 -- a provable no-op; call
ConvectiveScalMinMax on the raw density instead.

The do_denminmax fix makes ns.do_denminmax=1 change results for the
first time, so the eb_run2d and eb_run3d hotspot benchmarks need
regolding.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment