Constrained free-slip is partition-dependent (3.4%): boundary normals accumulated rank-locally — and the rotated tests compare against a golden recorded on another host

[lmoresi]

Seven partition-independence assertions fail on CI. All are pre-existing and none is caused by the PR that surfaced them (#561): measured at that PR's merge base with a single line changed — uncommenting tests/parallel/test_10*py in scripts/test.sh — the same seven fail with numbers identical to every digit.

test	serial vs np=2	relative
test_1063::test_constrained_freeslip_partition_independent[iso]	0.6194547793955 vs 0.6107410846031	1.4%
test_1063::…[ti]	0.3925981604039 vs 0.3937854671587	0.3%
test_1063::test_constrained_raw_gauge_partition_independent	0.6194547487092 vs 0.6194402844556	2.3e-05
test_1064::test_rotated_freeslip_annulus_partition_independent	0.01897011154231 vs 0.01897329151624	1.7e-04
test_1064::test_rotated_freeslip_spherical3d_partition_independent	0.004069689334228 vs 0.004074314572473	1.1e-03
test_1064::test_rotated_freeslip_spherical3d_topography_partition_independent	0.4149689252074 vs 0.4125278837958	5.9e-03
test_1066::test_rotated_datum_prescribed_normal_partition_independent	6.65602336 vs 6.6559607579	9.4e-06

Why this has been invisible

scripts/test.sh --p N ran only tests/parallel/test_075*py; the test_10*py line was commented out. test_1017 and test_1062–test_1068 have executed at no rank count in CI since they were written — the assertions that would have caught this were sitting dormant in the tree. PR #561 uncomments that line, which is why the failures appeared there first.

They also pass locally at np=2 and np=4 on macOS/arm64, so the dependence is partition- and geometry-specific rather than universal — CI's mesh and partitioning expose it.

Scope — wider than #495

#495 recorded this for Stokes_Constrained (the multiplier path) at 3.4%. The measurements above show the rotated free-slip path is affected too — annulus, spherical shell, and the prescribed wall-normal datum — which is a different code path entirely. #495 should be treated as one member of this family rather than the whole of it.

Notably the rotated cases here all pass an explicit analytic normal=, and their numbers are bit-identical before and after #560's nodal-normal fix, so this is not the #560 mechanism.

Severity

The spherical-shell topography case differs by 0.6% between rank counts, and 3-D spherical velocity by 0.1%. Anyone comparing a serial development run against a production run at scale gets different answers with no warning, and a benchmark validated at one rank count is not validated at another.

Suggested handling

The five still-failing assertions are xfail(strict=False) against this issue in #561 so the batch can stay enabled and the other twenty tests run — hiding them again by re-commenting the line would be the wrong trade. Fix order should start with the largest ([iso] at 1.4%) and with whichever path is shared, if any: establishing whether the constrained and rotated dependences have one root cause or two is the first question.

Related: #495 (the constrained-path instance), #560/#561 (surfaced it; not the cause).

Underworld development team with AI support from Claude Code

[lmoresi]

Split: the rotated rows are a TEST-DESIGN defect, not a solver defect. The constrained row is real.

Measured on CI, same host, same mesh, using test_1064's own _annulus_diagnostics() so the value is directly comparable to GOLDEN_ANNULUS:

	velocity L2	vs golden
CI np=1	1.897329151623790e-02	+1.676e-04
CI np=2	1.897329151623740e-02	+1.676e-04
golden (recorded on macOS)	1.897011154231000e-02	—

np=1 and np=2 agree to the 13th significant figure (2.6e-14 relative), and leak_lo / leak_up are identical to every digit at both rank counts. Both rank counts differ from the golden by exactly the same +1.676e-04.

So the rotated free-slip path is partition-independent to machine precision. The whole discrepancy is the golden having been recorded on a different host: GOLDEN_ANNULUS and friends are hardcoded constants from a macOS/arm64 machine, gmsh builds a different mesh on the Linux CI runner, and the tests are marked mpi(min_size=2) so CI never runs np=1 to notice.

The failure messages say "differs serial vs np=N", which is what misled this issue's original framing — they actually measure "differs from a number recorded elsewhere". Independently confirmed locally: np=1..8 agree to 2e-13 (annulus), 1e-09 (spherical3d), 7e-08 (spherical3d topography), 4e-11 (datum) — four to six orders tighter than the CI gaps, and every CI gap sits inside the mesh-noise band measured by re-meshing at the same resolution with a different gmsh algorithm.

Fix for the rotated rows is to the tests: compare np=1 against np=N in the same environment rather than against a constant recorded on someone's laptop. A self-referential comparison tests what the docstring claims and is immune to the host's mesh generator.

The constrained row is a genuine defect and is being fixed

test_1063 constrained free-slip [iso] reproduces locally: np=1 6.194547793939e-01, np=2 5.982807168537e-01 — −3.4%, #495's number to every digit. Located: Mesh._assemble_boundary_normal (discretisation_mesh.py:2977) accumulates area × n over this rank's facets then normalises, so a seam vertex gets a partial stencil — max ‖n − r̂‖ 3.0e-10 serial but 7.55e-02 (≈4.4°) at np=2 on the annulus Upper arc. Bisected: passing normal=unit_r instead drops the spread to 1.6e-09. The TODO(parallel) at :3014-3022 asserting the code is "parallel-SAFE as-is" is the bug.

Same defect class as #560/#561 (rank-local normal accumulation) in a different function, and add_nitsche_bc shares it. Curved boundaries only, which is why box tests never caught it.

Both solves are converged, so this is a genuine discrete-problem difference, not a convergence artefact: rotated cases take 1 Newton increment with true ‖F̂‖/‖F̂₀‖ 1e-11 to 2.1e-07 and nothing caps; the constrained [p,λ] Schur block does cap at 200 its, but its np=2 answer is stable at 5.9828071e-01 across tol 1e-6/1e-8/1e-10, so the cap is not the cause.

Underworld development team with AI support from Claude Code

[lmoresi]

Fixed by PR #568, merged to development as 1e43914.

The constrained partition dependence is closed: test_1063 velocity L2 now spreads 1.2e-10 across np=1/2/3/4, where it was −3.4% at np=2 (#495's number to every digit). Mesh._assemble_boundary_normal completes the weighted facet sum across ranks before normalising; the annulus Upper-arc normal error goes from 5.8e-02 at np=2 to identical-to-serial at every rank count.

A second, previously unknown defect was fixed with it: the old kd-tree DOF assignment was wrong in serial on 3-D shells — SphericalShell Lower off by 1.03e-01 (5.9°) against an independent global-facet-sum oracle, on ordinary uniform tet meshes. 2-D was unaffected. Every 3-D curved-boundary result using add_constraint_bc or add_nitsche_bc without an explicit normal= was built on those normals. The radial oracle could not see it (faceting dominates); only the facet-sum oracle could, which is why it survived. Now guarded by a dedicated serial 3-D assertion, wired into a batch CI actually runs — the review verified it FAILS (4.712e-02) when the old assembly is restored.

The rotated rows of this issue were never a solver defect: np=1 and np=2 agree to the 13th significant figure on one CI host. Those tests compared against goldens recorded on a developer machine while gmsh builds a different mesh on the Linux runner, and were marked mpi(min_size=2) so CI never ran np=1 to notice. They are now self-referential — same environment, np=1 vs np=N — with the absolute goldens retained separately behind a mesh-fingerprint gate, so partition independence and accuracy are asserted as the two different claims they are. All seven xfails are gone.

Also folded in: the fourth copy of the facet-orientation pattern (fault_contact) now consumes the shared helper, and deform()'s failure path is collective so a rank-local failure can no longer become a silent zero normal or a hang.

Follow-ups tracked separately: #569 (cell_size partition dependence via the add_nitsche_bc(local_h=True) default), #570 (the scripts/test.sh coverage audit — test_106* being unbatched is what hid the serial 3-D defect).