Three point-locator tests fail when xdist worker count changes the file grouping

[lmoresi]

Found while measuring parallel test runs (#566).

What happens

Running level 1 under pytest-xdist --dist loadfile at 16 workers on a 16-core box:

FAILED tests/test_0761_point_locator.py::test_p1_field_on_a_shared_edge_matches_the_closed_form[2d-0.0]
FAILED tests/test_0761_point_locator.py::test_p1_field_on_a_shared_edge_matches_the_closed_form[2d-0.5]
FAILED tests/test_0761_point_locator.py::test_p1_field_on_a_shared_edge_matches_the_closed_form[2d-0.25]

The same selection is green at 4 and at 8 workers, and green serially.

It is not load, and not telemetry

• Deterministic: identical three failures across two runs.
• Not telemetry: UW_ENABLE_TELEMETRY=0 (which test_levels.sh sets specifically to avoid a kdtree race) makes no difference — same three failures.
• The failure is an accuracy assertion in uw.function.evaluate at points on cell edges, not a crash or a resource error.

With --dist loadfile, the worker count determines which files share a process. So the most likely reading is test pollution: some other file landing in the same worker leaves state that changes what the locator returns. The test itself is explicit about what it is discriminating — "a locator that returns any cell CONTAINING the point gives the exact answer, and a locator that returns a cell merely NEAR it does not" — so whatever the leaked state is, it is plausibly steering cell selection.

Why it matters beyond the test

If global state can change which cell a point-location query resolves to, that is worth understanding even for serial users: it means an evaluation's accuracy depends on what the process did earlier. It may equally turn out to be a fixture or a cached mesh in the test file. Either answer is worth having.

Reproducing

OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 \
  pytest --config-file=tests/pytest.ini tests/ \
  -m "level_1 and not level_2 and not level_3" -q -n 16 --dist loadfile

--isolation (one worker, still per-file) is the tool for bisecting which preceding file is responsible.

Related: the locator suite is recent (#556, part of the #551 umbrella), and these tests had no prior parallel exposure.

Underworld development team with AI support from Claude Code

[lmoresi]

Now reproducing at FOUR workers, in CI, and it is total rather than marginal

Upgrading this from "seen at 16 workers on one machine". It fails in CI on #566 at 4 workers, and the failure is not a tolerance miss:

AssertionError: 2-D P1 evaluation at t=0.0 along cell edges is off by 1.890e+00
at 829 of 829 points — the query was answered in a cell that does not contain it (#432)

Every point, off by ~1.9. The locator is not slightly wrong, it is answering in the wrong cell for the entire query set.

What we have ruled out

• Not the per-worker mesh directory. Forcing all workers to share one directory (UW_MESH_CACHE_DIR=.meshes/shared) reproduces it identically — 3 failed, same three tests.
• Not telemetry. UW_ENABLE_TELEMETRY=0 makes no difference.
• Not load. Deterministic across every run, at 4 and at 16 workers.
• Not a simple later-file polluter. Running test_0762_bounding_surfaces, test_0762_fault_metric_tensor, test_0763_boundary_slip_correctness or test_0764_node_redistribution immediately before test_0761 in one process does not reproduce it.
• Not serial order. The whole level-1 selection passes serially in one process, test_0761 included.

So it is the file grouping under --dist loadfile that matters, and the polluting file is not one of the obvious neighbours.

Reproduce in about a minute

OMP_NUM_THREADS=1 pytest --config-file=tests/pytest.ini \
    tests/test_05*py tests/test_07*py -q -n 4 --dist loadfile

That is CI's own batch, and it fails in ~1:06 locally, which makes this a fast bisect loop for whoever picks it up.

Why it was never seen before

On development the same command dies in 9 seconds with petsc4py.PETSc.Error on test_0502 — the mesh-file race (#563/#565). xdist could not run far enough to reach this. Once #565 lands, this becomes reachable, which is why it is surfacing now rather than being new.

Consequence

This currently blocks #566. The parallel change itself works — CI went 49m43s to 30m53s at 4 workers — but the job is red on these three tests, and they are pointing at something real about cell selection rather than at a flaky tolerance. Worth fixing rather than skipping.

Underworld development team with AI support from Claude Code