Repository navigation
spawn-safe and order-safe parallel insty and watfinder fixes - #2277
Open
jamesmkrieger wants to merge 11 commits into
Open
jamesmkrieger wants to merge 11 commits into
jamesmkrieger wants to merge 11 commits into
Conversation
Three separate reasons the parallel paths of `calcInteractionsMultipleFrames` and `InteractionsTrajectory.calcProteinInteractionsTrajectory` could not be relied on. The per-frame workers were nested functions. multiprocessing pickles the target of a `Process` under the "spawn" start method, which is the default on macOS since Python 3.8 and on Windows always, and a nested function cannot be pickled, so `p.start()` raised `AttributeError: Can't pickle local object` and the whole parallel path only ever ran where "fork" is still the default. The three workers move to module level and take what they closed over as arguments. They also take coordinates as a plain array rather than a `Frame`, which would drag its whole `Trajectory` into the child. Because `max_proc` defaults to `mp.cpu_count()//2`, whether the parallel path is taken at all depends on the machine: a two- or three-core runner gets 1 and goes serial, a four-core one gets 2 and does not. That is why this surfaced as a CI failure that moved between matrix cells rather than a reproducible one. The workers appended their results to the shared list, which records them in the order the processes happen to finish, so the parallel path silently returned frames out of order -- six frames at max_proc=3 came back as 2, 0, 1, 3, 4, 5. Each worker now writes into its own slot of a pre-sized list, indexed by frame, as the workers in waterbridges.py already were. Finally the parallel multi-model loop started its counter at `start_frame` while the serial loop starts at 0, so the worker's `i+start_frame` applied the offset twice, and its inner bound dropped the `+1` the outer bound has and stopped a model early. Both were invisible at the default `start_frame` of 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The same nested-worker problem as in interactions.py, in `calcWaterBridgesTrajectory` and `savePDBWaterBridgesTrajectory`: a nested function cannot be pickled, so every parallel path here raised `AttributeError: Can't pickle local object` under the "spawn" start method, the default on macOS since Python 3.8 and on Windows always. All three workers move to module level and take what they closed over as arguments -- `filename` in the case of the frame writer. These workers already indexed the shared list by frame rather than appending, so ordering was right; that idiom is now the one interactions.py follows too. The parallel multi-model loop had the same double-applied offset, starting its counter at `start_frame` when the worker adds `start_frame` itself, which indexed past the end of the list whenever `start_frame` was not 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`TestInteractions.tearDownClass` removed three .npy files unconditionally, so when a test failed before saving its file the teardown raised FileNotFoundError on top of it. That turned one failure into a failure plus an error and buried which test was the real one -- as it did on the macOS CI runs where the parallel interaction tests failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`max_proc` defaults to `mp.cpu_count()//2`, so a test that does not pass it runs whichever path the hardware picks: serial on a two- or three-core runner, parallel on a four-core one. That is how a bug that made the parallel path unusable sat in a CI matrix passing on some runners and failing on others. Each path now gets its own test with `max_proc` given explicitly. - `testSerialPath` pins `max_proc=1`. It is the reference the parallel runs are compared against, and it also asserts the frames actually contain hydrogen bonds, so a run that found nothing cannot pass by comparing empty to empty. - `testParallelPathIsTaken` pins `max_proc=2` and counts `Process` constructions, so the test cannot quietly degrade into a second serial run and still pass. - `testParallelKeepsFrameOrder` compares several `max_proc` values against the serial result, which catches a right result recorded against the wrong frame as well as a wrong one. - `testParallelPathUnderSpawn` swaps the module's multiprocessing for a "spawn" context, so the start method macOS and Windows actually use is exercised here too. Spawn pickles the Process target, so this is what catches a nested worker on Linux, where "fork" hides it. - `testWorkersArePicklable` pickles the six workers directly. It needs no multiprocessing, so it is the cheap guard that fails the moment one is moved back inside its caller. All but the last fail without the two preceding commits; the last fails with an AttributeError, the workers not existing at module level. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 10, 2026
Contributor
Author
A :class:`multiprocessing.Process` reports failure only through its exitcode. The exception is raised in the child, where its traceback goes to stderr, and the parent sees nothing -- so a worker that writes its result into a shared list leaves its slot untouched and a crash is indistinguishable from a genuinely empty result. Every parallel loop in interactions.py and waterbridges.py joined its processes without looking at the exit codes. `joinProcesses` joins them and raises if any died, naming how many and their exit codes, and pointing at max_proc=1 to get the exception itself in this process. All six loops use it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`calcWaterBridges` raised `ValueError` when the structure had no water, and `findClusterCenters` raised when nothing passed the clustering filter. Neither is an error: a structure with no water has no water bridges, and that is the answer. Raising aborted a whole batch on one such structure, whether the batch was a script looping serially or a parallel run over frames -- and in the parallel case the two paths disagreed, since the child's exception was swallowed and the frame came back empty anyway. Both now warn and return empty. Every output type of `calcWaterBridges` -- indices, atomic, info, and both the chain and cluster methods -- is a list, so an empty list is the right shape for all of them. `findClusterCenters` communicates through the PDB it writes and returns nothing, so it writes no file rather than an empty one, which would look like a result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`stop_frame` is the index of the last frame to read, inclusive: that is what `calcWaterBridgesTrajectory` does for a trajectory, and what interactions.py does throughout and documents as "index of last frame to read". Its multi-model branch sliced `[start_frame:stop_frame]` instead and so silently dropped the last model, and it never resolved the `-1` default there, which with the `+1` would have become an empty slice and processed nothing. Both are fixed, and the docstring now says the bound is inclusive. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- `testNoWaterGivesAnEmptyResult` runs the waterless structure on both paths and requires one empty result per frame from each, so the two cannot drift apart again. - `testStopFrameIsTheLastFrame` pins the inclusive bound at several values in both modules, and checks that -1 still means all of them rather than none. - `TestJoinProcesses` tests the exit-code guard directly, with workers that exit 1 and workers that exit cleanly, rather than through a caller -- so it does not depend on any particular analysis failing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rom insty-fix `InteractionsTrajectory.calcProteinInteractionsTrajectory` had its own copy of the per-frame loop -- pre-sized lists, process pool, slot bookkeeping -- for all seven interaction types at once, duplicating what `calcInteractionsMultipleFrames` already does for one type. Two code paths for the same job meant each fix had to be made twice, and in practice was not. AnthonyBogetti's insty-fix branch replaced it with one call per interaction type, which is the better shape, so this takes that and drops the now-unused `_analyseFrameAll` worker and the fourteen per-type lists, `interactions_dic`, `atoms_copy` and the frame slicing that only fed them. The caller's frame index is still saved and restored; the slicing itself belongs to `calcInteractionsMultipleFrames`, which resets the trajectory on its own. Also from that branch: the no-trajectory early exit returned `interactions_nb_traj`, a local not assigned until further down the function, so it raised `UnboundLocalError` rather than returning. It returns None. Verified with a linked DCD `Trajectory` at max_proc 1, 2 and 3 under both the "fork" and "spawn" start methods: seven types, three frames each, and the trajectory's frame index back where it started. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`testWorkersArePicklable` named `_analyseFrameAll`, which the previous commit removed, and the teardown list did not cover `test_3o21_disu.npy`, which the disulfide test on 3o21 writes -- so that file was left behind after every run. Both taken from insty-fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jamesmkrieger
force-pushed
the
jamesk-claude/spawn-safe-parallel
branch
from
September 10, 2026 18:08
f0a998a to
0f8ce99
Compare
… None The previous commit made this path return None, following insty-fix. Binding the variable it already meant to return is better: `interactions_nb_traj = []` goes up front, exactly as `calcInteractionsMultipleFrames` does with `interactions_all`, and the early exit returns it. Both functions now answer the same way when there is nothing to iterate -- an empty list rather than one None and one empty list -- and a caller need not special-case it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(This PR incorporates Anthony's #2200)
The parallel paths in
interactions.pyandwaterbridges.pycould not be relied on. Thisis the cause of the macOS CI failures that appear to move between matrix cells.
The workers could not be pickled
Each per-frame worker was a nested function. multiprocessing pickles the target of a
Processunder the "spawn" start method — the default on macOS since Python 3.8 and onWindows always — and a nested function cannot be pickled, so
p.start()raised:The parallel path therefore only ever ran where "fork" is still the default. Six workers
move to module level and take what they closed over as arguments; the frame workers take
coordinates as a plain array rather than a
Frame, which would drag its wholeTrajectoryinto the child.
Why it looks Python-version specific
max_procdefaults tomp.cpu_count()//2, so whether the parallel path is entered at alldepends on the machine: a two- or three-core runner gets 1 and goes serial, a four-core one
gets 2 and does not. macOS runner images differ in core count, so the same commit passes or
fails on the same matrix cell between runs. It is not a Python version boundary, and giving
CI more cores would make the failure deterministic rather than fix it.
Results came back in completion order
The
interactions.pyworkers appended to the shared list, recording results in the orderthe processes happened to finish, so the parallel path silently returned frames out of order
— six frames at
max_proc=3came back as 2, 0, 1, 3, 4, 5. Each worker now writes into itsown slot of a pre-sized list indexed by frame, which is what the
waterbridges.pyworkersalready did; that idiom is now used in both.
A dead worker looked like an empty result
A
Processreports failure only through its exitcode: the exception is raised in the child,whose traceback goes to stderr, and a worker that writes into a shared list leaves its slot
untouched. Every parallel loop joined its processes without checking. New
prody.utilities.joinProcessesjoins and raises if any died, naming how many and their exitcodes and pointing at
max_proc=1to get the exception itself in-process. All six loops useit.
No water is an empty result, not an error
calcWaterBridgesraisedValueErrorwhen the structure had no water, andfindClusterCentersraised when nothing passed the clustering filter. Neither is an error:a structure with no water has no water bridges, and that is the answer. Raising aborted a
whole batch on one such structure — whether that batch was a script looping serially or a
parallel run over frames — and in the parallel case the two paths disagreed, since the
child's exception was swallowed and the frame came back empty anyway. Both now warn and
return empty. Every output type of
calcWaterBridges— indices, atomic, info, and both thechain and cluster methods — is a list, so an empty list is the right shape for all of them.
Relatedly, the no-trajectory exit of
calcProteinInteractionsTrajectoryreturnedinteractions_nb_traj, a local not assigned until further down the function, so it raisedUnboundLocalErrorrather than returning. That variable is now bound up front, exactlywhere
calcInteractionsMultipleFramesbindsinteractions_all, and returned — so bothfunctions answer the same way when there is nothing to iterate.
stop_frameis the last frame, inclusiveThat is what
calcWaterBridgesTrajectorydoes for a trajectory, and whatinteractions.pydoes throughout and documents as "index of last frame to read".
waterbridges.py'smulti-model branch sliced
[start_frame:stop_frame]and so silently dropped the last model,and it never resolved the
-1default there, which with the+1would have become an emptyslice. Both parallel multi-model loops also started their counter at
start_framewhile theworker adds
start_frameitself, applying the offset twice — invisible only at the defaultof 0.
One implementation, not two
InteractionsTrajectory.calcProteinInteractionsTrajectoryhad its own copy of the per-frameloop for all seven interaction types at once, duplicating
calcInteractionsMultipleFrames.Two code paths for the same job meant each fix had to be made twice, and in practice was
not. AnthonyBogetti's insty-fix branch replaced it with one call per interaction type, which
is the better shape, so this takes that and drops the now-unused all-types worker along with
the fourteen per-type lists and the frame slicing that only fed them. Verified with a linked
DCD
Trajectoryatmax_proc1, 2 and 3 under both start methods: seven types, three frameseach, and the trajectory's frame index back where it started.
Tests
The guarantee comes from pinning
max_proc, not from the runner. Each path gets its owntest, with
max_procgiven explicitly:max_proctestSerialPathtestParallelPathIsTakenProcessconstructions and asserts one per frame, so the test cannot quietly degrade into a second serial run and still passtestParallelKeepsFrameOrdertestParallelPathUnderSpawnget_context('spawn'), so the start method macOS and Windows actually use is exercised on Linux tootestNoWaterGivesAnEmptyResulttestStopFrameIsTheLastFrametestWorkersArePicklableTestJoinProcessestestParallelPathUnderSpawnis the one that matters most: it makes this class of bugreproducible on every runner, rather than only on a macOS cell that happens to report enough
cores. Pinning cores in CI would not have given the same coverage, since Linux would still
run under "fork".
TestInteractions.tearDownClassalso removed four.npyfiles unconditionally, so a testthat failed before saving raised
FileNotFoundErroron top of its own failure — turning onefailure into a failure plus an error and burying which test was the real one, as on the macOS
runs. It now removes only what is there.
With "spawn" forced on Linux,
test_insty.pygoes from 12 failures plus a teardown error to26 passed, 0 failed. Full suite: 1111 passed, with only pre-existing environmental
failures remaining.
🤖 Generated with Claude Code