Skip to content

spawn-safe and order-safe parallel insty and watfinder fixes - #2277

Open
jamesmkrieger wants to merge 11 commits into
prody:mainfrom
jamesmkrieger:jamesk-claude/spawn-safe-parallel
Open

jamesmkrieger wants to merge 11 commits into
prody:mainfrom
jamesmkrieger:jamesk-claude/spawn-safe-parallel

Conversation

@jamesmkrieger

@jamesmkrieger jamesmkrieger commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

(This PR incorporates Anthony's #2200)

The parallel paths in interactions.py and waterbridges.py could not be relied on. This
is the cause of the macOS CI failures that appear to move between matrix cells.

The workers could not be pickled

Each per-frame worker was a nested function. multiprocessing pickles the target of a
Process under the "spawn" start method — the default on macOS since Python 3.8 and on
Windows always — and a nested function cannot be pickled, so p.start() raised:

AttributeError: Can't pickle local object
  'calcInteractionsMultipleFrames.<locals>.analyseFrame'

The parallel path therefore only ever ran where "fork" is still the default. Six workers
move to module level and take what they closed over as arguments; the frame workers take
coordinates as a plain array rather than a Frame, which would drag its whole Trajectory
into the child.

Why it looks Python-version specific

max_proc defaults to mp.cpu_count()//2, so whether the parallel path is entered at all
depends on the machine: a two- or three-core runner gets 1 and goes serial, a four-core one
gets 2 and does not. macOS runner images differ in core count, so the same commit passes or
fails on the same matrix cell between runs. It is not a Python version boundary, and giving
CI more cores would make the failure deterministic rather than fix it.

Results came back in completion order

The interactions.py workers appended to the shared list, recording results in the order
the processes happened to finish, so the parallel path silently returned frames out of order
— six frames at max_proc=3 came back as 2, 0, 1, 3, 4, 5. Each worker now writes into its
own slot of a pre-sized list indexed by frame, which is what the waterbridges.py workers
already did; that idiom is now used in both.

A dead worker looked like an empty result

A Process reports failure only through its exitcode: the exception is raised in the child,
whose traceback goes to stderr, and a worker that writes into a shared list leaves its slot
untouched. Every parallel loop joined its processes without checking. New
prody.utilities.joinProcesses joins and raises if any died, naming how many and their exit
codes and pointing at max_proc=1 to get the exception itself in-process. All six loops use
it.

No water is an empty result, not an error

calcWaterBridges raised ValueError when the structure had no water, and
findClusterCenters raised when nothing passed the clustering filter. Neither is an error:
a structure with no water has no water bridges, and that is the answer. Raising aborted a
whole batch on one such structure — whether that batch was a script looping serially or a
parallel run over frames — and in the parallel case the two paths disagreed, since the
child's exception was swallowed and the frame came back empty anyway. Both now warn and
return empty. Every output type of calcWaterBridges — indices, atomic, info, and both the
chain and cluster methods — is a list, so an empty list is the right shape for all of them.

Relatedly, the no-trajectory exit of calcProteinInteractionsTrajectory returned
interactions_nb_traj, a local not assigned until further down the function, so it raised
UnboundLocalError rather than returning. That variable is now bound up front, exactly
where calcInteractionsMultipleFrames binds interactions_all, and returned — so both
functions answer the same way when there is nothing to iterate.

stop_frame is the last frame, inclusive

That is what calcWaterBridgesTrajectory does for a trajectory, and what interactions.py
does throughout and documents as "index of last frame to read". waterbridges.py's
multi-model branch sliced [start_frame:stop_frame] and so silently dropped the last model,
and it never resolved the -1 default there, which with the +1 would have become an empty
slice. Both parallel multi-model loops also started their counter at start_frame while the
worker adds start_frame itself, applying the offset twice — invisible only at the default
of 0.

One implementation, not two

InteractionsTrajectory.calcProteinInteractionsTrajectory had its own copy of the per-frame
loop for all seven interaction types at once, duplicating calcInteractionsMultipleFrames.
Two code paths for the same job meant each fix had to be made twice, and in practice was
not. AnthonyBogetti's insty-fix branch replaced it with one call per interaction type, which
is the better shape, so this takes that and drops the now-unused all-types worker along with
the fourteen per-type lists and the frame slicing that only fed them. Verified with a linked
DCD Trajectory at max_proc 1, 2 and 3 under both start methods: seven types, three frames
each, and the trajectory's frame index back where it started.

Tests

The guarantee comes from pinning max_proc, not from the runner. Each path gets its own
test, with max_proc given explicitly:

test max_proc what it guarantees
testSerialPath 1, on purpose the reference the parallel runs are compared against; also asserts the frames actually contain hydrogen bonds, so an empty-vs-empty comparison cannot pass
testParallelPathIsTaken 2 counts Process constructions and asserts one per frame, so the test cannot quietly degrade into a second serial run and still pass
testParallelKeepsFrameOrder 2, 3, 6 catches a right result recorded against the wrong frame as well as a wrong one
testParallelPathUnderSpawn 2, under a spawn context swaps the module's multiprocessing for get_context('spawn'), so the start method macOS and Windows actually use is exercised on Linux too
testNoWaterGivesAnEmptyResult 1, 2, 3 one empty result per frame from both paths, so they cannot drift apart again
testStopFrameIsTheLastFrame 1, 2 the inclusive bound at several values in both modules, and that -1 still means all of them
testWorkersArePicklable n/a no multiprocessing at all; fails the moment a worker is moved back inside its caller
TestJoinProcesses n/a the exit-code guard directly, with workers that exit 1 and workers that exit cleanly

testParallelPathUnderSpawn is the one that matters most: it makes this class of bug
reproducible on every runner, rather than only on a macOS cell that happens to report enough
cores. Pinning cores in CI would not have given the same coverage, since Linux would still
run under "fork".

TestInteractions.tearDownClass also removed four .npy files unconditionally, so a test
that failed before saving raised FileNotFoundError on top of its own failure — turning one
failure into a failure plus an error and burying which test was the real one, as on the macOS
runs. It now removes only what is there.

With "spawn" forced on Linux, test_insty.py goes from 12 failures plus a teardown error to
26 passed, 0 failed. Full suite: 1111 passed, with only pre-existing environmental
failures remaining.

🤖 Generated with Claude Code

jamesmkrieger and others added 4 commits September 10, 2026 18:22
Three separate reasons the parallel paths of `calcInteractionsMultipleFrames` and
`InteractionsTrajectory.calcProteinInteractionsTrajectory` could not be relied on.

The per-frame workers were nested functions.  multiprocessing pickles the target
of a `Process` under the "spawn" start method, which is the default on macOS
since Python 3.8 and on Windows always, and a nested function cannot be pickled,
so `p.start()` raised `AttributeError: Can't pickle local object` and the whole
parallel path only ever ran where "fork" is still the default.  The three
workers move to module level and take what they closed over as arguments.  They
also take coordinates as a plain array rather than a `Frame`, which would drag
its whole `Trajectory` into the child.

Because `max_proc` defaults to `mp.cpu_count()//2`, whether the parallel path is
taken at all depends on the machine: a two- or three-core runner gets 1 and goes
serial, a four-core one gets 2 and does not.  That is why this surfaced as a CI
failure that moved between matrix cells rather than a reproducible one.

The workers appended their results to the shared list, which records them in the
order the processes happen to finish, so the parallel path silently returned
frames out of order -- six frames at max_proc=3 came back as 2, 0, 1, 3, 4, 5.
Each worker now writes into its own slot of a pre-sized list, indexed by frame,
as the workers in waterbridges.py already were.

Finally the parallel multi-model loop started its counter at `start_frame` while
the serial loop starts at 0, so the worker's `i+start_frame` applied the offset
twice, and its inner bound dropped the `+1` the outer bound has and stopped a
model early.  Both were invisible at the default `start_frame` of 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The same nested-worker problem as in interactions.py, in
`calcWaterBridgesTrajectory` and `savePDBWaterBridgesTrajectory`: a nested
function cannot be pickled, so every parallel path here raised
`AttributeError: Can't pickle local object` under the "spawn" start method,
the default on macOS since Python 3.8 and on Windows always.  All three
workers move to module level and take what they closed over as arguments --
`filename` in the case of the frame writer.

These workers already indexed the shared list by frame rather than appending,
so ordering was right; that idiom is now the one interactions.py follows too.
The parallel multi-model loop had the same double-applied offset, starting its
counter at `start_frame` when the worker adds `start_frame` itself, which
indexed past the end of the list whenever `start_frame` was not 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`TestInteractions.tearDownClass` removed three .npy files unconditionally, so
when a test failed before saving its file the teardown raised
FileNotFoundError on top of it.  That turned one failure into a failure plus an
error and buried which test was the real one -- as it did on the macOS CI runs
where the parallel interaction tests failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`max_proc` defaults to `mp.cpu_count()//2`, so a test that does not pass it runs
whichever path the hardware picks: serial on a two- or three-core runner,
parallel on a four-core one.  That is how a bug that made the parallel path
unusable sat in a CI matrix passing on some runners and failing on others.  Each
path now gets its own test with `max_proc` given explicitly.

- `testSerialPath` pins `max_proc=1`.  It is the reference the parallel runs are
  compared against, and it also asserts the frames actually contain hydrogen
  bonds, so a run that found nothing cannot pass by comparing empty to empty.
- `testParallelPathIsTaken` pins `max_proc=2` and counts `Process`
  constructions, so the test cannot quietly degrade into a second serial run and
  still pass.
- `testParallelKeepsFrameOrder` compares several `max_proc` values against the
  serial result, which catches a right result recorded against the wrong frame
  as well as a wrong one.
- `testParallelPathUnderSpawn` swaps the module's multiprocessing for a "spawn"
  context, so the start method macOS and Windows actually use is exercised here
  too.  Spawn pickles the Process target, so this is what catches a nested
  worker on Linux, where "fork" hides it.
- `testWorkersArePicklable` pickles the six workers directly.  It needs no
  multiprocessing, so it is the cheap guard that fails the moment one is moved
  back inside its caller.

All but the last fail without the two preceding commits; the last fails with an
AttributeError, the workers not existing at module level.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jamesmkrieger

Copy link
Copy Markdown
Contributor Author

This fixes sporadic insty parallel bugs like those found in #2275 and #2276

jamesmkrieger and others added 4 commits September 10, 2026 18:53
A :class:`multiprocessing.Process` reports failure only through its exitcode.
The exception is raised in the child, where its traceback goes to stderr, and
the parent sees nothing -- so a worker that writes its result into a shared
list leaves its slot untouched and a crash is indistinguishable from a
genuinely empty result.  Every parallel loop in interactions.py and
waterbridges.py joined its processes without looking at the exit codes.

`joinProcesses` joins them and raises if any died, naming how many and their
exit codes, and pointing at max_proc=1 to get the exception itself in this
process.  All six loops use it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`calcWaterBridges` raised `ValueError` when the structure had no water, and
`findClusterCenters` raised when nothing passed the clustering filter.  Neither
is an error: a structure with no water has no water bridges, and that is the
answer.  Raising aborted a whole batch on one such structure, whether the batch
was a script looping serially or a parallel run over frames -- and in the
parallel case the two paths disagreed, since the child's exception was
swallowed and the frame came back empty anyway.

Both now warn and return empty.  Every output type of `calcWaterBridges` --
indices, atomic, info, and both the chain and cluster methods -- is a list, so
an empty list is the right shape for all of them.  `findClusterCenters`
communicates through the PDB it writes and returns nothing, so it writes no
file rather than an empty one, which would look like a result.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`stop_frame` is the index of the last frame to read, inclusive: that is what
`calcWaterBridgesTrajectory` does for a trajectory, and what interactions.py
does throughout and documents as "index of last frame to read".  Its
multi-model branch sliced `[start_frame:stop_frame]` instead and so silently
dropped the last model, and it never resolved the `-1` default there, which
with the `+1` would have become an empty slice and processed nothing.  Both are
fixed, and the docstring now says the bound is inclusive.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- `testNoWaterGivesAnEmptyResult` runs the waterless structure on both paths and
  requires one empty result per frame from each, so the two cannot drift apart
  again.
- `testStopFrameIsTheLastFrame` pins the inclusive bound at several values in
  both modules, and checks that -1 still means all of them rather than none.
- `TestJoinProcesses` tests the exit-code guard directly, with workers that exit
  1 and workers that exit cleanly, rather than through a caller -- so it does
  not depend on any particular analysis failing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jamesmkrieger and others added 2 commits September 10, 2026 19:08
…rom insty-fix

`InteractionsTrajectory.calcProteinInteractionsTrajectory` had its own copy of
the per-frame loop -- pre-sized lists, process pool, slot bookkeeping -- for all
seven interaction types at once, duplicating what
`calcInteractionsMultipleFrames` already does for one type.  Two code paths for
the same job meant each fix had to be made twice, and in practice was not.

AnthonyBogetti's insty-fix branch replaced it with one call per interaction
type, which is the better shape, so this takes that and drops the now-unused
`_analyseFrameAll` worker and the fourteen per-type lists, `interactions_dic`,
`atoms_copy` and the frame slicing that only fed them.  The caller's frame index
is still saved and restored; the slicing itself belongs to
`calcInteractionsMultipleFrames`, which resets the trajectory on its own.

Also from that branch: the no-trajectory early exit returned
`interactions_nb_traj`, a local not assigned until further down the function, so
it raised `UnboundLocalError` rather than returning.  It returns None.

Verified with a linked DCD `Trajectory` at max_proc 1, 2 and 3 under both the
"fork" and "spawn" start methods: seven types, three frames each, and the
trajectory's frame index back where it started.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`testWorkersArePicklable` named `_analyseFrameAll`, which the previous commit
removed, and the teardown list did not cover `test_3o21_disu.npy`, which the
disulfide test on 3o21 writes -- so that file was left behind after every run.
Both taken from insty-fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jamesmkrieger
jamesmkrieger force-pushed the jamesk-claude/spawn-safe-parallel branch from f0a998a to 0f8ce99 Compare September 10, 2026 18:08
… None

The previous commit made this path return None, following insty-fix.  Binding
the variable it already meant to return is better: `interactions_nb_traj = []`
goes up front, exactly as `calcInteractionsMultipleFrames` does with
`interactions_all`, and the early exit returns it.  Both functions now answer
the same way when there is nothing to iterate -- an empty list rather than one
None and one empty list -- and a caller need not special-case it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant