Skip to content

Fix CHARMM PSF topology handling: cross-term width, term order, and topology under selection - #2275

Merged
jamesmkrieger merged 8 commits into
prody:mainfrom
jamesmkrieger:jamesk-claude/psf-topology-order
Sep 15, 2026
Merged

jamesmkrieger merged 8 commits into
prody:mainfrom
jamesmkrieger:jamesk-claude/psf-topology-order

Conversation

@jamesmkrieger

Copy link
Copy Markdown
Contributor

ProDy cannot currently round-trip a CHARMM PSF: parsePSF followed by writePSF produces a file with a different topology from the one it read. Two independent causes, plus a third that hides one of them.

1. A CMAP cross-term is eight atom indices, not four

A !NCRTERM record holds the two coupled dihedrals of a CMAP term — eight indices, one term per line. From msbA's PSF:

        20        22        24        31        22        24        31        33

that is φ (20-22-24-31) and ψ (22-24-31-33) of one residue. parsePSF read count=n_crossterms*4, which consumed only the first 571 of the file's 1142 records and split each one into two unrelated 4-tuples — filling the array to the expected count with terms that do not exist. writePSF then wrote those 4-tuples two to a line, so the output disagreed with its own !NCRTERM count.

Fixed in the parser, the writer, setCrossterms (which now requires the eight columns and documents what they are), and evalCrossterms, which now takes the term width from the array instead of assuming four, so its neighbour map covers all eight atoms.

2. Sorting the indices inside a term changes which term it is

The atom order within an angle, dihedral, improper or cross-term is what identifies it: an angle's middle index is its vertex, a torsion's four are a sequence, a cross-term's eight are two dihedrals. All four setters began with .sort(1). msbA's PSF gives its first angles as

         2         1         3         2         1         4         2         1         5

— all three about vertex 1 — and they were stored as 1 2 3, 1 2 4, 1 2 5, so Angle.getVectors, documented as returning the bond vectors that originate from the central atom, took atom 2 as the vertex of all three.

Donors and acceptors are ordered pairs too (!NDON is a donor and its hydrogen, !NACC an acceptor and its antecedent), so they are no longer sorted or deduplicated either. The term-level argsort chains go with the sorts: nothing downstream needs a canonical term order, and keeping the order read is what makes the round-trip exact.

setBonds and setNBExclusions keep their within-pair sort deliberately — those pairs really are symmetric, so sorting each is what collapses (i, j) and (j, i) in the np.unique that follows, and for bonds it is also what guarantees i < j in the '%d %d' keys of _bondIndex that Bond looks up. Their row-level argsort chains are dropped as dead code, since np.unique(..., axis=0) already sorts rows.

3. !NCRTERM was dropped when it directly follows the exclusions

The loop reading !NNB stops on the first line containing ! — which has already consumed the next section's header. CHARMM puts !NGRP and !NUMLP there, so real files survived, but a file that goes straight to !NCRTERM — including one written by writePSF — had the scan start past the header, and every cross-term was silently dropped with no error. The line the loop stopped on is now checked before scanning on.

Testing

  • New prody/tests/trajectory/test_psffile.py and a hand-made ten-atom fixture topology.psf with every section populated, laid out the way CHARMM writes one, with no term's indices already sorted. 5 of the 7 tests fail without these fixes.
  • prody/tests/atomic, trajectory and proteins: 701 passed. The 7 failures are pre-existing and environmental (no mmtf, no Hpb, network fetches) and are unaffected.
  • Real-world check on a 720,343-atom CHARMM-GUI system: an OpenMM System built from the original PSF and from the ProDy round-tripped PSF give identical potential energy, −9,820,146.3 kJ/mol, with the same 704,223 constraints.

Not addressed here: writePSF still omits !NGRP, !NUMLP and the per-atom !NNB pointer array. OpenMM tolerates that; CHARMM and NAMD may not.

🤖 Generated with Claude Code

@jamesmkrieger jamesmkrieger changed the title Fix PSF topology round-trip: cross-term width, and atom order within terms PSF/topology fixes Sep 10, 2026
jamesmkrieger and others added 8 commits September 10, 2026 17:53
A CHARMM `!NCRTERM` record holds the two coupled dihedrals of a CMAP term,
so eight atom indices, one term per line.  `parsePSF` read four, which
consumed only half of the section -- 571 of msbA's 1142 records -- and split
each record it did read into two unrelated 4-tuples.  `writePSF` then emitted
those 4-tuples two to a line, so a file written from a parsed PSF disagreed
with its own `!NCRTERM` count.

`parsePSF` now reads and reshapes eight per term, `writePSF` writes one term
per line, and `setCrossterms` requires the eight columns and says so.
`evalCrossterms` takes its term width from the array rather than assuming
four, so its neighbour map is built over all eight atoms of a term.

Also fix a second way cross-terms went missing: the loop that reads the
non-bonded exclusions stops on the first line containing `!`, which has
already consumed the header of the section that follows.  CHARMM puts `!NGRP`
and `!NUMLP` there, but a file -- including one written by `writePSF` -- may go
straight to `!NCRTERM`, in which case the scan for that header started past
it and every cross-term was silently dropped.  The line the loop stopped on
is now checked before scanning on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The atom order inside an angle, dihedral, improper or CMAP cross-term is what
identifies the term: the middle index of an angle is its vertex, a torsion's
four indices are a sequence, and a cross-term's eight are two dihedrals.
`setAngles`, `setDihedrals`, `setImpropers` and `setCrossterms` each began by
sorting every term's indices, which turns a term into a different one.

msbA's PSF gives its first angles as `2 1 3`, `2 1 4`, `2 1 5` -- all three
about vertex 1 -- and they were stored as `1 2 3`, `1 2 4`, `1 2 5`, so
`Angle.getVectors`, documented as returning the bond vectors that originate
from the central atom, took atom 2 as the vertex of all three.

Donors and acceptors are ordered pairs too -- a `!NDON` record is a donor and
its hydrogen, a `!NACC` record an acceptor and its antecedent -- so they are no
longer sorted or deduplicated either.

The term-level `argsort` chains go with them: nothing downstream requires a
canonical term order, and preserving the order read means a topology parsed
from a file is written back out unchanged.

`setBonds` and `setNBExclusions` keep their within-pair sort, which is
load-bearing rather than cosmetic: those pairs ARE symmetric, so sorting each
is what collapses (i, j) and (j, i) in the `np.unique` that follows, and for
bonds it is also what guarantees i < j in the '%d %d' keys of `_bondIndex`
that `Bond` looks up.  Their row-level `argsort` chains are dropped as dead
code, since `np.unique(..., axis=0)` already sorts the rows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`topology.psf` is a hand-made ten-atom fixture -- two ALA residues -- with
every topology section populated, laid out the way CHARMM writes one, and
chosen so that no term's indices are already in sorted order.  Five of the
seven tests fail without the two preceding commits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`evalBonds` and its siblings build `numbonds`, `numangles`, `numdihedrals`,
`numimpropers`, `numdonors`, `numacceptors`, `numnbexclusions` and
`numcrossterms` with `np.bincount`, which returns an array only as long as the
highest atom index that appears in a term.  Any atom after the last one taking
part in a term of that kind simply had no entry, so the arrays came out shorter
than the group -- a ten-atom test topology whose only donor is atoms 1-2 got a
`numdonors` of length 2.

That breaks the documented use of these arrays.  `setBonds` offers `'numbonds
0'` for selecting ions, which are exactly the atoms in no bond and so the ones
past the end of a short array, and `Atomic.copy` reads every data label through
`data[self._indices]`, which raised `IndexError` on any selection reaching past
the end.

Each of these functions is already given `n_atoms` and uses it to size the
neighbour map; the bincount now gets it too as `minlength`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`AtomSubset._iterAngles` and its siblings had two paths: filter the parent
group's terms by membership, or -- when the selection is small relative to the
group -- rebuild each term from the neighbour map.  The map records which atoms
share a term, not where in it they sit, so a rebuilt term came out with the
pivot atom first: the angle 2-1-3 arrived as 1-2-3, naming atom 2 as the vertex.
Which path ran depended on the ratio of selection size to group size, so the
same selection of a small group and of a large one disagreed.

Angles, dihedrals, impropers, donors, acceptors and cross-terms are all ordered,
so the map cannot be a faithful source for any of them and the shortcut is gone;
each now filters the term array, which is what the other path already did.  Only
bonds and non-bonded exclusions are symmetric, and they keep their map path.
`_iterCrossterms` also still assumed four atoms per term and unpacked its rows
into three, which raised `ValueError` on any group carrying CMAP terms.

Separately, `getBonds` built each `Bond` on the pointer rather than on the
parent group, as `iterBonds` does.  A `Bond` reads `_bondOrders`, which only an
`AtomGroup` has, so `selection.getBonds()` and `selection.numBonds()` raised
`AttributeError` for every selection.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`Atomic.copy` trimmed and carried over `_bonds` and stopped there, so copying a
selection of a CHARMM system -- what cropping a solvated box amounts to --
silently returned an AtomGroup with no angles, dihedrals, impropers, donors,
acceptors, non-bonded exclusions or CMAP cross-terms.  Nothing warned; the
topology was simply gone.

All eight sections are now carried, whole for a whole-group copy and trimmed
and renumbered for a selection.  `trimBonds` already did the renumbering and
was only two-atom in the one line that tests membership, so it becomes
`trimTerms`, which takes any term width and keeps both the order of the terms
and the atom order within each; `trimBonds` stays as its two-atom case.  A term
only partly inside the selection cannot be renumbered and is dropped, as bonds
already were.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The topology arrays hold indices into the AtomGroup.  For a selection those are
not the positions the atom records are written in -- they run up to the size of
the parent group -- and `writePSF` wrote them unchanged, so a PSF written from a
selection had bonds pointing past its own `!NATOM` count.  Writing four atoms
out of ten gave bonds `3 4 / 3 5 / 5 6`; on a 258,469-atom crop of a
720,343-atom system the indices would have run to 720,343.

Every section is now mapped onto the 1-based position of its atom in the file
being written.  A whole-group write is unaffected, its indices already being
its positions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ten more tests on the same fixture: that the num* arrays are per-atom, that a
selection delegates its terms in parent numbering and in order, that
`numBonds` works at all, that `copy()` carries all eight sections renumbered
and drops the partly-selected ones, that a written subset's indices stay inside
its own `!NATOM` count, and that copy-then-write agrees with writing the
selection directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jamesmkrieger
jamesmkrieger force-pushed the jamesk-claude/psf-topology-order branch from fb22b42 to 87a0fb7 Compare September 10, 2026 17:01
@jamesmkrieger jamesmkrieger changed the title PSF/topology fixes Fix CHARMM PSF topology handling: cross-term width, term order, and topology under selection Sep 10, 2026
@jamesmkrieger

Copy link
Copy Markdown
Contributor Author

the sporadic insty parallel bug is fixed in PR #2277

@karolamik13 karolamik13 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I tested on a few PSF files.

Code:
from prody import *
import numpy as np

name = 'test.psf'
p = parsePSF(name)
writePSF(name[:-4]+'_out.psf', p)
p2 = parsePSF(name[:-4]+'_out.psf')

for x in ['bonds', 'angles', 'dihedrals', 'impropers', 'donors', 'acceptors', 'crossterms']:
print(x, np.array_equal(getattr(p, '' + x), getattr(p2, '' + x)))

print(p._crossterms.shape)
print(p2._crossterms.shape)

Output:
bonds True
angles True
dihedrals True
impropers True
donors True
acceptors True
crossterms True
(1012, 8)
(1012, 8)

@jamesmkrieger
jamesmkrieger merged commit b18d285 into prody:main Sep 15, 2026
11 of 12 checks passed
@jamesmkrieger
jamesmkrieger deleted the jamesk-claude/psf-topology-order branch September 15, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants