Skip to content

feat(abi): every library reports its state-struct layout; wrappers refuse a mismatch (#246) - #261

Merged
k-yoshimi merged 2 commits into
developfrom
feat/abi-state-layout-check-246
Oct 17, 2026
Merged

k-yoshimi merged 2 commits into
developfrom
feat/abi-state-layout-check-246

Conversation

@k-yoshimi

Copy link
Copy Markdown
Owner

Fixes #246 (develop-targeted, so close by hand after merge). Design: docs/superpowers/specs/2026-10-17-c-abi-state-layout-check-design.md (option 2 of the issue; the size-negotiated _v2 getter is deferred until an out-of-tree C consumer exists).

Why

Each module fills one fixed-layout struct through X_get_state(X_state_t*), and the only rule was "append, bump the macro where there is one, rebuild in the same commit". Nothing checked it at run time, so a library and a wrapper from different commits failed silently: a library with a longer struct wrote past the wrapper's allocation (56 bytes in #233's case, during the unconditional zeroing even on an error return), a library with a shorter one left the appended fields as plausible 0.0 all the way to the MCP clients. tr and tot had *_STATE_ABI_VERSION as a compile-time #define only; the other five had nothing, and test_size_matches_header_math compares ctypes arithmetic with numbers typed into the test.

What (same shape ×7: tr fp eq ti wr wrx tot)

  • Fortran is the source of truth. X_state.f90: INTEGER(C_INT), PARAMETER :: X_STATE_ABI_VERSION next to the BIND(C) type (tr, tot = 2 as before; eq fp ti wr wrx = 1). X_api.f90: X_state_abi_version() returns it, X_state_size() returns C_SIZEOF of the type; both BIND(C), no lifecycle check, callable before X_init. The mono image exports all twelve automatically (libtotapi_mono_inspect gets a gate for them).
  • Header mirrors it. X_api.h: the #define (new for five modules, with the History block tr/tot already had), the two prototypes, <stddef.h> for size_t.
  • C drivers cross-check Fortran vs header. tests/c_abi/test_negative.c (first thing in main, before init) and test_run_so.c (through dlopen, exit 92 = symbols missing, 93 = mismatch) compare X_state_abi_version() with the macro and X_state_size() with sizeof(X_state_t). This is the BIND(C)-type-vs-C-struct check the issue asked for. CI gets a step running the six per-module X_api_check_so (tot's tot_api_check_all already ran); the static-link drivers stay local (tr's and fp's do not link on macOS, pre-existing g3d limit).
  • Python refuses a mismatch at load. New shared python/_abi_check.py::check_state_layout, called from every _ffi.py::load_library right after ctypes.CDLL and before any prototype or struct; X_STATE_ABI_VERSION constant in each _ffi.py next to the X_MAX_* constants; new XlibAbiError(XlibError) in each errors.py (exported from the package) so except XlibError keeps working. Three refusals, each naming the fix: the library exports neither symbol (built before C-ABI state structs have no size/version negotiation: struct growth silently corrupts old callers or returns zeros to new callers #246), the version differs (a reorder keeps the size), the size differs.
  • Tests. python/tests/test_abi_check.py (helper, no .so, 7 cases: match attaches prototypes, no symbols, one symbol, larger, smaller, version drift, error class). Per wrapper in tests/test_ffi.py::TestLibraryLayout (4 cases: real library matches; a ctypes struct one field short is refused with both sizes; version drift through load_library raises the wrapper's own XlibAbiError; a stand-in without the query is refused). Guards in setUp.
  • Docs. "State struct layout rule" section in all seven docs/X-library/architecture.md; "Library layout check" in all seven python/Xlib/README.md; extending-tr.md (en, ja) step 5 now says bump the PARAMETER and the macro together; CHANGELOG.

Scope boundary: X_state.f90 / X_api.f90 / X_api.h are the L-2 C-ABI layer; no module physics, so outside the lead-sign-off rule.

Sequencing

#233 (fp global scalars, external, open) grows fp_state_t: after this merges it bumps FP_STATE_ABI_VERSION 1 → 2 in fp_state.f90, fp_api.h and python/fplib/_ffi.py (one comment on #233 says so). #244 follows #233.

Verification (macOS gfortran)

  • All seven .so and libtotapi_mono.so rebuilt; every wrapper loads its library and reports the same version and size as its ctypes struct (tr v2/72128, fp v1/38432, eq v1/80816, ti v1/105640, wr v1/20448, wrx v1/70424, tot v2/72144), also through the mono image.
  • make -C X X_api_check_so → OK for all seven; libtotapi_mono_inspect gate 1b → 12 symbols; cc -std=c99 -Wall -Wextra -fsyntax-only clean on all 14 drivers; X_api_check_all static links hit the pre-existing macOS g3d limit for tr/fp (CI runs tot's and the six dlopen drivers).
  • pytest python/ test_run/scripts/tests --forked --timeout=120 --timeout-method=signal → see the first comment.
  • Reviews: in-house + Codex (first comment).

🤖 Generated with Claude Code

https://claude.ai/code/session_01TA2KahTs3qLMZL6yjSsWwT

@k-yoshimi

Copy link
Copy Markdown
Owner Author

Pre-push reviews (in-house + Codex, read-only, on the full 84-file diff; the final f86cfab folds their findings in).

Verified by the reviewers: in all seven X_api.f90 the new USE X_state, ONLY: X_STATE_ABI_VERSION is a standalone statement before the continued USE, the extra PUBLIC sits before CONTAINS, and no pre-existing symbol clashes; C_SIZEOF of the BIND(C) local returns INTEGER(C_SIZE_T) (the probes are 20 to 106 KB, above gfortran's 64 KiB stack-var default, so they go to static storage, never written); <stddef.h> is inside the guard and before extern "C", the new macro block sits between the last X_MAX_* and enum X_error; the test_run_so.c dlsym block is after the return 91 check and before the cycle in all seven, %lu casts are right; _ffi → .errors has no cycle, _abi_check is importable wherever any wrapper is, the check runs before _apply_prototypes; the stale-struct test shrinks every struct (so it cannot false-pass); the mock.patch.object test reads the module global at call time; no existing test mocks ctypes.CDLL through load_library; the mono gate's nm regex matches Linux (T eq_state_size) and macOS (_eq_state_size) and correctly omits wr (not in the mono image).

Findings, all applied:

  • HIGH (Codex, on the spec's wording): a query-only design cannot protect a caller that predates it (a pre-C-ABI state structs have no size/version negotiation: struct growth silently corrupts old callers or returns zeros to new callers #246 wrapper or C program never asks, and X_get_state still takes no buffer size). True, and the spec's "both failure modes" sentence overstated it; the spec, CHANGELOG and READMEs now say the check protects callers from this change on, that the only such callers are this repository's (rebuilt with the library), and that an out-of-tree C consumer is the case for the deferred _v2 getter.
  • MED (Codex): a wrapper routed to the mono image said make -C eq libtotapi_mono.so; the hint now says make -C tot libtotapi_mono.so (two helper tests).
  • MED (in-house): TotAbiError typo in the tot architecture doc → TotlibAbiError.
  • LOW: duplicate WrlibAbiError definition (left by an aborted scripted edit); __all__ in eq/ti/tot errors.py; the helper docstring said _apply_prototypes calls it (it is load_library); spec status / "fourteen symbols" / "~5 MB" corrected; test_run_so.c header comments list the new exit codes 92/93; extending-tr.md line references replaced by names; tr/tot X_api.h rule paragraphs now mention the PARAMETER and the runtime check; architecture tables say "5 (6) functions + 2 layout queries".

Local: pytest python/ test_run/scripts/tests --forked --timeout=120 --timeout-method=signal → 954 passed, 35 skipped, 1 failed (test_close_raises_exception_group_when_multiple_modules_fail, the known macOS / py3.10 case). X_api_check_so OK for all seven; libtotapi_mono_inspect gate 1b OK; the 14 drivers pass cc -std=c99 -Wall -Wextra -fsyntax-only. The six X_api_check_so run in CI for the first time in this PR: that run is what makes the docs' "CI runs the dlopen drivers" claim true.

@k-yoshimi
k-yoshimi force-pushed the feat/abi-state-layout-check-246 branch from f86cfab to 45dca14 Compare October 17, 2026 05:12
k-yoshimi and others added 2 commits October 17, 2026 15:02
Design for #246: each library exports X_state_abi_version() and
X_state_size() from the BIND(C) type; the C-ABI drivers cross-check them
against the header macro and sizeof(X_state_t); the Python wrappers
refuse a library whose layout differs from their ctypes struct (missing
symbols = pre-#246 library, size or version mismatch = rebuild). The
size-negotiated _v2 getter is deferred. Decisions requested in §7.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TA2KahTs3qLMZL6yjSsWwT
…fuse a mismatch (#246)

Each module fills one fixed-layout struct through X_get_state and the
only rule was "append, bump the macro, rebuild in the same commit";
nothing checked it at run time, so a library and a wrapper from
different commits failed silently (fields written past the struct, or
appended fields read as 0.0).

Fortran is the source of truth: X_STATE_ABI_VERSION is a PARAMETER in
X_state.f90 (tr, tot stay at 2; eq fp ti wr wrx start at 1), exported by
X_state_abi_version(); X_state_size() exports C_SIZEOF of the BIND(C)
type; both callable before X_init. X_api.h mirrors the macro and the
prototypes. The C-ABI drivers (test_negative.c before init, test_run_so.c
through dlopen) fail when either differs from the header. The Python
wrappers call the shared python/_abi_check.py from load_library, before
any struct is passed, and raise XlibAbiError when the library exports
neither symbol, or its version or size differs from XStateC. CI runs the
six per-module dlopen drivers; libtotapi_mono_inspect gates the twelve
symbols. Tests: helper (7 cases) + per-wrapper TestLibraryLayout (4 ×7).
Docs: architecture.md ×7, README ×7, extending-tr en/ja, CHANGELOG.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TA2KahTs3qLMZL6yjSsWwT
@k-yoshimi
k-yoshimi force-pushed the feat/abi-state-layout-check-246 branch from 45dca14 to 7be25ef Compare October 17, 2026 06:05
@k-yoshimi
k-yoshimi merged commit 2e1aae2 into develop Oct 17, 2026
9 checks passed
HengyuLi-Ozaki-lab added a commit to HengyuLi-Ozaki-lab/task that referenced this pull request Oct 18, 2026
develop gained k-yoshimi#256, k-yoshimi#257, k-yoshimi#258, k-yoshimi#259, k-yoshimi#261, k-yoshimi#263 and k-yoshimi#264 since the
last merge (f41fe5b).

Two conflicts:
- CHANGELOG.md: this branch's entry and develop's new entries were
  added at the same place under "### Added"; both kept.
- fp/fp_api.f90, the USE lists of the module: this branch reads NTG1,
  TVOLR and the six P*T arrays from fpcomm, k-yoshimi#259 imports
  fp_param_generation; both kept.

And one change that no line conflict shows. k-yoshimi#259 replaced the clear of
g_prepared in fp_api_set_param by a count of registry sets
(fp_param_generation, compared by fp_run with g_prep_generation). The
global scalars of fp_get_state were gated on g_prepared alone, so after
the merge a set no longer zeroed them, against what this branch
documents and tests ("zero ... and again after a set_param until the
next run that is made"). The gate is now "prepared, and no set since
the last prep" (g_prepared .AND. g_prep_generation ==
fp_param_generation()), which also covers sets through tot's "fp:"
dispatcher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HengyuLi-Ozaki-lab added a commit to HengyuLi-Ozaki-lab/task that referenced this pull request Oct 18, 2026
As asked on the pull request after k-yoshimi#261 (k-yoshimi#246) merged: the seven
doubles this branch appends to fp_state_t make it a new layout, so
FP_STATE_ABI_VERSION goes from 1 to 2 in its three places
(fp/fp_state.f90, fp/fp_api.h, python/fplib/_ffi.py), with a History
line in fp_api.h. A libfpapi.so built before this pull request is 56
bytes shorter, which load_library already refuses on the size since the
merge; one built from the merge alone has the same size at version 1,
and only the version tells it apart. test_ffi.py's
TestLibraryLayout and fp/tests/c_abi/test_negative.c compare the
library, the header and the ctypes struct as before. One test pins the
pairing of version 2 with the seven scalars. In
docs/fp-library/architecture.md the recipe for a layout change names
the three places (it said "in both files", leaving out the Python
constant without which load_library refuses the rebuilt library) and
"currently 2"; the CHANGELOG entry says 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HengyuLi-Ozaki-lab added a commit to HengyuLi-Ozaki-lab/task that referenced this pull request Oct 18, 2026
develop gained k-yoshimi#242, k-yoshimi#243, k-yoshimi#248, k-yoshimi#249, k-yoshimi#251, k-yoshimi#254, k-yoshimi#255, k-yoshimi#256, k-yoshimi#257,
k-yoshimi#258, k-yoshimi#259, k-yoshimi#261, k-yoshimi#263 and k-yoshimi#264 since this branch left it.

One conflict, in python/eqlib/_ffi.py, python/fplib/_ffi.py and
python/trlib/_ffi.py alike: the end of load_library. k-yoshimi#261 (k-yoshimi#246) checks
the state layout of the library just loaded and raises
<Mod>libAbiError on a mismatch; this branch closes the directory it
registered with os.add_dll_directory when the load fails. The check is
now inside the block that cleans up, so a library refused for its
layout does not leave its directory in the process-wide DLL search set
either (or an entry in _DLL_DIR_HANDLES). Order as on develop: the
layout check, then the prototypes.

With the check on the load path, the cases of test_platform_libname.py
that fake the loaded library (ctypes.CDLL patched) were refused by it:
24 of 39 failed. Each of the three files now has an autouse fixture that
patches check_state_layout out for them (the check has its own cases in
test_ffi.py, TestLibraryLayout), and one new case per module puts a
check back that raises AbiMismatchError (with the prototypes passing,
so that only the check can refuse the load) and asserts
<Mod>libAbiError, the directory handle closed and nothing registered;
with the check moved after the cleanup block, that case fails (measured
on eq).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HengyuLi-Ozaki-lab added a commit to HengyuLi-Ozaki-lab/task that referenced this pull request Oct 18, 2026
k-yoshimi#261's check_state_layout (python/_abi_check.py) tells the caller how to
rebuild a library whose state layout does not match: `make -C <mod>
<lib_name>`, or `make -C tot ...` for the mono image. With the .dll names
of this branch that became `make -C eq libeqapi.dll` and `make -C tot
libtotapi_mono.dll`, targets no Makefile has: the defect 75f1fb1 fixed
in the wrappers' not-found hint, back on the new path. A name ending in
.dll now gets "<lib_name> from the Fortran sources", in both messages
(the library without the query symbols, and the one whose version or
size differ). One case in python/tests/test_abi_check.py checks both
messages for a module library and the mono image; without the branch
it fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HengyuLi-Ozaki-lab added a commit to HengyuLi-Ozaki-lab/task that referenced this pull request Oct 21, 2026
No line conflict. What the merge changes for this branch without one:
- k-yoshimi#233 put a scalars object into fp_mcp's state schema, zero after a
  set until the next run; set_param_str opens that window too, which
  the schema text did not say yet (next commit).
- k-yoshimi#256 brought eq_mcp's save tool, which this branch's docs still
  called future, after k-yoshimi#226 (fixed in a later commit).
- k-yoshimi#232 and k-yoshimi#257 moved fp_mcp's fd isolation into python/_fd_isolation;
  this branch's server.py keeps it.
- k-yoshimi#261 added the layout queries fp_state_abi_version and fp_state_size
  to libfpapi.so; fplib refuses a library without them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HengyuLi-Ozaki-lab added a commit to HengyuLi-Ozaki-lab/task that referenced this pull request Oct 21, 2026
With develop in this branch, a few sentences no longer hold:
- eq_mcp's save tool is in develop (k-yoshimi#256), so "once k-yoshimi#226 lands" goes
  from the module docstring, the set_param_str tool docstring and
  README section 8.4.
- The get_state tool docstring lists the returned keys and now names
  scalars too (the first LOW of the k-yoshimi#233 approval, left for this PR).
- libfpapi.so exports eight C symbols since k-yoshimi#261: README section 2 now
  names the six entry points and the two layout queries fplib checks.
- The Sphinx FP MCP pages (en, ja) still said nine tools and that the
  string setter is not reachable through MCP; they now list ten tools,
  with a set_param_str row, and point at README section 8.4.
- fp_mcp/__init__.py's docstring said nine tools.
- The comment above SupportedValue says that this server exposes the
  string setter as the set_param_str tool, as suggested on k-yoshimi#231.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HengyuLi-Ozaki-lab added a commit to HengyuLi-Ozaki-lab/task that referenced this pull request Oct 21, 2026
develop now carries the pieces split out of this branch (k-yoshimi#256 eq_save,
k-yoshimi#257 the MCP fd isolation, k-yoshimi#265 the mcp pin) and k-yoshimi#232, k-yoshimi#233, k-yoshimi#240, k-yoshimi#242,
k-yoshimi#243, k-yoshimi#248, k-yoshimi#249, k-yoshimi#251, k-yoshimi#254, k-yoshimi#255, k-yoshimi#258, k-yoshimi#259, k-yoshimi#261, k-yoshimi#263, k-yoshimi#264. 43
files conflicted; each was resolved as follows.

- Taken from develop (36 files): every file where this branch only
  changed the mcp pin to "mcp>=0.9,<2" (develop has ">=1.2,<2", k-yoshimi#265):
  the seven pyproject.toml, six READMEs, the fp, ti, tot, wr and wrx
  servers, the Sphinx MCP pages, the trlib presentation and its
  builder, the 2026-04-18 MCP plan; and the eq_save pieces now in
  develop in their reviewed form: eq/eq_api.f90, eq/eq_api.h,
  python/eqlib/tests/test_save.py, eq_mcp/tests/test_save_tool.py; and
  tr_mcp/server.py, whose changes here were this branch's own fd
  isolation (develop's _fd_isolation replaces it) and the os._exit(0)
  at the end of main().
- The os._exit(0) that ended main() of tr_mcp and eq_mcp on this branch
  (to skip the Python teardown, cited in k-yoshimi#227 item 5) is gone with
  them: develop's main() returns 0, as approved in k-yoshimi#257. The abort it
  avoided was a double DEALLOCATE, which this branch fixed in
  tr/trcomm.f90 (DEALLOCATE_TRCOMM returns when nothing is allocated);
  that fix stays, merged without a conflict.
- eq/eqfile.f90: develop's (EQSAVE without an argument, EQSAVE_IERR
  doing the work, and its kept EQDSKR error), plus this branch's
  MODELG=25 in EQ_READ's EQDSK branch (k-yoshimi#228-2). eq/eqmenu.f90 merged
  without a conflict but kept this branch's "CALL EQSAVE(IERR)"; it is
  develop's "CALL EQSAVE" again, as EQSAVE takes no argument.
- python/eqlib/_ffi.py, eqlib.py, tests/test_eqlib.py: develop's, plus
  this branch's get_psi_rz (the eq_common_get_psi_rz_ prototype, the
  Eq.get_psi_rz method, its closed-handle assertion).
- eq_mcp/server.py: develop's (the save tool of k-yoshimi#256, the shared fd
  isolation), plus this branch's get_psi_rz handler and tool and its
  --print-tools entry (13 tools). eq_mcp/tests/test_server.py: this
  branch's (13 tools with save and get_psi_rz), which contains
  develop's change.
- python/mcp-servers/eq_mcp/tests/test_import_isolation.py (this branch
  only, no conflict) is removed: it pinned this branch's own isolation
  helper (_install_fd_isolation, _FD_ISOLATION_INSTALLED in eq_mcp and
  tr_mcp), which develop's python/_fd_isolation.py replaces; develop's
  tests/test_fd_isolation.py of each of the seven servers checks the
  same contract (an import leaves fd 1 and sys.stdout alone).
- .github/workflows/python-tests.yml: develop's mcp pin; this branch's
  numpy install (the get_psi_rz tests import numpy) and ../task symlink
  step kept.
- CHANGELOG.md did not conflict. The three k-yoshimi#228 entries this branch held
  back in d8b6e4d are added to [Unreleased] Fixed, after its first
  entry, where no other open pull request inserts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant