Skip to content

feat(ffi): .dll names and DLL search directory on Windows - #240

Merged
k-yoshimi merged 7 commits into
k-yoshimi:developfrom
HengyuLi-Ozaki-lab:pr/ffi-windows-dll-names
Oct 20, 2026
Merged

k-yoshimi merged 7 commits into
k-yoshimi:developfrom
HengyuLi-Ozaki-lab:pr/ffi-windows-dll-names

Conversation

@HengyuLi-Ozaki-lab

Copy link
Copy Markdown
Collaborator

What

The ctypes loaders of eqlib, trlib and fplib (python/<mod>/_ffi.py) learn the two things Windows needs:

  • Library name. On Windows the default search looks for libeqapi.dll, libtrapi.dll and libfpapi.dll in
    <repo>/<mod>/ and <repo>/lib/. Every other platform keeps .so, macOS included (that is the name
    make lib<mod>api.so produces there). The "not found" errors name the file that was looked for.
  • Dependent DLLs. Since Python 3.8 the DLLs a library depends on are no longer found through PATH. On Windows
    load_library now makes the path absolute and registers its directory with os.add_dll_directory, once per
    directory (the handles are kept in a dict keyed by the normalised path, because the MCP servers open a fresh library
    handle after every finalize). The gfortran runtime DLLs (libgfortran, libquadmath, libgcc_s, libwinpthread)
    are expected next to the library. For a full path ctypes already searches the library's own directory, so the
    registration is a second line of defence, as the comment in the code says; the same comment explains why
    winmode=0 must not be used to "fix" a load error: it skips registered directories.

The three modules get the same change and differ only in the module name. Each package gets a
tests/test_platform_libname.py (8 tests per package, 24 in all) that fakes Windows (sys.platform,
os.add_dll_directory, ctypes.CDLL) so it runs on Linux and macOS.

Why

Downstream, the TASK Web Client's Windows installer ships lib<mod>api.dll beside the gfortran runtime DLLs. On
develop such a build loads only when EQLIB_PATH / TRLIB_PATH / FPLIB_PATH name it; the default search knows
only .so names.

In use downstream (TASK Web Client) since 2026-09-25.

Checked

  • python/eqlib python/trlib python/fplib, with upstream CI's flags (--forked --timeout=120 --timeout-method=signal): 172 passed, 8 skipped, 1 xpassed. macOS (Apple silicon), gfortran 15.2.0 (MacPorts,
    -O0 -fbounds-check -fcheck=all), Python 3.12.13, pytest 9.0.3, mcp 1.27.1, bpsd develop fb8414f.
  • The 8 skips are the equivalence tests of eqlib (2), trlib (3) and fplib (3), Linux-only since fplib + wrxlib equivalence tests fail 1e-10 tolerance on develop (mismatches ~1e-9) #213
    ("Equivalence tests are Linux-canonical"). They were not run here and are left to upstream's Linux CI; this change
    does not alter what they load on Linux. The xpass is the existing test_NSMAX_in_range (tr_tst2 fixture: NSMAX>2 mutation produces NaN scalars (RNU/RTU species 3+ unpopulated) #189).
  • The Windows branch of load_library was not run on Windows here; the new tests simulate it. Against
    develop's _ffi.py six of the eight tests per package fail; the other two (macOS and Linux keep .so; nothing is
    registered off Windows) are regression guards.

Things to know

  • Off Windows nothing changes. With sys.platform set to linux, darwin, cygwin and freebsd, the candidate paths,
    the default path and the "not found" message of each of the three wrappers are byte-identical to develop's (12
    cases compared).
  • On Windows this is not a regression. Upstream documents an Ubuntu install (INSTALL_Ubuntu.md) and its
    Makefiles build only lib<mod>api.so (there is no .dll target); a library named through EQLIB_PATH and its
    siblings loads as before, since those variables are read before the candidate names.
  • Size. The 519 added lines are 399 of tests (133 per package) and 120 of loader (40 per wrapper).
  • One wart. On Windows the "not found" message now reads libeqapi.dll not found at ... Tried EQLIB_PATH and [...]. Build it via make -C eq libeqapi.so or set EQLIB_PATH. The hint names the .so target because that is the
    only one the Makefiles have. Either of two small changes would remove the oddity, and the maintainers can pick: add
    .so as a second candidate on Windows (after .dll), or reword the hint on Windows. Say which and it will be done.
  • tilib, wrlib, wrxlib and totlib are not changed (still .so only), and neither is the MONO_LIB_PATH probe
    in python/_runtime_mode.py; it loads by the path it is given, which works for an absolute path. This pull request
    is limited to the three wrappers a Windows installer needs first.

Scope

Based on current develop (4a5d977) and standing alone on it: the results above were obtained on develop plus
these three commits. Independent of #226 and of #231, #232 and #233. The change shares python/eqlib/_ffi.py with #226
and python/fplib/_ffi.py with #233, in different hunks; trial merges with #226's head (682d9af), #231, #232 and #233
are conflict-free.

🤖 Generated with Claude Code

HengyuLi-Ozaki-lab and others added 3 commits October 3, 2026 13:01
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 6a96e56)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit d5dd446)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit d4b3cc2)

@k-yoshimi k-yoshimi left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 63b1f425 (based on current develop tip). Approving.

Verified locally (macOS arm64, stock .sos linked into a worktree of the PR head, pytest 9.1.1 --forked --timeout=120 --timeout-method=signal, strict-subTest override from #243 active): python/eqlib python/trlib python/fplib → 172 passed, 8 skipped (Linux-canonical equivalence), 1 xfailed (#189), matching the PR body. The three loader diffs and the three test files are identical after name normalisation (only the <repo>/<mod>/ docstring path differs), so there is one implementation to reason about.

Decision on the "one wart": please take the second option, reword the hint on Windows (e.g. "Build lib<mod>api.dll from the Fortran sources or set <MOD>LIB_PATH"). Adding .so as a second candidate on Windows would advertise a file ctypes cannot load there. Happy to merge once that is pushed (CI will re-run on the fork push; ping me and I will approve the workflow run).

For the record: ctypes.CDLL(..., mode=...) ignores mode on Windows, so passing the RTLD_LAZY fallback through is harmless. #226 also edits python/eqlib/_ffi.py (prototype additions in a different hunk); this PR goes in first, so #226 will need a small rebase.

@k-yoshimi

Copy link
Copy Markdown
Owner

Independent Codex review (second reviewer, read-only) found no HIGH items. Two things to fold into the same push as the hint reword, then this merges:

  1. MED — MONO_LIB_PATH with a bare relative value bypasses the new normalisation on Windows. python/_runtime_mode.py:40-51 probes with ctypes.CDLL(str(p)) before any _ffi.load_library runs; Path.exists() succeeds from cwd, but a bare libtotapi_mono.dll goes through Windows' secure search (no cwd, no add_dll_directory) and fails to find the DLL or its gfortran neighbours. Either make p absolute there on Windows (p = p.resolve() before the probe, and return the absolute path), or state in the PR/README that mono mode is not supported on Windows yet. The PR body already says the probe "works for an absolute path", so the one-liner is the natural choice.
  2. LOW — a failed load leaves its directory registered. The os.add_dll_directory handle is cached before ctypes.CDLL succeeds (python/eqlib/_ffi.py:302-310 and the tr/fp twins). If the file exists but is wrong-arch or has missing dependencies, the directory stays in the process-wide search set and the dict grows per distinct failing path. Register, try the load, and on failure handle.close() without caching.
  3. The docstrings of _default_lib_path() / load_library() and the module header still describe .so only; one clause each (".dll on Windows") keeps them honest alongside the hint reword.

Confirmed by Codex (matches my review): ctypes.CDLL ignores mode on Windows and, for a path containing a separator, adds LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR; cygwin stays on the .so path by design; repeated MCP reopen cycles are unaffected; the tests exercise the real _candidate_paths / load_library (not a copy), though they cannot see native Windows semantics — a Windows smoke run downstream remains the real validation.

…er a failed load

From the review of k-yoshimi#240, in eqlib, trlib and fplib alike:

- The "not found" message on Windows ends "Build lib<mod>api.dll from the
  Fortran sources or set <MOD>LIB_PATH." It named `make -C <mod>
  lib<mod>api.so`, a target that does not make the file being looked for.
  Off Windows the message is unchanged, byte for byte (a test holds it).
- `load_library` registered the library's directory with
  os.add_dll_directory and cached the handle before ctypes.CDLL had
  succeeded: a file that exists but does not load (wrong architecture, a
  missing dependency) left its directory in the process-wide search set and
  an entry in the dict. The handle is now cached only after the library
  has loaded and its prototypes applied; on a failure it is closed. A
  directory registered by an earlier successful load is not touched.
- The docstrings of the module, `_default_lib_path` and `load_library` say
  `.dll` on Windows.

python/_runtime_mode.py: on Windows `mono_lib_path()` makes MONO_LIB_PATH
absolute before it probes the image, and returns that path. A bare relative
name passed Path.exists() from the working directory, where Windows' DLL
search does not look. os.path.abspath, as the loaders use; off Windows the
value is used as given, as before.

Tests (5 more per package, 3 for the mono path; Windows is simulated, as in
the rest of this branch): 10 of them fail against the previous commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@HengyuLi-Ozaki-lab

Copy link
Copy Markdown
Collaborator Author

Pushed 75f1fb17, the same change in the three loaders:

  1. The hint on Windows reads "Build lib<mod>api.dll from the Fortran sources or set <MOD>LIB_PATH." Off Windows the message is unchanged byte for byte, and a test holds it.
  2. MONO_LIB_PATH (python/_runtime_mode.py): on Windows the path is made absolute before the probe, and that path is returned. I used os.path.abspath, as the loaders do, rather than Path.resolve(): it does not follow links or substituted drives. Off Windows the value is used as given.
  3. A failed load leaves nothing registered. The directory handle is cached only after ctypes.CDLL and the prototypes succeeded; on a failure it is closed. A directory registered by an earlier successful load is not touched.
  4. Docstrings: the module header, _default_lib_path() and load_library() say .dll on Windows.

Tests: five more per package and three for the mono path (Windows simulated, as in the rest of the branch); ten of the eighteen fail against 63b1f425. python/ on this branch: 765 passed, 35 skipped, 1 xpassed with CI's flags; with the mono image, the four test_mono_* files: 17 passed. develop has moved to a0b58408 (#242, #243) since your review: this branch merges into it without a conflict, and on develop with #231, #232, #233 and #240 merged together python/ gives 831 passed, 35 skipped, 1 xfailed (#243's strict subTest active).

Not changed, for you to say whether it belongs here: the MCP servers wrap the loader's FileNotFoundError and append their own "Build it via make -C <mod> lib<mod>api.so" (eq_mcp, tr_mcp, fp_mcp: _wrap_*_error), so on Windows an MCP client still reads the reworded hint followed by the old one.

The description above predates this push in three figures ("519 added lines", "8 tests per package", and "neither is the MONO_LIB_PATH probe [changed]"). Still true: none of this has run on Windows here. The workflow run needs your approval.

HengyuLi-Ozaki-lab and others added 2 commits October 18, 2026 18:55
develop gained k-yoshimi#242, k-yoshimi#243, k-yoshimi#248, k-yoshimi#249, k-yoshimi#251, k-yoshimi#254, k-yoshimi#255, k-yoshimi#256, k-yoshimi#257,
k-yoshimi#258, k-yoshimi#259, k-yoshimi#261, k-yoshimi#263 and k-yoshimi#264 since this branch left it.

One conflict, in python/eqlib/_ffi.py, python/fplib/_ffi.py and
python/trlib/_ffi.py alike: the end of load_library. k-yoshimi#261 (k-yoshimi#246) checks
the state layout of the library just loaded and raises
<Mod>libAbiError on a mismatch; this branch closes the directory it
registered with os.add_dll_directory when the load fails. The check is
now inside the block that cleans up, so a library refused for its
layout does not leave its directory in the process-wide DLL search set
either (or an entry in _DLL_DIR_HANDLES). Order as on develop: the
layout check, then the prototypes.

With the check on the load path, the cases of test_platform_libname.py
that fake the loaded library (ctypes.CDLL patched) were refused by it:
24 of 39 failed. Each of the three files now has an autouse fixture that
patches check_state_layout out for them (the check has its own cases in
test_ffi.py, TestLibraryLayout), and one new case per module puts a
check back that raises AbiMismatchError (with the prototypes passing,
so that only the check can refuse the load) and asserts
<Mod>libAbiError, the directory handle closed and nothing registered;
with the check moved after the cleanup block, that case fails (measured
on eq).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
k-yoshimi#261's check_state_layout (python/_abi_check.py) tells the caller how to
rebuild a library whose state layout does not match: `make -C <mod>
<lib_name>`, or `make -C tot ...` for the mono image. With the .dll names
of this branch that became `make -C eq libeqapi.dll` and `make -C tot
libtotapi_mono.dll`, targets no Makefile has: the defect 75f1fb1 fixed
in the wrappers' not-found hint, back on the new path. A name ending in
.dll now gets "<lib_name> from the Fortran sources", in both messages
(the library without the query symbols, and the one whose version or
size differ). One case in python/tests/test_abi_check.py checks both
messages for a module library and the mono image; without the branch
it fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@HengyuLi-Ozaki-lab

Copy link
Copy Markdown
Collaborator Author

develop (963ae3dc) is merged in, 0ec0f008, and one fix follows, e92c2183.

  • The merge. The conflict was the end of load_library in the eq, fp and tr wrappers. feat(abi): every library reports its state-struct layout; wrappers refuse a mismatch (#246) #261's layout check now sits inside the block that closes the newly registered DLL directory, so a library refused for its layout leaves no directory in the search set either; the order is develop's (the check, then the prototypes). The cases of test_platform_libname.py fake the loaded library, which the check refuses (24 of 39 failed), so they now patch it out (it has its own tests), and one case per module checks the refusal path: the error, the handle closed, nothing registered.
  • e92c2183: with .dll names, the layout check's rebuild hint said make -C eq libeqapi.dll (and make -C tot libtotapi_mono.dll), targets no Makefile has: what this pull request fixed in the not-found hint, back on the new path. For a .dll name the hint is now "Rebuild it: libeqapi.dll from the Fortran sources" (and the same in the mismatch message); one case in python/tests/test_abi_check.py.

macOS, clean build: 1009 passed, 38 skipped with the CI flags. Nothing was run on Windows.

@k-yoshimi k-yoshimi left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta review of 0ec0f008 (merge of develop 963ae3dc) + e92c2183, in-house + independent Codex. No HIGH, no MED that this PR introduces. Approving the delta; two LOWs to fold into the same push, then this merges.

Verified

  • The hand resolution in the three _ffi.py is the right one: check_state_layout call and AbiMismatchError → <Mod>libAbiError re-raise byte-identical to develop, now inside the try/except BaseException so a refused layout closes the just-registered DLL directory cookie and writes no dict entry. POSIX path behaviourally identical to develop (new_dir_handle is None). The three modules differ only in the module token.
  • test_platform_libname.py: the autouse no-op of check_state_layout makes nothing vacuous (every case still asserts on its own events). The new refusal-path case is real: moving the layout check back outside the cleanup block makes exactly that case fail.
  • e92c2183: the .dll branch precedes the mono branch, so libtotapi_mono.dll gets it too; the test covers both names × both refusal branches.
  • Local: 98 passed (abi_check + 3× platform_libname + eqlib/fplib/trlib suites) with the CI flags, macOS, stock .sos.

LOW, please fold in

  1. python/_abi_check.py:67 lib_name.endswith(".dll") is case-sensitive; an explicit EQLIB_PATH ending in .DLL (valid on Windows, basename keeps the case) falls through to the make -C eq LIBEQAPI.DLL hint this commit removes. .lower().endswith(".dll").
  2. python/_abi_check.py:81-82 the no-query message reads "Rebuild it: libeqapi.dll from the Fortran sources." — a fragment; the wrappers' not-found hint says "Build libeqapi.dll from the Fortran sources". Align the wording.

Not this PR (Codex MED, pre-existing from #261): on a layout refusal the directory cookie is closed but the CDLL handle is never FreeLibrary'd / dlclose'd, so on Windows a refused DLL stays mapped and file-locked until the process exits, which blocks the rebuild the hint asks for. Filed as a follow-up issue; not blocking here.

…ence

The two LOW points of the review of e92c218:
1. python/_abi_check.py tested lib_name.endswith(".dll"): an explicit
   EQLIB_PATH ending in .DLL (valid on Windows; the base name keeps its
   case) fell through to `make -C eq LIBEQAPI.DLL`. Now
   lib_name.lower().endswith(".dll").
2. The message for a library without the query symbols read "Rebuild
   it: libeqapi.dll from the Fortran sources." It now reads "Build
   libeqapi.dll from the Fortran sources.", as the wrappers' not-found
   hint; the mismatch message says "build libeqapi.dll again from the
   Fortran sources or update python/eqlib/_ffi.py". The messages for a
   .so name are unchanged (compared with develop's for a module library
   and the mono image, both refusals).

The case of python/tests/test_abi_check.py checks both messages for
libxxapi.dll, libtotapi_mono.dll and LIBXXAPI.DLL, now without subTest
(the expected text names the library); with the case-sensitive test it
fails under the CI flags (--forked), with and without subTest (measured;
develop's strict subTest handling, k-yoshimi#241 (PR k-yoshimi#243), reports the
subtest).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@HengyuLi-Ozaki-lab

Copy link
Copy Markdown
Collaborator Author

Both LOW points are in c268696d: _abi_check.py tests .lower().endswith(".dll"), so an explicit EQLIB_PATH ending in .DLL gets the Windows hint too, and the hint is a sentence: "Build libeqapi.dll from the Fortran sources." for a library without the query symbols, "... build libeqapi.dll again from the Fortran sources or update python/eqlib/_ffi.py." for a mismatch. The messages for a .so name are unchanged (compared with develop's for a module library and the mono image). The test covers libxxapi.dll, libtotapi_mono.dll and LIBXXAPI.DLL, without subTest; with the case-sensitive test it fails under --forked. #266 is not touched here.

macOS: the whole suite 1009 passed, 38 skipped with the CI flags. This head merges cleanly with develop 4b5463e4.

@k-yoshimi k-yoshimi left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta review of c268696d (the two LOW items), run through the full review cycle (Codex + in-house domain/test-oracle lenses + docs/workflow review), with the suite run locally under the CI flags.

Verdict: approve. Both LOW points are closed as described: .lower().endswith(".dll") catches LIBXXAPI.DLL, the hints are sentences, the non-.dll wording is byte-identical to before on both error paths (every loader passes os.path.basename), and the rewritten test fails on the previous case-sensitive code (the reviewers ran it against a reconstruction of the old helper).

Local: 1008 passed, 38 skipped, 1 failed (the known Python 3.10 ExceptionGroup case, same on develop). CI green at c268696d.

Two small things the reviewers raised; neither blocks the merge:

  • python/tests/test_abi_check.py:113-123 — without subTest, CI (which installs pytest-subtests) stops at the first failing case and hides the other five; attribution is preserved because the expected text embeds the name, so this is a convenience only.
  • CHANGELOG.md has no entry for the Windows .dll support. Every PR merged recently carries one; a short Added line (the .dll lookup, os.add_dll_directory registration with rollback on a failed load, the MONO_LIB_PATH normalisation) would be welcome, in this PR or a follow-up.

Noted, not requested: tilib/wrlib/wrxlib/totlib keep .so-only loaders, as the PR text says; a follow-up when there is a Windows user for them.

Merge waits for the maintainers' instruction, as agreed.

@k-yoshimi
k-yoshimi merged commit 889394c into k-yoshimi:develop Oct 20, 2026
3 checks passed
HengyuLi-Ozaki-lab added a commit to HengyuLi-Ozaki-lab/task that referenced this pull request Oct 21, 2026
develop now carries the pieces split out of this branch (k-yoshimi#256 eq_save,
k-yoshimi#257 the MCP fd isolation, k-yoshimi#265 the mcp pin) and k-yoshimi#232, k-yoshimi#233, k-yoshimi#240, k-yoshimi#242,
k-yoshimi#243, k-yoshimi#248, k-yoshimi#249, k-yoshimi#251, k-yoshimi#254, k-yoshimi#255, k-yoshimi#258, k-yoshimi#259, k-yoshimi#261, k-yoshimi#263, k-yoshimi#264. 43
files conflicted; each was resolved as follows.

- Taken from develop (36 files): every file where this branch only
  changed the mcp pin to "mcp>=0.9,<2" (develop has ">=1.2,<2", k-yoshimi#265):
  the seven pyproject.toml, six READMEs, the fp, ti, tot, wr and wrx
  servers, the Sphinx MCP pages, the trlib presentation and its
  builder, the 2026-04-18 MCP plan; and the eq_save pieces now in
  develop in their reviewed form: eq/eq_api.f90, eq/eq_api.h,
  python/eqlib/tests/test_save.py, eq_mcp/tests/test_save_tool.py; and
  tr_mcp/server.py, whose changes here were this branch's own fd
  isolation (develop's _fd_isolation replaces it) and the os._exit(0)
  at the end of main().
- The os._exit(0) that ended main() of tr_mcp and eq_mcp on this branch
  (to skip the Python teardown, cited in k-yoshimi#227 item 5) is gone with
  them: develop's main() returns 0, as approved in k-yoshimi#257. The abort it
  avoided was a double DEALLOCATE, which this branch fixed in
  tr/trcomm.f90 (DEALLOCATE_TRCOMM returns when nothing is allocated);
  that fix stays, merged without a conflict.
- eq/eqfile.f90: develop's (EQSAVE without an argument, EQSAVE_IERR
  doing the work, and its kept EQDSKR error), plus this branch's
  MODELG=25 in EQ_READ's EQDSK branch (k-yoshimi#228-2). eq/eqmenu.f90 merged
  without a conflict but kept this branch's "CALL EQSAVE(IERR)"; it is
  develop's "CALL EQSAVE" again, as EQSAVE takes no argument.
- python/eqlib/_ffi.py, eqlib.py, tests/test_eqlib.py: develop's, plus
  this branch's get_psi_rz (the eq_common_get_psi_rz_ prototype, the
  Eq.get_psi_rz method, its closed-handle assertion).
- eq_mcp/server.py: develop's (the save tool of k-yoshimi#256, the shared fd
  isolation), plus this branch's get_psi_rz handler and tool and its
  --print-tools entry (13 tools). eq_mcp/tests/test_server.py: this
  branch's (13 tools with save and get_psi_rz), which contains
  develop's change.
- python/mcp-servers/eq_mcp/tests/test_import_isolation.py (this branch
  only, no conflict) is removed: it pinned this branch's own isolation
  helper (_install_fd_isolation, _FD_ISOLATION_INSTALLED in eq_mcp and
  tr_mcp), which develop's python/_fd_isolation.py replaces; develop's
  tests/test_fd_isolation.py of each of the seven servers checks the
  same contract (an import leaves fd 1 and sys.stdout alone).
- .github/workflows/python-tests.yml: develop's mcp pin; this branch's
  numpy install (the get_psi_rz tests import numpy) and ../task symlink
  step kept.
- CHANGELOG.md did not conflict. The three k-yoshimi#228 entries this branch held
  back in d8b6e4d are added to [Unreleased] Fixed, after its first
  entry, where no other open pull request inserts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants