Repository navigation
feat(ffi): .dll names and DLL search directory on Windows - #240
Conversation
k-yoshimi
left a comment
There was a problem hiding this comment.
Reviewed at 63b1f425 (based on current develop tip). Approving.
Verified locally (macOS arm64, stock .sos linked into a worktree of the PR head, pytest 9.1.1 --forked --timeout=120 --timeout-method=signal, strict-subTest override from #243 active): python/eqlib python/trlib python/fplib → 172 passed, 8 skipped (Linux-canonical equivalence), 1 xfailed (#189), matching the PR body. The three loader diffs and the three test files are identical after name normalisation (only the <repo>/<mod>/ docstring path differs), so there is one implementation to reason about.
Decision on the "one wart": please take the second option, reword the hint on Windows (e.g. "Build lib<mod>api.dll from the Fortran sources or set <MOD>LIB_PATH"). Adding .so as a second candidate on Windows would advertise a file ctypes cannot load there. Happy to merge once that is pushed (CI will re-run on the fork push; ping me and I will approve the workflow run).
For the record: ctypes.CDLL(..., mode=...) ignores mode on Windows, so passing the RTLD_LAZY fallback through is harmless. #226 also edits python/eqlib/_ffi.py (prototype additions in a different hunk); this PR goes in first, so #226 will need a small rebase.
|
Independent Codex review (second reviewer, read-only) found no HIGH items. Two things to fold into the same push as the hint reword, then this merges:
Confirmed by Codex (matches my review): |
…er a failed load From the review of k-yoshimi#240, in eqlib, trlib and fplib alike: - The "not found" message on Windows ends "Build lib<mod>api.dll from the Fortran sources or set <MOD>LIB_PATH." It named `make -C <mod> lib<mod>api.so`, a target that does not make the file being looked for. Off Windows the message is unchanged, byte for byte (a test holds it). - `load_library` registered the library's directory with os.add_dll_directory and cached the handle before ctypes.CDLL had succeeded: a file that exists but does not load (wrong architecture, a missing dependency) left its directory in the process-wide search set and an entry in the dict. The handle is now cached only after the library has loaded and its prototypes applied; on a failure it is closed. A directory registered by an earlier successful load is not touched. - The docstrings of the module, `_default_lib_path` and `load_library` say `.dll` on Windows. python/_runtime_mode.py: on Windows `mono_lib_path()` makes MONO_LIB_PATH absolute before it probes the image, and returns that path. A bare relative name passed Path.exists() from the working directory, where Windows' DLL search does not look. os.path.abspath, as the loaders use; off Windows the value is used as given, as before. Tests (5 more per package, 3 for the mono path; Windows is simulated, as in the rest of this branch): 10 of them fail against the previous commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Pushed
Tests: five more per package and three for the mono path (Windows simulated, as in the rest of the branch); ten of the eighteen fail against Not changed, for you to say whether it belongs here: the MCP servers wrap the loader's The description above predates this push in three figures ("519 added lines", "8 tests per package", and "neither is the |
develop gained k-yoshimi#242, k-yoshimi#243, k-yoshimi#248, k-yoshimi#249, k-yoshimi#251, k-yoshimi#254, k-yoshimi#255, k-yoshimi#256, k-yoshimi#257, k-yoshimi#258, k-yoshimi#259, k-yoshimi#261, k-yoshimi#263 and k-yoshimi#264 since this branch left it. One conflict, in python/eqlib/_ffi.py, python/fplib/_ffi.py and python/trlib/_ffi.py alike: the end of load_library. k-yoshimi#261 (k-yoshimi#246) checks the state layout of the library just loaded and raises <Mod>libAbiError on a mismatch; this branch closes the directory it registered with os.add_dll_directory when the load fails. The check is now inside the block that cleans up, so a library refused for its layout does not leave its directory in the process-wide DLL search set either (or an entry in _DLL_DIR_HANDLES). Order as on develop: the layout check, then the prototypes. With the check on the load path, the cases of test_platform_libname.py that fake the loaded library (ctypes.CDLL patched) were refused by it: 24 of 39 failed. Each of the three files now has an autouse fixture that patches check_state_layout out for them (the check has its own cases in test_ffi.py, TestLibraryLayout), and one new case per module puts a check back that raises AbiMismatchError (with the prototypes passing, so that only the check can refuse the load) and asserts <Mod>libAbiError, the directory handle closed and nothing registered; with the check moved after the cleanup block, that case fails (measured on eq). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
k-yoshimi#261's check_state_layout (python/_abi_check.py) tells the caller how to rebuild a library whose state layout does not match: `make -C <mod> <lib_name>`, or `make -C tot ...` for the mono image. With the .dll names of this branch that became `make -C eq libeqapi.dll` and `make -C tot libtotapi_mono.dll`, targets no Makefile has: the defect 75f1fb1 fixed in the wrappers' not-found hint, back on the new path. A name ending in .dll now gets "<lib_name> from the Fortran sources", in both messages (the library without the query symbols, and the one whose version or size differ). One case in python/tests/test_abi_check.py checks both messages for a module library and the mono image; without the branch it fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
macOS, clean build: 1009 passed, 38 skipped with the CI flags. Nothing was run on Windows. |
k-yoshimi
left a comment
There was a problem hiding this comment.
Delta review of 0ec0f008 (merge of develop 963ae3dc) + e92c2183, in-house + independent Codex. No HIGH, no MED that this PR introduces. Approving the delta; two LOWs to fold into the same push, then this merges.
Verified
- The hand resolution in the three
_ffi.pyis the right one:check_state_layoutcall andAbiMismatchError → <Mod>libAbiErrorre-raise byte-identical to develop, now inside thetry/except BaseExceptionso a refused layout closes the just-registered DLL directory cookie and writes no dict entry. POSIX path behaviourally identical to develop (new_dir_handle is None). The three modules differ only in the module token. test_platform_libname.py: the autouse no-op ofcheck_state_layoutmakes nothing vacuous (every case still asserts on its own events). The new refusal-path case is real: moving the layout check back outside the cleanup block makes exactly that case fail.e92c2183: the.dllbranch precedes the mono branch, solibtotapi_mono.dllgets it too; the test covers both names × both refusal branches.- Local: 98 passed (abi_check + 3× platform_libname + eqlib/fplib/trlib suites) with the CI flags, macOS, stock
.sos.
LOW, please fold in
python/_abi_check.py:67lib_name.endswith(".dll")is case-sensitive; an explicitEQLIB_PATHending in.DLL(valid on Windows,basenamekeeps the case) falls through to themake -C eq LIBEQAPI.DLLhint this commit removes..lower().endswith(".dll").python/_abi_check.py:81-82the no-query message reads "Rebuild it: libeqapi.dll from the Fortran sources." — a fragment; the wrappers' not-found hint says "Build libeqapi.dll from the Fortran sources". Align the wording.
Not this PR (Codex MED, pre-existing from #261): on a layout refusal the directory cookie is closed but the CDLL handle is never FreeLibrary'd / dlclose'd, so on Windows a refused DLL stays mapped and file-locked until the process exits, which blocks the rebuild the hint asks for. Filed as a follow-up issue; not blocking here.
…ence The two LOW points of the review of e92c218: 1. python/_abi_check.py tested lib_name.endswith(".dll"): an explicit EQLIB_PATH ending in .DLL (valid on Windows; the base name keeps its case) fell through to `make -C eq LIBEQAPI.DLL`. Now lib_name.lower().endswith(".dll"). 2. The message for a library without the query symbols read "Rebuild it: libeqapi.dll from the Fortran sources." It now reads "Build libeqapi.dll from the Fortran sources.", as the wrappers' not-found hint; the mismatch message says "build libeqapi.dll again from the Fortran sources or update python/eqlib/_ffi.py". The messages for a .so name are unchanged (compared with develop's for a module library and the mono image, both refusals). The case of python/tests/test_abi_check.py checks both messages for libxxapi.dll, libtotapi_mono.dll and LIBXXAPI.DLL, now without subTest (the expected text names the library); with the case-sensitive test it fails under the CI flags (--forked), with and without subTest (measured; develop's strict subTest handling, k-yoshimi#241 (PR k-yoshimi#243), reports the subtest). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Both LOW points are in macOS: the whole suite 1009 passed, 38 skipped with the CI flags. This head merges cleanly with |
k-yoshimi
left a comment
There was a problem hiding this comment.
Delta review of c268696d (the two LOW items), run through the full review cycle (Codex + in-house domain/test-oracle lenses + docs/workflow review), with the suite run locally under the CI flags.
Verdict: approve. Both LOW points are closed as described: .lower().endswith(".dll") catches LIBXXAPI.DLL, the hints are sentences, the non-.dll wording is byte-identical to before on both error paths (every loader passes os.path.basename), and the rewritten test fails on the previous case-sensitive code (the reviewers ran it against a reconstruction of the old helper).
Local: 1008 passed, 38 skipped, 1 failed (the known Python 3.10 ExceptionGroup case, same on develop). CI green at c268696d.
Two small things the reviewers raised; neither blocks the merge:
python/tests/test_abi_check.py:113-123— withoutsubTest, CI (which installspytest-subtests) stops at the first failing case and hides the other five; attribution is preserved because the expected text embeds the name, so this is a convenience only.CHANGELOG.mdhas no entry for the Windows.dllsupport. Every PR merged recently carries one; a short Added line (the.dlllookup,os.add_dll_directoryregistration with rollback on a failed load, theMONO_LIB_PATHnormalisation) would be welcome, in this PR or a follow-up.
Noted, not requested: tilib/wrlib/wrxlib/totlib keep .so-only loaders, as the PR text says; a follow-up when there is a Windows user for them.
Merge waits for the maintainers' instruction, as agreed.
develop now carries the pieces split out of this branch (k-yoshimi#256 eq_save, k-yoshimi#257 the MCP fd isolation, k-yoshimi#265 the mcp pin) and k-yoshimi#232, k-yoshimi#233, k-yoshimi#240, k-yoshimi#242, k-yoshimi#243, k-yoshimi#248, k-yoshimi#249, k-yoshimi#251, k-yoshimi#254, k-yoshimi#255, k-yoshimi#258, k-yoshimi#259, k-yoshimi#261, k-yoshimi#263, k-yoshimi#264. 43 files conflicted; each was resolved as follows. - Taken from develop (36 files): every file where this branch only changed the mcp pin to "mcp>=0.9,<2" (develop has ">=1.2,<2", k-yoshimi#265): the seven pyproject.toml, six READMEs, the fp, ti, tot, wr and wrx servers, the Sphinx MCP pages, the trlib presentation and its builder, the 2026-04-18 MCP plan; and the eq_save pieces now in develop in their reviewed form: eq/eq_api.f90, eq/eq_api.h, python/eqlib/tests/test_save.py, eq_mcp/tests/test_save_tool.py; and tr_mcp/server.py, whose changes here were this branch's own fd isolation (develop's _fd_isolation replaces it) and the os._exit(0) at the end of main(). - The os._exit(0) that ended main() of tr_mcp and eq_mcp on this branch (to skip the Python teardown, cited in k-yoshimi#227 item 5) is gone with them: develop's main() returns 0, as approved in k-yoshimi#257. The abort it avoided was a double DEALLOCATE, which this branch fixed in tr/trcomm.f90 (DEALLOCATE_TRCOMM returns when nothing is allocated); that fix stays, merged without a conflict. - eq/eqfile.f90: develop's (EQSAVE without an argument, EQSAVE_IERR doing the work, and its kept EQDSKR error), plus this branch's MODELG=25 in EQ_READ's EQDSK branch (k-yoshimi#228-2). eq/eqmenu.f90 merged without a conflict but kept this branch's "CALL EQSAVE(IERR)"; it is develop's "CALL EQSAVE" again, as EQSAVE takes no argument. - python/eqlib/_ffi.py, eqlib.py, tests/test_eqlib.py: develop's, plus this branch's get_psi_rz (the eq_common_get_psi_rz_ prototype, the Eq.get_psi_rz method, its closed-handle assertion). - eq_mcp/server.py: develop's (the save tool of k-yoshimi#256, the shared fd isolation), plus this branch's get_psi_rz handler and tool and its --print-tools entry (13 tools). eq_mcp/tests/test_server.py: this branch's (13 tools with save and get_psi_rz), which contains develop's change. - python/mcp-servers/eq_mcp/tests/test_import_isolation.py (this branch only, no conflict) is removed: it pinned this branch's own isolation helper (_install_fd_isolation, _FD_ISOLATION_INSTALLED in eq_mcp and tr_mcp), which develop's python/_fd_isolation.py replaces; develop's tests/test_fd_isolation.py of each of the seven servers checks the same contract (an import leaves fd 1 and sys.stdout alone). - .github/workflows/python-tests.yml: develop's mcp pin; this branch's numpy install (the get_psi_rz tests import numpy) and ../task symlink step kept. - CHANGELOG.md did not conflict. The three k-yoshimi#228 entries this branch held back in d8b6e4d are added to [Unreleased] Fixed, after its first entry, where no other open pull request inserts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What
The ctypes loaders of
eqlib,trlibandfplib(python/<mod>/_ffi.py) learn the two things Windows needs:libeqapi.dll,libtrapi.dllandlibfpapi.dllin<repo>/<mod>/and<repo>/lib/. Every other platform keeps.so, macOS included (that is the namemake lib<mod>api.soproduces there). The "not found" errors name the file that was looked for.PATH. On Windowsload_librarynow makes the path absolute and registers its directory withos.add_dll_directory, once perdirectory (the handles are kept in a dict keyed by the normalised path, because the MCP servers open a fresh library
handle after every finalize). The gfortran runtime DLLs (
libgfortran,libquadmath,libgcc_s,libwinpthread)are expected next to the library. For a full path ctypes already searches the library's own directory, so the
registration is a second line of defence, as the comment in the code says; the same comment explains why
winmode=0must not be used to "fix" a load error: it skips registered directories.The three modules get the same change and differ only in the module name. Each package gets a
tests/test_platform_libname.py(8 tests per package, 24 in all) that fakes Windows (sys.platform,os.add_dll_directory,ctypes.CDLL) so it runs on Linux and macOS.Why
Downstream, the TASK Web Client's Windows installer ships
lib<mod>api.dllbeside the gfortran runtime DLLs. Ondevelopsuch a build loads only whenEQLIB_PATH/TRLIB_PATH/FPLIB_PATHname it; the default search knowsonly
.sonames.In use downstream (TASK Web Client) since 2026-09-25.
Checked
python/eqlib python/trlib python/fplib, with upstream CI's flags (--forked --timeout=120 --timeout-method=signal):172 passed, 8 skipped, 1 xpassed. macOS (Apple silicon), gfortran 15.2.0 (MacPorts,-O0 -fbounds-check -fcheck=all), Python 3.12.13, pytest 9.0.3, mcp 1.27.1, bpsddevelopfb8414f.eqlib(2),trlib(3) andfplib(3), Linux-only since fplib + wrxlib equivalence tests fail 1e-10 tolerance on develop (mismatches ~1e-9) #213("Equivalence tests are Linux-canonical"). They were not run here and are left to upstream's Linux CI; this change
does not alter what they load on Linux. The xpass is the existing
test_NSMAX_in_range(tr_tst2 fixture: NSMAX>2 mutation produces NaN scalars (RNU/RTU species 3+ unpopulated) #189).load_librarywas not run on Windows here; the new tests simulate it. Againstdevelop's_ffi.pysix of the eight tests per package fail; the other two (macOS and Linux keep.so; nothing isregistered off Windows) are regression guards.
Things to know
sys.platformset to linux, darwin, cygwin and freebsd, the candidate paths,the default path and the "not found" message of each of the three wrappers are byte-identical to
develop's (12cases compared).
INSTALL_Ubuntu.md) and itsMakefiles build only
lib<mod>api.so(there is no.dlltarget); a library named throughEQLIB_PATHand itssiblings loads as before, since those variables are read before the candidate names.
libeqapi.dll not found at ... Tried EQLIB_PATH and [...]. Build it viamake -C eq libeqapi.soor set EQLIB_PATH.The hint names the.sotarget because that is theonly one the Makefiles have. Either of two small changes would remove the oddity, and the maintainers can pick: add
.soas a second candidate on Windows (after.dll), or reword the hint on Windows. Say which and it will be done.tilib,wrlib,wrxlibandtotlibare not changed (still.soonly), and neither is theMONO_LIB_PATHprobein
python/_runtime_mode.py; it loads by the path it is given, which works for an absolute path. This pull requestis limited to the three wrappers a Windows installer needs first.
Scope
Based on current
develop(4a5d977) and standing alone on it: the results above were obtained ondevelopplusthese three commits. Independent of #226 and of #231, #232 and #233. The change shares
python/eqlib/_ffi.pywith #226and
python/fplib/_ffi.pywith #233, in different hunks; trial merges with #226's head (682d9af), #231, #232 and #233are conflict-free.
🤖 Generated with Claude Code