Auto update non py ports + bump arrow, py-arrow and gdal - #270
Merged
Merged
Conversation
The updater only handled vcpkg_from_pythonhosted and vcpkg_from_github
ports, so a port that fetches a release archive directly was invisible
to it - arrow being the one in this registry.
Add update_distfile_ports(). Such a port has no REPO to derive the
upstream from, so it declares where its releases are published:
# GITHUB_REPO apache/arrow
# GITHUB_REF apache-arrow-${VERSION}
The latest non-prerelease tag is resolved through the existing
get_latest_tag(), and the archive is downloaded from the port's own URLS
value with ${VERSION} substituted, so a port keeps using whatever
mirror it already points at rather than the GitHub tarball.
Also extract get_version_key() and reuse it in update_github_ports().
No port declares the markers yet, so this changes nothing on its own.
update_github_ports() only looked at ports named py-*, so gdal was never
considered even though it has everything the updater needs. Dropping the
name filter outright would also enrol pybind11 and python3, and
auto-bumping the interpreter under the whole registry is not wanted.
Replace the filter with is_auto_update_enabled(): py-* ports are still
updated by default, any other port opts in with a marker comment:
# AUTO_UPDATE
The same gate now applies to update_distfile_ports(), so one grep over
ports/*/portfile.cmake lists everything the bot manages. A port without
# GITHUB_REPO is still skipped silently there rather than reported as a
failure, since py-* sdists fetched with vcpkg_download_distfile pass the
gate but belong to update_pypi_ports().
The github loop now iterates ports/ directly instead of walking it
recursively, which the name filter used to make safe.
Opt gdal in, which bumps it 3.12.4 -> 3.13.3, and rebase sqlite3.diff:
upstream renamed the imported target SQLite::SQLite3 to SQLite3::SQLite3
and kept the old name only as an ALIAS, so the patch's property writes
had to move to the real target - set_target_properties() on an alias is
an error. All five gdal patches apply cleanly to 3.13.3.
arrow opts in separately, in the commit that ports its pyarrow build.
pyarrow dropped setup.py in 24.0.0 and moved to a PEP 517 build over scikit-build-core, so `python setup.py build_ext ... install` fails with "No such file or directory" on anything newer than 23.x. The C++ side of 25.0.1 is fine - upstream vcpkg ships exactly this version with the same five patches - only this registry's python feature was left behind. Build the wheel with pip instead. Build isolation stays off because x_vcpkg_get_python_packages() has already put requirements-build.txt in the venv, and --no-index keeps the build from reaching PyPI on its own. Two details the migration does not carry over directly: - There is no cmake.generator config-setting; scikit-build-core takes the generator from the environment. It also shells out to cmake and ninja, and when it cannot find them on PATH it adds cmake and ninja wheels to the build requirements, which --no-index then refuses - so vcpkg's own copies are prepended to PATH. - setup.py took --rpath. The replacement CMakeLists sets CMAKE_INSTALL_RPATH unconditionally and, on macOS, appends the absolute directory libarrow happens to sit in at build time, which for vcpkg is the staging tree. A plain set() cannot be overridden with -D, hence 0008-pyarrow-relative-rpath.patch, which lets the caller pass the relative RPATH the old flag used to supply. The port also declares the # AUTO_UPDATE and # GITHUB_REPO markers, which are inert comments until the updater that reads them lands.
3nids
changed the title
isort 9 declares a hard runtime dependency on mypy-extensions>=1.1.0, which 8.0.1 did not have and which has no port here, so the nightly bump to 9.x fails with "No module named 'mypy_extensions'". The updater only ever rewrites versions, never dependencies, so it reproposes the same broken bump every night. Add the missing port - mypy_extensions is a single pure-python module built with flit_core - and pick up the dependency along with the bump the bot was already trying to make.
gmp's portfile pins an exact MSYS2 package, and MSYS2 rebuilt
autoconf2.71 from pkgrel 3 to 4 and dropped the old one - their mirrors
only carry current versions. Every mirror now 404s on
autoconf2.71-2.71-3-any.pkg.tar.zst
so gmp fails to configure, taking py-pysfcgal down with it. Nothing in
this registry is involved; any PR would hit it.
Upstream fixed the URL in 37bb045f3c7a, "[gmp] Update autoconf 2.71
download URL" (#53437). The commit we pinned, 9e593bb1, is the 2026.07.29
release tag and there is no newer tag, so no tagged release carries the
fix - leaving the release line is unavoidable either way. Given that, go
to current master rather than to the fix commit alone. That moves 333
upstream commits, so expect a cold binary cache and a long run.
The sha lived in both workflows, and they had already drifted apart once:
6d9c233 "Bump vcpkg baseline to latest" updated macos.yml and left
windows.yml behind. Move it to .github/vcpkg-baseline.txt, which each
workflow now reads into a step output between checking out the registry
and checking out vcpkg, so a bump is a one-line edit in one file. This
is what the TODO on those lines was asking for.
3nids
force-pushed
the
auto-update-non-py-ports
branch
from
August 28, 2026 16:12
a47573c to
0a5b0c3
Compare
The Windows build got as far as configuring the pyarrow wheel and failed:
Could NOT find Python3 (missing: Development.Module NumPy)
Development: Cannot find the directory
".../installed/x64-windows/tools/python3/Include"
CMake derives the header and library locations from the interpreter's
prefix, but the python3 port puts them elsewhere: headers are copied to
include/python<X.Y>, and on Windows the import library stays at
lib/pythonXY.lib. The venv's base prefix is tools/python3, so CMake looks
for tools/python3/Include, which does not exist.
Pass Python3_EXECUTABLE, Python3_INCLUDE_DIR and Python3_NumPy_INCLUDE_DIR
explicitly, plus Python3_LIBRARY on Windows, deriving the version and the
NumPy include dir from the venv interpreter rather than hard-coding them.
Verified against a tree whose headers were moved out of the interpreter
prefix: find_package now reports "found components: Interpreter
Development.Module NumPy" and uses the supplied paths, where before it
failed exactly as CI did.
This was reached only now because arrow[python] on x64-windows had been
restoring 23.0.1 from the binary cache; bumping to 25.0.1 forces a real
rebuild, so it is the first time this path has been exercised since
pyarrow moved to scikit-build-core.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.