Skip to content

Repository navigation

[REFACTOR](build) Migrate workspace to uv_build and PEP 420 namespace packages - #623

Merged
Victor Schappert (vcschapp) merged 4 commits into
mainfrom
pep-420
Aug 12, 2026
Merged

[REFACTOR](build) Migrate workspace to uv_build and PEP 420 namespace packages#623
Victor Schappert (vcschapp) merged 4 commits into
mainfrom
pep-420

Conversation

@sethfitz

@sethfitz Seth Fitzsimmons (sethfitz) commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Three commits.

refactor(build): migrate all packages from hatchling to uv_build

Switch every workspace package's build backend to uv_build. Because the packages share the overture.schema namespace, each declares an explicit [tool.uv.build-backend] module-name (e.g. overture.schema.common); name normalization would derive the wrong module. The overture-schema aggregator uses module-name = "overture.schema" with namespace = true and ships only overture/schema/py.typed — uv_build requires a module root, so a deps-only package still owns the namespace marker.

Convert the pkgutil namespaces to PEP 420: uv_build ships shared namespaces implicitly and never packages an overture/__init__.py, so the extend_path shims are removed. Wheels and editable installs both merge the namespace natively.

Deleting those __init__.py files removes the signal ruff's isort uses to detect overture as first-party (detect-same-package walks __init__ chains), so add src = ["packages/*/src"]. With overture now first-party everywhere, imports across the test suite and a few pyspark modules recanonicalize into their own group; that reformat is folded in here because it cannot land as a separately-green commit.

refactor(build): emit PEP 420 generated pyspark tree

Stop the pyspark codegen from emitting empty __init__.py files so the generated expression and test trees are PEP 420 namespace packages, matching the rest of the workspace. The mirrored layout (generated/overture/schema/<theme>/) is kept — only the init emission is dropped.

Removing the inits broke three things, each given a real fix:

  • pytest collection: generated conformance tests reached _support via deep relative imports. Under PEP 420 the module resolves as generated.*, so those imports overshoot the top level. Emit absolute from _support.X import ..., delete the _support_prefix machinery, and move the hand-written tests to the same form. Set --import-mode=importlib and consider_namespace_packages = true.
  • type checking: add the pyspark tests dir to mypy_path; keep the hand-written tests/__init__.py so pyspark's test_cli doesn't collide with codegen's under importlib.
  • runtime registry: pkgutil.walk_packages skips directories without __init__.py, finding zero generated modules. Walk the namespace roots as files instead; test_registry.py covers this — no existing test exercised the on-disk walk, so an empty registry would have passed silently.

chore(build): guard PEP 420 namespace roots against __init__.py

Add a check-namespace gate that fails if overture/ or overture/schema/ gains an __init__.py, which would turn a namespace root back into a regular package and shadow other workspace packages' contributions at import. Wired into the check gate so the PEP 420 invariant this stack establishes cannot silently regress.

Stacked on #620. Closes #618

@github-actions

github-actions Bot commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown

🗺️ Schema reference docs preview is live!
🌍 Preview https://staging.overturemaps.org/schema/pr/623/schema/index.html
🕐 Updated Aug 12, 2026 16:22 UTC
📝 Commit 674959e
🔧 env SCHEMA_PREVIEW true

Note

♻️ This preview updates automatically with each push to this PR.

lowlydba
John McCall (lowlydba) previously approved these changes Aug 11, 2026
View reviewed changes

@lowlydba John McCall (lowlydba) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥

John McCall (lowlydba) added a commit that referenced this pull request Aug 12, 2026
@lowlydba
[REFACTOR](ci) Un-hoist pyspark expression generation into the packag…
fff7195 
…e's own prebuild script

overture-schema-pyspark ships generated validation expressions that
aren't committed to git. main-publish.yaml and release-publish.yaml
special-cased it directly (if: ... == 'overture-schema-pyspark'
generate step, plus a post-build wheel-unzip check), so both workflows
had to know about one package's build quirk.

Moves that into packages/overture-schema-pyspark/scripts/prebuild.sh,
invoked generically as "run the package's prebuild.sh if it has one".
Neither workflow references pyspark by name anymore, and the script's
own empty-output guard replaces the wheel-content check. This also
keeps the convention backend-agnostic ahead of #623's hatchling ->
uv_build migration, which drops hatchling's custom build-hook support
entirely.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
stack merge was automatically disabled August 12, 2026 16:15

Pull Request is not mergeable

stack merge was automatically disabled August 12, 2026 16:19

Pull Request is not mergeable

Base automatically changed from extract-validation-package to main August 12, 2026 16:20
@sethfitz
Seth Fitzsimmons (sethfitz) dismissed stale reviews from Victor Schappert (vcschapp) and John McCall (lowlydba) August 12, 2026 16:20

The merge-base changed after approval.

@sethfitz @vcschapp
refactor(build): migrate all packages from hatchling to uv_build
39f7db6 
Switch every workspace package's build backend to uv_build. Because the
packages share the overture.schema namespace, each declares an explicit
[tool.uv.build-backend] module-name (e.g. overture.schema.common); name
normalization would derive the wrong module. The overture-schema
aggregator uses module-name = "overture.schema" with namespace = true
and ships only overture/schema/py.typed -- uv_build requires a module
root, so a deps-only package still owns the namespace marker.

Convert the pkgutil namespaces to PEP 420: uv_build ships shared
namespaces implicitly and never packages an overture/__init__.py, so the
extend_path shims are removed. Wheels and editable installs both merge
the namespace natively.

Deleting those __init__.py files removes the signal ruff's isort uses to
detect overture as first-party (detect-same-package walks __init__
chains), so add src = ["packages/*/src"]. With overture now first-party
everywhere, imports across the test suite and a few pyspark modules
recanonicalize into their own group; that reformat is folded in here
because it cannot land as a separately-green commit.

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
@sethfitz @vcschapp
refactor(build): emit PEP 420 generated pyspark tree
da72c78 
Stop the pyspark codegen from emitting empty `__init__.py` files so the
generated expression and test trees are PEP 420 namespace packages,
matching the rest of the workspace after the hatchling->uv_build
migration. The mirrored layout (generated/overture/schema/<theme>/) is
kept -- only the init emission is dropped.

Removing the inits broke three things the bead's plan assumed would keep
working; each needed a real fix:

- pytest collection: generated conformance tests reached `_support` via
  deep relative imports whose depth `_support_prefix` computed. Under
  PEP 420 the module resolves as `generated.*` (tests/ is on
  pythonpath), not `tests.generated.*`, so those relative imports
  overshoot the top level. Emit absolute `from _support.X import ...`
  instead and delete the `_support_prefix` machinery. To keep one import
  style across the test tree, the hand-written tests move to the same
  absolute form. Set `--import-mode=importlib` (via addopts; it has no
  ini form) and `consider_namespace_packages = true` so collection walks
  the namespace.

- type checking: mypy resolves `_support` as top-level only when its
  parent is a base, so add the pyspark tests dir to `mypy_path`. The
  hand-written `tests/__init__.py` stays: dropping it would make
  pyspark's `test_cli` a top-level module colliding with codegen's under
  importlib.

- runtime registry: `pkgutil.walk_packages` skips subdirectories without
  `__init__.py`, finding zero generated modules under the PEP 420 tree.
  Walk the namespace roots as files instead. test_registry.py covers
  this -- no existing test exercised the on-disk walk, so an empty
  registry would have passed the suite silently.

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
@sethfitz @vcschapp
chore(build): guard PEP 420 namespace roots against __init__.py
77150e8 
check-namespace fails if overture/ or overture/schema/ gains an
__init__.py, which would turn a namespace root into a regular package
and shadow other workspace packages' contributions at import. Wired into
the check gate.

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
@sethfitz @vcschapp
chore(build): add changelog fragments for uv_build/PEP 420 migration
674959e 
Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
@vcschapp
Victor Schappert (vcschapp) merged commit 293b395 into main Aug 12, 2026
25 checks passed
John McCall (lowlydba) added a commit that referenced this pull request Aug 24, 2026
@lowlydba
[FEATURE] Branching strategy Phase 3 - PyPI and CodeArtifact publish …
5dc32f3 
…workflows (#638)

* [FEATURE](ci) Branching strategy Phase 3 - PyPI and CodeArtifact publish workflows

Adds the publish side of the versioning pipeline built in Phase 2.B:

- main-publish.yaml: on push to main, diffs changed packages/** files
  (via the new detect-affected-packages action) to find packages touched
  without a version bump, stamps a .postN+main.<sha> build via
  compute-version, and publishes to CodeArtifact. Also runs as a
  build-only smoke test on PRs touching the composite actions or itself,
  replacing compute-versions-dry-run.yaml.
- 
elease-publish.yaml: on 
elease: published, parses <package>-v<version>
  from the tag, builds, and publishes to PyPI via
  `pypa/gh-action-pypi-publish` (OIDC trusted publishing + attestations),
  gated by the pypi-release environment's required reviewers.
- 
elease-trigger.yaml: releases now get created with an
  overture-release-publisher app installation token instead of
  GITHUB_TOKEN, since GITHUB_TOKEN-created releases don't fire
  
elease: published for other workflows to pick up. App provisioning is
  tracked separately in #637 and has to happen before this path works.

Deletes compute-versions-dry-run.yaml and publish-python-packages.yaml,
both superseded by the two workflows above.

docs/versioning.md gets a workflow-name reference table and the
pypi-release approval-gate mechanics.

Still open, all external/manual, tracked on #509:
- provisioning overture-release-publisher (#637)
- PyPI Trusted Publisher config per package
- pypi-release environment + reviewers
- p3-dev-builds-ca, blocked on ops-team#299

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [REFACTOR](ci) Use tj-actions/changed-files in detect-affected-packages

Replaces the hand-rolled `git diff --name-only` + path-splitting with
tj-actions/changed-files' dir_names output, pinned by commit SHA. Fixes a
real gap in the old parser: git diff --name-only quotes non-ASCII/unusual
filenames, which naive line.split("/") didn't account for. Uses
�ll_modified_files (ACMRD) rather than �ll_changed_files (ACMR) so file
deletions still count as a package change, matching the old diff's behavior.

detect_affected_packages.py drops its subprocess/git plumbing entirely and
just reads the directory list from CHANGED_DIRS; the bump/removed
exclusion logic (a three-way set difference) stays in Python rather than
jq/bash, that's a different complexity class than this repo's existing
jq usage in enforce-change-type-label.yaml.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [REFACTOR](ci) Reorder detect-affected-packages steps so both Python scripts run back to back

The action wasn't wrong, just awkward to read: python, jump to a JS
action, jump back to python. Steps 1 (version diff) and 2 (changed
dirs) don't depend on each other, only step 3 does, so nothing stops
them running adjacent.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [REFACTOR](ci) Extract diff-package-versions composite action

package_versions.py diff was inlined at three call sites
(detect-version-bumps, detect-affected-packages, and

eusable-check-python-package-versions.yaml), each reimplementing the
temp-file-then-$GITHUB_OUTPUT plumbing slightly differently (one even
wrote to a different temp path and used heredoc-style output).

diff-package-versions is now the one place that knows how to run the
script and expose it as count/diff outputs. The three consumers just
pipe steps.diff.outputs.diff into their own filter logic:
detect_version_bumps.py and detect_affected_packages.py are unchanged,
they already read JSON from stdin. 
eusable-check-python-package-versions.yaml's
CodeArtifact existence check now reads the diff from an env var instead of
a temp file; its changed_packages output is compact JSON now instead of
pretty-printed, no consumer depends on the formatting.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [REFACTOR](ci) Move package_versions.py into diff-package-versions

Last remaining dangling script: package_versions.py lived in
.github/workflows/scripts/ and was reached into by relative path.
Every other action already keeps its script(s) alongside its own
action.yml (detect-version-bumps, detect-affected-packages,
create-package-release); this was the one holdout, and after the last
refactor diff-package-versions is its only remaining caller anyway.

Moved it in, switched the reference to ${GITHUB_ACTION_PATH}, and
removed the now-empty scripts/ directory. No behavior change: git
commands inside the script still run with the repo root as CWD, only
the path used to invoke it changed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [REFACTOR](docs) Polish CONTRIBUTING.md for the finished branching/release flow

Phase 3 was the last phase (#509); Phase 4 got folded into it instead of
staying a separate doc-polish pass (see #490). Drops the "rolling out in
phases" banner pointing at the tracking issue, since the flow it describes
is now fully implemented rather than in progress.

Also corrects two release-flow descriptions that predated the pypi-release
approval gate this phase added: a release no longer lands on PyPI
"immediately", it starts a maintainer-gated publish first.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [FIX](ci) Add pull_request trigger to test-schema

push never fires for fork PRs (only same-repo branch pushes), so a
required check relying solely on push would never run for external
contributors, a required status check that can never be satisfied by
the PRs GitHub Actions actually needs to gate.

Add pull_request with the same path filters; push stays for post-merge
validation on main.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [FEATURE](ci) Read overture-releaser PEM from Secrets Manager

release-trigger.yaml assumes the narrow gha-releaser-secrets-reader OIDC
role and fetches the PEM from omf-github-terraform/releaser/pem instead
of reading GHA repo secrets, matching the safe-settings and
project-manager app pattern. Client ID is inlined (not sensitive) instead
of a secret. Companion Terraform PR wires the role and Secrets Manager
entry in omf-github-terraform.

Fixes #637.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* fix(actions): treat a package's first version as a releasable bump

detect_version_bumps.py no longer skips before:null entries; a
brand-new package now flows through release-trigger to PyPI on its
first version instead of sitting on CodeArtifact-only .postN builds.

detect_affected_packages.py's bumped set is widened to match (any
package with a non-null after version) so the same push doesn't also
queue an internal build for a package that just got its first release.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [REFACTOR](ci) Un-hoist pyspark expression generation into the package's own prebuild script

overture-schema-pyspark ships generated validation expressions that
aren't committed to git. main-publish.yaml and release-publish.yaml
special-cased it directly (if: ... == 'overture-schema-pyspark'
generate step, plus a post-build wheel-unzip check), so both workflows
had to know about one package's build quirk.

Moves that into packages/overture-schema-pyspark/scripts/prebuild.sh,
invoked generically as "run the package's prebuild.sh if it has one".
Neither workflow references pyspark by name anymore, and the script's
own empty-output guard replaces the wheel-content check. This also
keeps the convention backend-agnostic ahead of #623's hatchling ->
uv_build migration, which drops hatchling's custom build-hook support
entirely.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [FEATURE](ci) Add a workflow_dispatch dry-run to Test PyPI on release-publish

Lets a package publish its current on-disk version to Test PyPI on
demand, exercising the build-and-publish pipeline without waiting on
a real release or touching the production index.

Each package gets its own pypi-release-<package> / test-pypi-<package>
Environment pair rather than one shared pypi-release/test-pypi pair: a
PyPI Trusted Publisher's identity is (repo, workflow filename,
environment), so a shared name across packages would let only one
package's project name ever bind to it (see #653).

Also replaces the post-build wheel-filename check with an upfront
`uv version --short` comparison against the release tag: uv has no
built-in check for this (astral-sh/uv#9653), and failing before the
build runs is cheaper than after.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [REFACTOR](ci) Drop the per-environment reviewer gate on PyPI publish

12 packages each needing their own pypi-release-<package> environment
(see prior commit, #653) makes per-environment required-reviewer
config unruly to maintain. The version-bump PR review is already the
approval; nothing further needs to gate the publish once a release
exists.

Environments stay, scoping only the Trusted Publisher identity (repo,
workflow filename, environment) that PyPI's OIDC matching requires.
Updates release-publish.yaml's comments, docs/versioning.md, and
CONTRIBUTING.md to match, and drops a stale token-scoping comment from
release-trigger.yaml that the `permission-contents: write` line already
says plainly.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [FIX](ci) Correct release-publisher app name and pin publish checkout to the release tag

overture-releaser was never the app's real name; overture-release-publisher
is, per #637. Fix it everywhere it drifted in.

release-publish's checkout also used target_commitish, which can be a
branch name that moves past the release. Use tag_name instead so the
publish always builds the exact released commit.

Also drops the "explicitly scope the app token" comment above
permission-contents: write per feedback -- the field name already says that.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* Add changelog fragment for pyspark prebuild un-hoisting

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* Expand changelog fragment with CI-only scope note

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* Rename check-python-code's default resolution matrix cell to locked

'default' didn't say what it defaulted to. locked names what actually
distinguishes it from lowest-direct: it runs against the committed uv.lock.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [FIX](ci) Address review feedback on publish workflows

- Downscope main-publish's PR smoke test to the CodeArtifact read-only
  IAM role instead of the publish role.
- Validate the parsed package directory exists for both release-publish
  arms (release tag and workflow_dispatch), not just workflow_dispatch.
- Standardize on the actions/checkout v7.0.1 pin used elsewhere in the
  repo; main-publish/release-publish had drifted to a mix of v7.0.0/v7.0.1.
- Fix wording nit: "in-memory checkout" -> temporary checkout.
- detect_affected_packages.py: read version_diff + changed_dirs from a
  single stdin JSON payload instead of splitting across stdin and an env
  var, and raise a clear error instead of an IndexError on a malformed
  changed-directory path.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* Fix optional pyspark import guard behavior

Co-authored-by: lowlydba <16843041+lowlydba@users.noreply.github.com>

* [FEATURE](ci) Add a gated real-PyPI priming path to release-publish's workflow_dispatch

PyPI rate-limits pending Trusted Publisher registrations to ~3 at a time, on
both pypi.org and test.pypi.org independently (see #653). Priming a publisher
past "pending" requires an actual publish, and workflow_dispatch previously
only supported Test PyPI, so there was no way to prime real PyPI ahead of a
package's first release.

workflow_dispatch now takes a target input (test-pypi default, or pypi) and
always publishes a synthetic <on-disk-version>.dev0 instead of the real
version: valid PEP 440, ignored by default resolvers, and never collides with
the eventual human-owned release. Dispatching to real PyPI uses a new
pypi-dispatch-<package> environment with its own required-reviewer gate,
since it's the one path that publishes to production PyPI without a
version-bump PR behind it; pypi-release-<package> (the automated release
path) and test-pypi-<package> are unchanged.

This is a bootstrapping tool, not a permanent fixture. Once all 12 packages
are primed for the v2.0 launch, the pypi target and its dispatch environments
should come out.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [REFACTOR](ci) Simplify the priming dispatch to reuse pypi-release-<package>, flag temp blocks for #688

Drops the separate pypi-dispatch-<package> environment: workflow_dispatch
already requires repo write access to trigger at all, which is gate enough
for a disposable .dev0 priming publish, so a real-PyPI priming dispatch now
reuses the same pypi-release-<package> environment the automated release
path uses. One environment set per package instead of a third variant.

Also marks every pypi-target-specific block with TODO(#688) so the temporary
priming path (vs. the permanent test-pypi dispatch) is obvious at the code
site, not just in the tracking issue.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

---------

Signed-off-by: John McCall <john@overturemaps.org>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lowlydba <16843041+lowlydba@users.noreply.github.com>

This branch was successfully deployed

1 active deployment
staging — 674959e4 Deployed Aug 12, 2026 by vcschapp via Deploy #390
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vcschapp Victor Schappert (vcschapp) vcschapp approved these changes
@lowlydba John McCall (lowlydba) lowlydba approved these changes

Assignees

No one assigned

Labels

change type - cosmetic 🌹 Cosmetic change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[CHORE] Migrate workspace to uv_build with PEP 420 namespace packages

3 participants

@sethfitz @vcschapp @lowlydba