[Devops] Configure PyPI Trusted Publisher + pypi-release environment (manual, one-time)

[lowlydba]

Context

release-publish.yaml publishes to public PyPI via Trusted Publishing (OIDC) and supports a workflow_dispatch dry-run to Test PyPI. Neither side exists yet: PyPI has no trusted publisher registered for any of these packages, and none of the required GitHub Environments exist in this repo (gh api repos/OvertureMaps/schema/environments returns copilot, github-pages, staging). The workflow will fail closed until both are set up. Both are one-time manual steps (PyPI's UI, GitHub's repo settings), not something CI can provision.

Note

#650 renamed the six theme packages (overture-schema-<theme>-theme → overture-schema-theme-<theme>), and the PyPI distribution name changed with it, not just the package directory. The list below reflects the current names.

Important

A PyPI Trusted Publisher's identity is (repo, workflow filename, environment); it doesn't see release-publish.yaml's package input. A single shared pypi-release/test-pypi environment across all 12 packages means only the first package to register can ever bind to it: every other package's registration fails with A pending trusted publisher matching this configuration has already been registered for a different project name. Each package needs its own environment (pypi-release-<package>, test-pypi-<package>), which release-publish.yaml now computes per-package.

These environments scope the Trusted Publisher identity only, not an approval gate: with 12 of them, per-environment required reviewers would be unruly to maintain. The version-bump PR review is the approval; nothing further blocks the publish once a release exists.

Important

PyPI and Test PyPI each cap pending Trusted Publishers (ones registered before a project's first upload) at 3 per account. A pending publisher only converts to a real, uncapped one once its package's first publish succeeds. With 12 packages on each of two sites, this forces a batched rollout instead of registering all 24 up front:

1. Merge this PR's workflow changes to main (environments + per-package release-publish.yaml need to exist there first).
2. Register 3 packages' pending publishers, publish each once (dry-run to Test PyPI, real release to PyPI) to convert them, then register the next 3. Repeat until all 12 are done, on both sites.
3. Once every package has published at least once, all 24 Trusted Publisher configs are finalized and none are pending anymore.

Note

GitHub Environments referenced by a workflow but not yet created are auto-created on first run, empty and with no protection rules (docs). Since these 24 environments need no required reviewers (see the approval-gate note above), there's nothing to manually create: each pypi-release-<package>/test-pypi-<package> environment comes into existence as a side effect of that package's first priming dispatch.

Tasks

• Create 12 pypi-release-<package> GitHub Environments on OvertureMaps/schema (auto-created by each package's first target: pypi dispatch; nothing to do manually)
• Create 12 test-pypi-<package> GitHub Environments (mirrors overturemaps-py's test-pypi environment). release-publish.yaml's workflow_dispatch dry-run path targets these to exercise the build-and-publish pipeline without a real release or the production index. Auto-created via the Test PyPI priming batches below.
• release-publish.yaml's workflow_dispatch also supports target: pypi: a real-PyPI priming dispatch that always publishes a synthetic <version>.dev0 through the existing pypi-release-<package> environment (no separate environment needed, workflow_dispatch already requires repo write access to trigger). This is how priming actually converts a pending Trusted Publisher to real on pypi.org, not just test.pypi.org. Temporary: see [Devops] Remove the real-PyPI priming dispatch path once all packages are primed #688 for removing this path once priming is done. Shipped in [FEATURE] Branching strategy Phase 3 - PyPI and CodeArtifact publish workflows #638.
• Register and publish the 12 packages in batches of 3 (per the pending-publisher cap above) on test.pypi.org, pointing each at:
  • Repo: OvertureMaps/schema
  • Workflow: release-publish.yaml
  • Environment: test-pypi-<package>
  • All 12 packages registered and primed (workflow_dispatch target: test-pypi), converting every Test PyPI Trusted Publisher from pending to real.
• Register and publish the 12 packages in batches of 3 on pypi.org, pointing each at:
  • Repo: OvertureMaps/schema
  • Workflow: release-publish.yaml
  • Environment: pypi-release-<package>
  • Packages:
    • overture-schema
    • overture-schema-theme-addresses
    • overture-schema-theme-base
    • overture-schema-theme-buildings
    • overture-schema-cli
    • overture-schema-codegen
    • overture-schema-common
    • overture-schema-theme-divisions
    • overture-schema-theme-places
    • overture-schema-pyspark
    • overture-schema-system
    • overture-schema-theme-transportation
  • Use PyPI's pending publisher flow for each (no existing PyPI project yet); a package's slot frees up once its first publish converts the pending entry to a real one

Definition of done

• All 12 pypi-release-<package> environments exist
• All 12 test-pypi-<package> environments exist
• All 12 packages have published at least once to PyPI and Test PyPI, converting all 24 Trusted Publisher configs from pending to real (Test PyPI: done, 12/12; PyPI: not started, 0/12)
• One real release exercises the path end-to-end (first release covered by #509's open question on before: null first-release handling)
• Real-PyPI priming dispatch path removed once priming is done ([Devops] Remove the real-PyPI priming dispatch path once all packages are primed #688)

Parent: #509

[lowlydba]

Hit a snag where we can only queue up to 3 to-be-published packages before we have to start...publishing the package in PyPI - I had hoped we could just manually prep them all in Test/PyPI and then just point CI at them.

This will be blocked until we can actually trigger the publishes via a manual workflow_dispatch, ahead of automated releases to ensure everything works.

[lowlydba]

Update: #638 now ships the priming path for this issue's Test PyPI limit.

release-publish.yaml's workflow_dispatch takes a target input (test-pypi default, or pypi) and always publishes a synthetic <on-disk-version>.dev0, never the real release version. That's what actually converts a package's Trusted Publisher from pending to normal on either site: PEP 440 dev-releases are valid, default resolvers ignore them, and they never collide with the eventual human-owned release version.

Real-PyPI dispatch reuses the existing pypi-release-<package> environment, no separate approval-gated environment needed: workflow_dispatch already requires repo write access to trigger, which is gate enough for a disposable .dev0. So the environment list in this issue's tasks is unchanged, still 12 pypi-release-<package> + 12 test-pypi-<package>, nothing extra to create.

This is a bootstrapping tool, not a permanent fixture. #688 tracks removing the pypi target (and simplifying workflow_dispatch back to Test-PyPI-only) once all 12 packages are primed for the v2.0 launch.

Once #638 merges, the batched rollout described above is unblocked: register 3 pending publishers per site, dispatch to convert them, repeat.

[lowlydba]

Seth Fitzsimmons (@sethfitz) Victor Schappert (@vcschapp) for visibility:

All 12 packages are primed on Test PyPI now (pending Trusted Publishers converted to real, batched 3 at a time per PyPI's rate limit).

Next batch is the same cycle against real pypi.org. Each priming dispatch publishes a synthetic <version>.dev0, never the real release version, PEP 440 dev-releases are excluded by default resolvers so nothing consumes one unknowingly. These get superseded by the actual v2.0 releases once we cut them, and we can tear the .dev0 artifacts down afterward if you'd rather not leave them on the index. But we need them published now, stubbed or not, since that first successful publish is what converts each package's Trusted Publisher off pending.

[vcschapp]

John McCall (@lowlydba) Are there any actions we need to take before the big first push of v2.0.0 on Wed 9/2?

Register and publish the 12 packages in batches of 3 on pypi.org, pointing each at:

The above seems action-y.

[lowlydba]

Are there any actions we need to take before the big first push of v2.0.0 on Wed 9/2?

Victor Schappert (@vcschapp) Yup! Those need a "primer" publish to complete the Trusted Publishing setup. So they'll get a non-prod marked version (similar to the Test PyPI versions that are live now) to finish the creation of the package in PyPI. Then the next release will seemingly be the v2, after which we can remove the other version manually from PyPI if desired for hygiene.