Validation registry empty when overture-schema-pyspark loaded via zipimport

[Rachmanin0xFF]

Type

Bug

Scope

Multi-theme or Platform

Skillset

engineering

Description

What: overture.schema.pyspark._registry.REGISTRY comes back empty on AWS Glue. Every feature type then silently reports as unregistered and the task erroneously succeeds 🫠

Why: _registry.py discovers generated modules by walking the expressions/generated/ PEP 420 namespace package with pathlib.Path(root_path).rglob("*.py"), which works... only when a real filesystem directory exists! When the package is added to sys.path as a raw .whl (as Glue does in data-platform when we call it with --extra-py-files), the namespace package's __path__ portions point inside the zip archive, and pathlib can't traverse into one, so the walk silently returns nothing.

Constraints: importlib.resources.files() looks like the obvious fix (its Traversable API is meant to be zipimport-aware), but its MultiplexedPath implementation (at least through Python 3.10) raises NotADirectoryError the moment any namespace portion isn't a real directory -- confirmed by testing it directly against a zip-imported wheel. It can't be used here as-is.

Acceptance criteria:

• REGISTRY populates correctly whether overture-schema-pyspark is installed normally (real directory) or loaded via zipimport (raw wheel on sys.path).
• The documented behavior for a missing expressions/generated/ subtree (empty registry, no crash) is unchanged.
• No regression in the existing overture-schema-pyspark test suite.

[sethfitz]

Adam Lastowka (@Rachmanin0xFF) what does the PySpark invocation that uses --extra-py-files actually look like? Are you injecting everything, or are you using pip or pyspark's --additional-python-modules to install using pip (from CodeArtifact (CA) or elsewhere) when the job runs?

I'm asking because --additional-python-modules combined with CA (--python-modules-installer-option --index-url <codeartifact-url>) might do the trick (given access to CA, possibly using a VPC Endpoint), as it installs packages onto the filesystem. zipimport being broken is a separate issue (and I'm inclined toward #663).

Claude tells me that importlib.resources in stdlib is broken (files() on a zip-backed namespace portion) until 3.13, but that there's a backport available: importlib_resources >= 6.2. Can you check the Glue image to see if that package comes pre-installed (dump importlib_resources.__file__), and if so, which version? #666 proposes this fix.

[Rachmanin0xFF]

It's sorta awkward. We use --additional-python-modules; for modules with pre-compiled dependencies (pyarrow, numpy, etc.). Anything that's plain Python gets zipped, dropped in S3, and installed with --extra-py-files. Claude told me that we do actually have a force_pip_packages parameter, though (with a total of one (1) active use case)! Right now it just resolves to public PyPI, but I think we could give it an --index-url like you suggest (John McCall (@lowlydba), can you confirm?)

Can you check the Glue image to see if that package comes pre-installed (dump importlib_resources.file),

Glue 5.0 (amzn-awsgluelibs==5.0.1063) does not come with importlib`. rip. : (

[sethfitz]

Glue 5.0 (amzn-awsgluelibs==5.0.1063) does not come with importlib.

This is good, actually (because it means that the dependency we declare won't need to fight), although depending on how you need to inject/install sub-dependencies it may add something to the invocation list.

Does passing a .whl to --extra-py-files automatically pull dependencies declared by that package?

[sethfitz]

There are currently 3 options we need to choose from that resolve this:

• [BUG] Fix validation registry discovery when loaded via zipimport #662
• [BUG] Discover validation registry via a generated index module #663
• [BUG] Discover validation registry through importlib.resources #666

Roel Bollens (@RoelBollens-TomTom) and Victor Schappert (@vcschapp): mind weighing in on the approaches?

[Rachmanin0xFF]

Does passing a .whl to --extra-py-files automatically pull dependencies declared by that package?

Not by default, but we have a step where we manually resolve the dependency tree for any requested plain-python packages, download them all from PyPI, then manually include them via --extra-py-files.

[sethfitz]

Ok, so an additional dependency will flow through the manual resolution step without much (any?) change?

If that's true, my inclination is toward #666 (which is my inclination anyway, since it fixes the problem using stdlib(ish)).

[vcschapp]

Some LLM slop material I made to help me understand what we're talking about:

• What it is: REGISTRY is an in-memory dictionary in overture.schema.pyspark._registry — one entry per Overture feature type (building, segment, place, …), each pointing at that type's PySpark validation logic. It's built once, the first time the package is imported.
• Who fills it: the _registry module fills it at import time by finding the code-generated validation modules (one per feature type, emitted by make generate-pyspark) and pulling each module's MODEL_VALIDATION into the dict under its ENTRY_POINT key.
• Who reads it: the pyspark validation API (get_feature_validation() / validate_model()) looks a feature type up in it when checking data — usually a Spark/Glue job validating Overture Parquet columns and types.
• Why the bug matters: if that build step finds nothing (the zipimport case in Validation registry empty when overture-schema-pyspark loaded via zipimport #661), the dict is empty, so every type reads back as "unknown" and validation quietly passes while checking nothing.

[vcschapp]

Solution space diagram:

(More AI slop.)

[vcschapp]

Here's my high-level understanding of the mechanism that's at play here:

1. overture-schema-pyspark has two distinct roles:
   1. Provides the runtime on which any generated PySpark validation relies.
   2. For convenience, it also hosts the PySpark validation rules that we generate for the Overture core schema (common + themes).
2. The core validation rules (1.2) are vended via this REGISTRY.
3. The registry doesn't work right when the package is loaded as a wheel.
4. None of this relates to the case of a 3P who wants to generate and
   use PySpark validation its own non-core schema.
   1. The generated PySpark validations can be sent to an arbitrary filesystem location.
   2. The 3P can role those validations into its own Python package and implement its
      own discovery mechanism for them.

Correct me if I'm wrong, that this sets the scene in which these three competing PR options are being profferred.

[vcschapp]

Thanks everyone for diving in and providing three excellent and usable fixes.

Of the three, I would prefer we go with #666.

Below the line is my ranking system.

---

Rank	PR	Mechanism	Factored considered
1.	#666	quasi-stdlib	Fewest net lines added (+20), least semantic change, path to stdlib reliance, only 1 package touched
2.	#662	zip-aware FS walk	Most net lines added (+40) but second lowest semantic change, only 1 package touched
3.	#663	codegen-emitted index	Mid-ground net lines added (+32), most semantic change, touches 2 packages and changes codegen package (general) just because of what pyspark wants (specific)

---

Seth Fitzsimmons (@sethfitz) Adam Lastowka (@Rachmanin0xFF) Can you align on which one of the three you want to put forward based on the above and any other discussion you want to have amongst each other?

[Rachmanin0xFF]

It seems like we're all leaning towards #666. Unless Seth Fitzsimmons (@sethfitz) has objections, I can close out the other two.

[sethfitz]

#666 is also my preference.