[CHORE] Make the PySpark test suite hermetic

[Roel Bollens (RoelBollens-TomTom)]

The PySpark test suite depended on machine state in two ways: ambient Spark/JVM configuration leaked into the test JVM, and the JVM itself came from whatever Java the system resolves. This makes the venv own both.

Environment scrubbing (tests/conftest.py)

• Scrubs PYSPARK_SUBMIT_ARGS, SPARK_CONF_DIR, HADOOP_CONF_DIR, YARN_CONF_DIR, JAVA_TOOL_OPTIONS, _JAVA_OPTIONS, JDK_JAVA_OPTIONS at conftest import.
• Pins SPARK_HOME to the venv's bundled pyspark distribution and PYSPARK_PYTHON/PYSPARK_DRIVER_PYTHON to sys.executable (hard-set, not setdefault).
• SPARK_LOCAL_IP is deliberately left alone — it exists to make local sessions work (macOS/VPN hostname resolution).
• Test-suite only: the CLI and library keep honoring ambient configuration.

Bundled JDK (jdk4py dev dependency)

jdk4py>=17.0.9.2,<22, marker-gated to darwin/linux:

• The committed lock resolves the maintained 21.x line (JDK 21, all five wheel platforms) for pyspark 4.x.
• CI's lowest-direct resolution picks 17.0.9.2 (JDK 17), matching pyspark 3.4's supported ceiling — so the conditional setup-java step in the workflow is gone entirely.

Caveat worth reviewing: jdk4py's runtime is jlink-stripped and lacks jdk.incubator.vector, which Spark 4.x passes to every JVM via --add-modules (fatal when missing; upstream declined to bundle it). The conftest therefore exposes the JDK through a mirrored JAVA_HOME whose bin/java is a 7-line POSIX wrapper dropping that single flag — the module only enables vectorized BLAS, which Spark treats as optional at runtime. On platforms without a jdk4py wheel or a POSIX shell (Windows, BSD, musl) the marker skips the install and the conftest falls back to the system Java, i.e. today's behavior.

Verified locally

• Full package suite (493 passed, 4 skipped) under a poisoned environment: SPARK_HOME → a real Spark 3.5.3 install, PYSPARK_SUBMIT_ARGS='--master yarn', JAVA_TOOL_OPTIONS='-Xmx1m'. Before this change that environment produces mass 'JavaPackage' object is not callable errors.
• Same suite with JAVA_HOME=/nonexistent-java — still green, proving the bundled JDK is the one running.
• Same suite with jdk4py uninstalled — still green on system Java (fallback path).
• pyspark 3.4.0 boots and computes on jdk4py 17.0.9.2 (the lowest-direct combination).

[github-actions]

❌ PR Title Validation Failed

Error: Unknown release type "Chore" found in pull request title "[Chore] Make the PySpark test suite hermetic".

Available types:

• BUG
• FEATURE
• ENHANCEMENT
• DOCS
• REFACTOR
• TEST
• CHORE
• PERFORMANCE
• SECURITY
• INVESTIGATION

---

Examples:

• [BUG] Fix null pointer exception in geometry validator
• [REFACTOR](schema) Consolidate validation logic

See the Overture Maps Foundation GitHub usage guidelines for full documentation.

[github-actions]

🗺️ Schema reference docs preview is live!

🌍 Preview	https://staging.overturemaps.org/schema/pr/654/schema/index.html
🕐 Updated	Aug 11, 2026 19:37 UTC
📝 Commit	bfa1422
🔧 env SCHEMA_PREVIEW	true

Note

♻️ This preview updates automatically with each push to this PR.