Design incremental and planet-scale Overture ingestion

[Medformatik]

Problem

OpenMapX can ingest regional Overture snapshots, but refreshing a large region still requires snapshot-scale work and the storage layout assumes a bounded Geofabrik-style region. A planet-scale deployment needs measured partitioning and release-delta semantics without ever exposing a mixed release.

Current state

• The data manager discovers STAC releases, pulls regional partitions with DuckDB, validates them, and atomically promotes staging tables.
• Country-scale OSM/GERS conflation is resumable and has capacity, retention, and admin controls.
• Scheduled refresh currently treats releases as immutable snapshots.
• docs/docs/features/overture-places.md explicitly says the regional database does not use the global changelog and is designed for country scale.

Proposed solution

Design and prove a release-aware ingestion mode using Overture changelog data (added, removed, data_changed, unchanged) plus a partitioned planet-scale storage contract. Keep full snapshot ingestion as the recovery/fallback path.

Acceptance criteria

• An ADR compares regional snapshot, regional delta, and planet-scale layouts across object storage, DuckDB, and Postgres/PostGIS.
• Benchmarks record release churn, download/compute time, peak disk/memory, index cost, query latency, and recovery time for representative country and planet slices.
• A release contract records source release, schema version, partitions, checksums, and predecessor.
• Delta ingestion handles added, changed, and removed records idempotently and can resume from durable checkpoints.
• No query can observe tables or indexes from mixed Overture releases.
• Missing predecessors, schema changes, corrupt partitions, or excessive churn trigger a safe full-snapshot fallback.
• OSM/GERS links, search documents, and embeddings are invalidated only for affected records where correctness permits.
• Reconciliation samples or counts prove the delta result matches the equivalent full snapshot.
• Capacity preflight, retention, rollback, and failed-staging cleanup work for the new layout.
• Operator UI/metrics expose current release, predecessor, lag, mode, changed counts, and fallback reason.
• Planet mode has an explicit supported hardware/storage profile or a documented no-go result.

Alternatives considered

Always rebuilding snapshots is simpler and remains the fallback, but scales poorly with global data. In-place updates without release staging risk partial/mixed data.

Area

Overture ingestion, data manager, PostgreSQL/PostGIS, operations.

References

• docs/docs/features/overture-places.md
• services/data-manager/src/jobs/overture/
• https://docs.overturemaps.org/gers/changelog/
• https://docs.overturemaps.org/getting-data/cloud-sources/