A production-focused operational reference for the geospatial data lifecycle —
cold-storage optimization, format conversion, and compliance automation.
Spatial Data Archival is a hands-on field guide for the engineers and archivists who keep petabyte-scale spatial data affordable, retrievable, and audit-ready. Raster mosaics, LiDAR point clouds, historical basemaps, and continuous sensor telemetry each demand distinct lifecycle handling — and treating them as a single low-cost bucket is how archives quietly hemorrhage money and durability.
Every page favours concrete configuration over theory: copy-ready CLI commands, PyArrow and DuckDB snippets, Terraform lifecycle rules, validation thresholds, and root-cause troubleshooting tables you can apply directly to production pipelines. External references are restricted to primary sources — OGC, NIST, AWS, GDAL, and the GeoParquet/FlatGeobuf specifications.
It is written for data engineers, GIS archivists, cloud architects, and compliance/operations teams. Across 50 in-depth articles the material drills from strategic overviews down to task-level playbooks, with a hand-authored diagram on every page.
The knowledge base is organised into three connected disciplines plus a cross-cutting cost model:
Shrink cold-tier footprint and egress cost without breaking retrieval SLAs — ZSTD level selection, row-group sizing, dictionary encoding for categorical GIS attributes, spatial partitioning (H3 / S2 / Quadkey), and the ZSTD vs LZ4 vs Snappy trade-off analysis.
Design hot/warm/cold tiers, select object storage, catalog metadata (STAC, ISO 19115), and codify retention with policy-as-code for legally defensible archives — including telemetry-driven lifecycle thresholds and S3 Object Lock for compliance.
Migrate legacy shapefiles to GeoParquet and FlatGeobuf, validate schemas and CRS, and automate idempotent, compliance-ready conversion pipelines — plus the head-to-head GeoParquet vs FlatGeobuf for cold-archive retrieval decision guide.
One auditable model tying together storage-class pricing, retrieval and request fees, early-deletion penalties, and compression-ratio tables — the numbers behind every tiering decision, referenced from all three disciplines.
- Runnable, not hand-wavy — real bucket names, real EPSG codes, real GDAL/DuckDB/Terraform flags.
- Decision-oriented — comparison matrices and cost models that produce an actual recommendation.
- Standards-grounded — every external link points to an authoritative specification or vendor doc.
- Fast & accessible — a static site that meets WCAG 2 AA, with an original inline SVG on every page.
Built as a static site with Eleventy and deployed on Cloudflare. Content is plain Markdown; page metadata, breadcrumbs, JSON-LD, and navigation are derived automatically at build time. To build locally:
npm install
npm run build # output in _site/
npm run serve # local preview© Spatial Data Archival & Cold Storage Optimization. Content is published for reference at www.spatialdataarchival.org.