Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Contributing

Maintainer procedures for this repository. For what the repository contains and how to run it locally, see the [README](README.md).

- [Release calendar](#release-calendar)
- [Schema reference](#schema-reference)
- [Adding a schema version](#adding-a-schema-version)
- [Schema PR previews](#schema-pr-previews)
- [OG image cache](#og-image-cache)

## Release calendar

`static/release-calendar.json` is the source of truth for release dates, and the `/release-calendar` page renders from it (`src/components/ReleaseCalendar.jsx`). Its shape is documented in the [README](README.md#release-calendar), and `static/release-calendar.schema.json` validates it.

Each month, add an entry for the release that's three to four months out, so at least three upcoming releases are listed. Set `majorChangeMonth` to true for March, June, September, and December. After a release ships, fill in `schemaVersion` if it was still `null`, and publish the release notes as `blog/YYYY-MM-DD-release-notes.mdx`. The history table links the data version to that post automatically, and shows plain text when there isn't one. Entries move from the schedule to the history table by date, so nothing else needs editing. The change shows up on the next deploy.

The fallback release in `docusaurus.config.js` (`getLatestOvertureRelease()`) is the newest shipped entry in this file. It's only used when the STAC catalog is unreachable at build time.

Don't hardcode the current release in docs examples. Use the `__OVERTURE_RELEASE` placeholder, which resolves the latest release from [STAC](https://stac.overturemaps.org/) at build time. It works in plain fenced code blocks and in the `QueryBuilder` component. Pin a version only when the surrounding text depends on that exact release.

## Schema reference

The reference pages under `docs.overturemaps.org/schema` are generated from the Pydantic models in [OvertureMaps/schema](https://github.com/OvertureMaps/schema).

The schema reference is its own [versioned Docusaurus docs instance](https://docusaurus.io/docs/versioning) (plugin id `schema`, config in `docusaurus.config.js`, sidebar in `sidebars-schema.js`), separate from the main `docs/` instance. Only released schema tags are published; there is no rolling "latest" built from `main`. The newest tag in `schema_versions.json` is served at `/schema/`, older tags at `/schema/vX.Y.Z/`, and a version dropdown appears in the navbar on schema pages only.

Snapshots are generated once per tag and committed under `schema_versioned_docs/version-vX.Y.Z/`. Builds do not call the schema generator, so a production deploy only ever changes the schema pages when a new snapshot lands here.

**A typo or error under `docs.overturemaps.org/schema` is not fixable in this repository.** Open an issue or PR against the docstrings/models in [OvertureMaps/schema](https://github.com/OvertureMaps/schema) instead; the fix appears in the next release's snapshot. Re-snapshotting an existing tag is possible (delete its three artifacts and re-run the script below) but pointless unless the tag itself moved.

The overview page (`schema/index.md`) is copied into each snapshot when it's created. Edits to it need to be applied to the `index.md` in each `schema_versioned_docs/version-*/` too.

### Adding a schema version

Run this after a `vX.Y.Z` tag is published in [OvertureMaps/schema](https://github.com/OvertureMaps/schema/releases). It needs `git`, [`uv`](https://docs.astral.sh/uv/), and `npm install` already done.

```shell
npm run add-schema-version -- v2.0.0
```

The script (`scripts/add-schema-version.mjs`) clones the schema repo at that tag, runs its `overture-codegen` into `schema/reference/`, then runs `docusaurus docs:version:schema <tag>`, which writes:

- `schema_versioned_docs/version-<tag>/` (with links into the schema repo pinned to the tag)
- `schema_versioned_sidebars/version-<tag>-sidebars.json`
- an entry in `schema_versions.json` (kept sorted newest-first; the first entry is what `/schema/` serves)

Commit those three and open a PR. `schema/reference/` is cleaned up afterwards.

Only tags that ship the `overture-schema-codegen` package (v1.17.0 and later) can be added; earlier releases were JSON Schema and have no generator. The script refuses tags that don't match `vX.Y.Z` or are already in `schema_versions.json`.

### Schema PR previews

The schema repo's PR preview workflow checks out this repo, generates Markdown from the PR branch into `schema/reference/`, and builds with `SCHEMA_PREVIEW=true`. In that mode the `schema` instance builds only the `current` version from `schema/reference/` at `/schema/`; committed snapshots, the version dropdown, blog, and community pages are skipped.

## OG image cache

The community page displays project cards with images. Each entry in `community/community-projects.json` can include an optional `"image"` field. For entries without one, the site falls back to a cached `og:image` fetched from the project's URL.

The cache lives in `community/og-image-cache.json` and is committed to the repository so CI builds never make external HTTP requests.

**When to run it:** after adding or updating entries in `community-projects.json`.

```shell
npm run fetch-og
```

The script (`scripts/fetch-og-images.mjs`):

1. Skips entries that already have an explicit `"image"` field
2. Re-validates any previously cached non-empty URLs via a HEAD request (`Content-Type: image/*`) and clears invalid ones
3. Fetches the HTML for uncached entries, extracts `og:image`, and validates the URL before writing it to the cache
4. Is idempotent - safe to re-run at any time

Cards with no image (neither explicit nor cached) display a branded gradient placeholder.
69 changes: 17 additions & 52 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Overture Documentation

[![WCAG 2.2 AA](https://img.shields.io/badge/WCAG_2.2-AA-green)](https://www.w3.org/WAI/WCAG22/quickref/)
[![release-calendar.json](https://img.shields.io/badge/%F0%9F%94%97-release--calendar.json-4051CC)](https://docs.overturemaps.org/release-calendar.json)

This repository uses [Docusaurus](https://docusaurus.io/) to publish the documentation pages seen at [docs.overturemaps.org](https://docs.overturemaps.org)
This repository uses [Docusaurus](https://docusaurus.io/) to publish the documentation pages seen at [docs.overturemaps.org](https://docs.overturemaps.org). Maintainer procedures (monthly release calendar updates, schema snapshots, OG image cache) are in [CONTRIBUTING](CONTRIBUTING.md).

<p align="center">
<a href="../../issues/new?template=community-project.yaml">
Expand All @@ -15,44 +16,28 @@ This repository uses [Docusaurus](https://docusaurus.io/) to publish the documen
- `blog/`: Entries for the Overture engineering blog available at docs.overturemaps.org/blog
- `community/`: The community page that showcases Overture data being used in the wild.
- `community-projects.json` - source data for all community project cards
- `og-image-cache.json` - cached `og:image` URLs for entries without an explicit `image` field (see [OG Image Cache](#og-image-cache) below)
- `og-image-cache.json` - cached `og:image` URLs for entries without an explicit `image` field (see [CONTRIBUTING](CONTRIBUTING.md#og-image-cache))
- `docs/`: The main documentation pages available at docs.overturemaps.org/. The sidebar for these pages is manually curated in the `sidebars.js` file.
- `schema/`: Source for the schema reference overview page (`index.md`) and scratch output of the schema doc generator. See below.
- `schema_versioned_docs/`, `schema_versioned_sidebars/`, `schema_versions.json`: committed snapshots of the schema reference for each released schema tag. See below.
- `schema/`: Source for the schema reference overview page (`index.md`) and scratch output of the schema doc generator.
- `schema_versioned_docs/`, `schema_versioned_sidebars/`, `schema_versions.json`: committed snapshots of the schema reference for each released schema tag.
- `static/release-calendar.json`: the release calendar data (see [Release Calendar](#release-calendar))

## Schema Reference (`docs.overturemaps.org/schema`)

The Overture schema repository [OvertureMaps/schema](https://github.com/OvertureMaps/schema) maintains the official Overture schema as Pydantic models, and the reference pages under `docs.overturemaps.org/schema` are generated directly from those models.
The reference pages under `docs.overturemaps.org/schema` are generated from the Pydantic models in [OvertureMaps/schema](https://github.com/OvertureMaps/schema). Only released schema tags are published; the newest is served at `/schema/` and older ones at `/schema/vX.Y.Z/`.

The schema reference is its own [versioned Docusaurus docs instance](https://docusaurus.io/docs/versioning) (plugin id `schema`, config in `docusaurus.config.js`, sidebar in `sidebars-schema.js`), separate from the main `docs/` instance. Only released schema tags are published; there is no rolling "latest" built from `main`. The newest tag in `schema_versions.json` is served at `/schema/`, older tags at `/schema/vX.Y.Z/`, and a version dropdown appears in the navbar on schema pages only.
**If you spot a typo or error under `docs.overturemaps.org/schema`, it is not fixable in this repository.** Open an issue or PR against the docstrings/models in [OvertureMaps/schema](https://github.com/OvertureMaps/schema) instead; the fix appears in the next release's snapshot. How snapshots are added is in [CONTRIBUTING](CONTRIBUTING.md#schema-reference).

Snapshots are generated once per tag and committed under `schema_versioned_docs/version-vX.Y.Z/`. Builds do not call the schema generator, so a production deploy only ever changes the schema pages when a new snapshot lands here.
## Release Calendar

**If you spot a typo or error under `docs.overturemaps.org/schema`, it is not fixable in this repository.** Open an issue or PR against the docstrings/models in [OvertureMaps/schema](https://github.com/OvertureMaps/schema) instead; the fix appears in the next release's snapshot. Re-snapshotting an existing tag is possible (delete its three artifacts and re-run the script below) but pointless unless the tag itself moved.
Release dates are published as JSON at `https://docs.overturemaps.org/release-calendar.json`, with a JSON Schema alongside it at `release-calendar.schema.json` (referenced by the file's `$schema` key, so editors validate edits as you type). `releases` is a flat list, and each entry has:

The overview page (`schema/index.md`) is copied into each snapshot when it's created. Edits to it need to be applied to the `index.md` in each `schema_versioned_docs/version-*/` too.
- `date`: release date, ISO 8601
- `dataVersion`: the release's data version
- `schemaVersion`: the schema version, or `null` when not yet determined (renders as TBD)
- `majorChangeMonth`: optional, `true` for the quarterly major breaking change release

### Adding a schema version

Run this after a `vX.Y.Z` tag is published in [OvertureMaps/schema](https://github.com/OvertureMaps/schema/releases). It needs `git`, [`uv`](https://docs.astral.sh/uv/), and `npm install` already done.

```shell
npm run add-schema-version -- v2.0.0
```

The script (`scripts/add-schema-version.mjs`) clones the schema repo at that tag, runs its `overture-codegen` into `schema/reference/`, then runs `docusaurus docs:version:schema <tag>`, which writes:

- `schema_versioned_docs/version-<tag>/` (with links into the schema repo pinned to the tag)
- `schema_versioned_sidebars/version-<tag>-sidebars.json`
- an entry in `schema_versions.json` (kept sorted newest-first; the first entry is what `/schema/` serves)

Commit those three and open a PR. `schema/reference/` is cleaned up afterwards.

Only tags that ship the `overture-schema-codegen` package (v1.17.0 and later) can be added; earlier releases were JSON Schema and have no generator. The script refuses tags that don't match `vX.Y.Z` or are already in `schema_versions.json`.

### Schema PR previews

The schema repo's PR preview workflow checks out this repo, generates Markdown from the PR branch into `schema/reference/`, and builds with `SCHEMA_PREVIEW=true`. In that mode the `schema` instance builds only the `current` version from `schema/reference/` at `/schema/`; committed snapshots, the version dropdown, blog, and community pages are skipped.
An entry dated after today is upcoming, and anything else has shipped. A release dated today counts as shipped. The file has no status field, so compare `date` against the current date. For which releases have actually shipped, use [STAC](https://stac.overturemaps.org/). The `/release-calendar` page renders from this file, and the monthly update steps are in [CONTRIBUTING](CONTRIBUTING.md#release-calendar).

## Developing

Expand All @@ -77,32 +62,12 @@ Now navigate to <http://localhost:3000> to see the live preview.
- `npm run build` - Build the production site (also shows locale/translation warnings and broken link checks)
- `npm run serve` - Serve the built site locally
- `npm run deploy` - Deploy the site
- `npm run fetch-og` - Fetch and cache `og:image` metadata for community project entries (see [OG Image Cache](#og-image-cache) below)
- `npm run add-schema-version -- vX.Y.Z` - Snapshot the schema reference for a released schema tag (see [Adding a schema version](#adding-a-schema-version) above)
- `npm run fetch-og` - Fetch and cache `og:image` metadata for community project entries (see [CONTRIBUTING](CONTRIBUTING.md#og-image-cache))
- `npm run add-schema-version -- vX.Y.Z` - Snapshot the schema reference for a released schema tag (see [CONTRIBUTING](CONTRIBUTING.md#adding-a-schema-version))
- `npm run swizzle` - Customize Docusaurus components by "ejecting" them for modification
- `npm run write-translations` - Generate translation files for internationalization
- `npm run write-heading-ids` - Auto-generate heading IDs for better linking

## OG Image Cache

The community page displays project cards with images. Each entry in `community/community-projects.json` can include an optional `"image"` field. For entries without one, the site falls back to a cached `og:image` fetched from the project's URL.

The cache lives in `community/og-image-cache.json` and is committed to the repository so CI builds never make external HTTP requests.

**When to run it:** after adding or updating entries in `community-projects.json`.

```shell
npm run fetch-og
```

The script (`scripts/fetch-og-images.mjs`):
1. Skips entries that already have an explicit `"image"` field
2. Re-validates any previously cached non-empty URLs via a HEAD request (`Content-Type: image/*`) and clears invalid ones
3. Fetches the HTML for uncached entries, extracts `og:image`, and validates the URL before writing it to the cache
4. Is idempotent - safe to re-run at any time

Cards with no image (neither explicit nor cached) display a branded gradient placeholder.

## LLM-Friendly Content

Each production build generates [llmstxt.org](https://llmstxt.org)-standard files for use with LLMs and AI tools:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
---
title: 2024-07-22.0 release notes
slug: 2024-07-22.0
tags:
- releases
- addresses
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
---
title: 2024-08-20.0 release notes
slug: 2024-08-20.0
tags:
- releases
---
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
---
title: 2024-09-18.0 release notes
slug: 2024-09-18.0
tags:
- releases
---
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
---
title: 2024-10-23.0 release notes
slug: 2024-10-23.0
tags:
- releases
---
Expand Down Expand Up @@ -39,7 +38,7 @@ We removed the `connector_ids` property from the schema and replaced it with a n

## Deprecations

In the `2024-08-20.0` [release notes](https://docs.overturemaps.org/blog/2024-08-20.0/), we announced the deprecation of the `connector_ids` property in the transportation schema. We have removed that property in this release.
In the `2024-08-20.0` [release notes](https://docs.overturemaps.org/blog/2024/08/20/release-notes/), we announced the deprecation of the `connector_ids` property in the transportation schema. We have removed that property in this release.

## Theme-specific updates

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
---
title: 2024-11-13.0 release notes
slug: 2024-11-13.0
tags:
- releases
---
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
---
title: 2024-12-18.0 release notes
slug: 2024-12-18.0
tags:
- releases
---
Expand Down
2 changes: 1 addition & 1 deletion docs/examples/ibis.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ import ibis
from ibis import _

t = ibis.read_parquet(
"s3://overturemaps-us-west-2/release/2024-09-18.0/theme=base/type=infrastructure/*",
"s3://overturemaps-us-west-2/release/__OVERTURE_RELEASE/theme=base/type=infrastructure/*",
table_name="infra",
)

Expand Down
2 changes: 1 addition & 1 deletion docs/examples/pandas.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ SELECT
names.primary AS primary_name,
ST_AsText(geometry) as geometry
FROM
read_parquet('s3://overturemaps-us-west-2/release/2024-09-18.0/theme=base/type=water/*', filename=true, hive_partitioning=1)
read_parquet('s3://overturemaps-us-west-2/release/__OVERTURE_RELEASE/theme=base/type=water/*', filename=true, hive_partitioning=1)
WHERE
bbox.xmin >= -91.3994
and bbox.xmax <= -89.3864
Expand Down
2 changes: 1 addition & 1 deletion docs/examples/spark.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

# Define constants for our read
OVERTURE_RELEASE = "2025-01-22.0"
OVERTURE_RELEASE = "__OVERTURE_RELEASE"
COUNTRY_CODES_OF_INTEREST = ["US", "GH"]
SOURCE_DATA_URL = f"s3a://overturemaps-us-west-2/release/{OVERTURE_RELEASE}/theme=places/type=place"
OUTPUT_FILE = "my_super_cool_data.parquet"
Expand Down
2 changes: 1 addition & 1 deletion docs/examples/wherobots.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ Define your area of interest (in this case, Golden Gate Park) and a reusable hel
region_wkt = "POLYGON ((-122.5106 37.7736, -122.4549 37.7788, -122.4491 37.7656, -122.5103 37.7606, -122.5106 37.7736))"

# Define the release version as a variable for configurability
RELEASE_VERSION = "2025-06-25.0" # Update this value when a new release is available
RELEASE_VERSION = "__OVERTURE_RELEASE"

def process_overture_layer(theme, type, region_wkt):
"""
Expand Down
6 changes: 3 additions & 3 deletions docs/gers/gers-tutorial.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ D CREATE TABLE IF NOT EXISTS places AS
confidence
FROM (
SELECT *
FROM read_parquet('s3://overturemaps-us-west-2/release/2025-04-23.0/theme=places/type=place/*', filename=true, hive_partitioning=1),
FROM read_parquet('s3://overturemaps-us-west-2/release/__OVERTURE_RELEASE/theme=places/type=place/*', filename=true, hive_partitioning=1),
bounding_box
WHERE addresses[1] IS NOT NULL
AND bbox.xmin BETWEEN (bounding_box.min_lon - 0.01) AND (bounding_box.max_lon + 0.01)
Expand All @@ -196,7 +196,7 @@ We’re doing several things here that are worth breaking down:
2. Our following `SELECT` statement prepares the Places data to more closely align with the conventions of our inspections data: the `addresses` column is broken down into component columns and we use only the primary name from the `names` column and label it to match `Facility_Name`.
3. Finally, we obtain the data from the Overture S3 bucket by remotely reading the parquet. A `WHERE` statement limits our request to a slightly buffered bounding box (buffered to ensure we’re capturing places near those facilities on the edges of our dataset).

We now have three tables: `inspections` with 27,515 records, `facilities` with 4,340 records, and `places` with 73,985 records.
We now have three tables: `inspections` with 27,515 records, `facilities` with 4,340 records, and `places` with 73,985 records. These counts, and the match results below, come from the April 2025 release. Expect different numbers when you run this against current data.
Comment thread
danabauer marked this conversation as resolved.

Before we work on connecting them, we’ll add some spatial indexes to each table, making our lookups much faster:

Expand Down Expand Up @@ -471,7 +471,7 @@ D CREATE TABLE places AS
categories
FROM (
SELECT *
FROM read_parquet('s3://overturemaps-us-west-2/release/2025-04-23.0/theme=places/type=place/*', filename=true, hive_partitioning=1),
FROM read_parquet('s3://overturemaps-us-west-2/release/__OVERTURE_RELEASE/theme=places/type=place/*', filename=true, hive_partitioning=1),
bounding_box
WHERE addresses[1] IS NOT NULL AND
bbox.xmin BETWEEN (bounding_box.min_lon - 0.01) AND (bounding_box.max_lon + 0.01) AND
Expand Down
Loading
Loading