Add stac-table-columns renderer to overture-schema-codegen

[jjcfrancisco]

Type

Task

Scope

Multi-theme or Platform

Skillset

engineering

Description

Context

The STAC catalog at stac.overturemaps.org emits table:columns with only column names — no type or description. Consumers reading the catalog have no way to discover what a column means without leaving STAC and going to the schema docs. The information already lives in the Pydantic models; a new codegen renderer can surface it as machine-readable JSON that the STAC generator consumes at build time.

Proposed change

In packages/overture-schema-codegen/src/overture/schema/codegen/, alongside markdown/ and pyspark/:

• Add a stac_table_columns/ renderer that emits one JSON file per (theme, type), matching the STAC table extension table:columns shape: [{"name", "type", "description"}].
• type maps from the Pydantic field's resolved primitive to STAC/parquet types (string, int32, int64, double, boolean, struct, list). Reuse extraction logic already used by the pyspark renderer where possible.
• description comes from the field's docstring / description text.
• Wire into the CLI as a new --format option so overture-codegen generate --format stac-table-columns --output-dir out/ writes buildings/building.json, places/place.json, etc.

[sethfitz]

Francisco Jiménez (@jjcfrancisco) awesome. I was tinkering with this a bit while I was on vacation last week and will have a more detailed reply (and a draft PR) up soon.

[sethfitz]

I built a version of this while working out how much of our schema comes through into output targets that aren't shaped like it — STAC, JSON Schema (to see if we represent everything needed to emit it, despite Pydantic supporting it natively), Vecorel. Working code is at #724.

Generating from the models rather than from the Parquet footer is right: everything a Column Object asks for is already in the models, so the catalog is a projection with no authoring step. The footer route isn't stable — type there is whatever the describing engine's stringifier produced, so the same file yields a different catalog depending on who ran DESCRIBE.

Past that, every hard decision in this renderer is the same question: what can table:columns represent, and what happens to the rest?

The type vocabulary is an open question upstream. The extension defines type as "data type of the column" and recommends that "if using a file format with a type system (like Parquet), we recommend you use those types". The pending rewrite in stac-extensions/table#13 changes that to "native data type", and adds a separate data_type carrying the standardized STAC name. Native to Parquet is still ambiguous: Parquet has physical types (INT32, BYTE_ARRAY) and logical types (STRING, DECIMAL), and most tooling describes Parquet through Arrow — three candidate vocabularies, and the spec picks none of them. stac-extensions/table#1 has been open since 2021 asking exactly this, with no replies. Meanwhile the spec repo's own generated reference catalog emits DuckDB SQL type names, despite its examples generally being Parquet. I followed the reference catalog because it is the only executable example, but the right move is to follow the prior art and flag the divergence in #1 — we publish at a scale that thread has had no evidence from in five years. Mapping through the pyspark renderer's type names would quietly add a fourth vocabulary.

Discriminated unions like Segment don't fit a flat column list. A road class and a rail class are alternatives for the same column; a column list has one slot, so one is kept and the other disappears from the output. This happens seven times in segment. Nothing catches it: the alternatives have identical physical types, so every compatibility check passes and the catalog looks complete. Whichever way you walk the union you lose something different, and neither walk can raise. The branch logs every site where it happens, because otherwise you can generate a catalog and not be told.

Descriptions thin out with depth. Most top-level columns carry an authored description and some don't, which stays queryable so long as an absent one omits the key rather than emitting "". Nested fields and named types have nowhere to go at all — they collapse into anonymous struct(...) types. That is table:columns inheriting the traditional tabular view, where the only columns are top-level and their types are primitive.

So this closes the "names only" gap but not the "understand column meanings" one. A Column Object can't carry our constraints, optionality, defaults, or enum vocabularies, which is most of the richness in the schema. A catalog with type and description still beats one with neither. But the surplus needs a pointer — a collection-level link to the schema itself, probably the Pydantic representations, since for us those are the least lossy — rather than a fatter flattened copy. Same move as Chris Holmes's partition:glob note on our STAC PR: "at the collection level expose the value you'd pass into DuckDB". I did try routing the surplus into Ossie, a semantic-model extension, and it doesn't fit — it requires a physical source and per-dialect SQL per field, neither of which a schema has, so emitting anything at all meant inventing both.

Last, found by writing this: stac-extensions/table puts additionalProperties: false on the array schema for table:columns rather than on its items, so it is inert and a Column Object with arbitrary extra keys validates today. It misleads code generators into emitting a closed struct, which is how I hit it. I'll take that upstream separately.

[jjcfrancisco]

Thanks for this Seth Fitzsimmons (@sethfitz). I made some comments in the linked PR.

[jjcfrancisco]

Blocked by #724