[ENHANCEMENT](schema) Adopt a policy: schema fields carry no non-null default

[sethfitz]

Type

Task

Scope

Multi-theme or Platform

Skillset

engineering

Description

The policy

An Overture schema field carries no non-null default value. A field is either required, or it is optional and has no value when absent. Where a value would otherwise be supplied by a default, it is supplied by the publisher, in the published data.

default is not one concept. Each representation reads it differently, and the two that Overture consumers actually use cannot express it at all.

Where it is read	What default means there
Pydantic, at parse time	Injected at construction. The parsed object carries a value the input never had, and Pydantic does not check it against the field's own constraints unless validate_default=True.
JSON Schema	Annotation only. Validators must not modify the instance; code generators and form builders often do. Same keyword, opposite behavior.
Parquet, the published form	No such concept. A column is nullable or it is not. Absent is null.
SQL over a release (DuckDB, Athena, Spark)	Invisible. WHERE level = 0 misses every feature that relied on the default.

The defaults never reach the published data

Measured against release 2026-08-19.0, one partition per type. Every non-null default in the schema occurs in zero published rows.

Field	Default	Rows measured	Rows equal to the default	Null	Other
Building.level	0	4,778,001	0	4,777,954	47
Water.level	0	1,985,644	0	1,985,620	24
Land.level	0	1,174,629	0	1,174,529	100
SpeedLimitRule.is_max_speed_variable	False	601,764 rules	0	601,497	267 (true)

The only consumer who ever sees level = 0 is one parsing a feature through the Pydantic models -- which is not how Overture data is primarily consumed, notwithstanding that the schema is authored in Pydantic. Everyone reading the release sees level = null.

That the minority path is the affected one makes this worse rather than more tolerable. The default is invisible to the audience the schema exists to serve, and the one path that can see it is handed a value the release does not contain.

Where the default belongs

Publication. If ground level is the right value for a building that does not state one, the pipeline that produces the release writes it, once, where the decision is visible and every consumer sees the same answer.

A column that is overwhelmingly one value is dictionary- and RLE-encoded in Parquet, so populating level explicitly across billions of buildings costs close to nothing on disk.

If instead the right answer is "absent means ground level," that is a sentence in the field description, which every representation carries and every consumer can read.

A default is a claim about what the data says; a description is a claim about what absence means. The schema may make the second and not the first.

What the policy does not forbid

• = None on an optional field. That is absence, not a value.
• The MISSING sentinel (pydantic.experimental.missing_sentinel) on bbox and id. It expresses JSON Schema omitability rather than a data value; see packages/overture-schema-system/src/overture/schema/system/optionality.py.

default_factory is forbidden outright, including factories that produce None. A factory is a Python callable, and where a literal default is at least transportable -- JSON Schema can carry default: 0 even though consumers disagree about what it means -- a callable cannot be rendered into any representation but Python. No schema field uses it today.

Related

• Segments with no connectors fail validation after a round trip #669 -- connectors = [] is rejected by its own min_length=2; the bug this policy generalizes.
• [ENHANCEMENT](transportation) Make segment connectors required #692 -- makes segment connectors required; the first removal, already in flight.
• [ENHANCEMENT](codegen) Decide support-or-reject for Pydantic Field kwargs the extractor ignores or crashes on #640 -- support-or-reject decision for the Pydantic Field() kwargs the extractor ignores or crashes on; default is the entry this issue resolves.
• [Places] Initial commit of places operating status #354 -- where the problem was first named, on operating_status.

[danabauer]

Somewhat related: #698

[sethfitz]

Splitting out the enforcement question, because I think it's got a better answer than the one I was circling.

The obvious place to enforce this is the code generators: have each target warn when it meets a non-null default. That's the wrong venue, for three reasons. It fires at generation time rather than at authoring time, so the person who added the default isn't the person who sees the warning. It fires once per target, in whatever stream that target writes to, so the same violation gets reported N times or zero times depending on which targets someone happens to run. And it can only ever carry rules a renderer has some reason to care about (which is a small and fairly arbitrary subset of what an Overture-style schema wants to be held to).

Proposal: overture-schema validate-schema

A subcommand that sanity-checks the Pydantic models themselves against a set of stricter rules for Overture-style projects. It fires once, at the source, on the thing being checked, and it runs in CI beside the other gates to validate our schema. The same checks can be adopted by others in their environments.

The non-null-default policy above would be one rule among them. Others already exist as conventions, enforced today only by review:

• No default_factory anywhere, including factories that return None (above).
• A field constraint has to be satisfiable by the field's own default. Segments with no connectors fail validation after a round trip #669 is the case where it wasn't, and a linter catches that whole class outright.
• Deprecation metadata is well-formed and carries the fields [FEATURE] Mark schema fields and models as deprecated and propagate through docs, PySpark, and the changelog #639 needs to propagate.

The rule set is the interesting part, and it ought to be able to grow without anyone touching the CLI. #449 gives the mechanism: subcommands (and their contents) arrive through entry points from whichever distributions are installed, so a rule can live next to the thing it constrains instead of in a central list, surfaced either through pyproject.toml or some other mechanism.

On the name

overture-schema validate already exists and it takes data. A validate-schema that takes models is one letter of distinction away from it in --help, which isn't enough. overture-schema lint is one potentially better alternative, with the help text saying plainly that it checks the schema definitions and not instances of them. Either way the pair wants to read as two different jobs rather than two variations of the same thing.

What it isn't

Not a runtime check. None of this belongs in validate's path, which sees data and has no view of how the models were constructed. Not a replacement for the test suite either: these are conventions about how the schema is written, and a test asserting each one individually (which is what we'd otherwise end up with) is the thing this exists to stop having to write.