Repository navigation
feat(schema): one folder per source release, sector as a column (schema 0.3.0) - #5
Merged
Merged
Conversation
…ma 0.3.0) Reorganise data/ from eleven ranked sector folders into one folder per published release of one datasource, data/<source>-<source_version>/ (bafu-2026-v1). sector becomes a required enum column on processes.parquet and metadata.json describes the source (publisher, license, citation, sectors present, row counts). A delivery from sentier-importers now replaces its folder whole; sources never share a file. The validator checks one source and one version per folder, both equal to metadata.json and to the folder name, and the sectors list against the column. Row counts unchanged: 11,947 processes, 420,063 exchanges. Motivation: sector parquets did not stack across sources; a second source delivered into a sector folder would have overwritten the first.
This was referenced Oct 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reorganises
data/from eleven ranked sector folders into one folder per published release of one datasource,data/<source>-<source_version>/(today onlybafu-2026-v1/).sectorbecomes a required enum column onprocesses.parquet, and each folder'smetadata.jsondescribes the source: publisher, license, required citation, sectors present, row counts.Why: sector parquets did not stack across sources. sentier-importers rewrites a sector's files whole on every run, so a second source delivered into an existing sector folder would have erased the first. With one folder per source release a delivery replaces exactly its own folder; nothing is ever merged and two sources never share a file. Lineage is explicit per folder (metadata) and per row (
source,source_version,sector).This narrows the "organised by sector, not by source" rule to sentier-vocab, where terms are a shared concept space nobody owns. Inventory rows are owned by their source; sentier-methods and sentier-mappings already use per-dataset folders.
Changes
schema/common.yaml:sectorenum (11 values,99-obsoletebecomesobsolete).schema/process.yaml:sectorrequired;source_versionnow required.schema/metadata.schema.json: per-source metadata,schema_version0.3.0.scripts/validate.py: onesourceand onesource_versionper folder, equal to metadata and to the folder name;sectorslist equals the column's distinct values.process_id. No BAFU regeneration. Counts unchanged: 11,947 processes, 420,063 exchanges.Test plan
python scripts/validate.pypasses on the moved datasectorslist fails with the two new errorsbafu-2026-v1/and matches these row counts andprocess_idsetsPlease merge with a merge commit so the pinned sha stays reachable.