feat(schema): one folder per source release, sector as a column (schema 0.3.0)

.github/workflows/ci.yml

@@ -1,6 +1,6 @@
 ---
 # PR gate: every pull request to main (and every push to a non-main branch) validates
-# the committed sector data against the repo schemas before it can be merged. This is a
+# the committed source data against the repo schemas before it can be merged. This is a
 # data-only repo, so the check is a single self-contained validation script.
 name: CI checks
 

README.md

@@ -6,7 +6,7 @@ Standardized life cycle inventory (LCI) data for the Sentier platform.
 Processes and their exchanges, stored as parquet. A Data Layer repo: artifacts only.
 
 - Fed by `sentier-importers`, which opens PRs with validated parquet.
-- Read by `sentier-brightway`, which loads every sector folder.
+- Read by `sentier-brightway`, which loads one source folder into one Brightway database.
 - Cross-source flow ids resolve via `sentier-mappings`.
 - No Python package, no loader. The only code is the CI validator.
 
@@ -16,7 +16,7 @@ Nothing to install. Clone the repo and read the parquet files directly.
 
 ## Use
 
-Validate all sector data against the schema contracts. This is what CI runs on every PR.
+Validate all source folders against the schema contracts. This is what CI runs on every PR.
 
 ```bash
 uv run --with "jsonschema[format],pyyaml,pyarrow" python scripts/validate.py
@@ -25,33 +25,32 @@ uv run --with "jsonschema[format],pyyaml,pyarrow" python scripts/validate.py
 Inspect a delivered parquet payload.
 
 ```bash
-uv run --with pandas,pyarrow python -c "import pandas as pd; print(pd.read_parquet('data/01-agriculture/processes.parquet'))"
+uv run --with pandas,pyarrow python -c "import pandas as pd; print(pd.read_parquet('data/bafu-2026-v1/processes.parquet'))"
 ```
 
 ## Layout
 
 ```
 schema/                  # the data contract, read by sentier-importers
-data/<NN>-<sector>/      # one folder per sector, ranked by NN
+data/<source>-<version>/ # one folder per published release of one datasource
 scripts/validate.py      # CI validator, self-contained
 .github/workflows/ci.yml # runs scripts/validate.py on every PR and branch push
 ```
 
 ## Data
 
-Inventory is organized by sector, not by source. Each process row carries a `source` tag (`bafu-2026`) and each folder's `metadata.json` lists the `sources` it holds, so consumers select by source without a source-named folder. See [data/README.md](data/README.md) for the id convention.
+Inventory is organized by source. One folder per published release of one datasource, named `<source>-<source_version>` (`bafu-2026-v1`). A delivery replaces its source folder whole; sources never share a file. Every process row carries `source`, `source_version` and `sector`, so a consumer picks a folder by its `metadata.json` and filters rows by sector. See [data/README.md](data/README.md) for the id convention.
 
-Each `data/<NN>-<sector>/` folder holds:
+Each `data/<source>-<source_version>/` folder holds:
 
 | file | rows |
 |---|---|
 | `processes.parquet` | one per process |
 | `exchanges.parquet` | one per exchange (technosphere and biosphere edges) |
-| `metadata.json` | sector, title, rank, schema_version, sources, row_counts |
+| `metadata.json` | source, source_version, title, publisher, license, citation, schema_version, sectors, row_counts |
 
-- The `NN` prefix orders sectors. Lower wins when records overlap.
-- Sector ids are lower-kebab: `agriculture`, `electricity`, `chemicals`.
-- Background link targets such as ecoinvent are not sectors. They resolve through `sentier-mappings`.
+- Sector ids are lower-kebab enum values in `schema/common.yaml`: `agriculture`, `electricity`, `chemicals`, ...
+- Background link targets such as ecoinvent are not sources here. They resolve through `sentier-mappings`.
 - Parquet is committed directly. No git-LFS, no release artifacts.
 
 ## Schema
@@ -60,12 +59,12 @@ Plain YAML. Each file lists the columns of one parquet table. LinkML is used onl
 
 | file | describes |
 |---|---|
-| `schema/common.yaml` | shared enums: `flow_direction`, `flow_type`, `process_type` |
+| `schema/common.yaml` | shared enums: `flow_direction`, `flow_type`, `process_type`, `sector` |
 | `schema/process.yaml` | `processes.parquet`, primary key `process_id` |
 | `schema/exchange.yaml` | `exchanges.parquet`, joins on `process_id` |
-| `schema/metadata.schema.json` | JSON Schema for each sector's `metadata.json` |
+| `schema/metadata.schema.json` | JSON Schema for each source folder's `metadata.json` |
 
-CI checks columns, types, enums, `row_counts`, `process_id` uniqueness, and the exchange to process foreign key per sector.
+CI checks columns, types, enums, `row_counts`, `process_id` uniqueness, the exchange to process foreign key, one `source`/`source_version` per folder matching its name, and `sectors` against the column.
 
 ## Contributing
 

data/README.md

@@ -1,26 +1,28 @@
 # data/
 
-Raw **parquet** inventory artifacts, **one subfolder per sector / data type**,
-ranked by a numeric prefix:
+Raw **parquet** inventory artifacts, **one subfolder per published release of one
+datasource**:
 
 ```
 data/
-  01-agriculture/   # rank 1 — highest resolution precedence
-  02-electricity/   # Brightcon deliverable
-  03-chemicals/     # chemicals & materials
+  bafu-2026-v1/   # BAFU:2026 v1 LCI database
 ```
 
-Convention: `data/<NN>-<sector>/`, where `<sector>` is a lower-kebab sector /
-data-type id (`agriculture`, `electricity`, `chemicals`) and `<NN>` sets folder
-ordering / resolution precedence (lower wins when records overlap).
-
-Inventory is organized by **sector, not by datasource**. Upstream sources
-(BAFU, Agribalyse, …) each span many sectors, so a folder may hold rows from
-several sources. Every process row is tagged with its `source` (and optional
-`source_version`), and each folder's `metadata.json` lists the `sources` present,
-so a consumer can pick folders by metadata and rows by column. Precedence between
-folders is by rank; within a folder the consumer selects by `source` (there is no
-intra-folder precedence rule). The import log of which upstream file produced
+Convention: `data/<source>-<source_version>/`, where `<source>` is the lower-kebab
+datasource id `<publisher>-<edition>` and `<source_version>` is the publisher's release.
+A folder holds exactly one source and one version. A delivery from `sentier-importers`
+replaces the folder whole; sources never share a file, so nothing is ever merged. A
+re-release is a new folder; the PR that adds it removes the old one unless both are meant
+to be served.
+
+Inventory is organized by **source**, with the **sector as a column**. Every process
+row carries `source`, `source_version` and `sector` (enum in `../schema/common.yaml`:
+`agriculture`, `electricity`, `chemicals`, `transport`, `energy`, `waste`,
+`construction`, `materials`, `electronics`, `building-services`, `obsolete`). Each
+folder's `metadata.json` carries the publisher, licence, required citation, the
+`sectors` present and the row counts, so a consumer can pick a folder from metadata
+alone and filter rows by sector. Within one source there is no precedence rule;
+between sources the consumer chooses. The import log of which upstream file produced
 which rows stays in `sentier-importers`.
 
 ### Source ids
@@ -31,15 +33,12 @@ sentier-mappings pair-folder prefix. The publisher's release goes in
 `source_version`, never in the id, so a re-release does not rename the database a
 consumer's project depends on.
 
-| datasource | `source` | `source_version` |
-|---|---|---|
-| BAFU:2026 v1 LCI database | `bafu-2026` | `v1` |
+| datasource | `source` | `source_version` | folder |
+|---|---|---|---|
+| BAFU:2026 v1 LCI database | `bafu-2026` | `v1` | `bafu-2026-v1/` |
 
 > Background datasets that exist only as link targets (e.g. ecoinvent) are not a
-> sector; they are resolved through `sentier-mappings`, not stored here as their
-> own folder.
+> source here; they are resolved through `sentier-mappings`, not stored as a folder.
 
 Each folder holds `processes.parquet`, `exchanges.parquet`, and `metadata.json`.
-See [../schema/](../schema/) for column definitions. Parquet payloads are pushed
-by `sentier-importers` / `sentier-models`; the skeleton ships `.gitkeep` +
-`metadata.json` stubs only.
+See [../schema/](../schema/) for column definitions.

data/bafu-2026-v1/metadata.json

@@ -0,0 +1,29 @@
+{
+  "source": "bafu-2026",
+  "source_version": "v1",
+  "title": "Life Cycle Inventory database of the Swiss Federal Administration, BAFU:2026 (v1)",
+  "publisher": "Swiss Federal Office for the Environment (BAFU/FOEN)",
+  "license": "BAFU Terms of Use: free use and processing, no redistribution as or inside another database without agreement (§2.1/§2.2), share-alike and change documentation for modified data (§3), citation required (§2.3)",
+  "citation": "Life Cycle Inventory database of the Swiss Federal Administration, BAFU:2026. Version 1, EcoSpold v1 export (11,947 datasets).",
+  "source_url": null,
+  "provenance": "https://vocab.sentier.dev/sources/bafu-2026",
+  "retrieved": null,
+  "schema_version": "0.3.0",
+  "sectors": [
+    "agriculture",
+    "building-services",
+    "chemicals",
+    "construction",
+    "electricity",
+    "electronics",
+    "energy",
+    "materials",
+    "obsolete",
+    "transport",
+    "waste"
+  ],
+  "row_counts": {
+    "processes.parquet": 11947,
+    "exchanges.parquet": 420063
+  }
+}

schema/README.md

@@ -7,10 +7,10 @@ parquet repos describe columns directly.
 
 | file | describes |
 |---|---|
-| `common.yaml` | shared enums (`flow_direction`, `flow_type`, `process_type`) |
+| `common.yaml` | shared enums (`flow_direction`, `flow_type`, `process_type`, `sector`) |
 | `process.yaml` | `processes.parquet` |
 | `exchange.yaml` | `exchanges.parquet` |
-| `metadata.schema.json` | JSON Schema for each sector's `metadata.json` |
+| `metadata.schema.json` | JSON Schema for each source folder's `metadata.json` |
 
 Column types: `string` · `double` · `int` · `date` · `enum` (enum values live in
 `common.yaml`). `processes.parquet` is the parent; `exchanges.parquet` joins on

schema/common.yaml

@@ -16,4 +16,17 @@ enums:
     - system        # a system process aggregating multiple unit processes
     - lci_result    # a cradle-to-gate LCI result
 
+  sector:           # sector / data-type a process belongs to (was the folder level < 0.3.0)
+    - agriculture
+    - electricity
+    - chemicals
+    - transport
+    - energy
+    - waste
+    - construction
+    - materials
+    - electronics
+    - building-services
+    - obsolete      # legacy link targets kept so older references resolve
+
 # Column types used across schemas: string | double | int | date | enum