Data catalogue

A static catalogue of the datasets our group holds, built by crawling our GitLab group for Frictionless datapackage.json descriptors. It answers "what data do we have, where, and which versions exist" — the discovery layer on top of data-versioning-demo, which shows the versioning (DVC) and description (Frictionless) layers for a single repo.

What it does

For every repo in the mcc-apsis group that contains a datapackage.json, the catalogue records:

• the dataset's name, title, description and tags (keywords / themes);
• each resource's name, type, format and description;
• whether the data is DVC-tracked, and if so a version timeline reconstructed from the git history of the .dvc pointer file — date, author, message, content hash, size, file count, and any git-tag / GTO version/stage labels.

Each version row carries a copy-paste retrieval command:

dvc get --rev <git-sha> <repo-url> data    # download that exact version

The result is a single self-contained output/index.html with a client-side tag filter.

Why this design

We deliberately chose a static catalogue over a metadata platform (OpenMetadata, CKAN):

• the source of truth stays in each repo (datapackage.json, git-versioned next to the code and DVC pointers); the catalogue is a derived artifact, rebuilt on demand;
• it needs no server, database, or maintainer — appropriate for a 20-person group;
• the build is read-only and data-free: it reads only text (descriptors and .dvc pointers) via the GitLab API, so it needs only a read_api token and never touches the data on shared storage.

CKAN (with the Frictionless extensions) is the documented upgrade path if we later want a multi-user portal with accounts, search at scale, and web-based curation.

The Frictionless Catalog class is flagged experimental, so we don't use it — we aggregate over the stable Package API into our own output/catalogue.json.

Usage

uv sync
export GITLAB_TOKEN=<a token with read_api scope for gitlab.pik-potsdam.de>
uv run python scripts/build_catalogue.py
# open output/index.html

Configure the host, groups, and version cap in config.toml. The token is read from GITLAB_TOKEN and is never stored in the repo.

Next steps

• Run in CI (scheduled GitLab CI job) and publish output/ to GitLab Pages, so the catalogue refreshes itself whenever any repo's datapackage changes.
• Version labels: the timeline shows git-tag / GTO labels where they exist. To make them visible in the demo, register a version on a tracked repo (git tag, or gto register).
• Natural-language query (optional): a single small LLM call can map a query like "datasets dealing with climate and health" onto the tag vocabulary, then reuse the same deterministic filter — the corpus is small enough to need no vector search.
• More sources: the crawler is structured to accept additional groups (and, later, GitHub orgs).