Skip to content

taxonomy: offline NCBI taxonomy table, embedded in the Docker image - #19

Merged
hyphaltip merged 3 commits into
mainfrom
offline-taxonomy
Oct 4, 2026
Merged

hyphaltip merged 3 commits into
mainfrom
offline-taxonomy

Conversation

@hyphaltip

Copy link
Copy Markdown
Member

Implements spec item A1 (offline taxonomy; docs/superpowers/specs/2026-10-04-packaging-service-and-reports-design.md), per curator request "embed/use the slim table (all taxa) in the container".

What

  • matpredict curate-db build-taxonomy --taxdump <dir|taxdmp_*.zip|taxdump.tar.gz> --snapshot DATE --out ncbi_taxonomy.tsv.zst: slim all-taxa table (taxid, parent, rank, nuclear genetic code, scientific name; merged ids). Dump files may be plain, .gz or .zst.
  • src/MATPredict/db/local_taxonomy.py: lookups with efetch semantics (lineage root-first without root/self; phylum = phylum-rank ancestor; own genetic code; merged ids followed).
  • db/taxonomy.py: default_lineage_taxids, default_lineage_phylum_name, default_genetic_code use the table first when $MATPREDICT_TAXONOMY is set. $MATPREDICT_OFFLINE=1: never call NCBI; a missing taxid raises and the router records it in routing_error.
  • Reports gain taxonomy_source (e.g. local NCBI taxonomy snapshot 2026-10-01).
  • Dockerfile: builds the table from taxdmp_2026-10-01.zip (SHA-256 pinned; NCBI publishes no checksum), sets MATPREDICT_TAXONOMY and MATPREDICT_OFFLINE=1. CI smoke test checks a lookup with --network none.
  • README: configuration table + "Offline taxonomy" section. CHANGELOG: Unreleased.

Measured

  • Table: 3,016,750 taxa, 101,411 merged ids, 23.7 MB; build 53 s; load 1.96 s, 76 MB RSS (streamed into int arrays; a first version used 1.45 GB).
  • vs 8,185 cached NCBI efetch answers: phylum 8,185/8,185, genetic code 8,185/8,185, lineage 8,179/8,185 (the 6 were re-parented by NCBI between fetch and snapshot, e.g. C. gattii VGV 2268382); 21 taxids deleted in the snapshot.
  • End to end, offline, empty cache (P. tannophilus, --taxid 669874): 0 NCBI calls; genetic code 26 from the table; calls identical to the online run (results/2026-10-04_offline_taxonomy_check/).
  • Tests: 1017 passed (10 new in tests/db/test_local_taxonomy.py, on Python 3.12 without compression.zstd).

Not measured yet: image size with the table, and the CI image build (runs on this PR).

🤖 Generated with Claude Code
Claude-Session: https://claude.ai/code/session_01JkhYFEPaa1WenE5MtA8Qya

hyphaltip and others added 2 commits October 4, 2026 14:21
`curate-db build-taxonomy` writes a slim all-taxa table (taxid, parent, rank,
genetic code, scientific name; merged ids) from a taxdump directory, NCBI
taxdmp_*.zip or taxdump.tar.gz (plain, .gz or .zst). With $MATPREDICT_TAXONOMY
set, lineage, phylum and genetic code come from it; $MATPREDICT_OFFLINE=1
never calls NCBI (a missing taxid is reported in routing_error). Reports gain
`taxonomy_source`.

Measured: taxdmp_2026-10-01 -> 3,016,750 taxa, 101,411 merged, 23.7 MB; load
1.96 s, 76 MB RSS (streamed into int arrays). Against 8,185 cached efetch
answers: phylum and genetic code identical for all; lineage for 8,179 (6
re-parented by NCBI between fetch and snapshot); 21 deleted taxids. Offline
end-to-end run (P. tannophilus, code 26): no NCBI call, calls identical to
the online run.

Dockerfile builds the table from the dated archive (SHA-256 pinned) and sets
MATPREDICT_TAXONOMY and MATPREDICT_OFFLINE=1; CI smoke test checks a lookup
with --network none. Tests: 1017 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JkhYFEPaa1WenE5MtA8Qya
CI image build failed in build-taxonomy with "No such file or directory:
'zstd'": pixi.lock pins Python 3.13, which has no compression.zstd (3.14+),
and the fallback called the zstd binary, which is not on PATH in that build
step. Local checks had run on a 3.14 environment. zstandard (conda-forge
0.25.0) is now a dependency for Python < 3.14 (pixi.toml, pyproject,
environment.yml); the table is streamed through it.

Checked on the locked default environment (Python 3.13.15) with an empty
PATH: build 87 s, 2.3 GB peak; table content identical to the 3.14 build;
load 1.72 s, 74 MB. Tests: 1017 passed (3.12 test env: zstd-binary fallback path, no zstandard); pytest is
not in the default environment, so no test run on 3.13.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JkhYFEPaa1WenE5MtA8Qya
@hyphaltip

Copy link
Copy Markdown
Member Author

CI fix (018a549): the image build failed in build-taxonomy with No such file or directory: 'zstd'. pixi.lock pins Python 3.13, which has no compression.zstd (3.14+), and the fallback called the zstd binary, which is not on PATH in that RUN step. My earlier local checks ran on an older Python 3.14 environment, which hid this.

Fix: zstandard (conda-forge 0.25.0) is a dependency for Python < 3.14 (pixi.toml, pyproject, environment.yml; relock adds only zstandard, cffi, pycparser); the table is streamed through it.

Checked on the locked default environment (Python 3.13.15) with an empty PATH: build 87 s (2.3 GB peak), table content identical to the 3.14 build, load 1.72 s / 74 MB. Unit tests: 1017 passed on the 3.12 test env, which used the zstd-binary fallback path (no zstandard there); pytest is not in the default environment, so the tests did not run on 3.13.

@hyphaltip
hyphaltip merged commit 3ee6b83 into main Oct 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant