Repository navigation
taxonomy: offline NCBI taxonomy table, embedded in the Docker image - #19
Conversation
`curate-db build-taxonomy` writes a slim all-taxa table (taxid, parent, rank, genetic code, scientific name; merged ids) from a taxdump directory, NCBI taxdmp_*.zip or taxdump.tar.gz (plain, .gz or .zst). With $MATPREDICT_TAXONOMY set, lineage, phylum and genetic code come from it; $MATPREDICT_OFFLINE=1 never calls NCBI (a missing taxid is reported in routing_error). Reports gain `taxonomy_source`. Measured: taxdmp_2026-10-01 -> 3,016,750 taxa, 101,411 merged, 23.7 MB; load 1.96 s, 76 MB RSS (streamed into int arrays). Against 8,185 cached efetch answers: phylum and genetic code identical for all; lineage for 8,179 (6 re-parented by NCBI between fetch and snapshot); 21 deleted taxids. Offline end-to-end run (P. tannophilus, code 26): no NCBI call, calls identical to the online run. Dockerfile builds the table from the dated archive (SHA-256 pinned) and sets MATPREDICT_TAXONOMY and MATPREDICT_OFFLINE=1; CI smoke test checks a lookup with --network none. Tests: 1017 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JkhYFEPaa1WenE5MtA8Qya
fbe79f1 to
b0ed542
Compare
CI image build failed in build-taxonomy with "No such file or directory: 'zstd'": pixi.lock pins Python 3.13, which has no compression.zstd (3.14+), and the fallback called the zstd binary, which is not on PATH in that build step. Local checks had run on a 3.14 environment. zstandard (conda-forge 0.25.0) is now a dependency for Python < 3.14 (pixi.toml, pyproject, environment.yml); the table is streamed through it. Checked on the locked default environment (Python 3.13.15) with an empty PATH: build 87 s, 2.3 GB peak; table content identical to the 3.14 build; load 1.72 s, 74 MB. Tests: 1017 passed (3.12 test env: zstd-binary fallback path, no zstandard); pytest is not in the default environment, so no test run on 3.13. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JkhYFEPaa1WenE5MtA8Qya
b0ed542 to
018a549
Compare
|
CI fix (018a549): the image build failed in Fix: Checked on the locked default environment (Python 3.13.15) with an empty PATH: build 87 s (2.3 GB peak), table content identical to the 3.14 build, load 1.72 s / 74 MB. Unit tests: 1017 passed on the 3.12 test env, which used the zstd-binary fallback path (no zstandard there); pytest is not in the default environment, so the tests did not run on 3.13. |
Implements spec item A1 (offline taxonomy;
docs/superpowers/specs/2026-10-04-packaging-service-and-reports-design.md), per curator request "embed/use the slim table (all taxa) in the container".What
matpredict curate-db build-taxonomy --taxdump <dir|taxdmp_*.zip|taxdump.tar.gz> --snapshot DATE --out ncbi_taxonomy.tsv.zst: slim all-taxa table (taxid, parent, rank, nuclear genetic code, scientific name; merged ids). Dump files may be plain,.gzor.zst.src/MATPredict/db/local_taxonomy.py: lookups with efetch semantics (lineage root-first without root/self; phylum = phylum-rank ancestor; own genetic code; merged ids followed).db/taxonomy.py:default_lineage_taxids,default_lineage_phylum_name,default_genetic_codeuse the table first when$MATPREDICT_TAXONOMYis set.$MATPREDICT_OFFLINE=1: never call NCBI; a missing taxid raises and the router records it inrouting_error.taxonomy_source(e.g.local NCBI taxonomy snapshot 2026-10-01).taxdmp_2026-10-01.zip(SHA-256 pinned; NCBI publishes no checksum), setsMATPREDICT_TAXONOMYandMATPREDICT_OFFLINE=1. CI smoke test checks a lookup with--network none.Measured
--taxid 669874): 0 NCBI calls; genetic code 26 from the table; calls identical to the online run (results/2026-10-04_offline_taxonomy_check/).tests/db/test_local_taxonomy.py, on Python 3.12 withoutcompression.zstd).Not measured yet: image size with the table, and the CI image build (runs on this PR).
🤖 Generated with Claude Code
Claude-Session: https://claude.ai/code/session_01JkhYFEPaa1WenE5MtA8Qya