taxonomy: offline NCBI taxonomy table, embedded in the Docker image

.github/workflows/docker.yml

@@ -62,6 +62,9 @@ jobs:
           docker run --rm matpredict:test detect --help > /dev/null
           docker run --rm --entrypoint /bin/bash matpredict:test /shell-hook.sh \
             bash -c 'for t in tblastn diamond miniprot exonerate mafft augustus taxonkit; do command -v $t > /dev/null || { echo "missing $t"; exit 1; }; done; python -c "import pyhmmer, MATPredict"'
+          # embedded taxonomy answers with no network (P. tannophilus: code 26)
+          docker run --rm --network none --entrypoint /bin/bash matpredict:test /shell-hook.sh \
+            python -c "from MATPredict.db import taxonomy as t; assert t.default_genetic_code(4918) == 26; assert t.default_lineage_phylum_name(4918) == 'Ascomycota'; print('taxonomy ok')"
 
       - name: Log in to GHCR
         if: github.event_name != 'pull_request'

CHANGELOG.md

@@ -4,6 +4,20 @@ All notable changes to MATPredict. Versions follow [semantic versioning](https:/
 release tags may carry a clade suffix naming the lineage whose infrastructure that
 release completed.
 
+## [Unreleased]
+
+### Added
+- Offline NCBI taxonomy. `matpredict curate-db build-taxonomy` writes a slim
+  all-taxa table (taxid, parent, rank, genetic code, scientific name; merged
+  ids) from an NCBI taxdump directory, `taxdmp_*.zip` or `taxdump.tar.gz`
+  (plain, `.gz` or `.zst` files). With `$MATPREDICT_TAXONOMY` set, lineage,
+  phylum and genetic code come from it; `$MATPREDICT_OFFLINE=1` never calls
+  NCBI. Checked against 8,185 cached efetch answers: phylum and genetic code
+  identical for all; lineage identical for 8,179 (6 re-parented by NCBI between
+  fetch and snapshot). Reports gain `taxonomy_source`.
+- The Docker image builds the table from the dated archive
+  `taxdmp_2026-10-01.zip` (SHA-256 pinned) and runs offline by default.
+
 ## [0.6.1] — 2026-10-04 — `v0.6.1`
 
 ### Fixed

Dockerfile

@@ -3,8 +3,9 @@
 # Runtime stage: the environment, the package source and the curated database.
 #
 #   docker build -t matpredict .
-#   docker run --rm -v "$PWD":/data -e MATPREDICT_NCBI_EMAIL=you@example.org \
+#   docker run --rm -v "$PWD":/data \
 #     matpredict detect --genome /data/genome.fna --taxid 4837 --out-dir /data/out
+# Taxonomy lookups use the embedded NCBI snapshot (no network).
 
 FROM ghcr.io/prefix-dev/pixi:0.71.3-noble AS build
 WORKDIR /app
@@ -14,6 +15,20 @@ RUN pixi install --locked -e default \
  && pixi shell-hook -e default -s bash > /shell-hook.sh \
  && echo 'exec "$@"' >> /shell-hook.sh
 
+# Offline NCBI taxonomy (curator ruling 2026-10-04): a slim all-taxa table
+# (taxid, parent, rank, genetic code, scientific name; merged ids) built from a
+# DATED NCBI archive, so one image = one taxonomy snapshot and runs make no
+# E-utilities call. NCBI publishes no checksum for the archive; the SHA-256 is
+# the one measured when the date was pinned. Change both together.
+ARG TAXDUMP_DATE=2026-10-01
+ARG TAXDUMP_SHA256=d744af371c0b9fc7269d80b49546b6ac4c9ddc52a2fd97bcbfd02783edb5c9eb
+RUN /app/.pixi/envs/default/bin/python -c "import urllib.request,sys; urllib.request.urlretrieve(sys.argv[1], '/tmp/taxdmp.zip')" \
+      "https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_${TAXDUMP_DATE}.zip" \
+ && echo "${TAXDUMP_SHA256}  /tmp/taxdmp.zip" | sha256sum -c - \
+ && PYTHONPATH=/app/src /app/.pixi/envs/default/bin/python -m MATPredict curate-db build-taxonomy \
+      --taxdump /tmp/taxdmp.zip --snapshot "${TAXDUMP_DATE}" --out /app/taxonomy/ncbi_taxonomy.tsv.zst \
+ && rm /tmp/taxdmp.zip
+
 FROM ubuntu:24.04 AS runtime
 ARG VERSION=dev
 LABEL org.opencontainers.image.title="MATPredict" \
@@ -27,7 +42,13 @@ COPY --from=build /shell-hook.sh /shell-hook.sh
 COPY --from=build /app/src /app/src
 COPY pyproject.toml README.md /app/
 COPY db /app/db
+COPY --from=build /app/taxonomy /app/taxonomy
+# MATPREDICT_OFFLINE=1: a taxid not in the snapshot is reported (routing_error)
+# instead of fetched from NCBI. Run with -e MATPREDICT_OFFLINE=0 to allow
+# E-utilities for such taxids.
 ENV MATPREDICT_DB_ROOT=/app/db \
+    MATPREDICT_TAXONOMY=/app/taxonomy/ncbi_taxonomy.tsv.zst \
+    MATPREDICT_OFFLINE=1 \
     MATPREDICT_CACHE_DIR=/data/.matpredict_cache
 WORKDIR /data
 ENTRYPOINT ["/bin/bash", "/shell-hook.sh", "matpredict"]

README.md

@@ -174,11 +174,13 @@ pixi run matpredict --help
 ### Other ways to install
 
 **Docker.** Each release publishes an image to the GitHub Container Registry.
-The image holds the environment, the package and the curated database.
+The image holds the environment, the package, the curated database and an
+NCBI taxonomy snapshot (see [Offline taxonomy](#offline-taxonomy)), so a run
+needs no network.
 
 ```bash
 docker pull ghcr.io/stajichlab/matpredict:latest     # or a release version, :<version>
-docker run --rm -v "$PWD":/data -e MATPREDICT_NCBI_EMAIL=you@example.org \
+docker run --rm -v "$PWD":/data \
   ghcr.io/stajichlab/matpredict:latest \
   detect --genome /data/genome.fna --taxid 4837 --out-dir /data/out
 ```
@@ -339,6 +341,27 @@ tool still runs, sends no e-mail, and logs one warning.
 | `MATPREDICT_CONFIG` | `~/.config/matpredict/config.toml` | Config file path |
 | `MATPREDICT_DB_ROOT` | `./db` | Reference database to use |
 | `MATPREDICT_CACHE_DIR` | `./.matpredict_cache` | Cache for NCBI responses. The e-mail and API key are not part of the cache key, so users with different settings share one cache |
+| `MATPREDICT_TAXONOMY` | none (image: `/app/taxonomy/ncbi_taxonomy.tsv.zst`) | Offline NCBI taxonomy table; used before E-utilities |
+| `MATPREDICT_OFFLINE` | off (image: `1`) | `1`: never call NCBI; a taxid not in the table is reported in `routing_error` |
+
+### Offline taxonomy
+
+`detect` needs the lineage, phylum and genetic code of a `--taxid`. A local
+table answers these without NCBI E-utilities. Build it from a dated NCBI
+archive and point `MATPREDICT_TAXONOMY` at it:
+
+```bash
+curl -O https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2026-10-01.zip
+matpredict curate-db build-taxonomy --taxdump taxdmp_2026-10-01.zip \
+    --snapshot 2026-10-01 --out ncbi_taxonomy.tsv.zst
+export MATPREDICT_TAXONOMY=$PWD/ncbi_taxonomy.tsv.zst
+export MATPREDICT_OFFLINE=1        # optional: never call NCBI
+```
+
+All taxa, about 24 MB; loading it takes about 2 s and 80 MB of memory. Each
+report records the source in `taxonomy_source` (for example
+`local NCBI taxonomy snapshot 2026-10-01`). The Docker image builds this table
+at image build time and sets both variables.
 
 ### Tests
 

environment.yml

@@ -7,6 +7,8 @@ dependencies:
 - python >=3.11
 - pip *
 - taxonkit *
+- zstd *
+- zstandard *
 - diamond *
 - blast *
 - miniprot *