Skip to content

fix(conda): vendor only the needed dependencies and ship their data files (#1814) - #1821

Merged
JarryShaw merged 1 commit into
mainfrom
fix/1814-conda-vendored-deps
Oct 16, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
fix/1814-conda-vendored-deps

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Please follow the guide below

What is the purpose of your pull request?

Tick the commit type your subject line carries.

  • fix — corrects a defect
  • feat — adds a feature
  • perf — changes performance, not behaviour
  • refactor — changes neither behaviour nor performance
  • test — tests only
  • docs — documentation only
  • ci — workflows or build tooling
  • release — bumps the version or rolls up a distribution
  • chore — anything else

Description

Closes #1814. The conda package's vendored dependencies (pcapkit/_extern) reached the wheel as .py files only, so the package was broken. In an emulated build, chardet could not detect anything (models/idf.bin missing), lxml.etree and pyshark failed to import, emoji had no JSON, scapy misreported its version, and certifi.where() pointed at a missing file.

  • Ship everything: package-data now includes every file under pcapkit._extern except bytecode. MANIFEST.in cannot do this without warning in non-conda builds (measured). Clean sdist and wheel builds are unchanged apart from the files this PR adds or removes, with no new warnings. The ** glob needs setuptools 62.3: measured, 61.0 and 62.2 read it as *. The build floor in pyproject.toml and meta.yaml is raised to match.
  • Vendored list, per your note that it was stale: it is now dependencies plus the all extra and their requirements, for the supported CPython tier.
    • Dropped: the capture engines (dpkt, scapy, pyshark), which fall back at run time, and the pcapkit-vendor crawlers' requests and bs4.
    • Added: cryptography and pycrate (ESP and NGAP), plus cffi and pycparser.
  • Automation: util/conda_vendor.py derives the list from pyproject.toml, replacing util/conda-dist.py. --check flags a stale list, and --smoke checks a built package. Conda Update rewrites the list, and each conda-dist leg smoke-tests its package before upload. This is deliberately not a blocking PR check: an upstream release would otherwise turn unrelated PRs red.

Vendored set: option A, by your ruling on #1814: exactly all. The emulated .tar.bz2 is about 16 MB without bytecode, or about 39 MB if conda-build compiles .pyc (not checked). pycrate's ASN.1 data is about 154 MB unpacked without bytecode.

Not verified: no real conda build or Actions run was possible here. The macOS conda convert step ships arm64 binaries for osx-64, which already affected lxml and now affects cffi and cryptography too. That is tracked in #1822.

tests/project: 959 passed, 1 skipped. The new test_conda_vendor_1814.py fails 5 tests on main.

@JarryShaw JarryShaw added fix Pull requests that fix a defect (fix: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 16, 2026
@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 16, 2026
…iles (#1814)

- conda/requirements.txt: dependencies plus the all extra and what they
  require, for the supported CPython tier. The capture engines and the
  crawlers' requests and beautifulsoup4 go, as on PyPI since #910;
  cryptography and pycrate, with cffi and pycparser, come in; the
  pathlib2 and typing-extensions lines were dead on every conda build.
- util/conda_vendor.py derives that list, replacing util/conda-dist.py;
  --check reports a stale list, --smoke a built package's missing files.
- pyproject.toml: package-data ships every file under pcapkit/_extern but
  bytecode; before, chardet could not detect and emoji had no JSON. Its
  ** needs setuptools 62.3.0, so the floor there and in meta.yaml rises
  to it (61.0.0 and 62.2.0 read it as *, measured).
- cron-conda.yml: Conda Update installs .[all] and runs the script; each
  conda-dist leg smoke-tests its package before the upload, and no longer
  installs pathlib2 and typing_extensions.
- tests/project/test_conda_vendor_1814.py pins the package-data rule,
  the derivation, the committed list and the smoke.

Clean sdist and wheel: unchanged but for the files added and removed
here, with no new warning. tests/project passes.

Closes #1814
@JarryShaw
JarryShaw force-pushed the fix/1814-conda-vendored-deps branch from 7f01d49 to ea5a8bb Compare October 16, 2026 14:25
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 16, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Verdict: GOOD TO GO at ea5a8bb47. A Sonnet cross-review and re-check (author: Opus) checked the following:

  • Packaging: on a real vendored install, every non-bytecode file reaches the wheel and nested .pyc stays out. A normal build's wheel is unchanged.
  • Vendored list: it resolves on every conda target (3.11–3.15, Linux and arm64 macOS), and the script reproduces it byte-identically. --check flags main's old list.
  • Lazy-read fixture: with the RECORD check disabled, the test now fails.
  • setuptools floor: confirmed independently. 61.0 and 62.2 read ** as *, while 62.3 and 68.0 ship every file. 62.3 still builds on the KTLO 3.7 legs.
  • Test runs: tests/project gives 959 passed and 1 skipped. Nothing ran on Actions or with a real conda build, which is stated in the body.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 16, 2026
@JarryShaw
JarryShaw merged commit 24eb561 into main Oct 16, 2026
55 checks passed
@JarryShaw
JarryShaw deleted the fix/1814-conda-vendored-deps branch October 16, 2026 15:01
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Oct 16, 2026
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Pull requests that fix a defect (fix: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix(conda): vendored dependencies ship without their data files

1 participant