diff --git a/.github/workflows/cron-conda.yml b/.github/workflows/cron-conda.yml index 5a704d5ab..09a5d6fc7 100644 --- a/.github/workflows/cron-conda.yml +++ b/.github/workflows/cron-conda.yml @@ -576,16 +576,17 @@ jobs: echo EOF } >> "$GITHUB_OUTPUT" - # #1814: the package declares no run requirements, so the packages it - # vendors in `pcapkit/_extern` are all it has, and they once shipped as - # modules alone. This unpacks the package built for this runner and - # checks that every vendored distribution has every file its RECORD lists - # and imports from there, with no site-packages on the path, before - # anything is uploaded. #1822: it also fails unless the package, and every - # native binary in it, is for this runner's platform, so a relabelled - # package cannot pass on the runner that built it. setup-python's - # interpreter, not conda's: it is the matrix version the package was - # built for, which conda's base is not on 3.15. + # #1814: the package declares no run requirement but Python, so the + # packages it vendors in `pcapkit/_extern` are all it has, and package + # discovery on its own ships them as bare modules, without their data + # files. This unpacks the package built for this runner and checks that + # every vendored distribution has every file its RECORD lists and imports + # from there, with no site-packages on the path, before anything is + # uploaded. #1822: it also fails unless the package, and every native + # binary in it, is for this runner's platform, so a relabelled package + # cannot pass on the runner that built it. setup-python's interpreter, not + # conda's: it is the matrix version the package was built for, which + # conda's base is not on 3.15. - name: Smoke-test the vendored packages env: PCAPKIT_CONDA_PACKAGE: ${{ steps.conda_build.outputs.host_package }} diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 418b80a9f..f81992b8b 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -13,7 +13,7 @@ name: "Lint" # own numbers, and vermin too, which had moved before the issue's own text said # it "was not re-checked". Only bandit held. A written-down count invites # exactly this, with no way to tell staleness from disagreement, so each step -# writes its current count to the run summary instead; read the numbers there. +# writes its result to the run summary instead; read the numbers there. # # pylint's R class is the one that moves without the code moving. Repeated runs # on an identical tree were measured to disagree on it, entirely through @@ -54,7 +54,7 @@ name: "Lint" # # None of them is clean, so none can block: a job that is red the day # it lands teaches everyone to scroll past red, which costs more than the checks -# are worth. Advisory steps keep the numbers visible (each writes its count to +# are worth. Advisory steps keep the numbers visible (each writes its result to # the run summary) without that damage, and promoting one to blocking is a matter # of deleting its `continue-on-error` line once its count reaches zero. # diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit-tests.yml index 1b578deef..3ffcf256a 100644 --- a/.github/workflows/unit-tests.yml +++ b/.github/workflows/unit-tests.yml @@ -134,7 +134,7 @@ jobs: cache: pip # The DPKT extra is here so the tests gated on it (test_dpkt_unit.py, - # test_misc.py, test_extraction.py, test_core.py) actually execute + # test_misc.py, test_extraction.py and test_core.py among them) execute # instead of skipping -- see #729. It is a small, pure-Python # dependency (measured clean installs on 3.10-3.14), so it costs # nothing per leg and adds no new job or leg, unlike Scapy below. @@ -1392,11 +1392,11 @@ jobs: matrix: ${{ fromJSON(needs.python-versions.outputs.pypcap-parity-ktlo) }} steps: *pypcap-parity-steps - # GitHub issue #981: two test modules - # (tests/protocols/application/test_http_unit.py and - # tests/test_base_class_contract.py) each pass alone, and pytest -- the + # GitHub issue #981: the test modules + # tests/protocols/application/test_http_unit.py and + # tests/test_base_class_contract.py each pass alone, and pytest -- the # runner every job above uses -- reports the combined run green, while - # plain unittest over the same two modules in one process fails some of + # plain unittest over the same modules in one process fails some of # their subTest cases. pytest-subtests is not why -- it is not even installed in # this project (absent from the `test` extra in pyproject.toml), and plain # pytest already reports a failed subTest as its own top-level SUBFAILED @@ -1411,7 +1411,7 @@ jobs: # unittest.TextTestRunner -- that never loads conftest.py and so never # gets the reconciliation that hides the defect from pytest. # - # A cheaper, narrower alternative: `pytest --noconftest` over just those two + # A cheaper, narrower alternative: `pytest --noconftest` over just those # modules, in that order, disables that same autouse fixture directly and # reproduced the identical subTest failures in about 70s on the pre-fix # tree, under the pytest this CI already installs. Whether that single @@ -1638,8 +1638,8 @@ jobs: with: python-version: ${{ steps.python-versions.outputs.latest }} - # No install step: the generator imports argparse, difflib, pathlib, re, sys - # and typing and nothing else, so it runs against a bare interpreter. + # No install step: the generator imports nothing outside the standard + # library, so it runs against a bare interpreter. # Verified with ``python -S``. - name: Check CHANGELOG.md is in step with its source entry shell: bash @@ -1725,7 +1725,7 @@ jobs: # docs/ or a Markdown file, since a docs-only diff can still break those: # tests/project and the root-level modules, plus the modules elsewhere under # tests/ named in the second run step. tests/project/test_unit_tests_queue.py - # finds every module outside those two that names a docs/ or Markdown path + # finds every module outside those that names a docs/ or Markdown path # and fails until it is listed there, so a new reader cannot go unrun. Runs # only when `changes` says the diff is docs-only, so it never duplicates # `test` on a code change. It uses diff --git a/MANIFEST.in b/MANIFEST.in index 464ddb004..f9ccf204d 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -11,7 +11,7 @@ prune docs prune examples/captures prune temp # The engine stubs mypy reads (#1878) check this tree and are not for shipping. -# Not load-bearing today: ``global-include *.py`` does not match a ``.pyi``, +# Not load-bearing: ``global-include *.py`` does not match a ``.pyi``, # measured by building an sdist without this line. It keeps them out should a # pattern above come to match one. prune typings diff --git a/examples/benchmark/README.md b/examples/benchmark/README.md index f71452346..f3c6bdc79 100644 --- a/examples/benchmark/README.md +++ b/examples/benchmark/README.md @@ -292,8 +292,9 @@ Pinned exactly: - every engine package, in `requirements-common.txt`, `requirements-pypcap.txt` and `requirements-pcap_ct.txt`. Note that `pcap-ct` and `libpcap` are pre-release-only projects, so `pip install pcap-ct` with no version resolves to - nothing at all -- an exact pre-release pin is honoured without `--pre`, which is - why the versions are spelled out; + whichever beta is newest, through pip's fallback to pre-releases for a project + with no final release -- an exact pre-release pin is honoured without `--pre`, + which is why the versions are spelled out; - `pcapkit`'s own runtime dependencies, because they are on the baseline's hot path and the baseline is what every ratio divides by. diff --git a/examples/benchmark/requirements-pcap_ct.txt b/examples/benchmark/requirements-pcap_ct.txt index 3110e3f02..6c647df0e 100644 --- a/examples/benchmark/requirements-pcap_ct.txt +++ b/examples/benchmark/requirements-pcap_ct.txt @@ -5,9 +5,10 @@ -r requirements-common.txt # Both of these are pre-release-only projects -- every release ever published is a -# beta -- so `pip install pcap-ct` with no version resolves to nothing at all. An -# *exact* pre-release pin is honoured without `--pre`, which is why the versions -# are spelled out here rather than left to a range plus a flag. +# beta. pip falls back to pre-releases for a project with no final release, so +# `pip install pcap-ct` with no version does resolve, but to whichever beta is +# newest. An *exact* pre-release pin is honoured without `--pre`, which is why the +# versions are spelled out here rather than left to a range plus a flag. # # Both publish `py3-none-any` wheels, so neither needs a compiler on any # architecture. They do need a system libpcap at *run* time: `libpcap`'s published diff --git a/examples/generators/dispatch.py b/examples/generators/dispatch.py index 957e14380..e1eb4da76 100644 --- a/examples/generators/dispatch.py +++ b/examples/generators/dispatch.py @@ -772,10 +772,10 @@ class Outcome(NamedTuple): #: renamed-on-the-wire class (RARP reporting itself as ``'ARP'``) from a #: wrong one -- but wrong here, because :class:`~pcapkit.protocols.misc.raw.Raw` #: itself constructs with ``alias=`` set to the *dispatching* code's own - #: name (pcapkit/protocols/misc/raw.py:152), so a mis-pointed registry - #: entry still renders its original code's name and would satisfy the - #: string check while no instance of the pinned class exists anywhere in - #: the chain. + #: name (``Raw.__post_init__``, pcapkit/protocols/misc/raw.py), so a + #: mis-pointed registry entry still renders its original code's name and + #: would satisfy the string check while no instance of the pinned class + #: exists anywhere in the chain. reached: 'bool' #: Exception type and message, if building or extracting the envelope #: raised outright rather than merely dispatching somewhere else. Empty @@ -878,8 +878,8 @@ def probe(case: 'Case') -> 'Outcome': # on Type.id(). That is deliberate there (RARP reports itself as 'ARP' for # some ``oper`` values, and is still the RARP class), but it is exactly # the loophole that let a mis-pointed registry entry through here: several - # dissectors -- Raw chief among them, in - # pcapkit/protocols/misc/raw.py:152 -- construct with ``alias=`` set to + # dissectors -- Raw chief among them, in ``Raw.__post_init__`` + # (pcapkit/protocols/misc/raw.py) -- construct with ``alias=`` set to # the *dispatching* code's own name, so a code named ``AH`` retargeted at # Raw still renders as ``'...:AH'`` and still satisfies ``AH in # protochain`` on the string alone, even though no AH instance exists diff --git a/examples/generators/legacy.py b/examples/generators/legacy.py index dfc089c21..3a5b940f7 100644 --- a/examples/generators/legacy.py +++ b/examples/generators/legacy.py @@ -22,21 +22,14 @@ FIN-completed stream and an RST-aborted one. ``http6.cap`` - Read by ``examples/legacy_smoke/test_analyse.py``, which pretty-prints the - application layer :mod:`pcapkit` finds in each reassembled datagram. Two + Read by ``examples/legacy_smoke/test_analyse.py``, which extracts it with + ``tcp=True``, ``reasm_strict=True`` and ``reassembly=True`` and pretty-prints + the application layer :mod:`pcapkit` finds in each reassembled datagram. Two HTTP/1.1 connections over IPv6 to port 80: a page fetch whose response body spans three segments, and a conditional request for a stylesheet answered ``304 Not Modified`` with no body at all. Every datagram, requests and responses alike, analyses as HTTP. - That script does not run to completion even with this fixture in place, and - the reason is the script rather than the capture: it asks for ``tcp=True`` - and ``reasm_strict=True`` but never for ``reassembly=True``, so reading - ``extraction.reassembly`` raises ``UnsupportedCall: 'Extractor(reassembly= - False)' object has no attribute 'reassembly'``. It extracts every frame - first, and adding that one keyword makes the rest of it work; the fixture - is built for the script as it will read once repaired. - Every packet is built from protocol semantics: real handshakes, sequence and acknowledgement arithmetic that adds up, and checksums computed by :mod:`scapy` from the finished packet. Payload text is generated from a fixed @@ -50,26 +43,26 @@ look like for a datagram to come out of it at all: 1. A buffer is keyed by the connection's four-tuple *including direction*, so - each direction is a datagram of its own. + each direction is a buffer of its own. :meth:`~pcapkit.foundation.reassembly.tcp.TCP.submit` runs on that direction's RST; once its FIN has been seen and every octet below the FIN has arrived; when a SYN, or a segment farther than ``__window__`` from the buffer, opens a new one on the same four-tuple; on a timeout, if one is - set; and, for whatever is still buffered, when the capture ends. So a - half-closed connection still yields a datagram per direction. Both ends of - every connection here close anyway, so that each datagram is submitted by - its own FIN or RST rather than by that final flush. + set; and, for whatever is still buffered, when the datagrams are read -- + ``extraction.reassembly`` flushes every open buffer through + :meth:`~pcapkit.foundation.reassembly.reassembly.Reassembly.fetch`. So a + half-closed connection still yields the datagrams of both directions. Both + ends of every connection here close anyway, so that each datagram is + submitted by its own FIN or RST rather than by that final flush. 2. Inside a buffer, fragments are keyed by the segment's *acknowledgement* number, so consecutive data segments merge only while the peer sends no data of its own. Both fixtures therefore keep each direction of each connection to a single HTTP message, and let the peer answer only with pure acknowledgements until that message is complete. -3. A datagram counts as complete when no hole in the descriptor list falls - inside the octets that direction actually received. An ordered run of - segments leaves only the open-ended hole past the last octet, which is - outside the payload buffer and so does not count; an out-of-order segment - opens a hole in the middle, and the missing segment's arrival closes it - again. +3. A datagram counts as complete when its payload buffer has no gap left: + each payload buffer tracks the holes among the octets it holds. An ordered + run of segments opens none; an out-of-order segment opens one in the + middle, and the missing segment's arrival closes it again. 4. The payload of a complete datagram goes to :meth:`pcapkit.protocols.transport.transport.Transport.analyze`, which picks the application protocol from the two port numbers. Port 80 is what @@ -93,8 +86,8 @@ sequence number every slice started far beyond the end of the buffer, every fragment came out empty, and ``if data:`` discarded the datagram without a word. That is fixed -- ``submit`` converts each hole's absolute bounds into -offsets into the buffer it is reading, and decides completeness from whether any -hole survives that conversion -- and GitHub issue #349 records the whole of it. +offsets into the buffer it is reading, and decides completeness from that +payload buffer's own gap list -- and GitHub issue #349 records the whole of it. Regenerating these two fixtures to carry a permanently lost segment as well was considered and rejected: they are pinned byte-for-byte by the test suite, and a stream with a hole in it is cheaper to build segment by segment than to read out diff --git a/examples/legacy_smoke/Makefile b/examples/legacy_smoke/Makefile index 30b537432..a5cc27f51 100644 --- a/examples/legacy_smoke/Makefile +++ b/examples/legacy_smoke/Makefile @@ -4,11 +4,10 @@ #: `make fixtures PYTHON=../../.venv/bin/python`. PYTHON ?= python -#: Timezone the reference reports are generated in. Pinned because pcapkit renders -#: frame timestamps in the host's local zone, so an unpinned run rewrites every -#: timestamp line with wherever it happened to be run. UTC is also the only zone in -#: which PCAP-NG's `timestamp_epoch` comes out as the true UNIX epoch -- see the -#: README for the reason. +#: Timezone the reference reports are generated in. pcapkit renders timestamps in +#: UTC, or in a PCAP-NG interface's own `if_tzone`, never in the host's zone (#361, +#: #403), so the reports come out the same without it; the pin is belt-and-braces +#: -- see the README. FIXTURE_TZ ?= UTC ## Build the reference reports in ../captures/. Git tracks none of them -- they diff --git a/examples/legacy_smoke/README.md b/examples/legacy_smoke/README.md index a58cb8f58..044ef20c8 100644 --- a/examples/legacy_smoke/README.md +++ b/examples/legacy_smoke/README.md @@ -115,12 +115,8 @@ found by reading the file rather than by anything breaking. Generating them on d means the copy on your disk describes the tree you are standing in, which was the only thing that made them worth looking at. -`TZ=UTC` **is not decoration.** `pcapkit` renders frame timestamps in the host's -local zone, so an unpinned run rewrites every timestamp line in every report with -wherever it happened to be run -- which is how these reports came to disagree with -each other while they were still tracked, the PCAP reports having been generated at -UTC-05:00 and `pcapng.txt` at UTC+08:00. Pinning UTC makes the output reproducible on -any host, so two runs can be diffed against each other. It also happens to be the only -zone in which PCAP-NG's `timestamp_epoch` is the true UNIX epoch: -`PCAPNG._read_timestamp` adds the zone's UTC offset to the value it returns, so a -report generated anywhere else bakes in that offset. +`TZ=UTC` is a belt-and-braces pin, not a requirement. `pcapkit` renders a frame +timestamp in UTC, or in the zone a PCAP-NG interface declares in `if_tzone`, and +never in the host's; PCAP-NG's `timestamp_epoch` is the UNIX epoch with no zone +offset added (#361, #403). Measured by generating every report under `UTC`, +`Asia/Shanghai` and `America/New_York`: they are identical. diff --git a/examples/legacy_smoke/test_perf.py b/examples/legacy_smoke/test_perf.py index ffe6ab42e..06ff5d061 100644 --- a/examples/legacy_smoke/test_perf.py +++ b/examples/legacy_smoke/test_perf.py @@ -2,7 +2,7 @@ """Benchmark ``pcapkit.extract`` across engines with pyperf. ``pyperf`` is not a pcapkit dependency -- it is a benchmarking harness this one demo -uses -- so it will not be present in a plain ``pip install pcapkit`` environment. +uses -- so it will not be present in a plain ``pip install pypcapkit`` environment. When it is missing the demo says how to get it and stops there rather than dying on an import traceback. diff --git a/pyproject.toml b/pyproject.toml index 71834efd4..f78e7e44b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -143,9 +143,10 @@ crypto = [ "cryptography>=3.4" ] # ``crypto`` core addons ``all`` bundles it with. # # No version floor: NGAP has shipped precompiled in ``pycrate_asn1dir`` for many -# releases and the two entry points used here (``NGAP_PDU.from_aper`` and -# ``to_aper``) are long-standing. Verified against 0.8.1, which is pure Python -# with no compiled extension, so there is no toolchain prerequisite either. +# releases and the entry points used here (``NGAP_PDU.from_aper`` and +# ``to_aper`` among them) are long-standing. Verified against 0.8.1, which is +# pure Python with no compiled extension, so there is no toolchain prerequisite +# either. NGAP = [ "pycrate" ] # for normal users DPKT = [ "dpkt" ] diff --git a/util/conda_vendor.py b/util/conda_vendor.py index 4240d6891..66b840224 100644 --- a/util/conda_vendor.py +++ b/util/conda_vendor.py @@ -1,13 +1,13 @@ # -*- coding: utf-8 -*- """The third-party packages the conda recipe vendors into :file:`pcapkit/_extern` (#1814). -The conda package declares no run requirements. ``conda/pypcapkit/build.sh`` and -``bld.bat`` install every wheel that ``pip download -r conda/requirements.txt`` -fetched into :file:`pcapkit/_extern`, which :file:`pcapkit/__init__.py` appends -to :data:`sys.path`. So :file:`conda/requirements.txt` is the conda -distribution's whole dependency list, and this script derives it from -:file:`pyproject.toml` instead of leaving it to be edited by hand. Each package -falls in exactly one of these classes: +The conda package declares no run requirement but Python. +``conda/pypcapkit/build.sh`` and ``bld.bat`` install every wheel that +``pip download -r conda/requirements.txt`` fetched into :file:`pcapkit/_extern`, +which :file:`pcapkit/__init__.py` appends to :data:`sys.path`. So +:file:`conda/requirements.txt` is the conda distribution's whole dependency +list, and this script derives it from :file:`pyproject.toml` instead of leaving +it to be edited by hand. Each package falls in exactly one of these classes: ``needed`` ``[project] dependencies`` and the ``all`` extra. A conda user has no diff --git a/util/run_unittest_leg.py b/util/run_unittest_leg.py index 265b60595..8d1b23c1f 100644 --- a/util/run_unittest_leg.py +++ b/util/run_unittest_leg.py @@ -1,11 +1,11 @@ # -*- coding: utf-8 -*- """Run one unit-tier directory under plain :mod:`unittest`, in a single process. -GitHub issue #981: two test modules -- +GitHub issue #981: the test modules :mod:`tests.protocols.application.test_http_unit` and -:mod:`tests.test_base_class_contract` -- each pass alone and :program:`pytest` +:mod:`tests.test_base_class_contract` each pass alone and :program:`pytest` reports the whole run green when they are collected together, but plain -:mod:`unittest` over the same two modules in one process fails some of their +:mod:`unittest` over the same modules in one process fails some of their ``subTest`` cases. ``pytest-subtests`` is *not* why: that plugin is not installed in this project at all (absent from the ``test`` extra in ``pyproject.toml``), and plain :program:`pytest` (9.1.1 in this checkout) @@ -32,7 +32,7 @@ under it. A cheaper alternative exists, and is recorded here rather than left -undocumented: running ``pytest --noconftest`` over just those same two +undocumented: running ``pytest --noconftest`` over just those same modules, in that order, disables the very same autouse fixture directly and reproduces the identical ``subTest`` failures in about 70s on the pre-fix tree, under the exact :program:`pytest` version this CI already installs. @@ -54,7 +54,7 @@ after it was sharded), are tabulated in that workflow's own comment rather than restated here. At the other end sit :mod:`tests.dumpkit` and :mod:`tests.interface`, small legs within a few -seconds of each other in wall time (~39s and ~37s). Neither small leg is +seconds of each other in wall time (~39s and ~38s). Neither small leg is :mod:`tests.const` (~268s), which is larger by test count as well. Peak RSS was measured for the three cheapest legs only, via diff --git a/util/verdict_covers_checkout.py b/util/verdict_covers_checkout.py index 6011e8db0..65e3887fc 100644 --- a/util/verdict_covers_checkout.py +++ b/util/verdict_covers_checkout.py @@ -22,9 +22,9 @@ pushed with a deploy key, not with ``GITHUB_TOKEN``, whose events "will not create a new workflow run" (GitHub, "Triggering a workflow from a workflow"). Every bot commit on ``main`` since has its own ``push`` run of ``Unit Tests``, -f4f94bcad's being 38325349953; the ones before #401, 5d5d8af4d, 48bc11384 and -e4d0823db, have none. The second case only saves waiting for that run where it -cannot change the answer. +f4f94bcad's being 38325349953, while 5d5d8af4d, 48bc11384 and e4d0823db, +ancestors of bb900906b, have none. The second case only saves waiting for that +run where it cannot change the answer. Run with ``VERDICT_SHA`` set, from the checkout. Writes ``verified=true`` or ``verified=false`` to ``$GITHUB_OUTPUT``, and exits non-zero only when it cannot