Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .github/workflows/ci-timing.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
name: CI Timing

# The periodic CI-duration review #1538 asked for. util/ci_timing.py reads the
# last week of Unit Tests, Compat, Lint, CodeQL and Pages runs through the API,
# writes a report to the job summary and to an artifact, and fails the run when a
# last week of Unit Tests, Compat, Lint, CodeQL, Pages and Experimental Tests
# runs through the API, writes a report to the job summary and to an artifact,
# and fails the run when a
# HARD threshold fires (status 1; 2 and 3 are an unreadable API and a crash, each
# annotated as such). GitHub emails a scheduled workflow's failures to whoever
# last edited its cron line, so the owner hears about it with no `issues: write`.
Expand All @@ -29,7 +30,7 @@ on:

# Read-only: the runs, jobs and artifacts need `actions: read`, the workflow
# files at each run's sha need `contents: read`. GITHUB_TOKEN allows 1,000 API
# requests an hour per repository; a default run uses about 600.
# requests an hour per repository; a default run uses about 690.
permissions:
actions: read
contents: read
Expand Down
16 changes: 8 additions & 8 deletions .github/workflows/create-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ concurrency:
# an unintended publish -- a version number it has accepted cannot be reused, and
# ``skip-existing: true`` below means a re-upload then *succeeds* having published
# nothing, so the number the intended release wanted is gone regardless. ``github``
# and ``tag`` were split on the same principle: since #1634 ``github`` spends a
# and ``tag`` were split on the same principle (#1634): ``github`` spends a
# token of the release App, minted in the job from ``github-release``'s secret
# and revoked when it ends (the run's ``GITHUB_TOKEN`` while no App is set up),
# whereas ``tag`` spends a long-lived PAT and a deploy key to write to the
Expand Down Expand Up @@ -266,7 +266,7 @@ jobs:
version_check:
name: Check Version
runs-on: ubuntu-latest
# The outermost gate since #1052, moved here from ``unit-tests``: a
# The outermost gate, moved here from ``unit-tests`` by #1052: a
# ``workflow_run`` trigger proceeds only when the Vendor Update run that
# triggered it concluded ``success``, and a tag push always proceeds. This
# job is read-only -- it installs the package and reads three public
Expand Down Expand Up @@ -598,13 +598,13 @@ jobs:
name: GitHub Release
runs-on: ubuntu-latest
# Gated: publishes the Release, and creates the ``v*`` tag -- at
# ``github.sha`` since #1634 -- that every job below then verifies before
# publishing under it. Since #887 this is the release's only approval --
# ``github.sha`` (#1634) -- that every job below then verifies before
# publishing under it. This is the release's only approval (#887) --
# see the note above ``jobs:``. Still gated on ``PCAPKIT_TAG_EXISTS``,
# unlike the three jobs below it: the tag that output answers for is this
# job's own artefact, not a proxy for someone else's -- see the #888 note
# above ``jobs:``. ``unit-tests`` is listed since #1052, when the gate stopped
# preceding ``version_check``; this ``if:`` calls no status-check function,
# above ``jobs:``. ``unit-tests`` is listed because the gate follows
# ``version_check`` (#1052); this ``if:`` calls no status-check function,
# so the implicit ``success()`` requires the gate to have passed. The
# release App's private key is a ``github-release`` environment secret, so
# no job reads it before this approval, and no other job reads it at all.
Expand Down Expand Up @@ -1495,8 +1495,8 @@ jobs:
# why, in one place, regardless of which upstream job or gate is what made it
# release nothing. ``version_check`` can skip (a ``workflow_run`` whose Vendor
# Update did not conclude ``success``) before any evidence output the jobs
# above rely on even exists, and ``unit-tests`` can fail or -- since #1052 --
# skip because that evidence says nothing is left to publish, so this job
# above rely on even exists, and ``unit-tests`` can fail, or skip (#1052)
# because that evidence says nothing is left to publish, so this job
# cannot simply read those outputs the way
# ``tag``/``pypi``/``conda`` do -- it has to run unconditionally and
# reconstruct what happened from ``needs.*.result`` and the trigger context
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/cron-vendor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -163,12 +163,12 @@ jobs:
python -m pip install -U pip setuptools wheel packaging
python -m pip install -U isort

# `all` carries `pycrate` since #910 (a core addon there now, see its
# own comment in pyproject.toml), but not `requests`/`beautifulsoup4`
# any more -- #910 moved those to the `dev` extra instead, which is
# `all` carries `pycrate`, a core addon under #910 (see its
# own comment in pyproject.toml), but not `requests`/`beautifulsoup4`:
# #910 moved those to the `dev` extra instead, which is
# for the toolchain rather than for a vendor crawl. `vendor` is
# installed alongside `all` for those two: `pcapkit-vendor` below
# regenerates every crawler with no target given, and seven of them
# regenerates every crawler with no target given, and six of them
# import `bs4` at module scope while `pcapkit.vendor.default` imports
# `requests` the same way (#507). See the `vendor` extra's own
# comment in pyproject.toml for the full reasoning.
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/python-compatibility.yml
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ jobs:

# The install needs this leg's interpreter, and the check below needs the
# install, as in unit-tests.yml's `test` job: on a KTLO leg a failed step
# no longer stops the job, so each one waits on the `outcome` it needs.
# does not stop the job, so each one waits on the `outcome` it needs.
- name: Install package
id: install
if: ${{ steps.python.outcome == 'success' }}
Expand Down
14 changes: 7 additions & 7 deletions .github/workflows/unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ jobs:
# 7 of the 10 HAS_PYPCAPFILE-gated methods this comment used to count
# into failures, against a real bug in pcapkit/toolkit/pypcapfile.py's
# handling of pypcapfile 0.12.0's `IP.src`/`IP.dst`. #743 and #746 fixed
# that bug, and #751 gave the now-safe extra a home: the dedicated
# that bug, and #751 gave the extra a home: the dedicated
# `engine-tests` and `pypcap-parity` jobs below install it (15
# HAS_PYPCAPFILE-gated methods between them), deliberately apart from
# this job rather than added to it -- see either job's own comment for
Expand Down Expand Up @@ -489,7 +489,7 @@ jobs:
# PyPCAPFile is NOT added here either, even though this job's selection
# also reaches test_new_engine_parity_runtime.py's 6 HAS_PYPCAPFILE
# methods -- see the `test` job's comment above for the bug that used to
# make installing it here a regression, now fixed by #743/#746. #751's
# make installing it here a regression, fixed by #743/#746. #751's
# `pypcap-parity` job below covers those 6 methods instead, on the same
# fixture-tier selection as this job but in a venv of its own.
#
Expand Down Expand Up @@ -1111,7 +1111,7 @@ jobs:
# instruction was "try to build and if the CI is not a good suit, then we
# ripe it". This job is that attempt: a C toolchain plus libpcap headers, on
# the two Python versions its own marker allows
# ("pypcap; python_version < '3.12'", pyproject.toml:157). If a clean run of
# ("pypcap; python_version < '3.12'", pyproject.toml:168). If a clean run of
# this job's install step goes red -- not a flake -- the fix is deleting this
# job and returning HAS_PYPCAP to
# tests/_dependency_gates.DEPENDENCY_GATE_EXCLUSIONS with that run linked as
Expand Down Expand Up @@ -1428,8 +1428,8 @@ jobs:
# #981's own reproduction needs, since the defect is an earlier module
# polluting a later one. Running the whole suite in one process is not an
# option (it OOMs at 29 GB on the machine this was diagnosed on); a matrix
# of one leg per top-level directory (protocols, since #1029, being two
# cells) is the affordable approximation, and it
# of one leg per top-level directory (protocols being two cells, #1029)
# is the affordable approximation, and it
# is sized for wall time rather than for memory. Peak RSS was measured for
# the three cheapest legs only -- cli 152 MiB, dumpkit 237 MiB, interface
# 305 MiB (resource.getrusage on RUSAGE_CHILDREN, this venv) -- leaving the
Expand All @@ -1439,7 +1439,7 @@ jobs:
# 70s, protocols 847s (before the #1029 split), toolkit 97s, utilities 42s,
# vendor 104s, one leg per
# matrix cell -- see `timeout-minutes` below for why the job-level budget is
# no longer 20.
# 45 rather than 20.
#
# GitHub issue #1029: the protocols cell measured 30.4 minutes on a CI
# runner against this job's 45-minute cap (#1029 measured the next slowest
Expand Down Expand Up @@ -1798,7 +1798,7 @@ jobs:
# above, which each cover only the subset their own selection reaches.
# PyPCAPFile is deliberately NOT added here -- see the `test` job's
# comment above for the bug that used to make installing it a
# regression, now fixed by #743/#746. This job runs on `latest` only, where
# regression, fixed by #743/#746. This job runs on `latest` only, where
# PyPCAPFile's marker resolves to nothing regardless, so adding it here
# would be a no-op that misleadingly suggests the extra is exercised by
# `gate`; #751's `engine-tests` and `pypcap-parity` jobs cover it for
Expand Down
6 changes: 3 additions & 3 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ prune temp
# file mentions them, so neither line is load-bearing. An earlier version of this
# comment claimed the opposite -- that ``include README.md`` was "the only thing
# that puts it in an sdist" and that an sdist without it "cannot be installed at
# all" -- and both halves are false. Measured by deleting these four lines and the
# two below them and rebuilding from a clean tree: the ``tar tzf`` listing is
# identical, 861 entries either way with an empty ``diff``, and the resulting sdist
# all" -- and both halves are false. Measured by deleting the two ``include``
# lines below and rebuilding from a clean tree: the ``tar tzf`` listing is
# identical either way, with an empty ``diff``, and the resulting sdist
# installs with exit 0. ``README.md``, ``LICENSE``, ``CHANGELOG.md`` and
# ``CITATION.cff`` are all still in it.
#
Expand Down
6 changes: 3 additions & 3 deletions examples/benchmark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,8 +126,8 @@ never as a zero.
## One flaky engine must not cost the others their rows

This is not hypothetical; it is what the first full run of this harness did.
`pyshark` spawns a `tshark` process per extraction, so a default run spawns **six
thousand** of them, and one of them died -- `TSharkCrashException ... retcode: 255`,
`pyshark` spawns a `tshark` process per extraction, so a default run spawns **three
thousand** of them per environment, and one of them died -- `TSharkCrashException ... retcode: 255`,
on the third pass. The exception propagated, `benchmark.py` exited, no JSON was
written, and the run produced **no table at all**. Six working engines went unreported
because the seventh hiccupped once.
Expand Down Expand Up @@ -397,7 +397,7 @@ self-tests below.
## Python 3.15

Opt-in, and allowed to fail -- the benchmark's own choice, not CI's: the repository's CI
has run 3.15 as a blocking, supported leg since #1633. There is no released
runs 3.15 as a blocking, supported leg (#1633). There is no released
`python:3.15-slim-bookworm`; the
pins file names a `3.15-rc` digest, which freezes one particular release candidate
and will name an older one as time passes. Measure it with:
Expand Down
2 changes: 1 addition & 1 deletion examples/benchmark/benchmark.py
Original file line number Diff line number Diff line change
Expand Up @@ -481,7 +481,7 @@ def measure(engine: 'str', capture: 'str', rounds: 'int',

A small number of failed extractions is tolerated, and this is not
fastidiousness. ``pyshark`` spawns a :program:`tshark` process per extraction,
so a default run spawns six thousand of them, and measured here: one of them
so a default run spawns three thousand of them, and measured here: one of them
crashed with ``TSharkCrashException ... retcode: 255`` partway through the
third pass and took the entire run with it -- six working engines went
unreported because the seventh hiccupped once. A crashed extraction is a
Expand Down
4 changes: 2 additions & 2 deletions examples/benchmark/python-images.txt
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@
# `3.15-rc-slim-bookworm` is a release candidate and the tag moves with every rc, so
# the digest below freezes one particular candidate and will name an older rc as
# time passes -- bump it deliberately, or drop the row when 3.15 ships. Opt-in is
# this file's own tier, not CI's: the repository's CI has run 3.15 as a blocking,
# supported leg since #1633. `run.sh --pythons ...,3.15` measures it, a plain
# this file's own tier, not CI's: the repository's CI runs 3.15 as a blocking,
# supported leg (#1633). `run.sh --pythons ...,3.15` measures it, a plain
# `run.sh` does not, and a
# version that cannot be built is reported as a missing column with the reason
# rather than dropped.
Expand Down
6 changes: 3 additions & 3 deletions examples/generators/dispatch.py
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,7 @@ def _link_payload(code: 'int') -> 'bytes':
from pcapkit.protocols.application.rarp import RARP

# ARP/RARP/InARP/DRARP all report their alias from the wire ``oper``
# field rather than from the dispatching EtherType (link/arp.py:176-190
# field rather than from the dispatching EtherType (link/arp.py:191-202
# per #496), so ``oper`` has to land in the RARP range (3, 4) or this
# would silently probe ARP instead while still "passing".
return bytes(RARP(oper=3))
Expand Down Expand Up @@ -772,7 +772,7 @@ class Outcome(NamedTuple):
#: renamed-on-the-wire class (RARP reporting itself as ``'ARP'``) from a
#: wrong one -- but wrong here, because :class:`~pcapkit.protocols.misc.raw.Raw`
#: itself constructs with ``alias=`` set to the *dispatching* code's own
#: name (pcapkit/protocols/misc/raw.py:144), so a mis-pointed registry
#: name (pcapkit/protocols/misc/raw.py:152), so a mis-pointed registry
#: entry still renders its original code's name and would satisfy the
#: string check while no instance of the pinned class exists anywhere in
#: the chain.
Expand Down Expand Up @@ -879,7 +879,7 @@ def probe(case: 'Case') -> 'Outcome':
# some ``oper`` values, and is still the RARP class), but it is exactly
# the loophole that let a mis-pointed registry entry through here: several
# dissectors -- Raw chief among them, in
# pcapkit/protocols/misc/raw.py:144 -- construct with ``alias=`` set to
# pcapkit/protocols/misc/raw.py:152 -- construct with ``alias=`` set to
# the *dispatching* code's own name, so a code named ``AH`` retargeted at
# Raw still renders as ``'...:AH'`` and still satisfies ``AH in
# protochain`` on the string alone, even though no AH instance exists
Expand Down
10 changes: 5 additions & 5 deletions examples/generators/make_samples.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# -*- coding: utf-8 -*-
"""Regenerate the sample captures used by the test suite and the examples.

The captures under ``examples/captures/`` are not tracked in git (see
``.gitignore``), yet the runtime, regression and integration tests read them and
pin their contents, as do the demonstration scripts in
``examples/legacy_smoke/``. This script rebuilds the whole set, so a fresh clone
can run ``pytest`` without any ignore flags:
The captures under ``examples/captures/`` are not tracked in git, apart from the
two inputs ``in.pcap`` and ``dhcp.pcapng`` (see ``.gitignore``), yet the runtime,
regression and integration tests read them and pin their contents, as do the
demonstration scripts in ``examples/legacy_smoke/``. This script rebuilds the
generated captures, so a fresh clone can run ``pytest`` without any ignore flags:

.. code-block:: shell

Expand Down
14 changes: 7 additions & 7 deletions examples/generators/options.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,10 @@
:meth:`ProtocolBase._lookup_registry
<pcapkit.protocols.protocol.ProtocolBase._lookup_registry>` instead.

IPv4 and HIP have no such registry yet -- they dispatch on the attribute name
``_make_opt_${name}`` / ``_make_param_${name}`` -- so for those two the source
of truth is the enumeration crossed with the presence of the handler, which is
what :func:`_named_registry` builds.
IPv4 and HIP are read through :func:`_named_registry`, which returns the
protocol's own registry and, for a protocol without one, falls back to the
enumeration crossed with the presence of the ``_make_opt_${name}`` /
``_make_param_${name}`` handler.

Determinism
-----------
Expand Down Expand Up @@ -372,8 +372,8 @@ def _named_registry(owner: 'type', enum: 'Any', prefix: 'str',
enumeration crossed with the presence of the handler, which is what this
builds when the registry is absent.

GitHub pull request #434 gave both of them a real registry, so the fallback
is no longer taken for either -- it is kept because it costs nothing and is
Both of them have a real registry (#429), so the fallback is not taken
for either -- it is kept because it costs nothing and is
the only thing that would notice a protocol added tomorrow without one. Both
shapes are a :class:`collections.defaultdict`, so :func:`cases` and
:func:`handler` cannot tell which one they were given -- and in particular
Expand Down Expand Up @@ -1284,7 +1284,7 @@ def _pcapng_context() -> 'tuple[Any, Any]':
# ``Protocol.info`` is annotated as the generic data model for the protocol,
# so a checker cannot see that a Section Header Block's is the one ``Context``
# wants. The engine does exactly this, at
# pcapkit/foundation/engines/pcapng.py:148.
# pcapkit/foundation/engines/pcapng.py:181.
section_info = section.info # type: Any
context = Context(section_info)
interface = PCAPNG(num=1, sct=1, ctx=context,
Expand Down
11 changes: 5 additions & 6 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ PyPCAPFile = [ "pypcapfile; python_version < '3.12'" ]
# for developers -- installs pcapkit.vendor's crawlers, run via
# `pcapkit-vendor` to regenerate pcapkit.const. Includes ``pycrate`` so
# pcapkit.vendor.ngap's two crawlers (#880) can run, and ``requests``/
# ``beautifulsoup4`` for the seven crawlers that import them at module scope
# ``beautifulsoup4`` for the six crawlers that import them at module scope
# (c.f. the ``test`` extra's own comment below for which, and #507). #910
# narrowed ``all`` to core addons only, so this extra names all three
# itself rather than leaning on ``all`` to also carry them -- ``pycrate``
Expand Down Expand Up @@ -302,11 +302,11 @@ test = [
# rather than failing, so the byte-for-byte regeneration check that is the
# only guard on the hand-maintained IPX socket registry never runs on any CI
# path: every ``unit-tests.yml`` job's install line builds on ``.[test]``,
# and ``cron-vendor.yml`` installs ``.[all]`` but never invokes pytest.
# and ``cron-vendor.yml`` installs ``.[all,vendor]`` but never invokes pytest.
#
# Both are needed, not just ``requests``. ``pcapkit.vendor.default`` imports
# ``requests`` at module scope, and ``pcapkit.vendor.__init__`` then imports
# every crawler unconditionally -- seven of which import ``bs4`` at module
# every crawler unconditionally -- six of which import ``bs4`` at module
# scope. Its ``try: import bs4`` guard suppresses only the warning, not the
# ``ModuleNotFoundError`` that the crawler imports below it raise anyway.
#
Expand Down Expand Up @@ -346,9 +346,8 @@ test = [
# said nothing at all.
"isort",
# The second opinion on the hand-rolled YAML scanners: the ``yaml.safe_load``
# cross-checks in ``tests/project/test_project_status.py``,
# ``test_release_gates.py``, ``test_bump_version.py`` and
# ``test_workflow_apt_timeouts.py`` skip without it.
# cross-checks under ``tests/project/``, and the tests there that parse a
# workflow with it outright, skip without it.
# Test-only, as ``pcapkit`` never imports ``yaml``. Unpinned, since
# ``safe_load`` is all they use.
"PyYAML",
Expand Down
6 changes: 3 additions & 3 deletions util/bump_version.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,9 @@
actually going out. Until then this is the better of the two available answers, and
the ``version`` half is guarded by a test that fails when the file drifts from
``pcapkit.__version__`` -- which also covers the version changes made by hand,
which never run this script at all. Those are 40 of the 159 commits that have
moved ``__version__`` on ``main``, a quarter over the project's life and a rising
share lately: 11 of the most recent 25.
which never run this script at all. Those are 40 of the 167 commits that have
moved ``__version__`` on ``main``: a quarter over the project's life, and 11 of
the most recent 25.

The rewrite is line-oriented rather than a :mod:`yaml` load-and-dump. A round
trip through a YAML library would reorder keys, normalise quoting, and drop the
Expand Down
Loading
Loading