diff --git a/CHANGELOG.md b/CHANGELOG.md index fb75be38d8..acd2863c5a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,8 +1,8 @@ ## 1.5.0 -- unreleased -The largest release since 1.0, and the first recorded here as it happened rather than reconstructed. Three more extraction engines, ESP with payload decryption, SCTP and NGAP over SCTP, a library logger that no longer hijacks the consumer's, and a defect programme run through the issue tracker, citing 319 issues and pull requests from [#121](https://github.com/JarryShaw/PyPCAPKit/pull/121) to [#1195](https://github.com/JarryShaw/PyPCAPKit/issues/1195) as of this revision. The programme is still running, so those figures are a snapshot rather than a total. +The largest release since 1.0, and the first recorded here as it happened rather than reconstructed. Three more extraction engines, ESP with payload decryption, SCTP and NGAP over SCTP, a library logger that no longer hijacks the consumer's, and a defect programme run through the issue tracker, citing 611 issues and pull requests from [#340](https://github.com/JarryShaw/PyPCAPKit/pull/340) to [#1754](https://github.com/JarryShaw/PyPCAPKit/pull/1754). -Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) and `1.5.0b3` (2026-09-19), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` reshipped it -- an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels ([#497](https://github.com/JarryShaw/PyPCAPKit/pull/497), [#498](https://github.com/JarryShaw/PyPCAPKit/pull/498)). +Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18), `1.5.0b3` (2026-09-19) and `1.5.0b4` to `1.5.0b12` (2026-09-20 to 2026-10-13), all published as prereleases and so resolved only by `pip install --pre`. `1.5.0b1` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because `twine check` found a Sphinx-only `:mod:` role in `README.rst`, which `pyproject.toml` declares as the dynamic long description. `1.5.0b2` reshipped it -- an existing version cannot republish -- and `1.5.0b3` followed the CI change that stops a TestPyPI outage from costing a release its wheels ([#497](https://github.com/JarryShaw/PyPCAPKit/pull/497), [#498](https://github.com/JarryShaw/PyPCAPKit/pull/498)). ### pcapkit.const @@ -12,24 +12,34 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) an - the six `%`-formatted `raise ValueError(...)` calls in the generated `pcapkit.const.reg.apptype.apptype`, left by [#759](https://github.com/JarryShaw/PyPCAPKit/issues/759), are f-strings ([#792](https://github.com/JarryShaw/PyPCAPKit/issues/792)). The remainder followed: three `_missing_` guards (`pcapkit.vendor.tcp.flags`, `.ftp.command`, `.http.method`), `AppType`'s `__new__`/`__repr__`/`__str__`, and the `extend_enum` fallbacks in `get()` and in the span-handling tail. The `consider-using-f-string` disable drops from `tcp.flags` and `reg.apptype.apptype` and stays on `ftp.command` and `http.method`, whose `__repr__` still used `%`. Edited in the vendor templates and applied to the generated const files by hand, since a crawl would fetch IANA's live registry and rewrite unrelated rows ([#798](https://github.com/JarryShaw/PyPCAPKit/issues/798)). - the three `__repr__` methods [#798](https://github.com/JarryShaw/PyPCAPKit/issues/798) left on `%` (`pcapkit/const/ftp/command.py:40,119`, `pcapkit/const/http/method.py:42`) are f-strings, so the disable drops from both files and both `pcapkit/vendor/` templates. The other bespoke templates (`{const,vendor}/ftp/return_code.py`, `{const,vendor}/http/status_code.py`, `vendor/http/frame.py`) and `vendor/default.py` still carry it, so [#804](https://github.com/JarryShaw/PyPCAPKit/issues/804)'s claim that nothing in `pcapkit/{const,vendor}/{ftp,http}/` needs it holds for this pair only; `repr()` output is unchanged ([#804](https://github.com/JarryShaw/PyPCAPKit/issues/804)). - `test_const_enum_no_mint.py`'s `is_manufactured()` exemption for [#878](https://github.com/JarryShaw/PyPCAPKit/pull/878)'s (above) 53 preserved hex-suffixed names moves from a per-*file* to a per-*name-shape* one. The per-file `HEX_SUFFIXED_NAME_EXEMPT_PATHS` was shape-blind: rewriting a branch to `'Xyplex_%d' % value` -- exactly the value-derived shape the predicate exists to flag -- left the sweep green. The check now exempts only a `'..._0x%s' % hex(value)[2:].upper().zfill(4)` expression, and the exact `exempt_hits == 53` bound stays. Test-only. Closes [#879](https://github.com/JarryShaw/PyPCAPKit/issues/879) ([#881](https://github.com/JarryShaw/PyPCAPKit/pull/881)). +- **a breaking change to** `AppType.proto`: it is the single transport of the registry the member lives in, not every transport IANA assigned the service, and `register_apptype` takes its transports as varargs ([#806](https://github.com/JarryShaw/PyPCAPKit/issues/806), [#815](https://github.com/JarryShaw/PyPCAPKit/pull/815)). Aliases show in `repr()` and `str()`, and can be registered per transport ([#807](https://github.com/JarryShaw/PyPCAPKit/issues/807), [#837](https://github.com/JarryShaw/PyPCAPKit/pull/837)). +- **a breaking change to** `TransportProtocol`: its base moves from `aenum.IntFlag` to `aenum.IntEnum`, so members no longer compose with `|` ([#808](https://github.com/JarryShaw/PyPCAPKit/issues/808), [#836](https://github.com/JarryShaw/PyPCAPKit/pull/836)). +- **a breaking change to** `TLSKeyLabel`: the hand-written class in `pcapkit.protocols.misc.pcapng` becomes the generated `pcapkit.const.pcapng.TLSKeyLabel`, crawled from the IANA registry [RFC 9850](https://datatracker.ietf.org/doc/html/rfc9850) creates ([#886](https://github.com/JarryShaw/PyPCAPKit/issues/886), [#890](https://github.com/JarryShaw/PyPCAPKit/pull/890)), and the old path is removed ([#1744](https://github.com/JarryShaw/PyPCAPKit/issues/1744), [#1754](https://github.com/JarryShaw/PyPCAPKit/pull/1754)). +- **a breaking change to** `Method.get`: the method token is case-sensitive, per [RFC 9110 Section 9.1](https://datatracker.ietf.org/doc/html/rfc9110#section-9.1), so `Method.get('get')` no longer resolves to `GET` ([#896](https://github.com/JarryShaw/PyPCAPKit/issues/896), [#907](https://github.com/JarryShaw/PyPCAPKit/pull/907)). It also looks a key up by value, so `BASELINE-CONTROL` and `VERSION-CONTROL` resolve ([#908](https://github.com/JarryShaw/PyPCAPKit/issues/908), [#915](https://github.com/JarryShaw/PyPCAPKit/pull/915)). +- **a breaking change to** `ExtensionHeader`: it is generated from IANA's IPv6 Extension Header Types registry rather than the Protocol Numbers registry's flag, and the two disagree on 147 (`BIT-EMU`) ([#925](https://github.com/JarryShaw/PyPCAPKit/issues/925), [#926](https://github.com/JarryShaw/PyPCAPKit/pull/926)). +- the TCP and UDP `AppType` registries load on first use rather than at import ([#1538](https://github.com/JarryShaw/PyPCAPKit/issues/1538), [#1560](https://github.com/JarryShaw/PyPCAPKit/pull/1560)). #### Fixed - constant lookups that rejected a value the registry defines. `RouterAlert(0)` is the only value [RFC 2113](https://datatracker.ietf.org/doc/html/rfc2113) defines and the one IGMP, RSVP and MLD actually send, and it was discarded because the vendor crawler skipped a header row IANA's CSV does not have; IPX `Socket(0)` is that protocol's own default, so `bytes(IPX(...))` crashed on its own defaults; and two FTP `_missing_` overrides were plain methods rather than classmethods, so every unregistered value raised `TypeError` instead of extending the enumeration ([#492](https://github.com/JarryShaw/PyPCAPKit/issues/492), [#503](https://github.com/JarryShaw/PyPCAPKit/pull/503)). -- the string-keyed `get()` in `pcapkit.const.ftp.command` and `pcapkit.const.http.method` tested membership with the raw key but registered `key.upper()`, so the *first* lowercase or mixed-case token raised `TypeError: 'RETR' already in use`. Reachable from wire data: `pcapkit.protocols.application.ftp` matches with `re.I` and passes the match verbatim, and [RFC 959 Section 5](https://datatracker.ietf.org/doc/html/rfc959#section-5) makes FTP commands case-insensitive, so `retr file.txt` was a valid request the library could not parse. `get()` and `_missing_` now look the key up under the canonical upper-case name, so every casing resolves to the existing member; a duplicate `GET` would have lacked the real one's `safe` and `idempotent` attributes ([#582](https://github.com/JarryShaw/PyPCAPKit/issues/582), [#583](https://github.com/JarryShaw/PyPCAPKit/issues/583)). +- the string-keyed `get()` in `pcapkit.const.ftp.command` and `pcapkit.const.http.method` tested membership with the raw key but registered `key.upper()`, so the *first* lowercase or mixed-case token raised `TypeError: 'RETR' already in use`. Reachable from wire data: `pcapkit.protocols.application.ftp` matches with `re.I` and passes the match verbatim, and [RFC 959 Section 5](https://datatracker.ietf.org/doc/html/rfc959#section-5) makes FTP commands case-insensitive, so `retr file.txt` was a valid request the library could not parse. `get()` and `_missing_` now look the key up under the canonical upper-case name, so every casing resolves to the existing member; a duplicate `GET` would have lacked the real one's `safe` and `idempotent` attributes ([#582](https://github.com/JarryShaw/PyPCAPKit/issues/582), [#583](https://github.com/JarryShaw/PyPCAPKit/issues/583)). `http.method` no longer folds case: [RFC 9110 Section 9.1](https://datatracker.ietf.org/doc/html/rfc9110#section-9.1) makes the method token case-sensitive, so `Method.get` matches it exactly ([#896](https://github.com/JarryShaw/PyPCAPKit/issues/896), below). - `get()`'s documented `default` was ignored on the integer path across the generated `pcapkit.const` tree, because `_missing_` cannot see the caller's `default` -- `Hardware.get(99999, 0)` raised `ValueError` instead of returning `0`. The integer path now consults `default` before letting the error escape. `-1`, the placeholder the generated signature already carried, still separates "no default" from "a default"; [#859](https://github.com/JarryShaw/PyPCAPKit/pull/859) replaced it with an identity sentinel once the shared base [#858](https://github.com/JarryShaw/PyPCAPKit/pull/858) (below) put `get()` behind one call site. The sweep covers 110 of the 118 integer registries, not the three [#584](https://github.com/JarryShaw/PyPCAPKit/issues/584) named; `pcapng` `OptionType` and `reg` `AppType` carry a bespoke integer fallback and are deliberately left alone, since neither drops a default by raising. Not reachable from wire data, so a contract fix rather than a parse fix ([#584](https://github.com/JarryShaw/PyPCAPKit/issues/584)). - `_missing_` in the four Mobility Header flag enumerations (`BindingACKFlag`, `BindingUpdateFlag`, `HandoverACKFlag`, `HandoverInitiateFlag`) ended in `return cls(value)`, the constructor that had just failed, so every in-range value that is not already a member re-entered `_missing_` unbounded and raised `RecursionError` -- `F(0)`, no flags set, and every composite such as `BindingACKFlag(0x06)`. Defining `_missing_` at all caused it, by shadowing the `aenum` `Flag` machinery that resolves exactly those values; `pcapkit/const/tcp/flags.py` had none and so never had the defect (it has one now, from [#647](https://github.com/JarryShaw/PyPCAPKit/issues/647) below, with the same tail). All four now end in `return super()._missing_(value)`, as `pcapkit/vendor/default.py` emits and most const modules ([#647](https://github.com/JarryShaw/PyPCAPKit/issues/647)'s two new ones included) already did, so `F(0)` is an empty flag and `BindingACKFlag(0x06)` is `S|D`. Minting through `extend_enum` was rejected for a flag type: naming `0x06` `Unassigned_0x06` would hide the composite and add a member per bit pattern. The range guard is untouched ([#623](https://github.com/JarryShaw/PyPCAPKit/issues/623)). - six of the 123 constant registries rejected an invalid value differently from the built-in `enum`, and three did not reject it at all. `Flags`, `ftp.command.CommandType` and `reg.apptype.TransportProtocol` are `IntFlag` types with no `_missing_`, so `aenum` composed a pseudo-member for any integer: `Flags(-1)` returned `65520` (every TCP flag set, from a value no 16-bit field can hold) and `Flags(-65536)` none. `ftp.command.Command`, `FEATCode` and `http.method.Method` are `StrEnum` types whose `_missing_` called `value.upper()` before checking the type, so an integer raised `AttributeError`. All six now raise a bare `ValueError`, as `enum.IntEnum` does and 113 of the 117 modules already did. [#647](https://github.com/JarryShaw/PyPCAPKit/issues/647)'s proposal to replace those 113 with `pcapkit.utilities.exceptions.EnumError` was deliberately rejected: `EnumError` is `(BaseError, TypeError)`, not a `ValueError`, so it would diverge from the built-in, walk past the `except ValueError` in all 113 generated `get()` bodies (silently undoing [#584](https://github.com/JarryShaw/PyPCAPKit/issues/584)), and log CRITICAL from `BaseError.__init__` once per discarded default. The one deliberate divergence is kept and pinned: mutable registries look up, miss, and `extend_enum` rather than raising, so `Method('FROBNICATE')`, `ProtectionAuthority(1 << 70)` and `AppType.get(65000, proto=tcp)` still register. The three flag guards end in `return super()._missing_(value)`, so in-range composites still decompose and [#623](https://github.com/JarryShaw/PyPCAPKit/issues/623) stays fixed; `Flags` accepts `0 <= value <= 0xFFFF`. A new `tests/const/test_const_enum_builtin_parity.py` asserts that every one of the 123 registries raises the same exception type as a stdlib `enum.IntEnum`; it keys on `aenum.Enum` because an `IntFlag` is not an `IntEnum` subclass, which is how three of the six escaped the existing sweeps ([#647](https://github.com/JarryShaw/PyPCAPKit/issues/647)). -- **a breaking change to** `AppType.get`: an out-of-range port is refused rather than minted. Its `except ValueError` caught the range rejection from `_missing_` and minted regardless, so `AppType.get(-1, proto='tcp')` returned an unreachable `PORT_-1_tcp` while `TCP(-1)` raised. It now tests `_missing_` for `None` ("no row for this port") and lets the rejection through; a valid but unassigned port still mints. **`Transport._make_port` passes an unvalidated `int`, so `TCP.make(srcport=99999)` now raises where it minted junk.** Separately, the crawler rendered one `_missing_` branch per registry row, so the two spans IANA registers on more than one transport emitted the same condition twice and every registry answered with the first: `AppType.get(6010, proto='udp')` gave a member whose `proto` read `tcp`, `6666/udp` answered `ircu` rather than `reserved`, and `proto='sctp'` minted `x11` where IANA assigns nothing. A branch naming a transport now tests `cls.__transport__`; the 762 naming none are unchanged ([#764](https://github.com/JarryShaw/PyPCAPKit/pull/764)). +- **a breaking change to** `AppType.get`: an out-of-range port is refused rather than minted. Its `except ValueError` caught the range rejection from `_missing_` and minted regardless, so `AppType.get(-1, proto='tcp')` returned an unreachable `PORT_-1_tcp` while `TCP(-1)` raised. It now tests `_missing_` for `None` ("no row for this port") and lets the rejection through; a valid but unassigned port still mints. `Transport._make_port` **passes an unvalidated** `int`, **so** `TCP.make(srcport=99999)` **now raises where it minted junk.** Separately, the crawler rendered one `_missing_` branch per registry row, so the two spans IANA registers on more than one transport emitted the same condition twice and every registry answered with the first: `AppType.get(6010, proto='udp')` gave a member whose `proto` read `tcp`, `6666/udp` answered `ircu` rather than `reserved`, and `proto='sctp'` minted `x11` where IANA assigns nothing. A branch naming a transport now tests `cls.__transport__`; the 762 naming none are unchanged ([#764](https://github.com/JarryShaw/PyPCAPKit/pull/764)). - **a breaking change to** `AppType._dispatch`: it resolved a `proto` naming several transports by its lowest set bit, and `tcp` is lowest, so every composite containing it dispatched into the TCP registry -- silently, and undetectably to a caller checking equality, since `AppType.__eq__` compares on `port` alone. `TransportProtocol` is an `aenum.IntFlag` and a member carries the whole set IANA assigned the service, so `tcp | udp` is an ordinary value to pass back in. Over every multi-transport member this misrouted two: port 888 (`accessbuilder` resolved to TCP's `cddbp`) and port 999 (`puprouter` resolved to TCP's `garcon`). It now raises `ProtocolError`, a `ValueError` subclass, so the documented contract holds. No library caller passes a composite ([#759](https://github.com/JarryShaw/PyPCAPKit/issues/759)). - `get()`'s string-key miss and every generated registry's `_missing_` bounded-range branch called `aenum.extend_enum()` on an unrecognised value, so e.g. `Hardware(40)` minted a permanent member. Fixed at the two shared sites in `pcapkit/vendor/default.py`, plus a new `register(value, name)` base classmethod for the caller-named path that still mints. All 105 generated registries are updated; the non-minting `get()` miss applies to all 105, but the `_missing_` fix changes behaviour only for the **21** with a bounded-range `extend_enum` branch. `pcapkit/toolkit/scapy.py` and `pcapkit/toolkit/pyshark.py` fed `get()` a string key with no default and now raise `MissingKeyError` instead of receiving a placeholder; four reader sites in `hopopt.py`/`ipv6_opts.py` convert the `KeyError` into `ProtocolError`. Riding along: `pcapkit/toolkit/pyshark.py` gains a two-entry `FILTER_NAME_TO_LINKTYPE` (`eth` to `ETHERNET`, `tr` to `IEEE802_5`), consulted ahead of `LinkType.get` since pyshark reports a filter name, not a DLT name. The other minting call sites in `_missing_`, `get` and the hand-written `mh.py` and `ngap.py` were deferred; the 106 `register`/`register_alias` sites are not deferred but the by-design caller-named path ([#838](https://github.com/JarryShaw/PyPCAPKit/pull/838)). -- **a breaking change to** `pcapkit.const.ipx.socket.Socket`'s `_missing_`: three of its five archived range branches were strict subsets of a range tested earlier -- `(0x0020, 0x003F)` after `(0x0001, 0x0BB8)`, and `(0x4000, 0x4FFF)` and `(0x8000, 0xFFFF)` after `(0x0BB9, 0xFFFF)` -- so they never fired. The generator `pcapkit/vendor/ipx/socket.py` now lists each subset range ahead of the range containing it. **36891 of the 65521 `_missing_`-resolved values (56.3%) change name**: 32 from `Registered by Xerox` to `Experimental`, 4095 from `Dynamically Assigned` to `Dynamically Assigned Socket Numbers` and 32764 from `Dynamically Assigned` to `Statically Assigned Socket Numbers`. No defined member changes and every value still resolves ([#847](https://github.com/JarryShaw/PyPCAPKit/pull/847)). +- **a breaking change to** `pcapkit.const.ipx.socket.Socket`'s `_missing_`: three of its five archived range branches were strict subsets of a range tested earlier -- `(0x0020, 0x003F)` after `(0x0001, 0x0BB8)`, and `(0x4000, 0x4FFF)` and `(0x8000, 0xFFFF)` after `(0x0BB9, 0xFFFF)` -- so they never fired. The generator `pcapkit/vendor/ipx/socket.py` now lists each subset range ahead of the range containing it. **36891 of the 65521 values that** `_missing_` **resolves (56.3%) change name**: 32 from `Registered by Xerox` to `Experimental`, 4095 from `Dynamically Assigned` to `Dynamically Assigned Socket Numbers` and 32764 from `Dynamically Assigned` to `Statically Assigned Socket Numbers`. No defined member changes and every value still resolves ([#847](https://github.com/JarryShaw/PyPCAPKit/pull/847)). - `LinkType(209).name` returned `'IPMB_LINUX'`, which tcpdump's own table marks "Legacy names (do not use)", because a 2024-05-04 regeneration emitted that row above the current `I2C_LINUX = 209` and `aenum` gives a value to the member defined first. `pcapkit/vendor/reg/linktype.py` now sinks any row whose note mentions "legacy" after every current row, so the current name is defined first; today that affects value 209 alone. `IPMB_LINUX` stays a reachable alias (`LinkType['IPMB_LINUX']` still resolves to 209). Not breaking: only the canonical name for the shared value moves ([#848](https://github.com/JarryShaw/PyPCAPKit/pull/848)). - **a breaking change to** 82 more const registries' `_missing_`: applies the owner's mint/unmint ruling (settled on [#847](https://github.com/JarryShaw/PyPCAPKit/pull/847), confirmed on [#775](https://github.com/JarryShaw/PyPCAPKit/issues/775) -- *"a final concrete assigned name -> mint; a notation for the readers -> unmint"*) across the 89 `_missing_` files still calling `extend_enum`. 80 registries convert wholly (164 branches), each minting a bare status word (`Unassigned`, `Reserved [...]`, `Reserved for Private/Experimental Use`, `Unspecified in the IANA registry`, `Deprecated`), which the ruling treats as a notation rather than an assignment. Two are mixed: `EtherType`'s `DEC Unassigned` (3 branches) and "Old Xerox Experimental values. Invalid as an Ethertype since 1983." (1) convert, while 46 vendor-company names across 50 range blocks (Xyplex, Motorola, Walker Richer & Quinn and 43 more) keep minting, since a proprietary protocol has no public name beyond the company's; `IEEE802_3_Length_Field` and `Berkeley_Trailer_encap_IP` also keep minting but are not company names. `Socket` (ipx)'s allocation-policy names (`Experimental`, `Dynamically Assigned Socket Numbers`, `Statically Assigned Socket Numbers`, `Dynamically Assigned`) convert, while `Registered by Xerox` keeps minting, a real ownership fact. 7 of the 89 are left alone: `CGAType`'s `Tag_` mint is not an IANA-style range, and the other 6 files sit on 9 classes not yet on `EnumRegistry` (`AppType`, `StatusCode`, `ftp.return_code`, `ftp.command`, `Method`, `OptionType`), left to [#859](https://github.com/JarryShaw/PyPCAPKit/pull/859)/[#860](https://github.com/JarryShaw/PyPCAPKit/issues/860). Each conversion changes both the vendor crawler and the generated file, per [#838](https://github.com/JarryShaw/PyPCAPKit/pull/838)'s and [#858](https://github.com/JarryShaw/PyPCAPKit/pull/858)'s precedent. The worked examples in `docs/source/contributing/conventions/mint-criterion.rst` keep `Registered by Xerox` under Mint ([#861](https://github.com/JarryShaw/PyPCAPKit/pull/861)). - **a breaking change to** `EtherType`'s `_missing_` ordering: values `0x0101`-`0x01FF` now resolve to their Old Xerox Experimental name instead of an `IEEE802_3_Length_Field_0x0101`-style one. The generated method tested `0x0000 <= value <= 0x05DC` (IEEE802.3 Length Field) first, and the Old Xerox range sits wholly inside it, so that branch was unreachable. Unlike [#841](https://github.com/JarryShaw/PyPCAPKit/issues/841)'s IPX fix, these bounds come from the live IANA CSV, so the fix is general: `pcapkit/vendor/reg/ethertype.py`'s `process()` collects range rows and a new `EtherType._insert_range` places each ahead of the first already-placed range that fully contains it, leaving non-overlapping pairs in CSV order; this pair is the only containment among the 56 ranges. Closes [#862](https://github.com/JarryShaw/PyPCAPKit/issues/862) ([#865](https://github.com/JarryShaw/PyPCAPKit/pull/865)). - **a breaking change to** `Method`: `__new__` called `str.__new__(cls)` with no argument, so all 40 registered members' `str` payload was empty regardless of value -- `str(Method.GET) == ''` and `Method.GET == 'GET'` was `False`; `.value` was always correct. [#869](https://github.com/JarryShaw/PyPCAPKit/pull/869) (above) had fixed the *unregistered* path only, leaving `str(Method('frob')) == 'frob'` beside `str(Method.GET) == ''`. Fixed to `str.__new__(cls, value)`, mirroring `Command.__new__`, in the generator (`pcapkit/vendor/http/method.py`) and the generated module. No sibling hand-rolled `__new__` has the defect (`OptionType`/`AppType` deliberately store a display string). Consequences, all corrections: `Method.GET == 'GET'` is `True`; `bool(Method.GET)` flips `False` to `True`; `json.dumps(Method.GET)` emits `"GET"` not `""`; `sorted()` is lexicographic rather than input-order-stable; and `{m: ... for m in Method}` gains 40 entries where it had 1, because all 40 members previously hashed as `''` and compared equal. Nothing under `pcapkit/protocols` relied on the old behaviour. Closes [#870](https://github.com/JarryShaw/PyPCAPKit/issues/870) ([#871](https://github.com/JarryShaw/PyPCAPKit/pull/871)). - **a breaking change to** `TransportProtocol` and `AppType`: step 2 of [#860](https://github.com/JarryShaw/PyPCAPKit/issues/860), PR 2 of 2 (the 8 non-`AppType` bespoke registries were [#869](https://github.com/JarryShaw/PyPCAPKit/pull/869), above). `AppType` and its four transport subclasses move onto `EnumRegistry` and both mint sites -- `_missing_`'s 766 range branches and `get()`'s second mint (`PORT_{port}_{transport}` -> `'unknown'`) -- now build through a new `_unregistered_member` override that reconstructs `svc`/`port`/`proto`, per the owner's ruling that *"only IANA registered ones are legit values... get will not have sufficient information to create new ones"*. All 766+1 branches convert, including 8 naming a real service assigned to a whole port span (7 distinct names: `x11` twice, `active-net`, `satvid-datalnk`, `vrml-multi-use`, `ircu`, `swx`, `flex-lm`); unlike `FEATCode`'s fix ([#869](https://github.com/JarryShaw/PyPCAPKit/pull/869), above) there is no import-time self-mutation to cure by declaring members statically, so these stay deliberately unregistered lookups. `AppType` gains a working `register()`; the base's would have built a member with `svc=''`. **The breaking part**: `TransportProtocol` moves from power-of-two values (`tcp=1, udp=2, sctp=4, dccp=8`) to sequential `auto()` (`sctp=3, dccp=4`), per the owner's follow-up ruling once `|`-joined composites stopped being parsed. `AppType.get` treats whatever `int` it is given as a whole, so the renumbering changes what specific integers *mean*: on `AppType.get(80, proto=...)`, a bare `4` resolved to `SCTP.http` on `main` and now **silently** resolves to `DCCP.unknown`, because a real member sits at `4` under both numberings; a bare `8` used to mint `DCCP.PORT_80_dccp` and now raises `ValueError`. The composed-value case is deliberately untested, per the owner's ruling that it is "an obsoleted path from a breaking change" ([#874](https://github.com/JarryShaw/PyPCAPKit/pull/874)). - closes the remaining scope of [#775](https://github.com/JarryShaw/PyPCAPKit/issues/775): `EtherType` (52 branches) and `Socket` (1), the last registries whose `_missing_` minted a permanent member, now return a non-registering `cls._unregistered_member(value, name)`. The issue's AST census of `extend_enum` in `_missing_`/`get` goes from 54 sites to the single `CGAType` carve-out (its mint is not an IANA-style range), from the original 1,169. `EtherType(0x0888)` used to grow `__members__` from 160 to 161 and return the identical object on a repeat; it now returns `` with `__members__` unchanged, absent from `_value2member_map_`, equal but not identical across lookups -- likewise `Socket(0x0010)`. The regenerated `EtherType` was not checked against the live IANA CSV, only against the committed fixture. Keeping each branch's hex-suffixed name reintroduces the value-derived shape `test_const_enum_no_mint.py`'s `is_manufactured()` sweep forbids; rather than weaken the predicate, a narrow exemption covers exactly these two files, since the collision the check guards against cannot occur once `_unregistered_member` never registers ([#878](https://github.com/JarryShaw/PyPCAPKit/pull/878)). +- `OptionType.register()` and `Command.register()` built members that `get()` could not find again, and `OptionType.register_alias()` raised `not a registered OptionType` for every argument. Both registries now round-trip through `get()`, in the generated files and the vendor templates ([#1311](https://github.com/JarryShaw/PyPCAPKit/pull/1311), [#1330](https://github.com/JarryShaw/PyPCAPKit/issues/1330), [#1369](https://github.com/JarryShaw/PyPCAPKit/pull/1369)). +- the Quick-Start function registries rejected 9 to 15, though [RFC 4782](https://datatracker.ietf.org/doc/html/rfc4782) gives the Function field four bits; the whole 4-bit range now resolves ([#1350](https://github.com/JarryShaw/PyPCAPKit/issues/1350), [#1353](https://github.com/JarryShaw/PyPCAPKit/pull/1353)). +- every three-digit FTP reply and HTTP status code resolves, including those no registry row names ([#1443](https://github.com/JarryShaw/PyPCAPKit/issues/1443), [#1444](https://github.com/JarryShaw/PyPCAPKit/pull/1444)). +- the `AppType` registry's dict union is defined only on interpreters whose `dict` has one ([#1694](https://github.com/JarryShaw/PyPCAPKit/issues/1694), [#1703](https://github.com/JarryShaw/PyPCAPKit/pull/1703)). ### pcapkit.corekit @@ -37,13 +47,19 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) an - `tests/corekit/test_fields_numbers_width_repair.py`, covering each octet boundary in its own method, since the defect is a pattern and a single case would pass against a fix that special-cased the reported width ([#599](https://github.com/JarryShaw/PyPCAPKit/issues/599)). - `FieldBaseShortReadPaddingSideTests`, covering both byte orders at 2, 4 and 8 octets and every truncation point, including the two figures [#604](https://github.com/JarryShaw/PyPCAPKit/issues/604) reports, plus guard rails for a full read and for a read against an empty buffer, the [#431](https://github.com/JarryShaw/PyPCAPKit/issues/431) behaviour the option and list loops depend on ([#604](https://github.com/JarryShaw/PyPCAPKit/issues/604)). +- `EnumLookup`, the base carrying `get` and `get_all` that `EnumRegistry` and the non-registry enumerations share ([#877](https://github.com/JarryShaw/PyPCAPKit/issues/877), [#906](https://github.com/JarryShaw/PyPCAPKit/pull/906), [#921](https://github.com/JarryShaw/PyPCAPKit/pull/921), [#932](https://github.com/JarryShaw/PyPCAPKit/pull/932)). Its `get` raises pcapkit exceptions in the shape of the stdlib `Enum`: a name miss raises `EnumKeyError`, a `KeyError` ([#923](https://github.com/JarryShaw/PyPCAPKit/issues/923), [#927](https://github.com/JarryShaw/PyPCAPKit/pull/927)). +- `MultiInfo` and `OrderedMultiInfo`, immutable `Info` subclasses that are also a `MultiDict` and an `OrderedMultiDict`, so a repeated key keeps every value. The data models' option lists use them ([#1484](https://github.com/JarryShaw/PyPCAPKit/issues/1484), [#1523](https://github.com/JarryShaw/PyPCAPKit/pull/1523)). +- `pcapkit.corekit.fields.byteorder`, holding `packet_byteorder` and `byteorder_callback`, the byte-order helpers the PCAP and PCAP-NG schemas share ([#1528](https://github.com/JarryShaw/PyPCAPKit/pull/1528), [#1711](https://github.com/JarryShaw/PyPCAPKit/issues/1711), [#1715](https://github.com/JarryShaw/PyPCAPKit/pull/1715)). #### Changed - `Probe`, `CipherSuite` and `IntegritySuite` are `Info` subclasses rather than `typing.NamedTuple`, and no `NamedTuple` remains in the package. They are Mappings now, so `len()` and iteration yield field names rather than values. - `ModuleDescriptor.klass` reads an already-imported module out of `sys.modules` rather than re-entering `importlib.import_module`, since next layer dispatch resolves a descriptor there on a per-frame path. Neither a registry *hit* holding a `ModuleDescriptor` nor a *miss* writes the resolved class back: a memoised hit went stale on `importlib.reload` ([#1559](https://github.com/JarryShaw/PyPCAPKit/issues/1559)), and recording a miss in a class-level `collections.defaultdict` is the defect [#421](https://github.com/JarryShaw/PyPCAPKit/issues/421)/[#426](https://github.com/JarryShaw/PyPCAPKit/pull/426) fixed at this layer, [#425](https://github.com/JarryShaw/PyPCAPKit/issues/425)/[#428](https://github.com/JarryShaw/PyPCAPKit/pull/428) for the option, chunk and block registries, and [#555](https://github.com/JarryShaw/PyPCAPKit/issues/555)/[#560](https://github.com/JarryShaw/PyPCAPKit/pull/560) at the schema layer. A resolution now costs ~117 ns rather than ~436 ns, which is **not measurable in** `extract()` **wall clock**. The change is taken for its shape: `sys.modules` stays the only module cache, whereas a memo of the class would serve the pre-reload class after an `importlib.reload` forever and fail `isinstance` against the live one. Proposed by `@Ts-Boom` in [#563](https://github.com/JarryShaw/PyPCAPKit/pull/563); the implementation differs because that one added a second, never-invalidated cache of resolved classes ([#574](https://github.com/JarryShaw/PyPCAPKit/issues/574)). - `SeekableReader.truncate()` raises instead of returning a size, and the misspelled `writeable()` is now `writable()`. Both break external callers; nothing inside the package calls either. `io.IOBase` documents one gate over two methods -- "If False, write() and truncate() will raise OSError" -- and this reader's `writable()` answers False, yet `truncate` returned its new size, where `open(path, 'rb').truncate()` raises. The refusal is `UnsupportedOperation('truncate')`, the same one `write` raises; pcapkit's subclasses `io.UnsupportedOperation`. The old resizing is kept as the private `_truncate_buffer`: it resizes a private lookback window, not the stream, and it is the only route to the "position sits before the window" state that `seek` and the four buffered read paths must refuse, for which [#643](https://github.com/JarryShaw/PyPCAPKit/issues/643) and [#644](https://github.com/JarryShaw/PyPCAPKit/issues/644) landed tests. `writeable()` was never an override of anything, so `io`, `shutil` and third parties never saw it, and the value reported is unchanged ([#645](https://github.com/JarryShaw/PyPCAPKit/issues/645)). -- **a breaking change to** `@final`, now enforced at runtime on `Info` and `Schema`. `info_final` and `schema_final` both end `return final(cls)`, so every finalised class carried `typing.final`'s `__final__` marker and nothing read it. **`@final` alone on an `Info` or `Schema` subclass -- without `@info_final` or `@schema_final` -- now raises `InfoError` or `SchemaError` at first construction**, the class being marked final but never finalised and so carrying no generated `__init__`. `__init_subclass__` cannot catch that, since `final` applies after class creation, so the check sits in the existing one-shot `FinalisedState.NONE` branch and a finalised class pays nothing. **Deriving from a finalised class raises too**, from new `Info.__init_subclass__` and `Schema.__init_subclass__` hooks (before, only `EnumSchema` had one). **`SchemaError` is a `ValueError`**, so code catching `TypeError` around a subclass declaration will not see it. `@info_final @final` and `@final @info_final` both finalise silently; the re-entry check keys on `__finalised__` rather than `__final__`, because only the former records *the decorator* having run. `@info_final` twice warns and returns the class unchanged. `pcapkit.utilities.compat` takes `final` from `typing_extensions` below 3.11 rather than 3.8, because `typing.final` only records `__final__` from 3.11. The guard cannot fire on the library's own classes ([#778](https://github.com/JarryShaw/PyPCAPKit/issues/778)). +- **a breaking change to** `@final`, now enforced at runtime on `Info` and `Schema`. `info_final` and `schema_final` both end `return final(cls)`, so every finalised class carried `typing.final`'s `__final__` marker and nothing read it. `@final` **alone on an** `Info` **or** `Schema` **subclass -- without** `@info_final` **or** `@schema_final` **-- now raises** `InfoError` **or** `SchemaError` **at first construction**, the class being marked final but never finalised and so carrying no generated `__init__`. `__init_subclass__` cannot catch that, since `final` applies after class creation, so the check sits in the existing one-shot `FinalisedState.NONE` branch and a finalised class pays nothing. **Deriving from a finalised class raises too**, from new `Info.__init_subclass__` and `Schema.__init_subclass__` hooks (before, only `EnumSchema` had one). `SchemaError` **is a** `ValueError`, so code catching `TypeError` around a subclass declaration will not see it. `@info_final @final` and `@final @info_final` both finalise silently; the re-entry check keys on `__finalised__` rather than `__final__`, because only the former records *the decorator* having run. `@info_final` twice warns and returns the class unchanged. `pcapkit.utilities.compat` takes `final` from `typing_extensions` below 3.11 rather than 3.8, because `typing.final` only records `__final__` from 3.11. The guard cannot fire on the library's own classes ([#778](https://github.com/JarryShaw/PyPCAPKit/issues/778)). +- **a breaking change to** `Info.to_dict()`: it returns an `OrderedMultiDict` at every level rather than a `dict`, so a key that repeats, such as a second IPv6 extension header of one type, survives a `from_dict(to_dict())` rebuild. `type(d) is dict` no longer holds ([#1453](https://github.com/JarryShaw/PyPCAPKit/issues/1453), [#1466](https://github.com/JarryShaw/PyPCAPKit/pull/1466), [#1484](https://github.com/JarryShaw/PyPCAPKit/issues/1484), [#1523](https://github.com/JarryShaw/PyPCAPKit/pull/1523)). +- **a breaking change to** `pcapkit.utilities.decorators`: it moves to `pcapkit.corekit.decorators`, with no re-export. `seekset`, `beholder` and `prepare` are no longer importable from `pcapkit.utilities`, and the module logs as `pcapkit.corekit.decorators` ([#1735](https://github.com/JarryShaw/PyPCAPKit/issues/1735), [#1737](https://github.com/JarryShaw/PyPCAPKit/pull/1737)). +- **a breaking change to** the sentinels: all live in `pcapkit.corekit.sentinels`, which exports the objects `NULL`, `NO_VALUE` and `NO_DEFAULT` but not their types, and `NoValue` is spelled `NO_VALUE` ([#916](https://github.com/JarryShaw/PyPCAPKit/pull/916), [#911](https://github.com/JarryShaw/PyPCAPKit/issues/911), [#922](https://github.com/JarryShaw/PyPCAPKit/pull/922), [#937](https://github.com/JarryShaw/PyPCAPKit/issues/937), [#939](https://github.com/JarryShaw/PyPCAPKit/pull/939)). No moved name keeps a re-export, so `NoValueType` has only that home, not its 1.4.x one in `pcapkit.corekit.fields.field` ([#1744](https://github.com/JarryShaw/PyPCAPKit/issues/1744), [#1754](https://github.com/JarryShaw/PyPCAPKit/pull/1754)). #### Fixed @@ -58,11 +74,19 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) an - the 16-bit band the [#573](https://github.com/JarryShaw/PyPCAPKit/issues/573) entry above records as deliberately left open: a length declared by a 16-bit wire field could still be repeated without limit. The crafted PCAP-NG of 2,000 Enhanced Packet Blocks, each with an option declaring 65,535 octets against four real ones, synthesised 1,637x its size in zeros and now synthesises none, with every frame and option still parsed. The bound is taken one layer up, where the missing information exists: a PCAP-NG block's Block Total Length is authoritative and cross-checked against its trailing copy, so the option area is that length less the fixed fields, `captured_len` and its padding, and an option declaring more payload than the area has left is malformed. A snapshot-truncated capture says so through `captured_len` and leaves its options whole, so it never trips this -- which the [#571](https://github.com/JarryShaw/PyPCAPKit/pull/571) `len(buffer) < length` rejection could not distinguish, and why it was declined. A new `bounded_option` clamps the payload to the octets the area has left and warns with `SchemaWarning`; it is applied to all fifteen variable-width option and record payloads in the PCAP-NG schema, and deliberately not to the block-level payload fields, the 32-bit band the running ledger already budgets. A second, `bounded_area`, clamps a packet block's option area to the octets the block itself holds less the trailing Block Total Length: the area was otherwise sized from a declared length nothing checks against the file, so a block declaring 1,000,000 octets while holding 36 synthesised 65,535 zeros with no warning. Both are no-ops on a well-formed block. The five non-packet blocks' option areas keep the framing assumption; the general fix for a declared length reaching a read is [#678](https://github.com/JarryShaw/PyPCAPKit/issues/678). Clamping rather than refusing keeps the [#431](https://github.com/JarryShaw/PyPCAPKit/issues/431) accommodation, since a PCAP-NG block read has no catch point above `FieldBase.unpack` and one refusal would abort a whole extraction rather than one block; and the clamp reads only the block's own declared framing, never a running total, so identical input answers identically whatever preceded it. The skip for a remainder already past zero is load-bearing on the unpacking path, where clamping would give the field a negative template such as `'-2s'`. No clamp fires on the sample captures, and [#678](https://github.com/JarryShaw/PyPCAPKit/issues/678)'s uncaught `ValueError` on truncated input is unchanged ([#594](https://github.com/JarryShaw/PyPCAPKit/issues/594)). - **a breaking change to** `FieldBase.length`: it called `struct.calcsize` on a template built from a negative resolved length (e.g. `'-5s'`, from a `length=` callback whose running counter had been overdrawn), raising a bare `struct.error` that ordinary caller code does not catch. The chokepoint every affected schema shares now raises `ProtocolError`, covering `application/{ftp,httpv1,httpv2,ngap}.py`, `internet/{hip,ipv6_route,mh}.py`, `link/ethernet.py`, `misc/pcapng.py` and `transport/sctp.py` with no other file changed; a non-negative length is unaffected. This closes the follow-on the `httpv2` length-guard entry above filed as out of scope ([#805](https://github.com/JarryShaw/PyPCAPKit/issues/805)). - **a breaking change to** `EnumRegistry.register()`: it now raises `ValueError` naming the existing member and pointing at `register_alias()`, where it used to silently alias an already-registered value under the caller's new name -- `aenum` treats a taken value as an alias request, so `register(existing_value, 'TOTALLY_NEW_NAME')` returned the existing member, minted nothing and raised nothing, contradicting its docstring. `pcapkit.corekit.enums.EnumRegistry` is a new mixin implementing `get`, `get_all`, `register`, `register_alias`, `register_aliases` and `_unregistered_member` once, per the maintainer's ruling on [#842](https://github.com/JarryShaw/PyPCAPKit/issues/842); it converts the first six registries with no bespoke `__new__` (`ipv6.extension_header.ExtensionHeader`, `tcp.flags.Flags` and four `mh.*_flag` modules), and serves `IntEnum`, `IntFlag` and `StrEnum` alike. The guard reaches only those six: the other 105 const registries still carry the old generator's bare `extend_enum(cls, name, value)`, so `pcapkit.const` ships two contradictory `register` contracts until [#858](https://github.com/JarryShaw/PyPCAPKit/pull/858) (below). The six also gain the generated template's `KeyError` catching on a missing name, so `Flags.get('NOPE', 2048)` returns `Flags.ACK` rather than raising; and a key that is neither `int` nor `str`, which raised `TypeError` on the five `IntFlag` registries (`Flags.get(3.5)`), now raises `ValueError`, tried as a value. On `ExtensionHeader` it raised `KeyError`, the shift [#858](https://github.com/JarryShaw/PyPCAPKit/pull/858)'s entry attributes to the other 105 ([#855](https://github.com/JarryShaw/PyPCAPKit/pull/855)). -- **a breaking change to** `EnumRegistry.register()`, extended: the 105 const registries still on the generated template's bare `extend_enum` now inherit the guard [#855](https://github.com/JarryShaw/PyPCAPKit/pull/855) (above) added, so `TransType.register(6, 'TOTALLY_NEW_NAME')` raises `ValueError` naming the value and the member that holds it and pointing at `register_alias()`, where it used to mint and raise nothing. `pcapkit/vendor/default.py`'s template now emits `class {NAME}(EnumRegistry, IntEnum)` and drops its own `get`/`register`/`_unregistered_member`. The 10 modules with a bespoke `__new__` (`ftp/command`, `ftp/return_code`, `http/method`, `http/status_code`, `pcapng/option_type` and `reg/apptype`) are left alone, so **111 of the 127 classes** inherit `EnumRegistry`, up from 6 ([#855](https://github.com/JarryShaw/PyPCAPKit/pull/855)). `get()`'s dispatch shifts only for a key that is neither `int` nor `str` (`KeyError`, tried as a name; now `ValueError`, tried as a value), a key type nothing in the library passes. `pcapkit/corekit/enums.py` is renamed **`enum.py`**, the maintainer's ruling ([#858](https://github.com/JarryShaw/PyPCAPKit/pull/858)). +- **a breaking change to** `EnumRegistry.register()`, extended: the 105 const registries still on the generated template's bare `extend_enum` now inherit the guard [#855](https://github.com/JarryShaw/PyPCAPKit/pull/855) (above) added, so `TransType.register(6, 'TOTALLY_NEW_NAME')` raises `ValueError` naming the value and the member that holds it and pointing at `register_alias()`, where it used to mint and raise nothing. `pcapkit/vendor/default.py`'s template now emits `class {NAME}(EnumRegistry, IntEnum)` and drops its own `get`/`register`/`_unregistered_member`. The 10 modules with a bespoke `__new__` (`ftp/command`, `ftp/return_code`, `http/method`, `http/status_code`, `pcapng/option_type` and `reg/apptype`) are left alone, so **111 of the 127 classes** inherit `EnumRegistry`, up from 6 ([#855](https://github.com/JarryShaw/PyPCAPKit/pull/855)). `get()`'s dispatch shifts only for a key that is neither `int` nor `str` (`KeyError`, tried as a name; now `ValueError`, tried as a value), a key type nothing in the library passes. `pcapkit/corekit/enums.py` is renamed `enum.py`, the maintainer's ruling ([#858](https://github.com/JarryShaw/PyPCAPKit/pull/858)). - `EnumRegistry.get()`'s `str`-key branch tried only a name lookup (`cls._member_map_[key]`) and on a miss raised or returned `cls(default)`, never the *value* path, so on a `StrEnum` registry a key that resolved through the constructor (`_Str('known-value')`) raised `KeyError` through `get()`, and a supplied `default` overrode a good value match the code never attempted. It now falls back to a plain `_value2member_map_` lookup once the name lookup misses and before consulting `default`. Deliberately not `cls(key)`: `FEATCode._missing_` (`pcapkit/const/ftp/command.py`) mints via `extend_enum` for any unrecognised string, so routing through the constructor would let a failed name lookup mint a permanent member once a registry like it converts; a dict lookup never reaches `_missing_`. A name still wins when a string is both a name and a different member's value. Step 1 of [#860](https://github.com/JarryShaw/PyPCAPKit/issues/860); step 2, converting the bespoke registries, follows ([#869](https://github.com/JarryShaw/PyPCAPKit/pull/869), [#874](https://github.com/JarryShaw/PyPCAPKit/pull/874), below). The non-`str` path is unchanged. Widens rather than breaks: a call that raised `KeyError` now resolves, and no caller that got a correct result gets a different one ([#863](https://github.com/JarryShaw/PyPCAPKit/pull/863)). - **a breaking change to** `EnumRegistry.get()`: its docstring claimed "It never mints", but both `cls(default)` call sites reached `_missing_` for a `default` inside a still-minting registry's range, growing the registry as a side effect of resolving `default` rather than `key`. Per the owner's ruling (*"Only register can mint. get should not mint unless it falls through the _missing_'s minted ranges"*), both sites now resolve `default` through a plain `_value2member_map_` lookup; `key` resolution is unchanged, so `EtherType.get(0x0888)` still mints `Xyplex_0x0888` through its own `_missing_`, as [#775](https://github.com/JarryShaw/PyPCAPKit/issues/775)'s deliberate exception permits. A `default` naming no registered member now falls through to the error `key` itself would have raised, rather than a fresh error about the default -- the accepted cost of the ruling. The docstring's "It never mints" is qualified to `default` only. So `Hardware.get('Definitely-Not-A-Member', 40)`, with `40` in `Hardware`'s declared-but-unassigned range, now raises `KeyError` naming the original key; its test waited on [#865](https://github.com/JarryShaw/PyPCAPKit/pull/865) (above), then landed once [#865](https://github.com/JarryShaw/PyPCAPKit/pull/865) merged. Closes [#864](https://github.com/JarryShaw/PyPCAPKit/issues/864) ([#868](https://github.com/JarryShaw/PyPCAPKit/pull/868)). - `ProtoChain.__add__` returned the left operand's cached `protocols` and `aliases`: once `a.protocols` had been read, `a + ProtoChain(IPv4)` reported `chain == 'Ethernet:IPv4'` but `protocols == (Ethernet,)`. `copy.copy(self)` carried the cache across; the result is now built fresh, as `from_list` builds one, and `+=` follows, since it falls back to `__add__` ([#1099](https://github.com/JarryShaw/PyPCAPKit/issues/1099)). - a `SchemaField` with no declared length handed its `-1` placeholder to the nested schema as that schema's length, so every item of such a list warned `packet length < 0`: 12 `SchemaWarning` for a two-locator HIP `LOCATOR_SET`, 6 for one MH CGA Parameters item. `SchemaField.unpack` now passes what is left in the buffer as a declared length -- not `None`, which would turn a truncated item's `FieldValueError` into `prepare()`'s quiet `StreamEOFError`. Fixed-length items are unchanged; the HTTP/2 `SETTINGS` pair, whose size is fixed, is given it in its own schema instead ([#1140](https://github.com/JarryShaw/PyPCAPKit/issues/1140), below) ([#1148](https://github.com/JarryShaw/PyPCAPKit/issues/1148)). +- a signed `NumberField` decoded as unsigned, a zero-length `PayloadField` misread, and an empty `Raw` did not rebuild ([#1331](https://github.com/JarryShaw/PyPCAPKit/pull/1331)). +- `from_dict(to_dict())` lost nested values, and `len()` of an `Info` counted its hidden keys ([#1334](https://github.com/JarryShaw/PyPCAPKit/pull/1334)). +- `StringField` packs back exactly the octets it unpacked, where a guessed charset, a `U+FFFD` substitution or a character-count length used to change them; the decoded text is unchanged ([#1293](https://github.com/JarryShaw/PyPCAPKit/issues/1293), [#1336](https://github.com/JarryShaw/PyPCAPKit/pull/1336)). A charset that rejects `'replace'` falls back to UTF-8 ([#1706](https://github.com/JarryShaw/PyPCAPKit/issues/1706), [#1728](https://github.com/JarryShaw/PyPCAPKit/pull/1728)), and Python 3.8 unquotes through `pcapkit.utilities.compat.unquote` ([#1693](https://github.com/JarryShaw/PyPCAPKit/issues/1693), [#1707](https://github.com/JarryShaw/PyPCAPKit/pull/1707)). +- `IPv4InterfaceField` keeps a mask that is not a netmask, and a deep copy of an `OrderedMultiDict` keeps its order ([#1346](https://github.com/JarryShaw/PyPCAPKit/pull/1346)). +- **a breaking change to** padding: non-zero padding octets are kept on rebuild rather than zeroed ([#1223](https://github.com/JarryShaw/PyPCAPKit/issues/1223), [#1436](https://github.com/JarryShaw/PyPCAPKit/pull/1436)). +- fields keep the bits and octets they captured on rebuild: a `BitField`'s unnamed bits ([#1487](https://github.com/JarryShaw/PyPCAPKit/issues/1487), [#1495](https://github.com/JarryShaw/PyPCAPKit/pull/1495)), a `NumberField`'s bits outside `bit_length` ([#1488](https://github.com/JarryShaw/PyPCAPKit/issues/1488), [#1497](https://github.com/JarryShaw/PyPCAPKit/pull/1497)), and the bytes an untyped `ListField` unpacked ([#1489](https://github.com/JarryShaw/PyPCAPKit/issues/1489), [#1494](https://github.com/JarryShaw/PyPCAPKit/pull/1494)). An unnamed `ListField`, `PayloadField` or `SchemaField` gets a default name ([#1491](https://github.com/JarryShaw/PyPCAPKit/issues/1491), [#1493](https://github.com/JarryShaw/PyPCAPKit/pull/1493)). +- each non-final `Info` subclass is finalised on its own state ([#1490](https://github.com/JarryShaw/PyPCAPKit/issues/1490), [#1496](https://github.com/JarryShaw/PyPCAPKit/pull/1496)), and its constructor reads only the class's own annotations ([#1692](https://github.com/JarryShaw/PyPCAPKit/issues/1692), [#1702](https://github.com/JarryShaw/PyPCAPKit/pull/1702)). +- copying or pickling a stream proxy no longer recurses ([#1524](https://github.com/JarryShaw/PyPCAPKit/issues/1524), [#1544](https://github.com/JarryShaw/PyPCAPKit/pull/1544)). ### pcapkit.dumpkit @@ -72,6 +96,10 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) an - `tests/dumpkit/test_nameless_enum_rendering_unit.py` (added by [#670](https://github.com/JarryShaw/PyPCAPKit/pull/670)) carried two probes assuming every flag-enum registry is 16 bits wide: `test_scalar_return_renders_a_nameless_member_as_its_value` used one fixed tuple ending in `65536`, and `test_no_flag_registry_renders_the_literal_none` swept every registry but only against `registry(0)`. Widths differ: `ftp.command.CommandType` 3 bits, `reg.apptype.TransportProtocol` 4 (computed at runtime), the three `mh` ack/initiate flags 8, and `mh.binding_update_flag.BindingUpdateFlag` and `tcp.flags.Flags` 16. `tcp.flags.Flags(65536)` correctly raises (`0 <= value <= 0xFFFF`), so the sweep was failing on correct behaviour. The probes now derive each registry's bound from its declared members, and a new test pins that one past the widest in-field value is refused. Test-only ([#702](https://github.com/JarryShaw/PyPCAPKit/issues/702)). - the plist writer emitted mapping keys unescaped, so a key carrying `&`, `<` or `>` produced a report that is not well-formed XML. `dictdumper`'s `_append_dict` interpolates the key into `'{item}'` and calls `_encode_value` on the value only, so both branches of `pcapkit.dumpkit.common` that build a mapping now escape their keys through one `escape_key` helper -- a `MultiDict`, where [#575](https://github.com/JarryShaw/PyPCAPKit/issues/575) escaped an enum-derived key inline and nothing else, and a plain dict, where nothing was escaped. The json, tree and text writers are untouched. A non-`str` key is escaped too: a PCAP-NG decryption secrets block keys TLS key log entries by a raw bytes client random whose repr can carry all three characters. The upstream half is `JarryShaw/DictDumper#125`; this does not wait on it. The json writer's quoting defect stands, upstream as `JarryShaw/DictDumper#121` ([#772](https://github.com/JarryShaw/PyPCAPKit/issues/772)). - the JSON writer emitted mapping keys unescaped, so the report of `test.pcapng` did not parse (`Expecting ':' delimiter`): its TLS key log is keyed by a `bytes` client random whose repr holds `"` and `\`. `escape_key` in `pcapkit.dumpkit.common` now JSON-escapes keys for JSON output, as [#772](https://github.com/JarryShaw/PyPCAPKit/issues/772) made it XML-escape them for plist, and the JSON reports of all 23 `examples/captures/` files now parse. The upstream defect, `JarryShaw/DictDumper#121`, stands ([#1152](https://github.com/JarryShaw/PyPCAPKit/issues/1152)). +- **a breaking change to** the dump schema: the output writers dropped or corrupted what the packets carried. An `OrderedMultiDict`, which every option list is, now dumps as a list of single-key entries in wire order ([#1305](https://github.com/JarryShaw/PyPCAPKit/pull/1305)). +- tree output raised on a `complex` value, and is now documented as lossy ([#1266](https://github.com/JarryShaw/PyPCAPKit/issues/1266), [#1366](https://github.com/JarryShaw/PyPCAPKit/pull/1366)). +- plist dates are written to whole seconds, so `plistlib` reads them back ([#1448](https://github.com/JarryShaw/PyPCAPKit/issues/1448), [#1450](https://github.com/JarryShaw/PyPCAPKit/pull/1450)). +- a PCAP trace record scales its fraction to the trace's resolution ([#1500](https://github.com/JarryShaw/PyPCAPKit/issues/1500), [#1529](https://github.com/JarryShaw/PyPCAPKit/pull/1529)), and writes the octets it has rather than a truncated record's claimed length ([#1541](https://github.com/JarryShaw/PyPCAPKit/issues/1541), [#1543](https://github.com/JarryShaw/PyPCAPKit/pull/1543)). ### pcapkit.foundation @@ -92,6 +120,7 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) an - extraction is around 46% faster on a 1,117-frame HTTP capture, with byte-identical output ([#420](https://github.com/JarryShaw/PyPCAPKit/pull/420)). A reassembled datagram's payload is analysed on first read rather than eagerly, cutting IP reassembly's own cost by 90.7% and TCP's by 23.7% (IP reassembly submits a datagram for every frame, fragmented or not) ([#424](https://github.com/JarryShaw/PyPCAPKit/pull/424)). Flow tracing over the same capture went from 1416.6 ms to 744.0 ms, because the flow dumper handed each record to a `Frame` constructor that re-dissected the whole stack to return the bytes it had just been given; options are no longer parsed twice either ([#427](https://github.com/JarryShaw/PyPCAPKit/pull/427)). - **a breaking change to** `Extractor.engine` and `Extractor.record_header`: both are now declared to return `EngineBase[_P]` rather than `Engine`, and the private `_exeng` attribute follows. The built-in engines subclass `EngineBase` directly, so none was ever an `Engine`, which adds only the `__init_subclass__` registration hook. Two things are visible to a type checker. Code passing the result to an `Engine`-typed parameter must now accept `EngineBase`; and because the old declarations were *unparameterised*, `engine.read_frame()` and `record_header().read_frame()` used to be `Any` and are now the extractor's own frame type, so an assignment that relied on `Any` there no longer type-checks ([#1022](https://github.com/JarryShaw/PyPCAPKit/issues/1022)). - the documentation of the engine base classes: the member stubs (`name`, `module`, `registry`, `extractor`, `unsupported_reason`, `run`, `read_frame`, `close` and the rest) moved from `Engine` to `EngineBase`, where they are defined, and `Engine` now documents only the `__init_subclass__` registration hook it adds, so the `EngineBase[_P]` that `Extractor.engine` returns leads to an API. Cross-references to `Engine.run`, `Engine.read_frame`, `Engine.close` and `Engine.unsupported_reason` now point at `EngineBase`. Exporting `EngineBase` in `__all__` is left to [#1016](https://github.com/JarryShaw/PyPCAPKit/issues/1016) ([#1024](https://github.com/JarryShaw/PyPCAPKit/issues/1024)). +- **a breaking change to** `register_mh_extension`: `schema` is keyword-only, as in the other fourteen schema-taking registrars ([#516](https://github.com/JarryShaw/PyPCAPKit/issues/516), [#526](https://github.com/JarryShaw/PyPCAPKit/pull/526)). #### Fixed @@ -108,6 +137,20 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) an - three `Extractor` defects: `Extractor.output`'s `UnsupportedCall` named `format` rather than `output`; `make_name` looked an unknown format up in the `__output__` `defaultdict` and so inserted it; and the Scapy engine never set `_offmt`, so `Extractor.format` raised `AttributeError` under `engine='scapy'` ([#1095](https://github.com/JarryShaw/PyPCAPKit/issues/1095)). - the DPKT, Scapy and PyShark engines call `Extractor.record_header()` at setup again, as they did until 2023, so the global-header step is shared with the default engine and sets `_offmt` for all four. **Their dumped output changes**: JSON, plist and tree reports gain the leading `Global Header` record (`Section Header 1` for PCAP-NG) the default engine already writes, identical to its, and `files=True` writes that record's file too; frame records are unchanged. `Extractor.format` now also answers on a capture with no frames. `format='pcap'` failing on every engine is a separate defect, [#1127](https://github.com/JarryShaw/PyPCAPKit/issues/1127) (below) ([#1104](https://github.com/JarryShaw/PyPCAPKit/issues/1104)). - `format='pcap'` failed on every engine with a `TypeError` from `PCAPIO.__init__()`, missing `protocol`: `Extractor.__init__` created the writer before any global header was read, so `PCAPIO` had no link type. The header step now creates it through `Extractor._open_output`, and `run()` opens it for engines with no such step; a PCAP writer takes the frame itself and skips the `Global Header` record, so the default engine writes `in.pcap` back byte for byte. The third-party engines hand the writer no frame octets, so they fall back to JSON with a `FormatWarning`, as their flow tracing already does, and a PCAP-NG input raises `FormatError` ([#1127](https://github.com/JarryShaw/PyPCAPKit/issues/1127)). The written global header also dropped the input's `thiszone`, `sigfigs` and `snaplen`, so only 1 of 17 sample PCAP files came back byte for byte; the PCAP engine now passes all three through `_open_output` to `PCAPIO`, and all 17 do ([#1165](https://github.com/JarryShaw/PyPCAPKit/issues/1165)). +- IP reassembly reports only the octets it holds: a truncated fragment, a strict run and a zero-length datagram no longer report octets from the declared Total Length ([#1368](https://github.com/JarryShaw/PyPCAPKit/pull/1368)). Only the final fragment's partial block counts, and a datagram too long for its length field never completes ([#1582](https://github.com/JarryShaw/PyPCAPKit/pull/1582), [#1585](https://github.com/JarryShaw/PyPCAPKit/issues/1585), [#1598](https://github.com/JarryShaw/PyPCAPKit/pull/1598)). +- TCP reassembly judges completion from each buffer's own gaps, compares sequence numbers modulo `2**32`, and holds a FIN until reordered data arrives ([#1375](https://github.com/JarryShaw/PyPCAPKit/pull/1375)). A TSO fragment's length is its captured length, and `RCVBT` covers the last block ([#1564](https://github.com/JarryShaw/PyPCAPKit/pull/1564)). +- a second TCP flow with the same label truncated the first flow's file; a reused label now gets the next free suffix ([#1360](https://github.com/JarryShaw/PyPCAPKit/issues/1360), [#1371](https://github.com/JarryShaw/PyPCAPKit/pull/1371)). +- the engine, reassembly, traceflow and dumper registrars accept the built-in a name shipped with, which restores it ([#1363](https://github.com/JarryShaw/PyPCAPKit/issues/1363), [#1376](https://github.com/JarryShaw/PyPCAPKit/pull/1376)). +- PCAP-NG blocks are numbered across the whole file, not per section ([#1256](https://github.com/JarryShaw/PyPCAPKit/issues/1256), [#1426](https://github.com/JarryShaw/PyPCAPKit/pull/1426)). +- `format='pcap'` is kept when an unavailable engine falls back to the default one ([#1264](https://github.com/JarryShaw/PyPCAPKit/issues/1264), [#1430](https://github.com/JarryShaw/PyPCAPKit/pull/1430)). +- `extract()` accepts a file object with no `name` as `fin` ([#1506](https://github.com/JarryShaw/PyPCAPKit/issues/1506), [#1509](https://github.com/JarryShaw/PyPCAPKit/pull/1509)), and reads every frame from a pipe or other non-seekable stream ([#1586](https://github.com/JarryShaw/PyPCAPKit/issues/1586), [#1595](https://github.com/JarryShaw/PyPCAPKit/pull/1595)). +- the third-party engines read what the capture holds. The dpkt engine keeps a frame dpkt cannot parse rather than leaking `AttributeError` ([#1351](https://github.com/JarryShaw/PyPCAPKit/issues/1351), [#1377](https://github.com/JarryShaw/PyPCAPKit/pull/1377)), applies each PCAP-NG interface's link type and timestamp resolution rather than the first one's ([#1379](https://github.com/JarryShaw/PyPCAPKit/issues/1379), [#1396](https://github.com/JarryShaw/PyPCAPKit/pull/1396)), and reads float timestamps, Simple Packet Blocks and raw-IP frames ([#1550](https://github.com/JarryShaw/PyPCAPKit/pull/1550)). The scapy engine resets its interfaces at each PCAP-NG section ([#1522](https://github.com/JarryShaw/PyPCAPKit/issues/1522), [#1546](https://github.com/JarryShaw/PyPCAPKit/pull/1546)), takes a packet's link type and clock from its interface ([#1563](https://github.com/JarryShaw/PyPCAPKit/pull/1563)), and reads each record whole ([#1717](https://github.com/JarryShaw/PyPCAPKit/issues/1717), [#1733](https://github.com/JarryShaw/PyPCAPKit/pull/1733)). The pyshark engine skips non-packet PCAP-NG blocks ([#1515](https://github.com/JarryShaw/PyPCAPKit/issues/1515), [#1531](https://github.com/JarryShaw/PyPCAPKit/pull/1531)), and the pypcapfile engine reads the PCAP records pypcapfile drops ([#1512](https://github.com/JarryShaw/PyPCAPKit/issues/1512), [#1569](https://github.com/JarryShaw/PyPCAPKit/pull/1569)). dpkt and scapy write real PCAP traces ([#1507](https://github.com/JarryShaw/PyPCAPKit/issues/1507), [#1511](https://github.com/JarryShaw/PyPCAPKit/pull/1511)). + +### pcapkit.interface + +#### Added + +- `pcapkit.interface.registry`, making public the registry-entry helpers `same_entry`, `restore_shipped` and `restore_shipped_dumper` ([#1519](https://github.com/JarryShaw/PyPCAPKit/issues/1519), [#1568](https://github.com/JarryShaw/PyPCAPKit/pull/1568)). ### pcapkit.protocols @@ -124,6 +167,8 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` and `1.5.0b2` (both 2026-09-18) an - `tests/protocols/test_option_generator_tcp_base_unit.py`, asserting that every `TCP_BASE` key is a parameter `TCP.make` declares (derived from `inspect.signature`) and that the built segment carries the stated header in both data model and packed octets ([#602](https://github.com/JarryShaw/PyPCAPKit/issues/602)). - ILNP nonce sizing coverage in `tests/protocols/internet/test_ipv6_extension_unit.py`, one test per protocol, asserting the declared length, the exact packed octets and the construct-pack-parse cycle over ten nonces, most with a bit length that is *not* a multiple of eight, where floor division and the ceiling disagree ([#601](https://github.com/JarryShaw/PyPCAPKit/issues/601)). - `SOLUTION` parameter width coverage in `tests/protocols/internet/test_hip_unit.py`: eight widths asserting the declared length, the reader's acceptance and the construct-pack-parse cycle, plus a 57-bit case pinning the 20-octet length [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) requires of HIPv1. Five widths are deliberately *not* multiples of eight, where the defective `ceil(bits / 4)` coincides with the correct width ([#608](https://github.com/JarryShaw/PyPCAPKit/issues/608)). +- the BSD loopback layer for `LINKTYPE_NULL` and `LINKTYPE_LOOP`, and dissection of `LINKTYPE_RAW` by the version nibble ([#1594](https://github.com/JarryShaw/PyPCAPKit/pull/1594)). +- one home for each RFC- or format-fixed constant spelled out in more than one module -- the IPv4, IPv6, AH and TCP header lengths and offsets, `IP_MAX_LENGTH`, and the PCAP and PCAP-NG framing constants -- in the module that owns the protocol ([#1739](https://github.com/JarryShaw/PyPCAPKit/issues/1739), [#1752](https://github.com/JarryShaw/PyPCAPKit/pull/1752)). #### Changed @@ -143,10 +188,13 @@ Import paths, before to after: Layer-limited extraction: `layer='link'` no longer sets the termination flag on OSPF and `layer='application'` now does -- the same inversion applies to RARP -- but the extracted protochain is unchanged at every `layer=` value, including `'internet'` (`IPv4:OSPFIGP` before and after, as IPv4 ends the chain itself). - renames with no compatibility alias left behind: PCAP-NG `Option` subclasses spell the namespace class keyword `ns=` instead of `namespace=` ([#439](https://github.com/JarryShaw/PyPCAPKit/issues/439)). - `tests/protocols/transport/test_tcp_udp_unit.py` now reaches the MP_JOIN dispatchers through `TCP()` itself, instead of assigning a Python `set` to `_flags` on a bare `TCP.__new__(TCP)`. A `set` answers the same membership tests, so every branch ran while the attribute had neither the type production assigns nor the ordering that governs when it exists -- how [#587](https://github.com/JarryShaw/PyPCAPKit/issues/587) stayed invisible. Reverting [#587](https://github.com/JarryShaw/PyPCAPKit/issues/587)'s hoist, or restoring the `cast('Enum_Flags', 0)` seed, now fails the file. Test-only ([#603](https://github.com/JarryShaw/PyPCAPKit/issues/603)). -- building a protocol through its constructor with a keyword that names nothing now raises `UnsupportedCall` instead of discarding it. **This is a behaviour change to a public API**: every `make` in the tree ends its signature with `**kwargs` and reads nothing out of it, so a misspelled keyword was accepted, dropped, and the field it named kept its default -- wrong octets, nothing said, as in [#602](https://github.com/JarryShaw/PyPCAPKit/issues/602), [#541](https://github.com/JarryShaw/PyPCAPKit/issues/541) and [#556](https://github.com/JarryShaw/PyPCAPKit/issues/556). The check sits in `ProtocolBase.__init__` rather than `make`, because `__post_init__` passes one `**kwargs` to the construction *and* to the parse of what it has just constructed, so a keyword declared only by `read` legitimately travels through `make` (`HIP.read` declares `extension`). The accepted set is the union of every keyword-taking parameter of `make`, `read`, `pack`, `unpack`, `__post_init__` and `__init__` across the MRO. Parsing is deliberately untouched: a protocol cannot know which keywords its parent passed on. A new `__keywords__` opts a class out: a set, for a keyword read out of `**kwargs` by name (`ESP.read` with `packet`); and `None`, for a dispatcher whose real signature belongs to a class chosen at call time (only `HTTP.make`). The message names the near neighbour (`seq` reports *did you mean 'seq_no'?*). `from_data` warns `UnknownFieldWarning` rather than raising, because the defect is then a `_make_data` key the caller cannot fix. Four leftover copies of [#602](https://github.com/JarryShaw/PyPCAPKit/issues/602)'s misspelt TCP base, in `examples/generators/dispatch.py` and `tests/protocols/transport/`, are fixed. Three such `_make_data` keys are reported, not fixed: `Frame._make_data` returns `ts_src` where `make` declares `ts_sec`, `L2TPv2._make_data` returns `prio` where it declares `priority`, and `Header._make_data` returns a `magic_number` that `Header.make` does not take. A *direct* `SomeProtocol.make(...)` call is not checked and still discards silently ([#617](https://github.com/JarryShaw/PyPCAPKit/issues/617)). +- building a protocol through its constructor with a keyword that names nothing now raises `UnsupportedCall` instead of discarding it. **This is a behaviour change to a public API**: every `make` in the tree ends its signature with `**kwargs` and reads nothing out of it, so a misspelled keyword was accepted, dropped, and the field it named kept its default -- wrong octets, nothing said, as in [#602](https://github.com/JarryShaw/PyPCAPKit/issues/602), [#541](https://github.com/JarryShaw/PyPCAPKit/issues/541) and [#556](https://github.com/JarryShaw/PyPCAPKit/issues/556). The check sits in `ProtocolBase.__init__` rather than `make`, because `__post_init__` passes one `**kwargs` to the construction *and* to the parse of what it has just constructed, so a keyword declared only by `read` legitimately travels through `make` (`HIP.read` declares `extension`). The accepted set is the union of every keyword-taking parameter of `make`, `read`, `pack`, `unpack`, `__post_init__` and `__init__` across the MRO. Parsing is deliberately untouched: a protocol cannot know which keywords its parent passed on. A new `__keywords__` opts a class out: a set, for a keyword read out of `**kwargs` by name (`ESP.read` with `packet`); and `None`, for a dispatcher whose real signature belongs to a class chosen at call time (only `HTTP.make`). The message names the near neighbour (`seq` reports *did you mean 'seq_no'?*). `from_data` warns `UnknownFieldWarning` rather than raising for a key its own `_make_data` returned, which the caller cannot fix; a keyword passed to `from_data` itself raises, as for a direct construction ([#1147](https://github.com/JarryShaw/PyPCAPKit/pull/1147)). Four leftover copies of [#602](https://github.com/JarryShaw/PyPCAPKit/issues/602)'s misspelt TCP base, in `examples/generators/dispatch.py` and `tests/protocols/transport/`, are fixed. Three such `_make_data` keys are reported, not fixed: `Frame._make_data` returns `ts_src` where `make` declares `ts_sec`, `L2TPv2._make_data` returns `prio` where it declares `priority`, and `Header._make_data` returns a `magic_number` that `Header.make` does not take. A *direct* `SomeProtocol.make(...)` call is not checked and still discards silently ([#617](https://github.com/JarryShaw/PyPCAPKit/issues/617)). - **a breaking change to** `Application`: it no longer forbids payload dispatch outright. It encodes "no further protocol layer above this one", which undissected trailer bytes are not, so `_decode_next_layer` and `_import_next_layer` accept the `-1` sentinel -- the rest of the packet, resolving to `Raw`, or to `NoPayload` when nothing remains -- and `__post_init__` keeps a payload that `read` set instead of overwriting it with `NoPayload`. A real protocol number (any `proto` other than `-1`) is still refused with `UnsupportedCall`. A subclass that calls either method with `-1` changes from raising to succeeding; no subclass in the package does ([#719](https://github.com/JarryShaw/PyPCAPKit/issues/719)). - the `Protocol` and `ProtocolMeta` docstrings. `Protocol` claimed to be the base of every protocol family, though nothing in the package subclasses it, and `ProtocolMeta` claimed to generate attributes its empty body does not. `Protocol` is now documented as the base for third-party protocols, and `ProtocolBase` as the one the families derive from ([#1101](https://github.com/JarryShaw/PyPCAPKit/issues/1101)). - **a breaking change to** `OSPF_CrytographicAuthentication`: the misspelling is corrected to `OSPF_CryptographicAuthentication`, and `CrytographicAuthentication` in `pcapkit.protocols.schema.application.ospf` and `pcapkit.protocols.data.application.ospf` to `CryptographicAuthentication`, with no alias left for the old spelling. Both OSPF names also move to the application group of `schema.__all__` and `data.__all__`, after [#719](https://github.com/JarryShaw/PyPCAPKit/issues/719)'s move ([#1090](https://github.com/JarryShaw/PyPCAPKit/issues/1090)). +- **a breaking change to** a declared length that runs past the data: it is refused with `ProtocolError` rather than accepted and zero-filled on rebuild, so a malformed option looks different in `info` ([#1325](https://github.com/JarryShaw/PyPCAPKit/issues/1325), [#1398](https://github.com/JarryShaw/PyPCAPKit/pull/1398)). +- dissection is bounded: a frame at 64 layers ([#1610](https://github.com/JarryShaw/PyPCAPKit/issues/1610), [#1620](https://github.com/JarryShaw/PyPCAPKit/pull/1620)) and an IPv6 extension-header chain at 32 headers ([#1604](https://github.com/JarryShaw/PyPCAPKit/issues/1604), [#1608](https://github.com/JarryShaw/PyPCAPKit/pull/1608)). Both limits are `PCAPKIT_FRAME_LAYER_LIMIT` and `PCAPKIT_EXTENSION_HEADER_LIMIT` in `pcapkit.utilities.config` ([#1743](https://github.com/JarryShaw/PyPCAPKit/pull/1743)). Rebuilding a frame cut at the chain bound warns once ([#1619](https://github.com/JarryShaw/PyPCAPKit/issues/1619), [#1630](https://github.com/JarryShaw/PyPCAPKit/pull/1630)). +- **a breaking change to** IPv6 extension headers: an unrecognised or unparsable one is parsed generically, per [RFC 6564](https://datatracker.ietf.org/doc/html/rfc6564), rather than collapsing the packet to `Raw` ([#891](https://github.com/JarryShaw/PyPCAPKit/issues/891), [#904](https://github.com/JarryShaw/PyPCAPKit/pull/904)). `IPv6_Ext` does it, and is also the shared base of all eight extension headers ([#917](https://github.com/JarryShaw/PyPCAPKit/issues/917), [#924](https://github.com/JarryShaw/PyPCAPKit/pull/924)). #### Fixed @@ -177,9 +225,9 @@ This resolves [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548), which r - `main` went red the moment [#604](https://github.com/JarryShaw/PyPCAPKit/issues/604)'s `ljust()` landed, because that test still pinned the head-padded short read: `Reserved_79` declaring `length=12` over 6 real octets now reports `aabbccddeeff00000000` where the test expected `00000000aabbccddeeff`. [#621](https://github.com/JarryShaw/PyPCAPKit/pull/621) left the file alone because [#612](https://github.com/JarryShaw/PyPCAPKit/pull/612) owned it at the time ([#604](https://github.com/JarryShaw/PyPCAPKit/issues/604), [#621](https://github.com/JarryShaw/PyPCAPKit/pull/621)). - the HIP `SOLUTION` builder sized the parameter with `4 + math.ceil(max(random.bit_length(), solution.bit_length()) / 4)`, an invalid shorthand for two fields. [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5) gives `Length` as `4 + RHASH_len / 4` over a `Random #I` and a `Puzzle solution #J` of `RHASH_len / 8` octets **each**, and `/ 4` equals twice `/ 8` only because `RHASH_len` is a whole number of octets. For an arbitrary `bit_length()` the builder emitted an *odd* contents width, which its own reader refused: `_read_param_solution`'s `(schema.len - 4) % 2` guard exists because `SolutionParameter` splits that width into two equal halves. `random=0x1` with `solution=0xfff` declared `len=7` and raised `ProtocolError: HIPv2: [ParamNo 321] invalid format` on input the library had produced. The undersized length was worse: both fields take their width from that `len`, so `solution=0xfff` was packed into one octet and came back as `0xff`, silently. Fixed to `4 + 2 * math.ceil(max(...) / 8)`, the form the sibling `PUZZLE` builder uses, which is even by construction. Loosening the reader was rejected: an odd contents width has no meaning in the wire format (the RFC makes the fields equal-width and does not say which takes an extra octet), and a laxer guard would still have truncated the value ([#608](https://github.com/JarryShaw/PyPCAPKit/issues/608)). - `TCP.read` seeded its connection-flag accumulator with `cast('Enum_Flags', 0)`. `typing.cast` is a runtime no-op, so the accumulator began as the plain `int` `0`, and only the `|=` promoting it to a `pcapkit.const.tcp.flags.Flags` member ever changed that. A segment with an all-zero flags octet (an nmap NULL scan) left `self._flags` an `int`, so a membership test raised `TypeError: argument of type 'int' is not a container or iterable`, and `TCP.connection` returned an `int` where it annotates `Flags`. Any flag masked it, which is how it survived 100% statement and branch coverage. The seed is `Flags(0)`, as [#597](https://github.com/JarryShaw/PyPCAPKit/pull/597) already did for the sibling accumulator in `make`; `Flags` declares no `_missing_`, so `Flags(0)` is an ordinary `aenum.IntFlag` pseudo-member and does not meet the `RecursionError` the same construction hits on the `pcapkit.const.mh` flag enumerations ([#623](https://github.com/JarryShaw/PyPCAPKit/issues/623)). The MP_JOIN path was latent: `mptcp_data_selector` rejects a flagless MP_JOIN before [RFC 8684](https://datatracker.ietf.org/doc/html/rfc8684) section 3.2's three layouts are chosen. One dump-format change is observable: a flagless segment's `connection` renders as a string where it rendered as the number `0`, so it no longer switches JSON type with the flags. That string read `Flags::None [0]` until the `pcapkit.dumpkit` fix above ([#616](https://github.com/JarryShaw/PyPCAPKit/issues/616)). -- **a breaking change to a public attribute.** `Frame.len` and `Frame.cap_len` were filled from *opposite* wire fields depending on the container format, so `frame.len` meant the captured length out of a `.pcap` and the on-wire length out of a `.pcapng`. The PCAP reader moved to match PCAP-NG: `len` is the on-wire length (the record header's `orig_len`) and `cap_len` the octets stored (`incl_len`). **Code reading `frame.len` or `frame.cap_len` from a `.pcap` gets the other field's value than before**, and for a truncated frame that is a different number rather than a relabelling. Seniority did not settle it: the PCAP spelling is the *older* (`c43892af`, 2022-01-11) and PCAP-NG's arrived in `25f216f4` (2023-04-27), both internally consistent. The *names* settle it, being Wireshark's: `frame.len` is "Frame length on the wire" and `frame.cap_len` "Frame length stored into the capture file", and `frame.len_lt_caplen` expert info flags `frame_len < cap_len` as malformed, which it could not be if `len` were the smaller, captured one. `Frame.read` hands `_decode_next_layer` the octets *present*, now `frame.cap_len`; handing over the on-wire length would be the declared-length-exceeds-available-octets fault of [#554](https://github.com/JarryShaw/PyPCAPKit/issues/554), [#573](https://github.com/JarryShaw/PyPCAPKit/issues/573) and [#594](https://github.com/JarryShaw/PyPCAPKit/issues/594) on every snapped frame. The dumpers are untouched. The two lengths differ only for a snapshot-cut frame, and no such fixture existed until [#614](https://github.com/JarryShaw/PyPCAPKit/pull/614) ([#618](https://github.com/JarryShaw/PyPCAPKit/issues/618)). +- **a breaking change to a public attribute.** `Frame.len` and `Frame.cap_len` were filled from *opposite* wire fields depending on the container format, so `frame.len` meant the captured length out of a `.pcap` and the on-wire length out of a `.pcapng`. The PCAP reader moved to match PCAP-NG: `len` is the on-wire length (the record header's `orig_len`) and `cap_len` the octets stored (`incl_len`). **Code reading** `frame.len` **or** `frame.cap_len` **from a** `.pcap` **gets the other field's value than before**, and for a truncated frame that is a different number rather than a relabelling. Seniority did not settle it: the PCAP spelling is the *older* (`c43892af`, 2022-01-11) and PCAP-NG's arrived in `25f216f4` (2023-04-27), both internally consistent. The *names* settle it, being Wireshark's: `frame.len` is "Frame length on the wire" and `frame.cap_len` "Frame length stored into the capture file", and `frame.len_lt_caplen` expert info flags `frame_len < cap_len` as malformed, which it could not be if `len` were the smaller, captured one. `Frame.read` hands `_decode_next_layer` the octets *present*, now `frame.cap_len`; handing over the on-wire length would be the declared-length-exceeds-available-octets fault of [#554](https://github.com/JarryShaw/PyPCAPKit/issues/554), [#573](https://github.com/JarryShaw/PyPCAPKit/issues/573) and [#594](https://github.com/JarryShaw/PyPCAPKit/issues/594) on every snapped frame. The dumpers are untouched. The two lengths differ only for a snapshot-cut frame, and no such fixture existed until [#614](https://github.com/JarryShaw/PyPCAPKit/pull/614) ([#618](https://github.com/JarryShaw/PyPCAPKit/issues/618)). - the HIP `PUZZLE` and `SOLUTION` builders derived three wire-format values from the payload value instead of the data model, and all three were wrong. **This changes two public data models and the octets both parameters emit.** The width of `Random #I` and `Puzzle solution #J` came from `int.bit_length()` alone, so every leading zero octet was dropped on re-serialisation: a `SOLUTION` read with `Length = 20` rebuilt as `Length = 6`, a `PUZZLE` with `Length = 12` as `Length = 5`, silently. That width is `RHASH_len / 8` octets [[RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4), [RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5)], a property of the Responder's HIT Suite, so both data models now carry it as `rhash_len` and both builders prefer it; a new `rhash_len=` keyword states it for a build from scratch, which under HIPv2 is the only place it can come from, since `RSA,DSA/SHA-256` is the REQUIRED HIT Suite [[RFC 7401 Section 5.2.10](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.10)] and makes the field 32 octets rather than 8. On the mandatory path about one parameter in 256 has a zero top octet and lost it ([#653](https://github.com/JarryShaw/PyPCAPKit/issues/653)). `SOLUTION`'s second contents octet is `Reserved`, "zero when sent, ignored when received" [[RFC 7401 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.5), and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) identically], not the `Lifetime` that only [RFC 7401 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.4) defines, which pcapkit encoded there as `2^(value - 32)` seconds. It wrote `0x20`, `0x21`, `0x25` or `0x2b` into a field the RFC requires to be zero, and could not write that zero: a conformant `SOLUTION` with `Reserved` `0x00` parsed to `timedelta(0)` and could not be re-serialised, escaping a bare `ValueError` from `math.log2(0.0)` that was not a `BaseError`. The field is renamed `reserved`, defaults to zero, and is carried verbatim across a round trip ([#654](https://github.com/JarryShaw/PyPCAPKit/issues/654)). And neither builder read its own `version` keyword, so `version=1` and `version=2` computed identical lengths, where [RFC 5201 Section 5.2.4](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.4) and [RFC 5201 Section 5.2.5](https://datatracker.ietf.org/doc/html/rfc5201#section-5.2.5) state both fields as literally 8 bytes and `Length` as literally 12 and 20. Under HIPv1 each builder accepted only a `bit_length()` of 57..64 and built, for anything narrower, a parameter this library's reader rejects -- the shape of [#608](https://github.com/JarryShaw/PyPCAPKit/issues/608), in the `PUZZLE` builder that [#629](https://github.com/JarryShaw/PyPCAPKit/pull/629) never opened ([#655](https://github.com/JarryShaw/PyPCAPKit/issues/655)). The two remaining `math.log2` sites, both in `PUZZLE` where a lifetime is real, raise `ProtocolError` (`(BaseError, ValueError)`, so a caller written around the bare exception still catches it). Migration: `SolutionParameter.lifetime` is now `reserved` and an `int` rather than a `timedelta`; `PuzzleParameter` and `SolutionParameter` gained a required `rhash_len`; and `_make_param_solution` no longer takes `lifetime=`. All three were reachable end to end only once [#608](https://github.com/JarryShaw/PyPCAPKit/issues/608) was fixed in [#629](https://github.com/JarryShaw/PyPCAPKit/pull/629), which removed the parity guard that had failed these rebuilds loudly; the round trip then stopped raising and started quietly emitting a different parameter, which is why they were fixed together ([#653](https://github.com/JarryShaw/PyPCAPKit/issues/653), [#654](https://github.com/JarryShaw/PyPCAPKit/issues/654), [#655](https://github.com/JarryShaw/PyPCAPKit/issues/655)). -- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding aligned the *contents* rather than the record. Corrected for **45 of the 46** HIP parameters; `LOCATOR_SET` is excluded, below. **This changes the octets those 45 parameters write, and the `length` their data models report.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires every TLV parameter to have "a length (that includes the Type and Length fields), which is a multiple of 8 bytes" and gives `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites computed `(8 - (Length % 8)) % 8`, which aligns the contents alone. For every `Length` from 0 to 63 the result was never a multiple of eight and never the RFC's value, erring both ways: at `Length = 4` (a whole `SEQ`, which [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts on every `UPDATE`) pcapkit appended four octets that must not be there, and at `Length = 8` it appended none and went out four short. A conformant peer lands mid-parameter; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test could see it and why the fix is asserted against the RFC's arithmetic. 94 of the 95 sites (45 of 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and 49 of 49 record lengths in `pcapkit/protocols/internet/hip.py`) are now one `parameter_total_len` and one `parameter_padding_len` ([#651](https://github.com/JarryShaw/PyPCAPKit/issues/651)). `LOCATOR_SET` kept the old expression because two defects there cancelled exactly: its padding callback never received the parameter's `len` (the nested `Locator` schemas share a packet context whose own `len` shadowed it, so it saw 4 for an IPv6 locator), and the parameter's `len` was written in 4-octet units where the RFC's `Length` is a byte count; the result, `4 + 24n + 4`, equalled the RFC total `24n + 8`. [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) later fixed both sites. `EncryptedParameter`'s `data` length callback is fixed in the same change: it subtracted the sixteen `iv` octets but not the four `reserved` ones, which cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}`, so correcting the padding alone would have made `ENCRYPTED` four octets too long at all eight residues. `HIP.make`'s `len = total_length // 8 + 4` is not part of the defect: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length in 8-byte units excluding the first 8 bytes, so the floor division is exact once each parameter is a multiple of eight. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and the other 44 move likewise, so code comparing stored output byte for byte or asserting on `Data_*Parameter.length` sees the RFC's values. `LOCATOR_SET` is unchanged in both respects until [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) (below). `examples/generators/options.py`'s `HIP_COPIES` stayed at two for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had masked, until [#672](https://github.com/JarryShaw/PyPCAPKit/issues/672) widened `counter` and [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) fixed `LOCATOR_SET`'s `Length` unit, after which [#689](https://github.com/JarryShaw/PyPCAPKit/issues/689) dropped `HIP_COPIES` to one ([#651](https://github.com/JarryShaw/PyPCAPKit/issues/651)). +- every HIP parameter pcapkit emitted was `4 (mod 8)` octets long, because the padding aligned the *contents* rather than the record. Corrected for **45 of the 46** HIP parameters; `LOCATOR_SET` is excluded, below. **This changes the octets those 45 parameters write, and the** `length` **their data models report.** [RFC 7401 Section 5.2.1](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.1) requires every TLV parameter to have "a length (that includes the Type and Length fields), which is a multiple of 8 bytes" and gives `Total Length = 11 + Length - (Length + 3) % 8`; all 95 padding sites computed `(8 - (Length % 8)) % 8`, which aligns the contents alone. For every `Length` from 0 to 63 the result was never a multiple of eight and never the RFC's value, erring both ways: at `Length = 4` (a whole `SEQ`, which [RFC 7401 Section 5.3.5](https://datatracker.ietf.org/doc/html/rfc7401#section-5.3.5) puts on every `UPDATE`) pcapkit appended four octets that must not be there, and at `Length = 8` it appended none and went out four short. A conformant peer lands mid-parameter; pcapkit did not, because its reader consumed the same wrong count its writer wrote, which is why no round-trip test could see it and why the fix is asserted against the RFC's arithmetic. 94 of the 95 sites (45 of 46 `PaddingField` callbacks in `pcapkit/protocols/schema/internet/hip.py` and 49 of 49 record lengths in `pcapkit/protocols/internet/hip.py`) are now one `parameter_total_len` and one `parameter_padding_len` ([#651](https://github.com/JarryShaw/PyPCAPKit/issues/651)). `LOCATOR_SET` kept the old expression because two defects there cancelled exactly: its padding callback never received the parameter's `len` (the nested `Locator` schemas share a packet context whose own `len` shadowed it, so it saw 4 for an IPv6 locator), and the parameter's `len` was written in 4-octet units where the RFC's `Length` is a byte count; the result, `4 + 24n + 4`, equalled the RFC total `24n + 8`. [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) later fixed both sites. `EncryptedParameter`'s `data` length callback is fixed in the same change: it subtracted the sixteen `iv` octets but not the four `reserved` ones, which cancelled the padding's four at `Length % 8` in `{0, 5, 6, 7}`, so correcting the padding alone would have made `ENCRYPTED` four octets too long at all eight residues. `HIP.make`'s `len = total_length // 8 + 4` is not part of the defect: [RFC 7401 Section 5.1.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.1.3) defines Header Length in 8-byte units excluding the first 8 bytes, so the floor division is exact once each parameter is a multiple of eight. Migration: a `SEQ` parameter's `length` is now 8 where it was 12, and the other 44 move likewise, so code comparing stored output byte for byte or asserting on `Data_*Parameter.length` sees the RFC's values. `LOCATOR_SET` is unchanged in both respects until [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) (below). `examples/generators/options.py`'s `HIP_COPIES` stayed at two for `R1_COUNTER`'s four-octet `counter` where [RFC 7401 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7401#section-5.2.3) requires eight, which this defect had masked, until [#672](https://github.com/JarryShaw/PyPCAPKit/issues/672) widened `counter` and [#679](https://github.com/JarryShaw/PyPCAPKit/issues/679) fixed `LOCATOR_SET`'s `Length` unit, after which [#689](https://github.com/JarryShaw/PyPCAPKit/issues/689) dropped `HIP_COPIES` to one ([#651](https://github.com/JarryShaw/PyPCAPKit/issues/651)). - two HTTP/2 flag defects, one on each side of a round trip. **This changes the octets a reconstructed DATA frame emits, and the dumped value of a flagless frame's flags.** `_make_http_data` was the only one of the six `_make_http_*` methods that never read `frame.flags`, so a DATA frame parsed with `END_STREAM` set rebuilt with the bit clear (flags octet `0x01` to `0x00`): a well-formed frame saying the stream continues where the capture said it ended [[RFC 9113 Section 6.1](https://datatracker.ietf.org/doc/html/rfc9113#section-6.1)]. `PADDED` was re-derived from `pad_len`, and the five sibling builders already restored theirs, so this was a missing read-back ([#652](https://github.com/JarryShaw/PyPCAPKit/issues/652)). Separately, `FrameType.post_process` seeded its flag accumulator with a bare `0`, which `|=` promotes only as a side effect, so a frame with **no** bit set left `__flags__` a plain `int` where the schema and data model declare a `Flags`, and `END_STREAM in __value__` raised `TypeError: argument of type 'int' is not a container or iterable` rather than answering `False`. That is the common case: a `0x00` flags octet is every opening `SETTINGS`, every non-final `DATA` and every non-ACK `PING`. It now seeds `self.Flags(0)`, as the construct path already did at all six of its sites. In the dump, `__value__` was a JSON *number* for a flagless frame and a *string* for every other frame in the same capture; it is consistently a string now, `"Flags::None [0]"` where it read `0`, the same change [#616](https://github.com/JarryShaw/PyPCAPKit/issues/616) made to TCP's `connection`. The seed is guarded because `FrameType.Flags` declares no members and, from Python 3.11, a memberless `enum.Flag` subclass refuses `Flags(0)`, so the five frame schemas that inherit it (`UnassignedFrame`, `PriorityFrame`, `RSTStreamFrame`, `GoawayFrame`, `WindowUpdateFrame`) keep the plain `int`, which nothing observes since all five pass `flags=None`. A DATA round trip is **still** lossy for an unrelated reason found while measuring: three payload length callbacks are mis-parenthesised, so an unpadded `DATA`, `HEADERS` or `PUSH_PROMISE` frame parses with its whole payload dropped ([#668](https://github.com/JarryShaw/PyPCAPKit/issues/668)). ([#652](https://github.com/JarryShaw/PyPCAPKit/issues/652), [#650](https://github.com/JarryShaw/PyPCAPKit/issues/650)). - four MPTCP error messages carried a doubled separator, rendering as `TCP: : [OptNo 30] 1: invalid flags combination`. Cosmetic: the exception type, option number and subtype were right and nothing parses these strings, but an empty field reads as a value that failed to interpolate. Four sites (`_read_mptcp_join` and `_make_mptcp_join` for the invalid flag combination, both guards of `_make_mptcp_dss` for missing required fields), against 28 messages in the same file spelled correctly; now 32 correct against 0 doubled. Pre-existing since 2023-04-10. The new test asserts each message in full, since the existing `assertIn` on the body passes either side of the change ([#649](https://github.com/JarryShaw/PyPCAPKit/issues/649)). - three names appeared in string annotations their own module never imported, so a type checker or documentation build could not resolve them while every test passed: `Any` in `pcapkit/utilities/logging.py`, and in `pcapkit/protocols/schema/internet/ipv6_route.py` both `Protocol` in a `payload:` stub and `Optional` inside a `typing.cast`. All three are now in their module's `TYPE_CHECKING` block, `Protocol` spelled `ProtocolBase as Protocol` as twenty-one sibling schema modules already do. The `cast` case is the one worth naming: `typing.cast` never evaluates its first argument, so no test that runs the line can see a bad name in it. Nothing resolves at runtime that did not before (`TYPE_CHECKING` is `False`); making these annotations resolve at runtime is a separate decision about the idiom. A new `tests/project/test_annotation_names.py` pins the invariant without a type checker: it resolves every string annotation in the package against the names its own module binds, following nested forward references such as `'list["Nested"]'` and treating `Literal` members and `Annotated` metadata as values, and reports exactly those three findings on the unfixed tree. The same change corrects the `MANIFEST.in` comment asserting that `include README.md` was the only thing putting the README in a source distribution and that an sdist without it could not be installed; both halves are false, and the [#619](https://github.com/JarryShaw/PyPCAPKit/pull/619) entry above carried the same claim and is corrected with it ([#642](https://github.com/JarryShaw/PyPCAPKit/issues/642)). @@ -222,12 +270,40 @@ This resolves [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548), which r - **a breaking change to** the SMF_DPD option's maker in `HOPOPT` and `IPv6_Opts`: it overwrote the parsed DPD mode, so H-DPD rebuilt as I-DPD, and inferred the TaggerID type from the Tagger ID's length, so a 4-octet `DEFAULT` Tagger ID rebuilt as IPv4. Both now come from the parsed option, or from `mode` and a new `tid_type` keyword. A Tagger ID that does not fit its type raises `ProtocolError`, so a 17-octet one does where it raised `FieldValueError`, and `tid=b''` builds a NULL Tagger ID where it raised `ValueError` ([#1190](https://github.com/JarryShaw/PyPCAPKit/issues/1190)). An H-DPD option with an empty Hash Assist Value raises `ProtocolError` where it raised a bare `IndexError` ([#1193](https://github.com/JarryShaw/PyPCAPKit/issues/1193)). - a `ForwardMatchField` that read short at the end of the stream rewound by its declared width, not by the octets it read, so the fields after it re-read the preceding octet. An SMF_DPD option in the last two octets of a `HOPOPT` or `IPv6_Opts` header was re-read from the padding and raised a bare `TypeError`; it now raises `FieldValueError`, as CALIPSO and MPL do. A full read rewinds as before ([#1195](https://github.com/JarryShaw/PyPCAPKit/issues/1195)). - `HIP()` with no parameters wrote Header Length 0, below [RFC 7401](https://datatracker.ietf.org/doc/html/rfc7401) section 5.1's minimum of 4 for the two HITs, so it failed to read back with `ProtocolError`; it now writes 4. `read` names a Header Length below 4 in a new `invalid header length` message. Lengths 0 to 3 already raised `ProtocolError`, so only the message changes ([#1189](https://github.com/JarryShaw/PyPCAPKit/issues/1189)). +- the RPL option, type `0x23`, raised `AttributeError` in `HOPOPT` and `IPv6_Opts` ([#1196](https://github.com/JarryShaw/PyPCAPKit/issues/1196), [#1198](https://github.com/JarryShaw/PyPCAPKit/pull/1198)). +- **a breaking change to** the RPL option: it keeps its reserved flag bits on rebuild and reads [RFC 6553](https://datatracker.ietf.org/doc/html/rfc6553) sub-TLVs ([#1199](https://github.com/JarryShaw/PyPCAPKit/issues/1199), [#1201](https://github.com/JarryShaw/PyPCAPKit/pull/1201)). +- `make()` rejected an enum member name given as `str` unless a namespace was passed ([#1210](https://github.com/JarryShaw/PyPCAPKit/issues/1210), [#1227](https://github.com/JarryShaw/PyPCAPKit/pull/1227)); raw items in a `make()` list build a packet, and `str` member names resolve per argument ([#1352](https://github.com/JarryShaw/PyPCAPKit/pull/1352)). +- **a breaking change to** rebuilding a parsed packet, the round-trip programme: `from_data(info)` gives back the packet's own octets. Most of these fixes add the `make()` keywords and `info` keys the rebuild needs, which is what breaks: + +- ARP, InARP and DRARP, any Ethernet `hlen`, and a `ptype`/`plen` mismatch ([#1224](https://github.com/JarryShaw/PyPCAPKit/pull/1224)); L2TPv2 reserved flag bits, and a lone `ns` or `nr` in `make()` ([#1226](https://github.com/JarryShaw/PyPCAPKit/pull/1226)); octets past the L2TPv2 and OSPF Length fields ([#1455](https://github.com/JarryShaw/PyPCAPKit/issues/1455), [#1460](https://github.com/JarryShaw/PyPCAPKit/pull/1460)). +- SCTP chunk flags, reserved fields and final padding ([#1253](https://github.com/JarryShaw/PyPCAPKit/pull/1253)), octets after the last whole chunk ([#1468](https://github.com/JarryShaw/PyPCAPKit/issues/1468), [#1472](https://github.com/JarryShaw/PyPCAPKit/pull/1472)), a last chunk short of its padding ([#1474](https://github.com/JarryShaw/PyPCAPKit/issues/1474), [#1477](https://github.com/JarryShaw/PyPCAPKit/pull/1477)), a final parameter or cause short of its padding ([#1479](https://github.com/JarryShaw/PyPCAPKit/issues/1479), [#1481](https://github.com/JarryShaw/PyPCAPKit/pull/1481)), and a Host Name with no NUL terminator ([#1482](https://github.com/JarryShaw/PyPCAPKit/issues/1482), [#1483](https://github.com/JarryShaw/PyPCAPKit/pull/1483)). +- TCP: MPTCP options ([#1254](https://github.com/JarryShaw/PyPCAPKit/pull/1254)); reserved bits, octets after EOOL and the UTO granularity ([#1348](https://github.com/JarryShaw/PyPCAPKit/pull/1348)); and a header length declared past the data, with IPv4 ([#1411](https://github.com/JarryShaw/PyPCAPKit/pull/1411)). +- UDP: octets past the Length field ([#1461](https://github.com/JarryShaw/PyPCAPKit/issues/1461), [#1467](https://github.com/JarryShaw/PyPCAPKit/pull/1467)). +- FTP and HTTP/1 lines, byte for byte ([#1301](https://github.com/JarryShaw/PyPCAPKit/pull/1301)), keeping the raw command case, HTTP/1 whitespace and field octets, with whole multi-line FTP replies parsed ([#1347](https://github.com/JarryShaw/PyPCAPKit/pull/1347)). HTTP/2 frames lose no octets ([#1341](https://github.com/JarryShaw/PyPCAPKit/pull/1341)). +- OSPF keeps the parsed Packet Length, and NGAP refuses octets after the NGAP-PDU ([#1304](https://github.com/JarryShaw/PyPCAPKit/pull/1304)); OSPF cryptographic authentication rebuilds from its dict form ([#1464](https://github.com/JarryShaw/PyPCAPKit/issues/1464), [#1469](https://github.com/JarryShaw/PyPCAPKit/pull/1469)). +- IPv4 option areas, route slots, timestamps and reserved bits ([#1344](https://github.com/JarryShaw/PyPCAPKit/pull/1344)); the TS option's slots beyond the pointer ([#1349](https://github.com/JarryShaw/PyPCAPKit/issues/1349), [#1390](https://github.com/JarryShaw/PyPCAPKit/pull/1390)); TS and SEC options byte for byte ([#1399](https://github.com/JarryShaw/PyPCAPKit/pull/1399)). +- IPv6 core headers ([#1307](https://github.com/JarryShaw/PyPCAPKit/pull/1307)); `HOPOPT` and `IPv6_Opts` option areas ([#1310](https://github.com/JarryShaw/PyPCAPKit/pull/1310)); a CALIPSO option's trailing pad ([#1403](https://github.com/JarryShaw/PyPCAPKit/issues/1403), [#1407](https://github.com/JarryShaw/PyPCAPKit/pull/1407)), and its `make()` writing a corrupt length on overflow ([#1408](https://github.com/JarryShaw/PyPCAPKit/issues/1408), [#1409](https://github.com/JarryShaw/PyPCAPKit/pull/1409)); an extension header parsed with `extension=True` ([#1446](https://github.com/JarryShaw/PyPCAPKit/issues/1446), [#1449](https://github.com/JarryShaw/PyPCAPKit/pull/1449)); RPL source routes ([#1463](https://github.com/JarryShaw/PyPCAPKit/pull/1463)); and a cut extension-header chain ([#1476](https://github.com/JarryShaw/PyPCAPKit/pull/1476)). +- octets past IPv4 Total Length and IPv6 Payload Length ([#1209](https://github.com/JarryShaw/PyPCAPKit/issues/1209), [#1431](https://github.com/JarryShaw/PyPCAPKit/pull/1431)), and past the IPX Length field ([#1433](https://github.com/JarryShaw/PyPCAPKit/issues/1433), [#1435](https://github.com/JarryShaw/PyPCAPKit/pull/1435)). IPX `from_data` raised `struct.error`, and AH wrote its reserved field back as zero ([#1338](https://github.com/JarryShaw/PyPCAPKit/pull/1338)). +- HIP: Header Length, an empty parameter list, the PUZZLE lifetime and reserved bits ([#1342](https://github.com/JarryShaw/PyPCAPKit/pull/1342)); Locator flags ([#1365](https://github.com/JarryShaw/PyPCAPKit/issues/1365), [#1367](https://github.com/JarryShaw/PyPCAPKit/pull/1367)); three `BitField` namespaces' reserved bits ([#1417](https://github.com/JarryShaw/PyPCAPKit/issues/1417), [#1419](https://github.com/JarryShaw/PyPCAPKit/pull/1419)); and the two fixed header bits ([#1459](https://github.com/JarryShaw/PyPCAPKit/issues/1459), [#1462](https://github.com/JarryShaw/PyPCAPKit/pull/1462)). +- MH: zero-length PadN, reserved fields and BU/BA flag bits ([#1324](https://github.com/JarryShaw/PyPCAPKit/issues/1324), [#1345](https://github.com/JarryShaw/PyPCAPKit/pull/1345)); the reserved bits of 24 `BitField` flag namespaces ([#1418](https://github.com/JarryShaw/PyPCAPKit/issues/1418), [#1421](https://github.com/JarryShaw/PyPCAPKit/pull/1421)); and the reserved octets `PaddingField` sites capture ([#1442](https://github.com/JarryShaw/PyPCAPKit/pull/1442)). +- PCAP-NG: constructed and rebuilt blocks ([#1339](https://github.com/JarryShaw/PyPCAPKit/pull/1339)), lengths and padding ([#1373](https://github.com/JarryShaw/PyPCAPKit/pull/1373)), blocks and options as parsed ([#1378](https://github.com/JarryShaw/PyPCAPKit/pull/1378)), further follow-ups ([#1412](https://github.com/JarryShaw/PyPCAPKit/pull/1412)), malformed journal entries and short packet blocks ([#1420](https://github.com/JarryShaw/PyPCAPKit/pull/1420)), an SHB or IDB shorter than its fixed fields ([#1422](https://github.com/JarryShaw/PyPCAPKit/issues/1422), [#1425](https://github.com/JarryShaw/PyPCAPKit/pull/1425)), and captured padding octets ([#1441](https://github.com/JarryShaw/PyPCAPKit/pull/1441)). +- a cut PCAP header or PCAP-NG block ([#1470](https://github.com/JarryShaw/PyPCAPKit/issues/1470), [#1475](https://github.com/JarryShaw/PyPCAPKit/pull/1475)). + +- `Schema.pack` keeps option padding, `SchemaField` spans and iterated keys ([#1374](https://github.com/JarryShaw/PyPCAPKit/pull/1374)); a `SchemaField`'s unread octets survive `from_data` rather than rebuilding zero-filled ([#1380](https://github.com/JarryShaw/PyPCAPKit/issues/1380), [#1397](https://github.com/JarryShaw/PyPCAPKit/pull/1397)); a header cut inside a fixed-width field rebuilds as captured ([#1465](https://github.com/JarryShaw/PyPCAPKit/pull/1465)); and each non-final `Schema` subclass is finalised on its own state ([#1498](https://github.com/JarryShaw/PyPCAPKit/issues/1498), [#1499](https://github.com/JarryShaw/PyPCAPKit/pull/1499)). +- `from_data` rebuilds the payload from `info.to_dict()` ([#1447](https://github.com/JarryShaw/PyPCAPKit/issues/1447), [#1452](https://github.com/JarryShaw/PyPCAPKit/pull/1452)). +- building a PCAP frame with `snaplen` 0 wrote a record with no packet data ([#1385](https://github.com/JarryShaw/PyPCAPKit/issues/1385), [#1410](https://github.com/JarryShaw/PyPCAPKit/pull/1410)); an out-of-range `if_tzone` is kept as captured and every `fromtimestamp` failure is caught ([#1340](https://github.com/JarryShaw/PyPCAPKit/issues/1340), [#1429](https://github.com/JarryShaw/PyPCAPKit/pull/1429)). +- registration: the parser registrars warn only when the entry really changes ([#1362](https://github.com/JarryShaw/PyPCAPKit/issues/1362), [#1400](https://github.com/JarryShaw/PyPCAPKit/pull/1400)); re-registering a stored `ModuleDescriptor` keeps it ([#1364](https://github.com/JarryShaw/PyPCAPKit/issues/1364), [#1415](https://github.com/JarryShaw/PyPCAPKit/pull/1415)); restoring a registration puts back the shipped descriptor, including through `PCAPNG.register` ([#1533](https://github.com/JarryShaw/PyPCAPKit/pull/1533), [#1573](https://github.com/JarryShaw/PyPCAPKit/pull/1573)); and unregistering a custom protocol removes its name from `__proto__` ([#1505](https://github.com/JarryShaw/PyPCAPKit/issues/1505), [#1530](https://github.com/JarryShaw/PyPCAPKit/pull/1530)). +- fragments: an IPv6 extension header behind a first fragment is dissected as without it ([#1539](https://github.com/JarryShaw/PyPCAPKit/issues/1539), [#1540](https://github.com/JarryShaw/PyPCAPKit/pull/1540)), and a later IPv4 or IPv6 fragment's data is raw, not an upper-layer header ([#1545](https://github.com/JarryShaw/PyPCAPKit/issues/1545), [#1552](https://github.com/JarryShaw/PyPCAPKit/pull/1552), [#1553](https://github.com/JarryShaw/PyPCAPKit/issues/1553), [#1562](https://github.com/JarryShaw/PyPCAPKit/pull/1562)). +- IPv4: a Total Length of 0 is read as TSO, and one inside the header as bogus ([#1547](https://github.com/JarryShaw/PyPCAPKit/issues/1547), [#1554](https://github.com/JarryShaw/PyPCAPKit/pull/1554)). +- IPv6: a jumbogram's payload is dissected up to its Jumbo Payload Length ([#1434](https://github.com/JarryShaw/PyPCAPKit/issues/1434), [#1440](https://github.com/JarryShaw/PyPCAPKit/pull/1440)), a jumbogram [RFC 2675](https://datatracker.ietf.org/doc/html/rfc2675) calls an error is rejected ([#1687](https://github.com/JarryShaw/PyPCAPKit/issues/1687), [#1713](https://github.com/JarryShaw/PyPCAPKit/pull/1713)), and so is a header whose Version is not 6 ([#1680](https://github.com/JarryShaw/PyPCAPKit/issues/1680), [#1723](https://github.com/JarryShaw/PyPCAPKit/pull/1723)). +- the warnings of a next layer replaced by `Raw` are dropped ([#1580](https://github.com/JarryShaw/PyPCAPKit/issues/1580), [#1588](https://github.com/JarryShaw/PyPCAPKit/pull/1588)). ### pcapkit.toolkit #### Added - `util/pyshark_encap_map.py`, a generator that regenerates `ENCAP_TYPE_TO_LINKTYPE` (152 entries) and `FILTER_NAME_TO_LINKTYPE` (58) in place inside `pcapkit/toolkit/pyshark.py`, so the two tables [#850](https://github.com/JarryShaw/PyPCAPKit/pull/850) hand-built stop being hand-maintained. It lives under `util/` rather than `pcapkit/vendor/` because `Vendor`'s base class is hardwired to an HTTP fetch and a generated enum-class file, where this generator's source of truth is the local `tshark`/`editcap` binaries. It guards against its worst failure mode: a naive `LinkType(dlt)` cannot detect an unmapped DLT, because `_missing_` mints a placeholder rather than raising, so the generator snapshots every known `LinkType` value first and calls `LinkType(dlt)` only once a value is confirmed a member. The real-`tshark` sweep is gated on a new `HAS_WIRESHARK` flag, registered in `tests/_dependency_gates.py`'s `NON_DISTRIBUTION_FLAGS` next to `HAS_PROC_FD`, for things pip cannot install. The sweep arithmetic is version-pinned (`editcap -T` accepts 226 encapsulations on Wireshark 4.6.9, 224 on 4.2.2, CI's Ubuntu noble), so count assertions run only under exactly the measured version, with a separate `VersionIndependentInvariantTests` class for what holds regardless. One line of `pcapkit/toolkit/pyshark.py` changes: the `IPMB_LINUX` entry becomes `I2C_LINUX`, the canonical name [#848](https://github.com/JarryShaw/PyPCAPKit/pull/848) gave value 209 -- the same member either way ([#853](https://github.com/JarryShaw/PyPCAPKit/pull/853)). +- `tcp_segment` and its result type are public in `pcapkit.toolkit.pcap` ([#1570](https://github.com/JarryShaw/PyPCAPKit/issues/1570), [#1579](https://github.com/JarryShaw/PyPCAPKit/pull/1579)). #### Fixed @@ -236,12 +312,17 @@ This resolves [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548), which r - `pypcapfile`'s `_read_a_packet(layers=0)` hexlifies a whole frame into ASCII text and stops, but `_decode()` fed that into `Ethernet(packet.packet, layers=1)`, whose `__init__` unpacks the header with `struct.unpack`, so every field came out garbage without raising. Fixed with `binascii.unhexlify(packet.packet)` ahead of the decoder call, in preference to decoding eagerly at `layers>0` inside the lazy generator, which would lose `_decode()`'s per-frame `AttributeWarning` fallback. Link-layer dispatch resolves from the header's `ll_type` and is unaffected. Landed with [#743](https://github.com/JarryShaw/PyPCAPKit/issues/743): standalone, this turns a non-crashing extraction into a crashing one, since `Extractor.run()` catches only `EOFError`, `StopIteration` and `KeyboardInterrupt` and the `AddressValueError`/`struct.error` [#743](https://github.com/JarryShaw/PyPCAPKit/issues/743) fixes escape until both land ([#746](https://github.com/JarryShaw/PyPCAPKit/issues/746)). - **a breaking change to** pyshark link-type resolution in `tcp_traceflow`: an unrecognised encapsulation or filter name now raises `MissingKeyError` instead of substituting a plausible DLT. `pcapkit/toolkit/pyshark.py` resolved a frame's link type from `packet.layers[0].layer_name`, a Wireshark display-filter name, with a blanket `Enum_LinkType.get(name.upper())` fallback. One filter name serves several encapsulations, so that fallback returned *wrong DLTs silently* (`null` for a `DLT_LOOP` capture, `101` (`RAW`) for `rawip6`). Re-keyed onto `frame.encap_type`, which is distinct per encapsulation, and the fallback is gone. The two tables were measured rather than transcribed (Wireshark's source is not on the build host): each entry round-tripped through `editcap -F pcap -T ` and read back with `tshark`. `ENCAP_TYPE_TO_LINKTYPE` grows to 152 entries with no DLT serving two keys, and `FILTER_NAME_TO_LINKTYPE` from [#838](https://github.com/JarryShaw/PyPCAPKit/pull/838)'s two entries to 58, every root filter name single-DLT across the 157 (of 226) encapsulations this host's Wireshark can write. `pcapkit/toolkit/pyshark.py` reaches 100% coverage under new tests that fail 26 ways against the unfixed tree; live pyshark parsing stays unexercised, since `pyshark.FileCapture` does not run on Python 3.14 ([#850](https://github.com/JarryShaw/PyPCAPKit/pull/850)). - `pypcapfile`'s `tcp_reassembly` set `last` one past the segment. `TCP_Packet.last` is inclusive, and the four other adapters set `seqnum + len(payload) - 1`, as this one now does too ([#1100](https://github.com/JarryShaw/PyPCAPKit/issues/1100)). +- scapy's `packet2dict` dumped nested packets as `{}` and mutated its input ([#1287](https://github.com/JarryShaw/PyPCAPKit/pull/1287)); the dpkt reassembly adapters build their data from the record's own octets rather than dpkt's re-serialised bytes, and leave the packet alone ([#1297](https://github.com/JarryShaw/PyPCAPKit/issues/1297), [#1335](https://github.com/JarryShaw/PyPCAPKit/pull/1335)). +- the TCP adapters: pyshark accepts tshark's `True`/`False` TCP flags ([#1514](https://github.com/JarryShaw/PyPCAPKit/issues/1514), [#1527](https://github.com/JarryShaw/PyPCAPKit/pull/1527)); a TCP segment whose header the snapshot cut is traced and reassembled ([#1518](https://github.com/JarryShaw/PyPCAPKit/issues/1518), [#1556](https://github.com/JarryShaw/PyPCAPKit/pull/1556)); dpkt reads SYN, FIN and RST whatever the NS bit ([#1618](https://github.com/JarryShaw/PyPCAPKit/issues/1618), [#1621](https://github.com/JarryShaw/PyPCAPKit/pull/1621)); and the scapy adapters stop at the IP datagram's declared length ([#1584](https://github.com/JarryShaw/PyPCAPKit/issues/1584), [#1589](https://github.com/JarryShaw/PyPCAPKit/pull/1589)). +- timestamps: dpkt and pypcapfile hand over the correctly rounded timestamp ([#1668](https://github.com/JarryShaw/PyPCAPKit/issues/1668), [#1690](https://github.com/JarryShaw/PyPCAPKit/pull/1690)), and a copied `DecimalTimestamp` keeps `orig_len` under `_pydecimal` ([#1647](https://github.com/JarryShaw/PyPCAPKit/issues/1647), [#1653](https://github.com/JarryShaw/PyPCAPKit/pull/1653)). +- the third-party engines are best effort: the code each adapter carried only to match the default engine is reverted ([#1718](https://github.com/JarryShaw/PyPCAPKit/pull/1718), [#1721](https://github.com/JarryShaw/PyPCAPKit/pull/1721), [#1724](https://github.com/JarryShaw/PyPCAPKit/pull/1724), [#1727](https://github.com/JarryShaw/PyPCAPKit/pull/1727), closing [#1708](https://github.com/JarryShaw/PyPCAPKit/issues/1708) through [#1743](https://github.com/JarryShaw/PyPCAPKit/pull/1743)). What each engine leaves out, such as TCP behind a tunnel or an IPv6 extension header, is documented rather than warned about; the pypcapfile engine still warns once per capture that it leaves TCP over IPv6 out ([#1513](https://github.com/JarryShaw/PyPCAPKit/issues/1513), [#1535](https://github.com/JarryShaw/PyPCAPKit/pull/1535)). ### pcapkit.utilities #### Added - `pcapkit.utilities.logging` as a real interface: `get_logger()` for per-module children, `configure()` to set level, handler, stream, format or propagation at runtime, `reset()` to return to library-neutral, and `ensure_output()`. Seventeen modules now log under their own `__name__`, so a consumer can silence `pcapkit.foundation.registry` while keeping `pcapkit.foundation.extraction` ([#384](https://github.com/JarryShaw/PyPCAPKit/pull/384)). +- `pcapkit.utilities.config`, one module for every tunable limit, each read once from a `PCAPKIT_*` environment variable: the frame-layer and extension-header limits, the dumpkit fallback, the charset-detection cache, the read chunk and the three zero-padding guards. A bad value warns with `ConfigWarning` and falls back to the default; `-1` lifts a limit. **The old homes are removed**, with no re-export ([#1726](https://github.com/JarryShaw/PyPCAPKit/issues/1726), [#1743](https://github.com/JarryShaw/PyPCAPKit/pull/1743)). #### Changed @@ -251,6 +332,8 @@ This resolves [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548), which r #### Fixed - two more gaps the [#514](https://github.com/JarryShaw/PyPCAPKit/issues/514) keyword audit turned up, neither covered by a test: `StreamEOFError`'s docstring did not say that `@prepare` always raises it with `quiet=True` (the end-of-stream convention `StructError` follows via its own `eof=True`), so nothing pinned that silence against regression; and `register_extractor_engine`'s real keyword, `name`, was not itself under test, only its corrected docstring, so a rename could put the two out of step again ([#577](https://github.com/JarryShaw/PyPCAPKit/pull/577)). +- `hold_warnings` is in `pcapkit.utilities.warnings.__all__` ([#1602](https://github.com/JarryShaw/PyPCAPKit/issues/1602), [#1603](https://github.com/JarryShaw/PyPCAPKit/pull/1603)). +- the default threading excepthook is the fallback before Python 3.10, which has no `threading.__excepthook__` ([#1691](https://github.com/JarryShaw/PyPCAPKit/issues/1691), [#1701](https://github.com/JarryShaw/PyPCAPKit/pull/1701)). ### pcapkit.vendor @@ -258,9 +341,11 @@ This resolves [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548), which r - thirteen `re.sub` sites under `pcapkit/vendor/` passed `re.MULTILINE` as the fourth *positional* argument, which is `count`, not `flags`: substitution capped at 8 matches, and a `DeprecationWarning` on Python 3.13+ (a future `TypeError`). Found by an AST sweep rather than grep, since 8 of the 13 put the flag on a continuation line. The flag moves to `flags=` at all 13 sites (`default.py` plus one each in `hip/eddsa_curve.py`, `http/frame.py`, `http/method.py`, `http/status_code.py`, `ipv4/router_alert.py`, `ipv6/option.py`, `ipv6/router_alert.py`, `ipv6/tagger_id.py`, `reg/ethertype.py`, `tcp/flags.py`, `tcp/mp_tcp_option.py` and `tcp/option.py`). Behaviour preserving: the shared pattern (`r'\r*\n'`) has no MULTILINE-sensitive anchor, so no `const/` regeneration was needed ([#796](https://github.com/JarryShaw/PyPCAPKit/issues/796)). - `main` was red ([#866](https://github.com/JarryShaw/PyPCAPKit/issues/866)): 12 tests across three files, all `RecursionError` (`tests/const/test_const_enum_lookup.py`; two `test_const_enum_no_mint.py` methods, two registries each; 7 of the 12 in `tests/protocols/misc/test_pcapng_unit.py`). `pcapkit/vendor/pcapng/record_type.py` and `secrets_type.py` rendered a two-line `_missing_` body whose first line did not return: `cls._unregistered_member(value, 'Unassigned')` followed by `return cls(value)`. That was harmless while the first line was `extend_enum(...)`, which registers the member so `cls(value)` found it; [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861) (above) replaced it with `_unregistered_member`, which deliberately does not register, so `cls(value)` missed and re-entered `_missing_`. [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861) fixed the *generated* modules but not the two *crawlers*, so [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861)'s CI stayed green until a live regeneration reintroduced the defect; [#861](https://github.com/JarryShaw/PyPCAPKit/pull/861)'s review had sampled 5 of 82 regenerations, missing these two. Both crawlers now collapse to the single returning line the other 101 registries use, and the two const files are regenerated. A new guard at the crawler-emission layer requires a hard-coded `_missing_` body to end in a return and no emitted line to re-enter `_missing_` for the same value -- deliberately narrower than "every emitted line must return", which `Vendor.process`'s shape legitimately breaks. The guard sees only 35 of 96 crawlers; the other 61 are covered behaviourally. Closes [#866](https://github.com/JarryShaw/PyPCAPKit/issues/866) ([#867](https://github.com/JarryShaw/PyPCAPKit/pull/867)). -- `python -m pcapkit.vendor ` could not fail: `run()` swallowed every crawler exception as a filterable `VendorRuntimeWarning` and `main()` always returned `0`, so a broken crawler regenerating nothing looked like success to a human and to the cron job. Per the owner's ruling (*"only discard changes made by a non-zero sub-vendor... for cron jobs, at most warn about failed sub-vendors, but never fail the entire job"*), three pieces land. **Exit code** (`pcapkit/vendor/__main__.py`): `run()` returns whether its target succeeded; `main()` attempts every target and returns `1` if any raised, writing the failing target's qualified name and exception `repr` to stderr unconditionally via `print()` rather than the filterable warning. **Per-target snapshot and restore**: a new `_snapshot_and_restore` context manager backs up a target's const file with `tempfile.mkstemp` before it runs (resolving the destination through `Vendor._dest_path`, now a `classmethod` in `pcapkit/vendor/default.py` so it can be called before `Vendor.__init__` fetches, renders or writes), restores the backup via `os.replace` on any exception before re-raising, and discards it on success. A target whose destination cannot be resolved (a crawler outside the real tree) runs unprotected rather than failing the snapshot step, since `Vendor.__init__` hits the same resolution failure itself; a destination that does not exist yet is left alone. **Workflow** (`.github/workflows/cron-vendor.yml`): the "Update Vendor" step no longer aborts under `bash -e` when `pcapkit-vendor` exits non-zero. The exit code is captured around that one command, so targets that succeeded are still `isort`ed, diffed, committed and pushed, while a "Warning" block naming the failed target(s) is appended to `$GITHUB_STEP_SUMMARY`, a `::warning::` annotation is emitted per failure, and the step's own exit code is always `0`. A per-target `git checkout -- ` restore step was dropped: with snapshot/restore, a failed target's const file is never touched. Closes [#872](https://github.com/JarryShaw/PyPCAPKit/issues/872) ([#873](https://github.com/JarryShaw/PyPCAPKit/pull/873)). +- `python -m pcapkit.vendor ` could not fail: `run()` swallowed every crawler exception as a filterable `VendorRuntimeWarning` and `main()` always returned `0`, so a broken crawler regenerating nothing looked like success to a human and to the cron job. Per the owner's ruling (*"only discard changes made by a non-zero sub-vendor... for cron jobs, at most warn about failed sub-vendors, but never fail the entire job"*), three pieces land. **Exit code** (`pcapkit/vendor/__main__.py`): `run()` returns whether its target succeeded; `main()` attempts every target and returns `1` if any raised, writing the failing target's qualified name and exception `repr` to stderr unconditionally via `print()` rather than the filterable warning. **Per-target snapshot and restore**: a new `_snapshot_and_restore` context manager backs up a target's const file with `tempfile.mkstemp` before it runs (resolving the destination through `Vendor._dest_path`, now a `classmethod` in `pcapkit/vendor/default.py` so it can be called before `Vendor.__init__` fetches, renders or writes), restores the backup via `os.replace` on any exception before re-raising, and discards it on success. A target whose destination cannot be resolved (a crawler outside the real tree) runs unprotected rather than failing the snapshot step, since `Vendor.__init__` hits the same resolution failure itself; a destination that does not exist yet is left alone. **Workflow** (`.github/workflows/cron-vendor.yml`): the "Update Vendor" step no longer aborts under `bash -e` when `pcapkit-vendor` exits non-zero. The exit code is captured around that one command, so targets that succeeded are still sorted by `isort`, diffed, committed and pushed, while a "Warning" block naming the failed target(s) is appended to `$GITHUB_STEP_SUMMARY`, a `::warning::` annotation is emitted per failure, and the step's own exit code is always `0`. A per-target `git checkout -- ` restore step was dropped: with snapshot/restore, a failed target's const file is never touched. Closes [#872](https://github.com/JarryShaw/PyPCAPKit/issues/872) ([#873](https://github.com/JarryShaw/PyPCAPKit/pull/873)). - three follow-ups from [#873](https://github.com/JarryShaw/PyPCAPKit/pull/873)'s review, none changing merged behaviour: one unpinned invariant and two inaccurate pieces of prose/lint. **1.** `except BaseException:` in `_snapshot_and_restore` is pinned: a test asserts that a `KeyboardInterrupt` still restores the file and propagates. **2.** A false docstring clause is corrected: a crawler whose `_dest_path` is an instance-method-only override fails the *class* call `_snapshot_and_restore` makes but not the *instance* call inside `__init__`, so that target runs unprotected and **succeeds silently**. **3.** The `# pylint: disable=no-else-raise` pragma is **kept**: the belief that R1720 never applies to `try`/`except`/`else` was a message-control artefact (`--disable=all --enable=no-else-raise` does not re-enable the check the way `--enable=R` does), and under the `Makefile`'s flags, which CI runs, pylint flags the `else:`, which is deliberate because it makes the backup cleanup unreachable from the `except`. Left out of scope: an untested `os.close(fd)` leak (harmless, since `copy2` reopens the path) and the semantically identical unindented form of the same `else:`. Closes [#875](https://github.com/JarryShaw/PyPCAPKit/issues/875) ([#876](https://github.com/JarryShaw/PyPCAPKit/pull/876)). - `TLSKeyLabel` tracked the Mozilla NSS key-log wiki, but `draft-ietf-opsawg-pcapng-06` S4.7 now points at `[I-D.ietf-tls-keylogfile]` (draft-05), published as [RFC 9850](https://datatracker.ietf.org/doc/html/rfc9850), "The SSLKEYLOGFILE Format for TLS", whose S4.2 creates the IANA "TLS SSLKEYLOGFILE Labels" registry. Against it, `TLSKeyLabel` was missing `ECH_SECRET` and `ECH_CONFIG` ([RFC 9850](https://datatracker.ietf.org/doc/html/rfc9850) SS2.3/4.2), so `TLSKeyLabel('ECH_SECRET')` raised `ValueError` on a legitimate Decryption Secrets Block; both are added, `ECH_SECRET` keeping the file's `# nosec B105` bandit convention for `*_SECRET` members. `RSA` has no row in [RFC 9850](https://datatracker.ietf.org/doc/html/rfc9850)'s registry (it is an NSS-only label the NSS wiki records as removed in NSS 3.34) and is kept, since removing a public member is breaking, with a docstring comment marking it the historical exception. `pcapkit/vendor/pcapng/secrets_type.py`'s stale `# NSS Key Log Format` comment is repointed at RFC 9850, and `TLSKeyLabel`'s docstring cites it. A new test pins the member set against RFC 9850 plus the `RSA` exception. Closes [#882](https://github.com/JarryShaw/PyPCAPKit/issues/882) ([#883](https://github.com/JarryShaw/PyPCAPKit/pull/883)). +- both spellings of an IANA citation render as `:rfc:` roles in the generated `pcapkit.const` docstrings ([#1664](https://github.com/JarryShaw/PyPCAPKit/issues/1664), [#1673](https://github.com/JarryShaw/PyPCAPKit/pull/1673)). +- `MAX_RETRY` and `CI_MODE` stay in `pcapkit.vendor.default`, read from `PCAPKIT_VENDOR_RETRY` and `PCAPKIT_CI_MODE`; a bad value warns with `ConfigWarning` and falls back, where 1.4.x crashed on a non-integer and read `0` as `1` ([#1745](https://github.com/JarryShaw/PyPCAPKit/pull/1745)). ### Project infrastructure @@ -284,6 +369,8 @@ This resolves [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548), which r - `engine-tests` ran one Python-version leg per interpreter regardless of which engines that interpreter could install, so `pyproject.toml`'s `PyPCAPFile = ["pypcapfile; python_version < '3.12'"]` marker resolved to *nothing* on 3.12--3.14, yet the `Engines Python 3.12/3.13/3.14` legs passed and, once required, gated merges while exercising an engine that was never installed. Rebuilt as a Python x engine matrix (6 interpreters x 6 engines, 36 cells after 10 `include` overrides) with three outcomes: **26** `supported` cells that install through the real extra and must pass; **6** `unsupported` cells (`PyPCAPFile` on 3.12--3.15, `PyShark` on 3.14/3.15) installed by raw `pip install` to bypass the marker and assert the engine's own `PYTHON_CEILING` decline; and **4** `not-installable` cells (`PyPCAP` on 3.12--3.15) where a genuine C-extension build failure is logged and the test step skipped. The install step never passes silently in either direction. 3.15 is `continue-on-error` throughout. Job names change shape (`Engines Python X` becomes `Engines Python X (engine)`), so the branch-protection ruleset's required-check list was updated in the same window, lest a required check that stopped existing block every merge ([#849](https://github.com/JarryShaw/PyPCAPKit/pull/849)). - Ruleset `23497679`'s `required_status_checks` names 22 exact check contexts, and GitHub rulesets match literally with no wildcard, so the list needs hand-editing on every matrix change. Five of the 22, the old `Engines Python ` names, are dead since [#849](https://github.com/JarryShaw/PyPCAPKit/pull/849); the five `Compat Python 3.10`-`3.14` names stay live, emitted by `python-compatibility.yml`, and are untouched. Adds one job, `required-checks` (context `Required checks passed`), depending on `[test, integration, engine-tests, pypcap-parity]` with `if: always() && inputs.gate-only != true` and an explicit per-dependency check that fails loudly, naming the job, on anything but `success`: a bare `needs:` list would leave it *skipped* rather than failed when a dependency failed, and GitHub's docs list a skipped required check as passing. That one context is meant to replace 17 of the ruleset's 22 entries (`test`, `integration`, `Engines`, `pypcap-parity`) once the ruleset is updated separately, a repository setting this PR does not touch; the five `Compat` names are excluded from the replacement. One file, `.github/workflows/unit-tests.yml`, 167 lines added; no library or test code ([#856](https://github.com/JarryShaw/PyPCAPKit/pull/856)). - the prose the post-[#719](https://github.com/JarryShaw/PyPCAPKit/issues/719) defect wave added or touched, checked against the code; no code changes. `MesgIDOption.seconds` in the Mobility Header schema said seconds since 1970 where it holds an [RFC 1305](https://datatracker.ietf.org/doc/html/rfc1305) NTP count from 1900; two `HTTP._guess_version` comments predated the HTTP/2 gate; and the `_parse_next_layer` note miscounted the captured octets. `Extractor._open_output` now says only the PCAP engine passes header fields, `Extractor.run` and the PCAP-NG engine's `run` list the `FormatError` for PCAP output from a PCAP-NG input, and the extraction and engine pages document the PCAP-output path ([#1179](https://github.com/JarryShaw/PyPCAPKit/pull/1179), [#1180](https://github.com/JarryShaw/PyPCAPKit/pull/1180), [#1186](https://github.com/JarryShaw/PyPCAPKit/pull/1186), [#1103](https://github.com/JarryShaw/PyPCAPKit/issues/1103)). +- **a breaking change to** the `all` extra: it carries the core addons only -- `cli`, `crypto` and `NGAP`. The three third-party engines 1.4.x bundled in it (`dpkt`, `scapy` and `pyshark`) and the vendor crawler's `requests` and `beautifulsoup4` are no longer in it; the engines keep their own `DPKT`, `Scapy` and `PyShark` extras and the crawler its `vendor` extra, and `pypcapfile` has a new `PyPCAPFile` extra of its own. A new `dev` extra carries what CI needs ([#910](https://github.com/JarryShaw/PyPCAPKit/issues/910), [#914](https://github.com/JarryShaw/PyPCAPKit/pull/914)). +- the `Scapy` and `dev` extras require `scapy>=2.6.0` ([#1572](https://github.com/JarryShaw/PyPCAPKit/issues/1572), [#1578](https://github.com/JarryShaw/PyPCAPKit/pull/1578)), and the `docs` extra `Sphinx>=7.3`, the first release with `versionremoved` ([#1743](https://github.com/JarryShaw/PyPCAPKit/pull/1743)). #### Fixed @@ -299,6 +386,8 @@ This resolves [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548), which r - `test_every_tracked_name_exists_and_matches_git` in `tests/test_tier_guard.py` checked neither the git index nor the filesystem despite its name: it asserted only that the tracked-capture list was non-empty and no name contained a `/`, so a hardcoded list passed whether or not it matched reality. Sibling `test_capture_suggestions_are_captures` had the same gap. Fixed in [#703](https://github.com/JarryShaw/PyPCAPKit/pull/703); [#703](https://github.com/JarryShaw/PyPCAPKit/pull/703)'s description names both [#700](https://github.com/JarryShaw/PyPCAPKit/issues/700) and [#708](https://github.com/JarryShaw/PyPCAPKit/issues/708). The first test now re-derives the expected set from `git ls-files -z` under `examples/captures/`, asserts equality against it, and checks each name with `Path.is_file()`; its sibling re-implements `committed_capture_names()`'s filter inline, which catches a wrong filter but still trusts `committed_captures()` for the tracked set. A hardcoded literal that happens to match the tracked names still passes; the test catches a set that is wrong when it runs ([#708](https://github.com/JarryShaw/PyPCAPKit/issues/708)). - `pcapkit.all.__all__` listed `OSPF`, `RARP` and `DRARP` on its Link Layer line after [#719](https://github.com/JarryShaw/PyPCAPKit/issues/719) moved them ([#1124](https://github.com/JarryShaw/PyPCAPKit/issues/1124)), and omitted 18 names its packages export, among them `ESP`, `HTTPv1`, `HTTPv2`, `PCAPNG` and `EnumRegistry`, so `from pcapkit.all import *` did not bind them. `pcapkit.utilities` stays unlisted deliberately, so generic helpers such as `warn` do not arrive through `import *` ([#1136](https://github.com/JarryShaw/PyPCAPKit/issues/1136)). - `examples/generators/options.py` wrote every HTTP/2 sample frame on stream 0, which [RFC 9113](https://datatracker.ietf.org/doc/html/rfc9113) forbids for `DATA`, `HEADERS` and the other stream-level types, so six of the ten frames in `options-transport.pcap` decoded as `TCP:Raw`. `_httpv2_build` now puts `SETTINGS`, `PING` and `GOAWAY` on stream 0 and every other type on stream 1, with `PUSH_PROMISE` promising stream 2, and all ten decode as `TCP:HTTP/2`; no other generated capture changes ([#1156](https://github.com/JarryShaw/PyPCAPKit/issues/1156)). +- `make_samples.py` builds fixtures from the tree it runs in, not the shared checkout ([#1343](https://github.com/JarryShaw/PyPCAPKit/issues/1343), [#1388](https://github.com/JarryShaw/PyPCAPKit/pull/1388)), and retries the Wireshark sample downloads on 429, 5xx and timeouts ([#1658](https://github.com/JarryShaw/PyPCAPKit/issues/1658), [#1659](https://github.com/JarryShaw/PyPCAPKit/pull/1659)). +- the Conda Update workflow tags and uploads each bump exactly once ([#1632](https://github.com/JarryShaw/PyPCAPKit/issues/1632), [#1641](https://github.com/JarryShaw/PyPCAPKit/pull/1641)), stays green when a merge races its push ([#1649](https://github.com/JarryShaw/PyPCAPKit/issues/1649), [#1669](https://github.com/JarryShaw/PyPCAPKit/pull/1669)), and keeps the release's name and notes when it uploads ([#1657](https://github.com/JarryShaw/PyPCAPKit/issues/1657), [#1678](https://github.com/JarryShaw/PyPCAPKit/pull/1678)). Vendor Update commits the new modules a crawl adds ([#1685](https://github.com/JarryShaw/PyPCAPKit/issues/1685), [#1688](https://github.com/JarryShaw/PyPCAPKit/pull/1688)). --- diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 24267da1ac..91ad8066c9 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -4,12 +4,12 @@ The largest release since 1.0, and the first recorded here as it happened rather than reconstructed. Three more extraction engines, ESP with payload decryption, SCTP and NGAP over SCTP, a library logger that no longer hijacks the consumer's, -and a defect programme run through the issue tracker, citing 319 issues and pull -requests from :pr:`121` to :issue:`1195` as of this revision. The programme is still -running, so those figures are a snapshot rather than a total. +and a defect programme run through the issue tracker, citing 611 issues and pull +requests from :pr:`340` to :pr:`1754`. Preceded by ``1.5.0a1`` (2026-09-15), ``1.5.0b1`` and ``1.5.0b2`` (both -2026-09-18) and ``1.5.0b3`` (2026-09-19), all published as prereleases and so +2026-09-18), ``1.5.0b3`` (2026-09-19) and ``1.5.0b4`` to ``1.5.0b12`` (2026-09-20 to +2026-10-13), all published as prereleases and so resolved only by ``pip install --pre``. ``1.5.0b1`` half-shipped: the tag, the GitHub release and the Conda deployments landed, but PyPI rejected the wheel because ``twine check`` found a Sphinx-only ``:mod:`` role in ``README.rst``, @@ -62,6 +62,28 @@ Changed now exempts only a ``'..._0x%s' % hex(value)[2:].upper().zfill(4)`` expression, and the exact ``exempt_hits == 53`` bound stays. Test-only. Closes :issue:`879` (:pr:`881`). +* **a breaking change to** ``AppType.proto``: it is the single transport of the + registry the member lives in, not every transport IANA assigned the service, and + ``register_apptype`` takes its transports as varargs (:issue:`806`, :pr:`815`). + Aliases show in ``repr()`` and ``str()``, and can be registered per transport + (:issue:`807`, :pr:`837`). +* **a breaking change to** ``TransportProtocol``: its base moves from ``aenum.IntFlag`` + to ``aenum.IntEnum``, so members no longer compose with ``|`` (:issue:`808`, + :pr:`836`). +* **a breaking change to** ``TLSKeyLabel``: the hand-written class in + ``pcapkit.protocols.misc.pcapng`` becomes the generated + ``pcapkit.const.pcapng.TLSKeyLabel``, crawled from the IANA registry :rfc:`9850` + creates (:issue:`886`, :pr:`890`), and the old path is removed (:issue:`1744`, + :pr:`1754`). +* **a breaking change to** ``Method.get``: the method token is case-sensitive, per + :rfc:`9110#section-9.1`, so ``Method.get('get')`` no longer resolves to ``GET`` + (:issue:`896`, :pr:`907`). It also looks a key up by value, so + ``BASELINE-CONTROL`` and ``VERSION-CONTROL`` resolve (:issue:`908`, :pr:`915`). +* **a breaking change to** ``ExtensionHeader``: it is generated from IANA's IPv6 + Extension Header Types registry rather than the Protocol Numbers registry's flag, + and the two disagree on 147 (``BIT-EMU``) (:issue:`925`, :pr:`926`). +* the TCP and UDP ``AppType`` registries load on first use rather than at import + (:issue:`1538`, :pr:`1560`). Fixed ~~~~~ @@ -83,7 +105,9 @@ Fixed and ``_missing_`` now look the key up under the canonical upper-case name, so every casing resolves to the existing member; a duplicate ``GET`` would have lacked the real one's ``safe`` and ``idempotent`` attributes (:issue:`582`, - :issue:`583`). + :issue:`583`). ``http.method`` no longer folds case: :rfc:`9110#section-9.1` makes + the method token case-sensitive, so ``Method.get`` matches it exactly + (:issue:`896`, below). * ``get()``'s documented ``default`` was ignored on the integer path across the generated ``pcapkit.const`` tree, because ``_missing_`` cannot see the caller's ``default`` -- ``Hardware.get(99999, 0)`` raised ``ValueError`` instead of returning @@ -139,9 +163,9 @@ Fixed ``_missing_`` and minted regardless, so ``AppType.get(-1, proto='tcp')`` returned an unreachable ``PORT_-1_tcp`` while ``TCP(-1)`` raised. It now tests ``_missing_`` for ``None`` ("no row for this port") and lets the rejection - through; a valid but unassigned port still mints. **``Transport._make_port`` - passes an unvalidated ``int``, so ``TCP.make(srcport=99999)`` now raises where - it minted junk.** Separately, the crawler rendered one ``_missing_`` branch per + through; a valid but unassigned port still mints. ``Transport._make_port`` + **passes an unvalidated** ``int``, **so** ``TCP.make(srcport=99999)`` **now raises + where it minted junk.** Separately, the crawler rendered one ``_missing_`` branch per registry row, so the two spans IANA registers on more than one transport emitted the same condition twice and every registry answered with the first: ``AppType.get(6010, proto='udp')`` gave a member whose ``proto`` read ``tcp``, @@ -181,8 +205,8 @@ Fixed ``(0x0020, 0x003F)`` after ``(0x0001, 0x0BB8)``, and ``(0x4000, 0x4FFF)`` and ``(0x8000, 0xFFFF)`` after ``(0x0BB9, 0xFFFF)`` -- so they never fired. The generator ``pcapkit/vendor/ipx/socket.py`` now lists each subset range ahead of the - range containing it. **36891 of the 65521 ``_missing_``-resolved values (56.3%) - change name**: 32 from ``Registered by Xerox`` to ``Experimental``, 4095 from + range containing it. **36891 of the 65521 values that** ``_missing_`` **resolves + (56.3%) change name**: 32 from ``Registered by Xerox`` to ``Experimental``, 4095 from ``Dynamically Assigned`` to ``Dynamically Assigned Socket Numbers`` and 32764 from ``Dynamically Assigned`` to ``Statically Assigned Socket Numbers``. No defined member changes and every value still resolves (:pr:`847`). @@ -282,6 +306,18 @@ Fixed ``is_manufactured()`` sweep forbids; rather than weaken the predicate, a narrow exemption covers exactly these two files, since the collision the check guards against cannot occur once ``_unregistered_member`` never registers (:pr:`878`). +* ``OptionType.register()`` and ``Command.register()`` built members that ``get()`` + could not find again, and ``OptionType.register_alias()`` raised + ``not a registered OptionType`` for every argument. Both registries now round-trip + through ``get()``, in the generated files and the vendor templates (:pr:`1311`, + :issue:`1330`, :pr:`1369`). +* the Quick-Start function registries rejected 9 to 15, though :rfc:`4782` gives the + Function field four bits; the whole 4-bit range now resolves (:issue:`1350`, + :pr:`1353`). +* every three-digit FTP reply and HTTP status code resolves, including those no + registry row names (:issue:`1443`, :pr:`1444`). +* the ``AppType`` registry's dict union is defined only on interpreters whose + ``dict`` has one (:issue:`1694`, :pr:`1703`). pcapkit.corekit --------------- @@ -297,6 +333,17 @@ Added plus guard rails for a full read and for a read against an empty buffer, the :issue:`431` behaviour the option and list loops depend on (:issue:`604`). +* ``EnumLookup``, the base carrying ``get`` and ``get_all`` that ``EnumRegistry`` and + the non-registry enumerations share (:issue:`877`, :pr:`906`, :pr:`921`, + :pr:`932`). Its ``get`` raises pcapkit exceptions in the shape of the stdlib + ``Enum``: a name miss raises ``EnumKeyError``, a ``KeyError`` (:issue:`923`, + :pr:`927`). +* ``MultiInfo`` and ``OrderedMultiInfo``, immutable ``Info`` subclasses that are also + a ``MultiDict`` and an ``OrderedMultiDict``, so a repeated key keeps every value. + The data models' option lists use them (:issue:`1484`, :pr:`1523`). +* ``pcapkit.corekit.fields.byteorder``, holding ``packet_byteorder`` and + ``byteorder_callback``, the byte-order helpers the PCAP and PCAP-NG schemas share + (:pr:`1528`, :issue:`1711`, :pr:`1715`). Changed ~~~~~~~ @@ -334,15 +381,15 @@ Changed * **a breaking change to** ``@final``, now enforced at runtime on ``Info`` and ``Schema``. ``info_final`` and ``schema_final`` both end ``return final(cls)``, so every finalised class carried ``typing.final``'s ``__final__`` marker and nothing - read it. **``@final`` alone on an ``Info`` or ``Schema`` subclass -- without - ``@info_final`` or ``@schema_final`` -- now raises ``InfoError`` or ``SchemaError`` - at first construction**, the class being marked final but never finalised and so + read it. ``@final`` **alone on an** ``Info`` **or** ``Schema`` **subclass -- without** + ``@info_final`` **or** ``@schema_final`` **-- now raises** ``InfoError`` **or** + ``SchemaError`` **at first construction**, the class being marked final but never finalised and so carrying no generated ``__init__``. ``__init_subclass__`` cannot catch that, since ``final`` applies after class creation, so the check sits in the existing one-shot ``FinalisedState.NONE`` branch and a finalised class pays nothing. **Deriving from a finalised class raises too**, from new ``Info.__init_subclass__`` and ``Schema.__init_subclass__`` hooks (before, only ``EnumSchema`` had one). - **``SchemaError`` is a ``ValueError``**, so code catching ``TypeError`` around a + ``SchemaError`` **is a** ``ValueError``, so code catching ``TypeError`` around a subclass declaration will not see it. ``@info_final @final`` and ``@final @info_final`` both finalise silently; the re-entry check keys on ``__finalised__`` rather than ``__final__``, because only the former records *the @@ -350,6 +397,21 @@ Changed ``pcapkit.utilities.compat`` takes ``final`` from ``typing_extensions`` below 3.11 rather than 3.8, because ``typing.final`` only records ``__final__`` from 3.11. The guard cannot fire on the library's own classes (:issue:`778`). +* **a breaking change to** ``Info.to_dict()``: it returns an ``OrderedMultiDict`` at + every level rather than a ``dict``, so a key that repeats, such as a second IPv6 + extension header of one type, survives a ``from_dict(to_dict())`` rebuild. + ``type(d) is dict`` no longer holds (:issue:`1453`, :pr:`1466`, :issue:`1484`, + :pr:`1523`). +* **a breaking change to** ``pcapkit.utilities.decorators``: it moves to + ``pcapkit.corekit.decorators``, with no re-export. ``seekset``, ``beholder`` and + ``prepare`` are no longer importable from ``pcapkit.utilities``, and the module + logs as ``pcapkit.corekit.decorators`` (:issue:`1735`, :pr:`1737`). +* **a breaking change to** the sentinels: all live in ``pcapkit.corekit.sentinels``, + which exports the objects ``NULL``, ``NO_VALUE`` and ``NO_DEFAULT`` but not their + types, and ``NoValue`` is spelled ``NO_VALUE`` (:pr:`916`, :issue:`911`, + :pr:`922`, :issue:`937`, :pr:`939`). No moved name keeps a re-export, so + ``NoValueType`` has only that home, not its 1.4.x one in + ``pcapkit.corekit.fields.field`` (:issue:`1744`, :pr:`1754`). Fixed ~~~~~ @@ -533,7 +595,7 @@ Fixed ``get()``'s dispatch shifts only for a key that is neither ``int`` nor ``str`` (``KeyError``, tried as a name; now ``ValueError``, tried as a value), a key type nothing in the library passes. - ``pcapkit/corekit/enums.py`` is renamed **``enum.py``**, the maintainer's ruling + ``pcapkit/corekit/enums.py`` is renamed ``enum.py``, the maintainer's ruling (:pr:`858`). * ``EnumRegistry.get()``'s ``str``-key branch tried only a name lookup (``cls._member_map_[key]``) and on a miss raised or returned ``cls(default)``, @@ -579,6 +641,28 @@ Fixed ``FieldValueError`` into ``prepare()``'s quiet ``StreamEOFError``. Fixed-length items are unchanged; the HTTP/2 ``SETTINGS`` pair, whose size is fixed, is given it in its own schema instead (:issue:`1140`, below) (:issue:`1148`). +* a signed ``NumberField`` decoded as unsigned, a zero-length ``PayloadField`` + misread, and an empty ``Raw`` did not rebuild (:pr:`1331`). +* ``from_dict(to_dict())`` lost nested values, and ``len()`` of an ``Info`` counted + its hidden keys (:pr:`1334`). +* ``StringField`` packs back exactly the octets it unpacked, where a guessed charset, + a ``U+FFFD`` substitution or a character-count length used to change them; the + decoded text is unchanged (:issue:`1293`, :pr:`1336`). A charset that rejects + ``'replace'`` falls back to UTF-8 (:issue:`1706`, :pr:`1728`), and Python 3.8 + unquotes through ``pcapkit.utilities.compat.unquote`` (:issue:`1693`, :pr:`1707`). +* ``IPv4InterfaceField`` keeps a mask that is not a netmask, and a deep copy of an + ``OrderedMultiDict`` keeps its order (:pr:`1346`). +* **a breaking change to** padding: non-zero padding octets are kept on rebuild + rather than zeroed (:issue:`1223`, :pr:`1436`). +* fields keep the bits and octets they captured on rebuild: a ``BitField``'s unnamed + bits (:issue:`1487`, :pr:`1495`), a ``NumberField``'s bits outside ``bit_length`` + (:issue:`1488`, :pr:`1497`), and the bytes an untyped ``ListField`` unpacked + (:issue:`1489`, :pr:`1494`). An unnamed ``ListField``, ``PayloadField`` or + ``SchemaField`` gets a default name (:issue:`1491`, :pr:`1493`). +* each non-final ``Info`` subclass is finalised on its own state (:issue:`1490`, + :pr:`1496`), and its constructor reads only the class's own annotations + (:issue:`1692`, :pr:`1702`). +* copying or pickling a stream proxy no longer recurses (:issue:`1524`, :pr:`1544`). pcapkit.dumpkit --------------- @@ -631,6 +715,17 @@ Fixed now JSON-escapes keys for JSON output, as :issue:`772` made it XML-escape them for plist, and the JSON reports of all 23 ``examples/captures/`` files now parse. The upstream defect, ``JarryShaw/DictDumper#121``, stands (:issue:`1152`). +* **a breaking change to** the dump schema: the output writers dropped or corrupted + what the packets carried. An ``OrderedMultiDict``, which every option list is, now + dumps as a list of single-key entries in wire order + (:pr:`1305`). +* tree output raised on a ``complex`` value, and is now documented as lossy + (:issue:`1266`, :pr:`1366`). +* plist dates are written to whole seconds, so ``plistlib`` reads them back + (:issue:`1448`, :pr:`1450`). +* a PCAP trace record scales its fraction to the trace's resolution (:issue:`1500`, + :pr:`1529`), and writes the octets it has rather than a truncated record's claimed + length (:issue:`1541`, :pr:`1543`). pcapkit.foundation ------------------ @@ -755,6 +850,8 @@ Changed ``Engine.read_frame``, ``Engine.close`` and ``Engine.unsupported_reason`` now point at ``EngineBase``. Exporting ``EngineBase`` in ``__all__`` is left to :issue:`1016` (:issue:`1024`). +* **a breaking change to** ``register_mh_extension``: ``schema`` is keyword-only, as + in the other fourteen schema-taking registrars (:issue:`516`, :pr:`526`). Fixed ~~~~~ @@ -906,6 +1003,47 @@ Fixed dropped the input's ``thiszone``, ``sigfigs`` and ``snaplen``, so only 1 of 17 sample PCAP files came back byte for byte; the PCAP engine now passes all three through ``_open_output`` to ``PCAPIO``, and all 17 do (:issue:`1165`). +* IP reassembly reports only the octets it holds: a truncated fragment, a strict run + and a zero-length datagram no longer report octets from the declared Total Length + (:pr:`1368`). Only the final fragment's partial block counts, and a datagram too + long for its length field never completes (:pr:`1582`, :issue:`1585`, + :pr:`1598`). +* TCP reassembly judges completion from each buffer's own gaps, compares sequence + numbers modulo ``2**32``, and holds a FIN until reordered data arrives + (:pr:`1375`). A TSO fragment's length is its captured length, and ``RCVBT`` covers + the last block (:pr:`1564`). +* a second TCP flow with the same label truncated the first flow's file; a reused + label now gets the next free suffix (:issue:`1360`, :pr:`1371`). +* the engine, reassembly, traceflow and dumper registrars accept the built-in a name + shipped with, which restores it (:issue:`1363`, :pr:`1376`). +* PCAP-NG blocks are numbered across the whole file, not per section + (:issue:`1256`, :pr:`1426`). +* ``format='pcap'`` is kept when an unavailable engine falls back to the default one + (:issue:`1264`, :pr:`1430`). +* ``extract()`` accepts a file object with no ``name`` as ``fin`` (:issue:`1506`, + :pr:`1509`), and reads every frame from a pipe or other non-seekable stream + (:issue:`1586`, :pr:`1595`). +* the third-party engines read what the capture holds. The dpkt engine keeps a frame + dpkt cannot parse rather than leaking ``AttributeError`` (:issue:`1351`, + :pr:`1377`), applies each PCAP-NG interface's link type and timestamp resolution + rather than the first one's (:issue:`1379`, :pr:`1396`), and reads float + timestamps, Simple Packet Blocks and raw-IP frames (:pr:`1550`). The scapy engine + resets its interfaces at each PCAP-NG section (:issue:`1522`, :pr:`1546`), takes a + packet's link type and clock from its interface (:pr:`1563`), and reads each record + whole (:issue:`1717`, :pr:`1733`). The pyshark engine skips non-packet PCAP-NG + blocks (:issue:`1515`, :pr:`1531`), and the pypcapfile engine reads the PCAP + records pypcapfile drops (:issue:`1512`, :pr:`1569`). dpkt and scapy write real PCAP traces + (:issue:`1507`, :pr:`1511`). + +pcapkit.interface +----------------- + +Added +~~~~~ + +* ``pcapkit.interface.registry``, making public the registry-entry helpers + ``same_entry``, ``restore_shipped`` and ``restore_shipped_dumper`` (:issue:`1519`, + :pr:`1568`). pcapkit.protocols ----------------- @@ -1002,6 +1140,12 @@ Added case pinning the 20-octet length :rfc:`5201#section-5.2.5` requires of HIPv1. Five widths are deliberately *not* multiples of eight, where the defective ``ceil(bits / 4)`` coincides with the correct width (:issue:`608`). +* the BSD loopback layer for ``LINKTYPE_NULL`` and ``LINKTYPE_LOOP``, and dissection of + ``LINKTYPE_RAW`` by the version nibble (:pr:`1594`). +* one home for each RFC- or format-fixed constant spelled out in more than one module + -- the IPv4, IPv6, AH and TCP header lengths and offsets, ``IP_MAX_LENGTH``, and the + PCAP and PCAP-NG framing constants -- in the module that owns the protocol + (:issue:`1739`, :pr:`1752`). Changed ~~~~~~~ @@ -1066,8 +1210,9 @@ Changed out of ``**kwargs`` by name (``ESP.read`` with ``packet``); and ``None``, for a dispatcher whose real signature belongs to a class chosen at call time (only ``HTTP.make``). The message names the near neighbour (``seq`` reports *did you mean - 'seq_no'?*). ``from_data`` warns ``UnknownFieldWarning`` rather than raising, because - the defect is then a ``_make_data`` key the caller cannot fix. Four leftover copies + 'seq_no'?*). ``from_data`` warns ``UnknownFieldWarning`` rather than raising for a key + its own ``_make_data`` returned, which the caller cannot fix; a keyword passed to + ``from_data`` itself raises, as for a direct construction (:pr:`1147`). Four leftover copies of :issue:`602`'s misspelt TCP base, in ``examples/generators/dispatch.py`` and ``tests/protocols/transport/``, are fixed. Three such ``_make_data`` keys are reported, not fixed: ``Frame._make_data`` returns ``ts_src`` where ``make`` declares @@ -1095,6 +1240,18 @@ Changed ``pcapkit.protocols.data.application.ospf`` to ``CryptographicAuthentication``, with no alias left for the old spelling. Both OSPF names also move to the application group of ``schema.__all__`` and ``data.__all__``, after :issue:`719`'s move (:issue:`1090`). +* **a breaking change to** a declared length that runs past the data: it is refused + with ``ProtocolError`` rather than accepted and zero-filled on rebuild, so a + malformed option looks different in ``info`` (:issue:`1325`, :pr:`1398`). +* dissection is bounded: a frame at 64 layers (:issue:`1610`, :pr:`1620`) and an IPv6 + extension-header chain at 32 headers (:issue:`1604`, :pr:`1608`). Both limits are + ``PCAPKIT_FRAME_LAYER_LIMIT`` and ``PCAPKIT_EXTENSION_HEADER_LIMIT`` in + ``pcapkit.utilities.config`` (:pr:`1743`). Rebuilding a + frame cut at the chain bound warns once (:issue:`1619`, :pr:`1630`). +* **a breaking change to** IPv6 extension headers: an unrecognised or unparsable one + is parsed generically, per :rfc:`6564`, rather than collapsing the packet to ``Raw`` + (:issue:`891`, :pr:`904`). ``IPv6_Ext`` does it, and is also the shared base of all + eight extension headers (:issue:`917`, :pr:`924`). Fixed ~~~~~ @@ -1426,8 +1583,8 @@ Fixed ``frame.len`` meant the captured length out of a ``.pcap`` and the on-wire length out of a ``.pcapng``. The PCAP reader moved to match PCAP-NG: ``len`` is the on-wire length (the record header's ``orig_len``) and ``cap_len`` the octets stored - (``incl_len``). **Code reading ``frame.len`` or ``frame.cap_len`` from a ``.pcap`` - gets the other field's value than before**, and for a truncated frame that is a + (``incl_len``). **Code reading** ``frame.len`` **or** ``frame.cap_len`` **from a** ``.pcap`` + **gets the other field's value than before**, and for a truncated frame that is a different number rather than a relabelling. Seniority did not settle it: the PCAP spelling is the *older* (``c43892af``, 2022-01-11) and PCAP-NG's arrived in ``25f216f4`` (2023-04-27), both internally consistent. The *names* settle it, being @@ -1482,7 +1639,7 @@ Fixed * every HIP parameter pcapkit emitted was ``4 (mod 8)`` octets long, because the padding aligned the *contents* rather than the record. Corrected for **45 of the 46** HIP parameters; ``LOCATOR_SET`` is excluded, below. **This changes the octets - those 45 parameters write, and the ``length`` their data models report.** + those 45 parameters write, and the** ``length`` **their data models report.** :rfc:`7401#section-5.2.1` requires every TLV parameter to have "a length (that includes the Type and Length fields), which is a multiple of 8 bytes" and gives ``Total Length = 11 + Length - (Length + 3) % 8``; all 95 padding sites computed @@ -2086,6 +2243,90 @@ Fixed ``ProtocolError``; it now writes 4. ``read`` names a Header Length below 4 in a new ``invalid header length`` message. Lengths 0 to 3 already raised ``ProtocolError``, so only the message changes (:issue:`1189`). +* the RPL option, type ``0x23``, raised ``AttributeError`` in ``HOPOPT`` and + ``IPv6_Opts`` (:issue:`1196`, :pr:`1198`). +* **a breaking change to** the RPL option: it keeps its reserved flag bits on rebuild + and reads :rfc:`6553` sub-TLVs (:issue:`1199`, :pr:`1201`). +* ``make()`` rejected an enum member name given as ``str`` unless a namespace was + passed (:issue:`1210`, :pr:`1227`); raw items in a ``make()`` list build a packet, + and ``str`` member names resolve per argument (:pr:`1352`). +* **a breaking change to** rebuilding a parsed packet, the round-trip programme: + ``from_data(info)`` gives back the packet's own octets. Most of these fixes add the + ``make()`` keywords and ``info`` keys the rebuild needs, which is what breaks: + + * ARP, InARP and DRARP, any Ethernet ``hlen``, and a ``ptype``/``plen`` mismatch + (:pr:`1224`); L2TPv2 reserved flag bits, and a lone ``ns`` or ``nr`` in + ``make()`` (:pr:`1226`); octets past the L2TPv2 and OSPF Length fields + (:issue:`1455`, :pr:`1460`). + * SCTP chunk flags, reserved fields and final padding (:pr:`1253`), octets after the + last whole chunk (:issue:`1468`, :pr:`1472`), a last chunk short of its padding + (:issue:`1474`, :pr:`1477`), a final parameter or cause short of its padding + (:issue:`1479`, :pr:`1481`), and a Host Name with no NUL terminator + (:issue:`1482`, :pr:`1483`). + * TCP: MPTCP options (:pr:`1254`); reserved bits, octets after EOOL and the UTO + granularity (:pr:`1348`); and a header length declared past the data, with IPv4 + (:pr:`1411`). + * UDP: octets past the Length field (:issue:`1461`, :pr:`1467`). + * FTP and HTTP/1 lines, byte for byte (:pr:`1301`), keeping the raw command case, + HTTP/1 whitespace and field octets, with whole multi-line FTP replies parsed + (:pr:`1347`). HTTP/2 frames lose no octets (:pr:`1341`). + * OSPF keeps the parsed Packet Length, and NGAP refuses octets after the NGAP-PDU + (:pr:`1304`); OSPF cryptographic authentication rebuilds from its dict form + (:issue:`1464`, :pr:`1469`). + * IPv4 option areas, route slots, timestamps and reserved bits (:pr:`1344`); the TS + option's slots beyond the pointer (:issue:`1349`, :pr:`1390`); TS and SEC options + byte for byte (:pr:`1399`). + * IPv6 core headers (:pr:`1307`); ``HOPOPT`` and ``IPv6_Opts`` option areas + (:pr:`1310`); a CALIPSO option's trailing pad (:issue:`1403`, :pr:`1407`), and + its ``make()`` writing a corrupt length on overflow (:issue:`1408`, :pr:`1409`); an + extension header parsed with ``extension=True`` (:issue:`1446`, :pr:`1449`); + RPL source routes (:pr:`1463`); and a cut extension-header chain (:pr:`1476`). + * octets past IPv4 Total Length and IPv6 Payload Length (:issue:`1209`, + :pr:`1431`), and past the IPX Length field (:issue:`1433`, :pr:`1435`). IPX + ``from_data`` raised ``struct.error``, and AH wrote its reserved field back as + zero (:pr:`1338`). + * HIP: Header Length, an empty parameter list, the PUZZLE lifetime and reserved bits + (:pr:`1342`); Locator flags (:issue:`1365`, :pr:`1367`); three ``BitField`` + namespaces' reserved bits (:issue:`1417`, :pr:`1419`); and the two fixed header + bits (:issue:`1459`, :pr:`1462`). + * MH: zero-length PadN, reserved fields and BU/BA flag bits (:issue:`1324`, + :pr:`1345`); the reserved bits of 24 ``BitField`` flag namespaces + (:issue:`1418`, :pr:`1421`); and the reserved octets ``PaddingField`` sites + capture (:pr:`1442`). + * PCAP-NG: constructed and rebuilt blocks (:pr:`1339`), lengths and padding + (:pr:`1373`), blocks and options as parsed (:pr:`1378`), further follow-ups + (:pr:`1412`), malformed journal entries and short packet blocks (:pr:`1420`), an + SHB or IDB shorter than its fixed fields (:issue:`1422`, :pr:`1425`), and + captured padding octets (:pr:`1441`). + * a cut PCAP header or PCAP-NG block (:issue:`1470`, :pr:`1475`). + +* ``Schema.pack`` keeps option padding, ``SchemaField`` spans and iterated keys + (:pr:`1374`); a ``SchemaField``'s unread octets survive ``from_data`` rather than + rebuilding zero-filled (:issue:`1380`, :pr:`1397`); a header cut inside a + fixed-width field rebuilds as captured (:pr:`1465`); and each non-final ``Schema`` + subclass is finalised on its own state (:issue:`1498`, :pr:`1499`). +* ``from_data`` rebuilds the payload from ``info.to_dict()`` (:issue:`1447`, + :pr:`1452`). +* building a PCAP frame with ``snaplen`` 0 wrote a record with no packet data + (:issue:`1385`, :pr:`1410`); an out-of-range ``if_tzone`` is kept as captured and + every ``fromtimestamp`` failure is caught (:issue:`1340`, :pr:`1429`). +* registration: the parser registrars warn only when the entry really changes + (:issue:`1362`, :pr:`1400`); re-registering a stored ``ModuleDescriptor`` keeps it + (:issue:`1364`, :pr:`1415`); restoring a registration puts back the shipped + descriptor, including through ``PCAPNG.register`` (:pr:`1533`, :pr:`1573`); and + unregistering a custom protocol removes its name from ``__proto__`` + (:issue:`1505`, :pr:`1530`). +* fragments: an IPv6 extension header behind a first fragment is dissected as without + it (:issue:`1539`, :pr:`1540`), and a later IPv4 or IPv6 fragment's data is raw, + not an upper-layer header (:issue:`1545`, :pr:`1552`, :issue:`1553`, :pr:`1562`). +* IPv4: a Total Length of 0 is read as TSO, and one inside the header as bogus + (:issue:`1547`, :pr:`1554`). +* IPv6: a jumbogram's payload is dissected up to its Jumbo Payload Length + (:issue:`1434`, :pr:`1440`), a jumbogram :rfc:`2675` calls an error is rejected + (:issue:`1687`, :pr:`1713`), and so is a header whose Version is not 6 + (:issue:`1680`, :pr:`1723`). +* the warnings of a next layer replaced by ``Raw`` are dropped (:issue:`1580`, + :pr:`1588`). pcapkit.toolkit --------------- @@ -2112,6 +2353,8 @@ Added line of ``pcapkit/toolkit/pyshark.py`` changes: the ``IPMB_LINUX`` entry becomes ``I2C_LINUX``, the canonical name :pr:`848` gave value 209 -- the same member either way (:pr:`853`). +* ``tcp_segment`` and its result type are public in ``pcapkit.toolkit.pcap`` + (:issue:`1570`, :pr:`1579`). Fixed ~~~~~ @@ -2166,6 +2409,24 @@ Fixed * ``pypcapfile``'s ``tcp_reassembly`` set ``last`` one past the segment. ``TCP_Packet.last`` is inclusive, and the four other adapters set ``seqnum + len(payload) - 1``, as this one now does too (:issue:`1100`). +* scapy's ``packet2dict`` dumped nested packets as ``{}`` and mutated its input + (:pr:`1287`); the dpkt reassembly adapters build their data from the record's own + octets rather than dpkt's re-serialised bytes, and leave the packet alone + (:issue:`1297`, :pr:`1335`). +* the TCP adapters: pyshark accepts tshark's ``True``/``False`` TCP flags + (:issue:`1514`, :pr:`1527`); a TCP segment whose header the snapshot cut is traced + and reassembled (:issue:`1518`, :pr:`1556`); dpkt reads SYN, FIN and RST whatever + the NS bit (:issue:`1618`, :pr:`1621`); and the scapy adapters stop at the IP + datagram's declared length (:issue:`1584`, :pr:`1589`). +* timestamps: dpkt and pypcapfile hand over the correctly rounded timestamp + (:issue:`1668`, :pr:`1690`), and a copied ``DecimalTimestamp`` keeps ``orig_len`` + under ``_pydecimal`` (:issue:`1647`, :pr:`1653`). +* the third-party engines are best effort: the code each adapter carried only to + match the default engine is reverted (:pr:`1718`, :pr:`1721`, :pr:`1724`, + :pr:`1727`, closing :issue:`1708` through :pr:`1743`). What each engine leaves out, + such as TCP behind a tunnel or an IPv6 extension header, is documented rather + than warned about; the pypcapfile engine still warns once per + capture that it leaves TCP over IPv6 out (:issue:`1513`, :pr:`1535`). pcapkit.utilities ----------------- @@ -2179,6 +2440,12 @@ Added modules now log under their own ``__name__``, so a consumer can silence ``pcapkit.foundation.registry`` while keeping ``pcapkit.foundation.extraction`` (:pr:`384`). +* ``pcapkit.utilities.config``, one module for every tunable limit, each read once + from a ``PCAPKIT_*`` environment variable: the frame-layer and extension-header + limits, the dumpkit fallback, the charset-detection cache, the read chunk and the + three zero-padding guards. A bad value warns with ``ConfigWarning`` and falls back + to the default; ``-1`` lifts a limit. **The old homes are removed**, with no + re-export (:issue:`1726`, :pr:`1743`). Changed ~~~~~~~ @@ -2212,6 +2479,10 @@ Fixed ``register_extractor_engine``'s real keyword, ``name``, was not itself under test, only its corrected docstring, so a rename could put the two out of step again (:pr:`577`). +* ``hold_warnings`` is in ``pcapkit.utilities.warnings.__all__`` (:issue:`1602`, + :pr:`1603`). +* the default threading excepthook is the fallback before Python 3.10, which has no + ``threading.__excepthook__`` (:issue:`1691`, :pr:`1701`). pcapkit.vendor -------------- @@ -2273,7 +2544,7 @@ Fixed (``.github/workflows/cron-vendor.yml``): the "Update Vendor" step no longer aborts under ``bash -e`` when ``pcapkit-vendor`` exits non-zero. The exit code is captured around that one command, so targets that succeeded are still - ``isort``ed, diffed, committed and pushed, while a "Warning" block naming the + sorted by ``isort``, diffed, committed and pushed, while a "Warning" block naming the failed target(s) is appended to ``$GITHUB_STEP_SUMMARY``, a ``::warning::`` annotation is emitted per failure, and the step's own exit code is always ``0``. A per-target ``git checkout -- `` restore step was dropped: with @@ -2309,6 +2580,12 @@ Fixed stale ``# NSS Key Log Format`` comment is repointed at RFC 9850, and ``TLSKeyLabel``'s docstring cites it. A new test pins the member set against RFC 9850 plus the ``RSA`` exception. Closes :issue:`882` (:pr:`883`). +* both spellings of an IANA citation render as ``:rfc:`` roles in the generated + ``pcapkit.const`` docstrings (:issue:`1664`, :pr:`1673`). +* ``MAX_RETRY`` and ``CI_MODE`` stay in ``pcapkit.vendor.default``, read from + ``PCAPKIT_VENDOR_RETRY`` and ``PCAPKIT_CI_MODE``; a bad value warns with + ``ConfigWarning`` and falls back, where 1.4.x crashed on a non-integer and read + ``0`` as ``1`` (:pr:`1745`). Project infrastructure ---------------------- @@ -2508,6 +2785,16 @@ Changed the PCAP-NG engine's ``run`` list the ``FormatError`` for PCAP output from a PCAP-NG input, and the extraction and engine pages document the PCAP-output path (:pr:`1179`, :pr:`1180`, :pr:`1186`, :issue:`1103`). +* **a breaking change to** the ``all`` extra: it carries the core addons only -- + ``cli``, ``crypto`` and ``NGAP``. The three third-party engines 1.4.x bundled in it + (``dpkt``, ``scapy`` and ``pyshark``) and the vendor crawler's ``requests`` and + ``beautifulsoup4`` are no longer in it; the engines keep their own ``DPKT``, + ``Scapy`` and ``PyShark`` extras and the crawler its ``vendor`` extra, and + ``pypcapfile`` has a new ``PyPCAPFile`` extra of its own. A new ``dev`` extra + carries what CI needs (:issue:`910`, :pr:`914`). +* the ``Scapy`` and ``dev`` extras require ``scapy>=2.6.0`` (:issue:`1572`, + :pr:`1578`), and the ``docs`` extra ``Sphinx>=7.3``, the first release with + ``versionremoved`` (:pr:`1743`). Fixed ~~~~~ @@ -2686,3 +2973,11 @@ Fixed ``_httpv2_build`` now puts ``SETTINGS``, ``PING`` and ``GOAWAY`` on stream 0 and every other type on stream 1, with ``PUSH_PROMISE`` promising stream 2, and all ten decode as ``TCP:HTTP/2``; no other generated capture changes (:issue:`1156`). +* ``make_samples.py`` builds fixtures from the tree it runs in, not the shared + checkout (:issue:`1343`, :pr:`1388`), and retries the Wireshark sample downloads on + 429, 5xx and timeouts (:issue:`1658`, :pr:`1659`). +* the Conda Update workflow tags and uploads each bump exactly once + (:issue:`1632`, :pr:`1641`), stays green when a merge races its push + (:issue:`1649`, :pr:`1669`), and keeps the release's name and notes when it + uploads (:issue:`1657`, :pr:`1678`). Vendor Update commits the new modules a crawl + adds (:issue:`1685`, :pr:`1688`). diff --git a/docs/source/contributing/conventions/process.rst b/docs/source/contributing/conventions/process.rst index f4c0902ad1..ad3d56d273 100644 --- a/docs/source/contributing/conventions/process.rst +++ b/docs/source/contributing/conventions/process.rst @@ -100,15 +100,15 @@ commands down instead of a figure that will be stale by the next merge: The grouping scheme was settled on :issue:`918`: **a section per top-level module, with** ``Added``/``Changed``/``Fixed`` **nested inside each** -- module granularity, not -per-file and not per-subpackage. The file carries **9** module-level sections holding -200 entries, and no entry carries an inline kind label:: +per-file and not per-subpackage. The file carries **10** module-level sections holding +269 entries, and no entry carries an inline kind label:: $ grep -cE '^\* \*\*(Added|Changed|Fixed)\*\*' docs/source/changelog/1.5.0.rst 0 -Eight of those nine name a module; the ninth, *Project infrastructure*, is for what -belongs to none. Nor is the map one-to-one with the package list below -- -:mod:`pcapkit.interface` has no 1.5.0 entry, so it has no section of its own: +Nine of those ten name a module; the tenth, *Project infrastructure*, is for what +belongs to none. The map is one-to-one with the package list below -- every package, +:mod:`pcapkit.interface` included, has a section of its own: .. code-block:: shell