diff --git a/docs/source/pcapkit/protocols/application/application.rst b/docs/source/pcapkit/protocols/application/application.rst index 9cfb822a6..952e3e2a2 100644 --- a/docs/source/pcapkit/protocols/application/application.rst +++ b/docs/source/pcapkit/protocols/application/application.rst @@ -10,6 +10,15 @@ which is a base class for application layer protocols, eg. :class:`HTTP/2 ` and etc. +.. versionchanged:: 1.5.0 + + :meth:`~pcapkit.protocols.application.application.Application._decode_next_layer` + and + :meth:`~pcapkit.protocols.application.application.Application._import_next_layer` + accept the ``-1`` sentinel, so an application layer protocol can carry the + rest of its packet undissected as its payload; every call raised + :exc:`~pcapkit.utilities.exceptions.UnsupportedCall` (:issue:`719`). + .. autoclass:: pcapkit.protocols.application.application.Application :no-members: :show-inheritance: diff --git a/docs/source/pcapkit/protocols/application/ftp.rst b/docs/source/pcapkit/protocols/application/ftp.rst index 1a7043996..fbd965221 100644 --- a/docs/source/pcapkit/protocols/application/ftp.rst +++ b/docs/source/pcapkit/protocols/application/ftp.rst @@ -9,6 +9,14 @@ which implements extractor for File Transfer Protocol (FTP) [*]_, and :class:`~pcapkit.protocols.application.ftp.FTP_DATA` for its data channel. +.. versionchanged:: 1.5.0 + + A command or reply rebuilds byte for byte, its CRLF included, and keeps + the command's case and any octets that are not UTF-8 as captured. A line + with no argument and a multi-line reply, whole in one segment or not, + parse and rebuild as captured (:issue:`1238`, :issue:`1239`, + :issue:`1240`, :issue:`1241`, :issue:`1303`, :issue:`1333`). + .. autoclass:: pcapkit.protocols.application.ftp.FTP :no-members: :show-inheritance: diff --git a/docs/source/pcapkit/protocols/application/httpv1.rst b/docs/source/pcapkit/protocols/application/httpv1.rst index b97f1c319..658d49785 100644 --- a/docs/source/pcapkit/protocols/application/httpv1.rst +++ b/docs/source/pcapkit/protocols/application/httpv1.rst @@ -4,10 +4,11 @@ HTTP/1.* - Hypertext Transfer Protocol .. module:: pcapkit.protocols.application.httpv1 :mod:`pcapkit.protocols.application.httpv1` contains -:class:`~pcapkit.protocols.application.httpv1.HTTP` -only, which implements extractor for Hypertext Transfer -Protocol (HTTP/1.*) [*]_, whose structure is described -as below: +:class:`~pcapkit.protocols.application.httpv1.HTTP`, +which implements extractor for Hypertext Transfer +Protocol (HTTP/1.*) [*]_, and its +:class:`~pcapkit.protocols.application.httpv1.Type` enumeration. The +structure of an HTTP/1.* message is described as below: .. code-block:: text @@ -23,6 +24,13 @@ as below: \r\n :==: RESPONSE SEPARATOR :==: RESPONSE BODY (optional) +.. versionchanged:: 1.5.0 + + A message rebuilds byte for byte: the whitespace of the start line and + header lines, and octets in them that are not UTF-8, are kept as captured + rather than normalised or re-encoded (:issue:`1241`, :issue:`1302`, + :issue:`1333`). + .. autoclass:: pcapkit.protocols.application.httpv1.HTTP :no-members: :show-inheritance: diff --git a/docs/source/pcapkit/protocols/application/httpv2.rst b/docs/source/pcapkit/protocols/application/httpv2.rst index d3ca4b12c..fe1cbb6c1 100644 --- a/docs/source/pcapkit/protocols/application/httpv2.rst +++ b/docs/source/pcapkit/protocols/application/httpv2.rst @@ -20,6 +20,17 @@ Octets Bits Name Description 9 72 ``http.payload`` Frame Payload ======= ========= ===================== ========================== +.. versionchanged:: 1.5.0 + + A frame's Length counts its payload alone, not the 9-octet frame header, + as :rfc:`9113#section-4.1` defines it, and the payload is read to that + Length rather than to the end of the buffer (:issue:`1121`, + :issue:`1243`). A frame rebuilds byte for byte: ``END_STREAM``, undefined + flag bits, the reserved bits, a zero Pad Length, duplicate SETTINGS + entries and the payload of an unpadded DATA, HEADERS or PUSH_PROMISE frame + are kept (:issue:`652`, :issue:`668`, :issue:`1244`, :issue:`1245`, + :issue:`1246`, :issue:`1247`). + .. raw:: html
diff --git a/docs/source/pcapkit/protocols/application/ngap.rst b/docs/source/pcapkit/protocols/application/ngap.rst index 9271442f2..ffcbeeb2c 100644 --- a/docs/source/pcapkit/protocols/application/ngap.rst +++ b/docs/source/pcapkit/protocols/application/ngap.rst @@ -91,7 +91,7 @@ Known Limitations * **NGAP over a fragmented SCTP association is not reassembled.** A DATA chunk is decoded on its own, so an ``NGAP-PDU`` split across chunks by SCTP fragmentation fails to decode rather than being reassembled first. -* **Private IEs (``PrivateMessage``) carry no schema.** Their contents are +* **Private IEs** (``PrivateMessage``) **carry no schema.** Their contents are vendor defined, so the generic conversion reports whatever ASN.1 shape the encoding declares and cannot name the fields. diff --git a/docs/source/pcapkit/protocols/application/ospf.rst b/docs/source/pcapkit/protocols/application/ospf.rst index 22071ec53..496b95941 100644 --- a/docs/source/pcapkit/protocols/application/ospf.rst +++ b/docs/source/pcapkit/protocols/application/ospf.rst @@ -31,6 +31,25 @@ as below: 16 128 ``ospf.auth`` Authentication ====== ===== ================== =============================== +.. versionadded:: 1.5.0 + + Moved here from ``pcapkit.protocols.link.ospf``, with its data model and + header schema. :class:`~pcapkit.protocols.application.ospf.OSPF` derives + from :class:`~pcapkit.protocols.application.application.Application`, so + its layer is ``'Application'`` (:issue:`719`). + +.. versionremoved:: 1.5.0 + + ``pcapkit.protocols.link.ospf``, ``pcapkit.protocols.data.link.ospf`` and + ``pcapkit.protocols.schema.link.ospf``, with no alias (:issue:`719`). + +.. versionchanged:: 1.5.0 + + A rebuild keeps the parsed Packet Length and the octets past it, where + :meth:`~pcapkit.protocols.application.ospf.OSPF.make` recomputed the + length from the payload and dropped those octets; it takes them as + ``packet_length`` and ``trailer`` (:issue:`1249`, :issue:`1455`). + .. raw:: html
diff --git a/docs/source/pcapkit/protocols/application/rarp.rst b/docs/source/pcapkit/protocols/application/rarp.rst index 19d77bda3..a638d9065 100644 --- a/docs/source/pcapkit/protocols/application/rarp.rst +++ b/docs/source/pcapkit/protocols/application/rarp.rst @@ -24,6 +24,19 @@ Octets Bits Name Description 24 192 ``rarp.tpa`` Target Protocol Address ====== ========= ========================= ========================= +.. versionadded:: 1.5.0 + + Moved here from ``pcapkit.protocols.link.rarp``. + :class:`~pcapkit.protocols.application.rarp.RARP` and + :class:`~pcapkit.protocols.application.rarp.DRARP` derive from + :class:`~pcapkit.protocols.application.application.Application` as well as + :class:`~pcapkit.protocols.link.arp.ARP`, so their layer is + ``'Application'`` (:issue:`719`). + +.. versionremoved:: 1.5.0 + + ``pcapkit.protocols.link.rarp``, with no alias (:issue:`719`). + .. autoclass:: pcapkit.protocols.application.rarp.RARP :no-members: :show-inheritance: diff --git a/docs/source/pcapkit/protocols/misc/pcap.rst b/docs/source/pcapkit/protocols/misc/pcap.rst index d8f4a4ede..2d5aafa28 100644 --- a/docs/source/pcapkit/protocols/misc/pcap.rst +++ b/docs/source/pcapkit/protocols/misc/pcap.rst @@ -33,6 +33,11 @@ of PCAP, whose structure is described as below: guint32 network; /* data link type */ } pcap_hdr_t; +.. versionchanged:: 1.5.0 + + A global header cut short rebuilds as captured, where it rebuilt at full + width (:issue:`1470`). + .. autoclass:: pcapkit.protocols.misc.pcap.header.Header :no-members: :show-inheritance: @@ -54,8 +59,6 @@ of PCAP, whose structure is described as below: .. automethod:: _make_data - .. automethod:: __post_init__ - Header Schemas -------------- @@ -108,6 +111,14 @@ whose structure is described as below: guint32 orig_len; /* actual length of packet */ } pcaprec_hdr_t; +.. versionchanged:: 1.5.0 + + ``bytes(frame)`` and :attr:`~pcapkit.protocols.misc.pcap.frame.Frame.packet` + hold this frame's own octets, where they held the next frame's + (:issue:`357`), and the parsed ``time`` is timezone-aware UTC. The parsed + ``len`` is the on-wire length and ``cap_len`` the captured one, where the + two were swapped (:issue:`618`). + .. autoclass:: pcapkit.protocols.misc.pcap.frame.Frame :no-members: :show-inheritance: diff --git a/docs/source/pcapkit/protocols/misc/pcapng.rst b/docs/source/pcapkit/protocols/misc/pcapng.rst index dcf0c92bc..4a2ee0439 100644 --- a/docs/source/pcapkit/protocols/misc/pcapng.rst +++ b/docs/source/pcapkit/protocols/misc/pcapng.rst @@ -5,8 +5,28 @@ PCAP-NG File Format .. module:: pcapkit.protocols.misc.pcapng :mod:`pcapkit.protocols.misc.pcapng` contains -:class:`~pcapkit.protocols.misc.pcapng.PCAPNG` only, -which implements extractor for PCAP-NG file format [*]_. +:class:`~pcapkit.protocols.misc.pcapng.PCAPNG`, +which implements extractor for PCAP-NG file format [*]_, and the +:class:`~pcapkit.protocols.misc.pcapng.PacketDirection`, +:class:`~pcapkit.protocols.misc.pcapng.PacketReception` and +:class:`~pcapkit.protocols.misc.pcapng.WireGuardKeyLabel` enumerations. + +.. versionchanged:: 1.5.0 + + A block rebuilds as it was parsed, byte for byte: its lengths, padding, + byte order, flag bits and repeated options are kept, and so is a block + that is unknown, malformed, shorter than its fixed fields or cut short + (:issue:`1272`, :issue:`1273`, :issue:`1276`, :issue:`1277`, + :issue:`1278`, :issue:`1279`, :issue:`1280`, :issue:`1281`, + :issue:`1282`, :issue:`1283`, :issue:`1284`, :issue:`1382`, + :issue:`1383`, :issue:`1384`, :issue:`1405`, :issue:`1413`, + :issue:`1414`, :issue:`1422`, :issue:`1470`). A packet block keeps the + octets it captured (:issue:`646`); an option running past its block's + option area leaves the whole area kept as captured in ``options_raw``, + with a :exc:`~pcapkit.utilities.warnings.ProtocolWarning`, where it was + read as zeros (:issue:`594`, :issue:`1325`); a truncated capture does not + raise :exc:`ValueError` (:issue:`678`); and ``timestamp_epoch`` is not + shifted by the host's timezone (:issue:`361`). .. autoclass:: pcapkit.protocols.misc.pcapng.PCAPNG :no-members: @@ -206,7 +226,7 @@ IANA registry. .. versionremoved:: 1.5.0 ``pcapkit.protocols.misc.pcapng.TLSKeyLabel``, now - :class:`pcapkit.const.pcapng.tls_key_label.TLSKeyLabel` (:issue:`890`, + :class:`pcapkit.const.pcapng.tls_key_label.TLSKeyLabel` (:issue:`886`, :issue:`1744`). .. autoclass:: pcapkit.protocols.misc.pcapng.WireGuardKeyLabel diff --git a/docs/source/pcapkit/protocols/protocol.rst b/docs/source/pcapkit/protocols/protocol.rst index da2dfe91a..266d1cb36 100644 --- a/docs/source/pcapkit/protocols/protocol.rst +++ b/docs/source/pcapkit/protocols/protocol.rst @@ -18,6 +18,14 @@ derive from :class:`~pcapkit.protocols.protocol.ProtocolBase`, whose metaclass is :class:`~pcapkit.protocols.protocol.ProtocolMeta`; both are listed under Internal Definitions below. +.. versionchanged:: 1.5.0 + + Constructing a protocol with a keyword that none of its signatures + declares raises :exc:`~pcapkit.utilities.exceptions.UnsupportedCall`, + where it was discarded (:issue:`617`). A next layer captured shorter than + its header is kept as :class:`~pcapkit.protocols.misc.raw.Raw`, where it + was parsed with zero-filled fields (:issue:`1170`). + .. autoclass:: pcapkit.protocols.protocol.Protocol :no-members: :show-inheritance: diff --git a/pcapkit/protocols/application/application.py b/pcapkit/protocols/application/application.py index e767098cd..5a4dcf941 100644 --- a/pcapkit/protocols/application/application.py +++ b/pcapkit/protocols/application/application.py @@ -10,6 +10,15 @@ :class:`HTTP/1.* ` and :class:`HTTP/2 `. +.. versionchanged:: 1.5.0 + + :meth:`~pcapkit.protocols.application.application.Application._decode_next_layer` + and + :meth:`~pcapkit.protocols.application.application.Application._import_next_layer` + accept the ``-1`` sentinel, so an application layer protocol can carry the + rest of its packet undissected as its payload; every call raised + :exc:`~pcapkit.utilities.exceptions.UnsupportedCall` (:issue:`719`). + """ from typing import TYPE_CHECKING, Generic, overload diff --git a/pcapkit/protocols/application/ftp.py b/pcapkit/protocols/application/ftp.py index 3cc561c1d..5ab11f1c4 100644 --- a/pcapkit/protocols/application/ftp.py +++ b/pcapkit/protocols/application/ftp.py @@ -13,6 +13,14 @@ .. [*] https://en.wikipedia.org/wiki/File_Transfer_Protocol +.. versionchanged:: 1.5.0 + + A command or reply rebuilds byte for byte, its CRLF included, and keeps + the command's case and any octets that are not UTF-8 as captured. A line + with no argument and a multi-line reply, whole in one segment or not, + parse and rebuild as captured (:issue:`1238`, :issue:`1239`, + :issue:`1240`, :issue:`1241`, :issue:`1303`, :issue:`1333`). + """ import re from typing import TYPE_CHECKING @@ -113,7 +121,7 @@ def read(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_FTP': # NOTE: A multi-line reply (:rfc:`959#section-4`) may arrive whole in one # segment. Its first line is parsed as any reply line, and the lines # after it are kept as received in ``lines``. A segment holding anything - # else -- no closing line, or more after it -- is refused as before. + # else -- no closing line, or more after it -- is refused. line, lines = data, () # type: bytes, tuple[bytes, ...] if FTP_RESPONSE.match(data) is None and data.endswith(b'\r\n'): first, _, rest = data.partition(b'\r\n') diff --git a/pcapkit/protocols/application/http.py b/pcapkit/protocols/application/http.py index f0adf15d4..0e1725b50 100644 --- a/pcapkit/protocols/application/http.py +++ b/pcapkit/protocols/application/http.py @@ -494,7 +494,7 @@ def _guess_version(self, length: 'int', **kwargs: 'Any') -> 'HTTP': Two things this deliberately does **not** decide, because one payload with no flow context cannot: - * **An ``Upgrade: h2c`` exchange** (:rfc:`7540#section-3.2`, + * **An** ``Upgrade: h2c`` **exchange** (:rfc:`7540#section-3.2`, deprecated but not removed by :rfc:`9113#section-3.1`) stays HTTP/1.1 here, and is *correctly* HTTP/1.1: on the wire the upgrade request and its ``101 Switching Protocols`` response are HTTP/1.1 @@ -508,7 +508,7 @@ def _guess_version(self, length: 'int', **kwargs: 'Any') -> 'HTTP': preface ahead of it, or an opaque HTTP/1 message body -- is genuinely undecidable from one payload, and the honest answer is the ``Raw`` that an escaping ``ProtocolError`` becomes under - :func:`~pcapkit.protocols.misc.raw.beholder`. The nine-octet frame + :func:`~pcapkit.corekit.decorators.beholder`. The nine-octet frame header is therefore never used to *identify* HTTP/2 ahead of HTTP/1: as identification it misfires on binary HTTP/1 bodies, which is precisely how garbage text was classified HTTP/2 to begin diff --git a/pcapkit/protocols/application/httpv1.py b/pcapkit/protocols/application/httpv1.py index 1eaf70c14..f2448191e 100644 --- a/pcapkit/protocols/application/httpv1.py +++ b/pcapkit/protocols/application/httpv1.py @@ -5,10 +5,11 @@ .. module:: pcapkit.protocols.application.httpv1 :mod:`pcapkit.protocols.application.httpv1` contains -:class:`~pcapkit.protocols.application.httpv1.HTTP` -only, which implements extractor for Hypertext Transfer -Protocol (HTTP/1.*) [*]_, whose structure is described -as below: +:class:`~pcapkit.protocols.application.httpv1.HTTP`, +which implements extractor for Hypertext Transfer +Protocol (HTTP/1.*) [*]_, and its +:class:`~pcapkit.protocols.application.httpv1.Type` enumeration. The +structure of an HTTP/1.* message is described as below: .. code-block:: text @@ -26,6 +27,13 @@ .. [*] https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol +.. versionchanged:: 1.5.0 + + A message rebuilds byte for byte: the whitespace of the start line and + header lines, and octets in them that are not UTF-8, are kept as captured + rather than normalised or re-encoded (:issue:`1241`, :issue:`1302`, + :issue:`1333`). + """ import re from typing import TYPE_CHECKING diff --git a/pcapkit/protocols/application/httpv2.py b/pcapkit/protocols/application/httpv2.py index 04335ea06..322351532 100644 --- a/pcapkit/protocols/application/httpv2.py +++ b/pcapkit/protocols/application/httpv2.py @@ -23,6 +23,17 @@ .. [*] https://en.wikipedia.org/wiki/HTTP/2 +.. versionchanged:: 1.5.0 + + A frame's Length counts its payload alone, not the 9-octet frame header, + as :rfc:`9113#section-4.1` defines it, and the payload is read to that + Length rather than to the end of the buffer (:issue:`1121`, + :issue:`1243`). A frame rebuilds byte for byte: ``END_STREAM``, undefined + flag bits, the reserved bits, a zero Pad Length, duplicate SETTINGS + entries and the payload of an unpadded DATA, HEADERS or PUSH_PROMISE frame + are kept (:issue:`652`, :issue:`668`, :issue:`1244`, :issue:`1245`, + :issue:`1246`, :issue:`1247`). + """ import collections from typing import TYPE_CHECKING, cast diff --git a/pcapkit/protocols/application/ngap.py b/pcapkit/protocols/application/ngap.py index 835aaa225..b0b3c6a79 100644 --- a/pcapkit/protocols/application/ngap.py +++ b/pcapkit/protocols/application/ngap.py @@ -90,7 +90,7 @@ * **NGAP over a fragmented SCTP association is not reassembled.** A DATA chunk is decoded on its own, so an ``NGAP-PDU`` split across chunks by SCTP fragmentation fails to decode rather than being reassembled first. -* **Private IEs (``PrivateMessage``) carry no schema.** Their contents are +* **Private IEs** (``PrivateMessage``) **carry no schema.** Their contents are vendor defined, so the generic conversion reports whatever ASN.1 shape the encoding declares and cannot name the fields. @@ -187,7 +187,7 @@ def load_pycrate() -> 'Optional[Any]': # dependency this module needs to decode NGAP at all. They therefore live under # pcapkit.const.ngap, generated by pcapkit.vendor.ngap.procedure_code and # pcapkit.vendor.ngap.protocol_ie, and are imported here as Enum_ProcedureCode -# and Enum_ProtocolIE; this module no longer re-exports them under their +# and Enum_ProtocolIE; this module does not re-export them under their # original names (#1744). Only the vendor crawler that generates the const module # needs pycrate; neither this module nor the generated one does at import. # diff --git a/pcapkit/protocols/application/ospf.py b/pcapkit/protocols/application/ospf.py index b5bff2ec3..49c0f985a 100644 --- a/pcapkit/protocols/application/ospf.py +++ b/pcapkit/protocols/application/ospf.py @@ -34,6 +34,25 @@ .. [*] https://en.wikipedia.org/wiki/Open_Shortest_Path_First +.. versionadded:: 1.5.0 + + Moved here from ``pcapkit.protocols.link.ospf``, with its data model and + header schema. :class:`~pcapkit.protocols.application.ospf.OSPF` derives + from :class:`~pcapkit.protocols.application.application.Application`, so + its layer is ``'Application'`` (:issue:`719`). + +.. versionremoved:: 1.5.0 + + ``pcapkit.protocols.link.ospf``, ``pcapkit.protocols.data.link.ospf`` and + ``pcapkit.protocols.schema.link.ospf``, with no alias (:issue:`719`). + +.. versionchanged:: 1.5.0 + + A rebuild keeps the parsed Packet Length and the octets past it, where + :meth:`~pcapkit.protocols.application.ospf.OSPF.make` recomputed the + length from the payload and dropped those octets; it takes them as + ``packet_length`` and ``trailer`` (:issue:`1249`, :issue:`1455`). + """ import ipaddress import re diff --git a/pcapkit/protocols/application/rarp.py b/pcapkit/protocols/application/rarp.py index 2be5bcde6..22b035483 100644 --- a/pcapkit/protocols/application/rarp.py +++ b/pcapkit/protocols/application/rarp.py @@ -27,6 +27,19 @@ .. [*] http://en.wikipedia.org/wiki/Address_Resolution_Protocol +.. versionadded:: 1.5.0 + + Moved here from ``pcapkit.protocols.link.rarp``. + :class:`~pcapkit.protocols.application.rarp.RARP` and + :class:`~pcapkit.protocols.application.rarp.DRARP` derive from + :class:`~pcapkit.protocols.application.application.Application` as well as + :class:`~pcapkit.protocols.link.arp.ARP`, so their layer is + ``'Application'`` (:issue:`719`). + +.. versionremoved:: 1.5.0 + + ``pcapkit.protocols.link.rarp``, with no alias (:issue:`719`). + """ from typing import TYPE_CHECKING diff --git a/pcapkit/protocols/data/application/ospf.py b/pcapkit/protocols/data/application/ospf.py index e1fe7806c..cc9b60624 100644 --- a/pcapkit/protocols/data/application/ospf.py +++ b/pcapkit/protocols/data/application/ospf.py @@ -18,7 +18,20 @@ @info_final class CryptographicAuthentication(Data): - """Data model for OSPF cryptographic authentication.""" + """Data model for OSPF cryptographic authentication. + + .. versionadded:: 1.5.0 + + Renamed from ``CrytographicAuthentication``, which + :mod:`pcapkit.protocols.data` exported as + ``OSPF_CrytographicAuthentication`` (:issue:`1090`). + + .. versionremoved:: 1.5.0 + + ``CrytographicAuthentication`` and ``OSPF_CrytographicAuthentication``, + with no alias (:issue:`1090`). + + """ #: The two reserved octets, as captured. reserved: 'bytes' diff --git a/pcapkit/protocols/misc/__init__.py b/pcapkit/protocols/misc/__init__.py index f3baa62bd..65e395ed2 100644 --- a/pcapkit/protocols/misc/__init__.py +++ b/pcapkit/protocols/misc/__init__.py @@ -8,7 +8,7 @@ :mod:`pcapkit.protocols.misc` contains the auxiliary protocol implementations. Such includes the :class:`~pcapkit.protocols.misc.raw.Raw` class for not-supported protocols, the :class:`~pcapkit.protocols.misc.null.NoPayload` class for -indication of empty payload, and PCAP header classes. +indication of empty payload, and the PCAP and PCAP-NG file format classes. """ diff --git a/pcapkit/protocols/misc/pcap/frame.py b/pcapkit/protocols/misc/pcap/frame.py index d5c8ed700..0f49e5927 100644 --- a/pcapkit/protocols/misc/pcap/frame.py +++ b/pcapkit/protocols/misc/pcap/frame.py @@ -21,6 +21,14 @@ .. [*] https://wiki.wireshark.org/Development/LibpcapFileFormat#Record_.28Packet.29_Header +.. versionchanged:: 1.5.0 + + ``bytes(frame)`` and :attr:`~pcapkit.protocols.misc.pcap.frame.Frame.packet` + hold this frame's own octets, where they held the next frame's + (:issue:`357`), and the parsed ``time`` is timezone-aware UTC. The parsed + ``len`` is the on-wire length and ``cap_len`` the captured one, where the + two were swapped (:issue:`618`). + """ import collections import datetime @@ -62,7 +70,7 @@ class Frame(ProtocolBase[Data_Frame, Schema_Frame], schema=Schema_Frame, data=Data_Frame): """Per packet frame header extractor. - This class currently supports parsing of the following protocols, which are + This class supports parsing of the following protocols, which are registered in the :attr:`self.__proto__ ` attribute: @@ -142,11 +150,11 @@ def register(cls, code: 'Enum_LinkType', protocol: 'ModuleDescriptor[ProtocolBas Arguments: code: protocol code as in :class:`~pcapkit.const.reg.linktype.LinkType` protocol: module descriptor or a - :class:`~pcapkit.protocols.protocol.Protocol` subclass + :class:`~pcapkit.protocols.protocol.ProtocolBase` subclass Raises: pcapkit.utilities.exceptions.RegistryError: If ``protocol`` is not a - class, or not a :class:`~pcapkit.protocols.protocol.Protocol` subclass. + class, or not a :class:`~pcapkit.protocols.protocol.ProtocolBase` subclass. Warns: pcapkit.utilities.warnings.RegistryWarning: If this link type is diff --git a/pcapkit/protocols/misc/pcap/header.py b/pcapkit/protocols/misc/pcap/header.py index 1ec2c64bf..a4bccf661 100644 --- a/pcapkit/protocols/misc/pcap/header.py +++ b/pcapkit/protocols/misc/pcap/header.py @@ -23,6 +23,11 @@ .. [*] https://wiki.wireshark.org/Development/LibpcapFileFormat#Global_Header +.. versionchanged:: 1.5.0 + + A global header cut short rebuilds as captured, where it rebuilt at full + width (:issue:`1470`). + """ import io import operator @@ -51,7 +56,7 @@ #: The PCAP magic number for each byte order and nanosecond flag, the reverse of #: :data:`~pcapkit.protocols.schema.misc.pcap.header.MAGIC_NUMBER`, so that #: :meth:`Header._make_magic` looks one up rather than searching for it -- an -#: undeclared pair raises :exc:`KeyError`, as it did before :issue:`1739`. +#: undeclared pair raises :exc:`KeyError`. _DECLARED_MAGIC = {declared: magic for magic, declared in MAGIC_NUMBER.items()} @@ -145,8 +150,7 @@ def read(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_Header' _magn = schema.magic_number # NOTE: looked up as :obj:`bytes`, so that a :obj:`bytearray` magic - # number is matched as it was compared before :issue:`1739` rather - # than rejected as unhashable. + # number is matched rather than rejected as unhashable. _decl = (MAGIC_NUMBER.get(bytes(_magn)) if isinstance(_magn, (bytes, bytearray, memoryview)) else None) if _decl is None: diff --git a/pcapkit/protocols/misc/pcapng.py b/pcapkit/protocols/misc/pcapng.py index 4e3c5c6b5..a0c973a87 100644 --- a/pcapkit/protocols/misc/pcapng.py +++ b/pcapkit/protocols/misc/pcapng.py @@ -6,17 +6,37 @@ .. module:: pcapkit.protocols.misc.pcapng :mod:`pcapkit.protocols.misc.pcapng` contains -:class:`~pcapkit.protocols.misc.pcapng.PCAPNG` only, -which implements extractor for PCAP-NG file format [*]_. +:class:`~pcapkit.protocols.misc.pcapng.PCAPNG`, +which implements extractor for PCAP-NG file format [*]_, and the +:class:`~pcapkit.protocols.misc.pcapng.PacketDirection`, +:class:`~pcapkit.protocols.misc.pcapng.PacketReception` and +:class:`~pcapkit.protocols.misc.pcapng.WireGuardKeyLabel` enumerations. .. [*] https://www.ietf.org/archive/id/draft-tuexen-opsawg-pcapng-03.html .. versionremoved:: 1.5.0 ``pcapkit.protocols.misc.pcapng.TLSKeyLabel``, now - :class:`pcapkit.const.pcapng.tls_key_label.TLSKeyLabel` (:issue:`890`, + :class:`pcapkit.const.pcapng.tls_key_label.TLSKeyLabel` (:issue:`886`, :issue:`1744`). +.. versionchanged:: 1.5.0 + + A block rebuilds as it was parsed, byte for byte: its lengths, padding, + byte order, flag bits and repeated options are kept, and so is a block + that is unknown, malformed, shorter than its fixed fields or cut short + (:issue:`1272`, :issue:`1273`, :issue:`1276`, :issue:`1277`, + :issue:`1278`, :issue:`1279`, :issue:`1280`, :issue:`1281`, + :issue:`1282`, :issue:`1283`, :issue:`1284`, :issue:`1382`, + :issue:`1383`, :issue:`1384`, :issue:`1405`, :issue:`1413`, + :issue:`1414`, :issue:`1422`, :issue:`1470`). A packet block keeps the + octets it captured (:issue:`646`); an option running past its block's + option area leaves the whole area kept as captured in ``options_raw``, + with a :exc:`~pcapkit.utilities.warnings.ProtocolWarning`, where it was + read as zeros (:issue:`594`, :issue:`1325`); a truncated capture does not + raise :exc:`ValueError` (:issue:`678`); and ``timestamp_epoch`` is not + shifted by the host's timezone (:issue:`361`). + """ import base64 import builtins @@ -399,7 +419,7 @@ def _missing_(cls, value: 'object') -> 'PacketReception': # closed, hand-picked set, so the TLS key label enumeration is the generated # :class:`pcapkit.const.pcapng.tls_key_label.TLSKeyLabel`, imported above as # ``Enum_TLSKeyLabel`` like its seven :mod:`pcapkit.const.pcapng` siblings. This -# module no longer re-exports it under the unaliased name (:issue:`1744`). +# module does not re-export it under the unaliased name (:issue:`1744`). # :class:`WireGuardKeyLabel` below stays hand-written: it is a closed set # (draft-ietf-opsawg-pcapng-06 section 4.7's "is one of" four names), not a # registry, so filing it under :mod:`pcapkit.const` would misrepresent it as @@ -426,7 +446,7 @@ class PCAPNG(ProtocolBase[Data_PCAPNG, Schema_PCAPNG], schema=Schema_PCAPNG, data=Data_PCAPNG): """PCAP-NG file block extractor. - The class currently supports parsing of the following protocols, which are + The class supports parsing of the following protocols, which are registered in the :attr:`self.__proto__ ` attribute: @@ -450,7 +470,7 @@ class PCAPNG(ProtocolBase[Data_PCAPNG, Schema_PCAPNG], its data is dispatched as ``IPV4`` or ``IPV6`` by the version nibble of its first octet, see :meth:`_dispatch_linktype`. - The class currently supports parsing of the following block types, which + The class supports parsing of the following block types, which are registered in the :attr:`self.__block__ ` attribute: @@ -494,7 +514,7 @@ class PCAPNG(ProtocolBase[Data_PCAPNG, Schema_PCAPNG], - :meth:`~pcapkit.protocols.misc.pcapng.PCAPNG._read_block_packet` - :meth:`~pcapkit.protocols.misc.pcapng.PCAPNG._make_block_packet` - The class currently supports parsing of the following option types, which + The class supports parsing of the following option types, which are registered in the :attr:`self.__option__ ` attribute: @@ -619,7 +639,7 @@ class PCAPNG(ProtocolBase[Data_PCAPNG, Schema_PCAPNG], - :meth:`~pcapkit.protocols.misc.pcapng.PCAPNG._read_option_pack_hash` - :meth:`~pcapkit.protocols.misc.pcapng.PCAPNG._make_option_pack_hash` - The class currently supports parsing of the following name resolution + The class supports parsing of the following name resolution record types, which are registered in the :attr:`self.__record__ ` attribute: @@ -639,7 +659,7 @@ class PCAPNG(ProtocolBase[Data_PCAPNG, Schema_PCAPNG], - :meth:`~pcapkit.protocols.misc.pcapng.PCAPNG._read_record_ipv6` - :meth:`~pcapkit.protocols.misc.pcapng.PCAPNG._make_record_ipv6` - The class currently supports parsing of the following decryption secrets + The class supports parsing of the following decryption secrets types, which are registered in the :attr:`self.__secrets__ ` attribute: @@ -1018,11 +1038,11 @@ def register(cls, code: 'Enum_LinkType', protocol: 'ModuleDescriptor[ProtocolBas Arguments: code: protocol code as in :class:`~pcapkit.const.reg.linktype.LinkType` protocol: module descriptor or a - :class:`~pcapkit.protocols.protocol.Protocol` subclass + :class:`~pcapkit.protocols.protocol.ProtocolBase` subclass Raises: pcapkit.utilities.exceptions.RegistryError: If ``protocol`` is not a - class, or not a :class:`~pcapkit.protocols.protocol.Protocol` subclass. + class, or not a :class:`~pcapkit.protocols.protocol.ProtocolBase` subclass. Warns: pcapkit.utilities.warnings.RegistryWarning: If this link type is @@ -1618,7 +1638,7 @@ def _get_payload(self) -> 'bytes': Payload of :attr:`self.__header__ ` as :obj:`bytes`. See Also: - This is a wrapper function for :meth:`pcapkit.protocols.schema.Schema.get_payload`. + This is a wrapper function for :meth:`pcapkit.protocols.schema.schema.Schema.get_payload`. Note: A packet block kept as captured has no payload (see @@ -1889,8 +1909,8 @@ def _read_padding(schema: 'Schema', data: 'Any', name: 'str' = 'padding', form a fresh build writes -- rather than whenever there are any. The octets are kept as ``key`` only when there is something a rebuild - would otherwise lose, so a data model reading as before for well-formed - input gains no key (:issue:`1437`). :meth:`_make_padding` writes them back. + would otherwise lose, so the data model of well-formed input gains no + key (:issue:`1437`). :meth:`_make_padding` writes them back. """ padding = schema.__dict__.get(name) diff --git a/pcapkit/protocols/protocol.py b/pcapkit/protocols/protocol.py index 5fc80574b..5e44b1058 100644 --- a/pcapkit/protocols/protocol.py +++ b/pcapkit/protocols/protocol.py @@ -6,9 +6,18 @@ .. module:: pcapkit.protocols.protocol :mod:`pcapkit.protocols.protocol` contains -:class:`~pcapkit.protocols.protocol.Protocol` only, which is -an abstract base class for all protocol family, with pre-defined -utility arguments and methods of specified protocols. +:class:`~pcapkit.protocols.protocol.Protocol`, the abstract base class for a +new protocol stack, and :class:`~pcapkit.protocols.protocol.ProtocolBase`, +from which the built-in protocol families derive, with its metaclass +:class:`~pcapkit.protocols.protocol.ProtocolMeta`. + +.. versionchanged:: 1.5.0 + + Constructing a protocol with a keyword that none of its signatures + declares raises :exc:`~pcapkit.utilities.exceptions.UnsupportedCall`, + where it was discarded (:issue:`617`). A next layer captured shorter than + its header is kept as :class:`~pcapkit.protocols.misc.raw.Raw`, where it + was parsed with zero-filled fields (:issue:`1170`). .. versionremoved:: 1.5.0 @@ -346,9 +355,14 @@ def _check_construction_keywords(cls: 'type', kwargs: 'dict[str, Any]', Args: cls: Protocol class being constructed. - kwargs: Keywords remaining after :meth:`ProtocolBase.__init__ - ` has consumed the - out-of-band ones. + kwargs: Construction keywords to check, the out-of-band ones left out. + :meth:`ProtocolBase.__init__ + ` passes those + remaining after it has consumed them; :meth:`ProtocolBase.from_data + ` passes its + caller's own, less ``_layer``, ``_protocol`` and ``__context__``, + before :meth:`ProtocolBase._make_data + ` adds to them. strict: Whether an unexpected keyword is an error. :data:`True` for a caller's own construction; :data:`False` when the keywords were generated by :meth:`ProtocolBase._make_data @@ -357,8 +371,8 @@ def _check_construction_keywords(cls: 'type', kwargs: 'dict[str, Any]', `. Raises: - UnsupportedCall: If ``strict`` and any keyword matches no parameter of - :data:`_KEYWORD_CONSUMERS` anywhere in the MRO of ``cls``. + UnsupportedCall: If ``strict`` and any keyword is not among those + :func:`_declared_keywords` collects for ``cls``. Warns: UnknownFieldWarning: The same finding when not ``strict``. @@ -761,6 +775,13 @@ def pack(self, **kwargs: 'Any') -> 'bytes': global packet data to underlying methods. This is useful when the packet data is not available in the current instance. + The octets are built from ``kwargs`` alone, through :meth:`make`; + the instance's own parsed or constructed state is not read. A bare + ``SomeProtocol(...).pack()`` therefore returns the octets of a + default header, not the instance's: :attr:`data` holds those. The + schema made here also replaces the instance's + :attr:`__header__ `. + """ self.__header__ = self.make(**kwargs) #: bool: If the made header may hold raw items (see :meth:`unpack`). @@ -951,7 +972,7 @@ def register(cls, code: 'int', protocol: 'ModuleDescriptor | Type[ProtocolBase]' Arguments: code: protocol code protocol: module descriptor or a - :class:`~pcapkit.protocols.protocol.Protocol` subclass + :class:`~pcapkit.protocols.protocol.ProtocolBase` subclass Raises: pcapkit.utilities.exceptions.RegistryError: If ``protocol`` is not a @@ -1361,7 +1382,7 @@ def __init_subclass__(cls, /, schema: 'Optional[Type[_ST]]' = None, :class:`~pcapkit.protocols.schema.misc.raw.Raw` and :class:`~pcapkit.protocols.data.misc.raw.Raw` classes will be used. - Dispatch registration is **opt-in**, like the ``name=``/``protocol=``/ + Dispatch registration is **opt-in**, like the ``engine=``/``protocol=``/ ``fmt=`` keywords of :class:`~pcapkit.foundation.engines.engine.Engine`, :class:`~pcapkit.foundation.reassembly.reassembly.Reassembly`, :class:`~pcapkit.foundation.traceflow.traceflow.TraceFlow` and diff --git a/pcapkit/protocols/schema/application/ospf.py b/pcapkit/protocols/schema/application/ospf.py index c28108c74..58e482acf 100644 --- a/pcapkit/protocols/schema/application/ospf.py +++ b/pcapkit/protocols/schema/application/ospf.py @@ -41,7 +41,20 @@ def ospf_auth_data_selector(pkt: 'dict[str, Any]') -> 'Field': @schema_final class CryptographicAuthentication(Schema): - """Header schema for OSPF cryptographic authentication.""" + """Header schema for OSPF cryptographic authentication. + + .. versionadded:: 1.5.0 + + Renamed from ``CrytographicAuthentication``, which + :mod:`pcapkit.protocols.schema` exported as + ``OSPF_CrytographicAuthentication`` (:issue:`1090`). + + .. versionremoved:: 1.5.0 + + ``CrytographicAuthentication`` and ``OSPF_CrytographicAuthentication``, + with no alias (:issue:`1090`). + + """ #: Reserved bytes. reserved: 'bytes' = PaddingField(length=2) diff --git a/pcapkit/protocols/schema/misc/pcap/header.py b/pcapkit/protocols/schema/misc/pcap/header.py index 42c66a886..631d9d067 100644 --- a/pcapkit/protocols/schema/misc/pcap/header.py +++ b/pcapkit/protocols/schema/misc/pcap/header.py @@ -71,8 +71,7 @@ def magic_number_callback(field: 'NumberField', packet: 'dict[str, Any]') -> 'No """ magic_number = packet['magic_number'] # NOTE: looked up as :obj:`bytes`, so that a :obj:`bytearray` magic number - # is matched as it was compared before :issue:`1739` rather than rejected - # as unhashable. + # is matched rather than rejected as unhashable. declared = (MAGIC_NUMBER.get(bytes(magic_number)) if isinstance(magic_number, (bytes, bytearray, memoryview)) else None) if declared is None: diff --git a/pcapkit/protocols/schema/misc/pcapng.py b/pcapkit/protocols/schema/misc/pcapng.py index b1c1d36aa..d8e8c7081 100644 --- a/pcapkit/protocols/schema/misc/pcapng.py +++ b/pcapkit/protocols/schema/misc/pcapng.py @@ -737,8 +737,9 @@ def post_process(self, value: 'int | bytes', packet: 'dict[str, Any]') -> 'Enum_ instead of ``get()``'s own miss path, for pickle-safety: that method carries its own ``__reduce_ex__``, so a member built this way round-trips through :mod:`pickle` -- see its own docstring -- which - :meth:`~pcapkit.const.pcapng.option_type.OptionType. - _unregistered_member`'s override, the miss path, does not. The override's + the override of + :meth:`~pcapkit.const.pcapng.option_type.OptionType._unregistered_member`, + the miss path, does not. The override's :attr:`~pcapkit.const.pcapng.option_type.OptionType._value_` is the *formatted* display string (``'opt_unknown [8888]'``, not ``8888``), so ``pickle.dumps`` on one succeeds but ``pickle.loads`` of the result @@ -897,6 +898,11 @@ class NewOption(Option, ns='opt', code=Enum_OptionType.opt_new): See Also: - :class:`pcapkit.const.pcapng.option_type.OptionType` + .. versionchanged:: 1.5.0 + + The class keyword ``namespace`` is renamed ``ns``, with no alias + (:issue:`439`). + """ # NOTE: the base hook goes first, before any of the registration work # below -- see :meth:`EnumSchema.__init_subclass__`, which makes the same diff --git a/pcapkit/protocols/schema/schema.py b/pcapkit/protocols/schema/schema.py index e0db58131..7924216a6 100644 --- a/pcapkit/protocols/schema/schema.py +++ b/pcapkit/protocols/schema/schema.py @@ -78,6 +78,12 @@ class marked with a user's own ``from typing import final`` on 3.10 is :attr:`~pcapkit.corekit.infoclass.FinalisedState.FINAL` in its own ``__dict__``. The class is returned untouched. + .. versionchanged:: 1.5.0 + + A class finalised with ``_finalised=True`` refuses to be subclassed, + and a schema marked ``@final`` without this decorator cannot be + instantiated (:issue:`778`). + :meta decorator: """ # NOTE: keyed on ``__finalised__`` rather than on ``final``'s ``__final__``, @@ -378,7 +384,8 @@ class Schema(Mapping[str, _VT], Generic[_VT], metaclass=SchemaMeta): __fields__: 'OrderedDict[str, FieldBase]' #: Mapping of field names to packed values. __buffer__: 'dict[str, bytes]' - #: Flag for whether the schema is recently updated. + #: Whether the schema has changed since it was last packed, so that + #: :meth:`__bytes__` packs it again. __updated__: 'bool' #: Octets of an enclosing :class:`~pcapkit.corekit.fields.misc.SchemaField` #: span that this nested schema left unread; set only when non-empty. @@ -440,6 +447,16 @@ def __new__(cls, *args: '_VT', **kwargs: '_VT') -> 'Self': # pylint: disable=un the same signature as specified in class variables' type annotations, which is inspired by :pep:`557` (:mod:`dataclasses`). + Note: + Whether ``cls`` still needs finalising is read from its *own* + ``__dict__``, never inherited (GitHub issue :issue:`1498`). Each + class is finalised on its first construction, whatever its + ancestors' state, so a subclass of a + :attr:`~pcapkit.corekit.infoclass.FinalisedState.BASE` ancestor + still gets its own ``__builtin__``, ``__excluded__`` and generated + ``__init__``, and a ``@final`` class under a ``BASE`` ancestor is + refused like any other. + Args: *args: Arbitrary positional arguments. **kwargs: Arbitrary keyword arguments. @@ -451,16 +468,6 @@ def __new__(cls, *args: '_VT', **kwargs: '_VT') -> 'Self': # pylint: disable=un ``__dict__`` without :attr:`~pcapkit.corekit.infoclass.FinalisedState.FINAL`. - Note: - Whether ``cls`` still needs finalising is read from its *own* - ``__dict__``, never inherited (GitHub issue :issue:`1498`). Each - class is finalised on its first construction, whatever its - ancestors' state, so a subclass of a - :attr:`~pcapkit.corekit.infoclass.FinalisedState.BASE` ancestor - still gets its own ``__builtin__``, ``__excluded__`` and generated - ``__init__``, and a ``@final`` class under a ``BASE`` ancestor is - refused like any other. - """ # NOTE: first instantiation is the earliest point a bare ``@final`` can # be caught -- ``final`` runs after the class object exists, so @@ -828,7 +835,7 @@ class will consider negative value as a placeholder. if isinstance(field, PayloadField): # NOTE: ``ProtocolBase``, not ``Protocol``: ``Protocol`` is a thin # subclass that only adds auto-registration for externally defined - # engines, and no protocol in the library subclasses it. Testing + # protocols, and no protocol in the library subclasses it. Testing # against it would send any protocol instance -- which is exactly # what :meth:`ProtocolBase._make_payload # ` returns,