docs(protocols): sweep application, misc and the protocol base (#1103)

docs/source/pcapkit/protocols/application/application.rst

@@ -10,6 +10,15 @@ which is a base class for application layer protocols, eg.
 :class:`HTTP/2 <pcapkit.protocols.application.httpv2.HTTP>`
 and etc.
 
+.. versionchanged:: 1.5.0
+
+   :meth:`~pcapkit.protocols.application.application.Application._decode_next_layer`
+   and
+   :meth:`~pcapkit.protocols.application.application.Application._import_next_layer`
+   accept the ``-1`` sentinel, so an application layer protocol can carry the
+   rest of its packet undissected as its payload; every call raised
+   :exc:`~pcapkit.utilities.exceptions.UnsupportedCall` (:issue:`719`).
+
 .. autoclass:: pcapkit.protocols.application.application.Application
    :no-members:
    :show-inheritance:

docs/source/pcapkit/protocols/application/ftp.rst

@@ -9,6 +9,14 @@ which implements extractor for File Transfer Protocol
 (FTP) [*]_, and :class:`~pcapkit.protocols.application.ftp.FTP_DATA`
 for its data channel.
 
+.. versionchanged:: 1.5.0
+
+   A command or reply rebuilds byte for byte, its CRLF included, and keeps
+   the command's case and any octets that are not UTF-8 as captured. A line
+   with no argument and a multi-line reply, whole in one segment or not,
+   parse and rebuild as captured (:issue:`1238`, :issue:`1239`,
+   :issue:`1240`, :issue:`1241`, :issue:`1303`, :issue:`1333`).
+
 .. autoclass:: pcapkit.protocols.application.ftp.FTP
    :no-members:
    :show-inheritance:

docs/source/pcapkit/protocols/application/httpv1.rst

@@ -4,10 +4,11 @@ HTTP/1.* - Hypertext Transfer Protocol
 .. module:: pcapkit.protocols.application.httpv1
 
 :mod:`pcapkit.protocols.application.httpv1` contains
-:class:`~pcapkit.protocols.application.httpv1.HTTP`
-only, which implements extractor for Hypertext Transfer
-Protocol (HTTP/1.*) [*]_, whose structure is described
-as below:
+:class:`~pcapkit.protocols.application.httpv1.HTTP`,
+which implements extractor for Hypertext Transfer
+Protocol (HTTP/1.*) [*]_, and its
+:class:`~pcapkit.protocols.application.httpv1.Type` enumeration. The
+structure of an HTTP/1.* message is described as below:
 
 .. code-block:: text
 
@@ -23,6 +24,13 @@ as below:
    \r\n                        :==: RESPONSE SEPARATOR
    <body>                      :==: RESPONSE BODY (optional)
 
+.. versionchanged:: 1.5.0
+
+   A message rebuilds byte for byte: the whitespace of the start line and
+   header lines, and octets in them that are not UTF-8, are kept as captured
+   rather than normalised or re-encoded (:issue:`1241`, :issue:`1302`,
+   :issue:`1333`).
+
 .. autoclass:: pcapkit.protocols.application.httpv1.HTTP
    :no-members:
    :show-inheritance:

docs/source/pcapkit/protocols/application/httpv2.rst

@@ -20,6 +20,17 @@ Octets      Bits        Name                    Description
   9          72   ``http.payload``            Frame Payload
 ======= ========= ===================== ==========================
 
+.. versionchanged:: 1.5.0
+
+   A frame's Length counts its payload alone, not the 9-octet frame header,
+   as :rfc:`9113#section-4.1` defines it, and the payload is read to that
+   Length rather than to the end of the buffer (:issue:`1121`,
+   :issue:`1243`). A frame rebuilds byte for byte: ``END_STREAM``, undefined
+   flag bits, the reserved bits, a zero Pad Length, duplicate SETTINGS
+   entries and the payload of an unpadded DATA, HEADERS or PUSH_PROMISE frame
+   are kept (:issue:`652`, :issue:`668`, :issue:`1244`, :issue:`1245`,
+   :issue:`1246`, :issue:`1247`).
+
 .. raw:: html
 
    <br />

docs/source/pcapkit/protocols/application/ngap.rst

@@ -91,7 +91,7 @@ Known Limitations
 * **NGAP over a fragmented SCTP association is not reassembled.** A DATA chunk
   is decoded on its own, so an ``NGAP-PDU`` split across chunks by SCTP
   fragmentation fails to decode rather than being reassembled first.
-* **Private IEs (``PrivateMessage``) carry no schema.** Their contents are
+* **Private IEs** (``PrivateMessage``) **carry no schema.** Their contents are
   vendor defined, so the generic conversion reports whatever ASN.1 shape the
   encoding declares and cannot name the fields.
 

docs/source/pcapkit/protocols/application/ospf.rst

@@ -31,6 +31,25 @@ as below:
    16       128 ``ospf.auth``      Authentication
    ====== ===== ================== ===============================
 
+.. versionadded:: 1.5.0
+
+   Moved here from ``pcapkit.protocols.link.ospf``, with its data model and
+   header schema. :class:`~pcapkit.protocols.application.ospf.OSPF` derives
+   from :class:`~pcapkit.protocols.application.application.Application`, so
+   its layer is ``'Application'`` (:issue:`719`).
+
+.. versionremoved:: 1.5.0
+
+   ``pcapkit.protocols.link.ospf``, ``pcapkit.protocols.data.link.ospf`` and
+   ``pcapkit.protocols.schema.link.ospf``, with no alias (:issue:`719`).
+
+.. versionchanged:: 1.5.0
+
+   A rebuild keeps the parsed Packet Length and the octets past it, where
+   :meth:`~pcapkit.protocols.application.ospf.OSPF.make` recomputed the
+   length from the payload and dropped those octets; it takes them as
+   ``packet_length`` and ``trailer`` (:issue:`1249`, :issue:`1455`).
+
 .. raw:: html
 
    <br />

docs/source/pcapkit/protocols/application/rarp.rst

@@ -24,6 +24,19 @@ Octets      Bits        Name                    Description
   24        192   ``rarp.tpa``              Target Protocol Address
 ====== ========= ========================= =========================
 
+.. versionadded:: 1.5.0
+
+   Moved here from ``pcapkit.protocols.link.rarp``.
+   :class:`~pcapkit.protocols.application.rarp.RARP` and
+   :class:`~pcapkit.protocols.application.rarp.DRARP` derive from
+   :class:`~pcapkit.protocols.application.application.Application` as well as
+   :class:`~pcapkit.protocols.link.arp.ARP`, so their layer is
+   ``'Application'`` (:issue:`719`).
+
+.. versionremoved:: 1.5.0
+
+   ``pcapkit.protocols.link.rarp``, with no alias (:issue:`719`).
+
 .. autoclass:: pcapkit.protocols.application.rarp.RARP
    :no-members:
    :show-inheritance:

docs/source/pcapkit/protocols/misc/pcap.rst

@@ -33,6 +33,11 @@ of PCAP, whose structure is described as below:
        guint32 network;        /* data link type */
    } pcap_hdr_t;
 
+.. versionchanged:: 1.5.0
+
+   A global header cut short rebuilds as captured, where it rebuilt at full
+   width (:issue:`1470`).
+
 .. autoclass:: pcapkit.protocols.misc.pcap.header.Header
    :no-members:
    :show-inheritance:
@@ -54,8 +59,6 @@ of PCAP, whose structure is described as below:
 
    .. automethod:: _make_data
 
-   .. automethod:: __post_init__
-
 Header Schemas
 --------------
 
@@ -108,6 +111,14 @@ whose structure is described as below:
        guint32 orig_len;   /* actual length of packet */
    } pcaprec_hdr_t;
 
+.. versionchanged:: 1.5.0
+
+   ``bytes(frame)`` and :attr:`~pcapkit.protocols.misc.pcap.frame.Frame.packet`
+   hold this frame's own octets, where they held the next frame's
+   (:issue:`357`), and the parsed ``time`` is timezone-aware UTC. The parsed
+   ``len`` is the on-wire length and ``cap_len`` the captured one, where the
+   two were swapped (:issue:`618`).
+
 .. autoclass:: pcapkit.protocols.misc.pcap.frame.Frame
    :no-members:
    :show-inheritance:

docs/source/pcapkit/protocols/misc/pcapng.rst

@@ -5,8 +5,28 @@ PCAP-NG File Format
 .. module:: pcapkit.protocols.misc.pcapng
 
 :mod:`pcapkit.protocols.misc.pcapng` contains
-:class:`~pcapkit.protocols.misc.pcapng.PCAPNG` only,
-which implements extractor for PCAP-NG file format [*]_.
+:class:`~pcapkit.protocols.misc.pcapng.PCAPNG`,
+which implements extractor for PCAP-NG file format [*]_, and the
+:class:`~pcapkit.protocols.misc.pcapng.PacketDirection`,
+:class:`~pcapkit.protocols.misc.pcapng.PacketReception` and
+:class:`~pcapkit.protocols.misc.pcapng.WireGuardKeyLabel` enumerations.
+
+.. versionchanged:: 1.5.0
+
+   A block rebuilds as it was parsed, byte for byte: its lengths, padding,
+   byte order, flag bits and repeated options are kept, and so is a block
+   that is unknown, malformed, shorter than its fixed fields or cut short
+   (:issue:`1272`, :issue:`1273`, :issue:`1276`, :issue:`1277`,
+   :issue:`1278`, :issue:`1279`, :issue:`1280`, :issue:`1281`,
+   :issue:`1282`, :issue:`1283`, :issue:`1284`, :issue:`1382`,
+   :issue:`1383`, :issue:`1384`, :issue:`1405`, :issue:`1413`,
+   :issue:`1414`, :issue:`1422`, :issue:`1470`). A packet block keeps the
+   octets it captured (:issue:`646`); an option running past its block's
+   option area leaves the whole area kept as captured in ``options_raw``,
+   with a :exc:`~pcapkit.utilities.warnings.ProtocolWarning`, where it was
+   read as zeros (:issue:`594`, :issue:`1325`); a truncated capture does not
+   raise :exc:`ValueError` (:issue:`678`); and ``timestamp_epoch`` is not
+   shifted by the host's timezone (:issue:`361`).
 
 .. autoclass:: pcapkit.protocols.misc.pcapng.PCAPNG
    :no-members:
@@ -206,7 +226,7 @@ IANA registry.
 .. versionremoved:: 1.5.0
 
    ``pcapkit.protocols.misc.pcapng.TLSKeyLabel``, now
-   :class:`pcapkit.const.pcapng.tls_key_label.TLSKeyLabel` (:issue:`890`,
+   :class:`pcapkit.const.pcapng.tls_key_label.TLSKeyLabel` (:issue:`886`,
    :issue:`1744`).
 
 .. autoclass:: pcapkit.protocols.misc.pcapng.WireGuardKeyLabel

docs/source/pcapkit/protocols/protocol.rst

@@ -18,6 +18,14 @@ derive from :class:`~pcapkit.protocols.protocol.ProtocolBase`, whose metaclass
 is :class:`~pcapkit.protocols.protocol.ProtocolMeta`; both are listed under
 Internal Definitions below.
 
+.. versionchanged:: 1.5.0
+
+   Constructing a protocol with a keyword that none of its signatures
+   declares raises :exc:`~pcapkit.utilities.exceptions.UnsupportedCall`,
+   where it was discarded (:issue:`617`). A next layer captured shorter than
+   its header is kept as :class:`~pcapkit.protocols.misc.raw.Raw`, where it
+   was parsed with zero-filled fields (:issue:`1170`).
+
 .. autoclass:: pcapkit.protocols.protocol.Protocol
    :no-members:
    :show-inheritance:

pcapkit/protocols/application/application.py

@@ -10,6 +10,15 @@
 :class:`HTTP/1.* <pcapkit.protocols.application.httpv1.HTTP>` and
 :class:`HTTP/2 <pcapkit.protocols.application.httpv2.HTTP>`.
 
+.. versionchanged:: 1.5.0
+
+   :meth:`~pcapkit.protocols.application.application.Application._decode_next_layer`
+   and
+   :meth:`~pcapkit.protocols.application.application.Application._import_next_layer`
+   accept the ``-1`` sentinel, so an application layer protocol can carry the
+   rest of its packet undissected as its payload; every call raised
+   :exc:`~pcapkit.utilities.exceptions.UnsupportedCall` (:issue:`719`).
+
 """
 from typing import TYPE_CHECKING, Generic, overload
 

pcapkit/protocols/application/ftp.py

@@ -13,6 +13,14 @@
 
 .. [*] https://en.wikipedia.org/wiki/File_Transfer_Protocol
 
+.. versionchanged:: 1.5.0
+
+   A command or reply rebuilds byte for byte, its CRLF included, and keeps
+   the command's case and any octets that are not UTF-8 as captured. A line
+   with no argument and a multi-line reply, whole in one segment or not,
+   parse and rebuild as captured (:issue:`1238`, :issue:`1239`,
+   :issue:`1240`, :issue:`1241`, :issue:`1303`, :issue:`1333`).
+
 """
 import re
 from typing import TYPE_CHECKING
@@ -113,7 +121,7 @@ def read(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_FTP':
         # NOTE: A multi-line reply (:rfc:`959#section-4`) may arrive whole in one
         # segment. Its first line is parsed as any reply line, and the lines
         # after it are kept as received in ``lines``. A segment holding anything
-        # else -- no closing line, or more after it -- is refused as before.
+        # else -- no closing line, or more after it -- is refused.
         line, lines = data, ()  # type: bytes, tuple[bytes, ...]
         if FTP_RESPONSE.match(data) is None and data.endswith(b'\r\n'):
             first, _, rest = data.partition(b'\r\n')

pcapkit/protocols/application/http.py

@@ -494,7 +494,7 @@ def _guess_version(self, length: 'int', **kwargs: 'Any') -> 'HTTP':
             Two things this deliberately does **not** decide, because one payload
             with no flow context cannot:
 
-            * **An ``Upgrade: h2c`` exchange** (:rfc:`7540#section-3.2`,
+            * **An** ``Upgrade: h2c`` **exchange** (:rfc:`7540#section-3.2`,
               deprecated but not removed by :rfc:`9113#section-3.1`) stays
               HTTP/1.1 here, and is *correctly* HTTP/1.1: on the wire the upgrade
               request and its ``101 Switching Protocols`` response are HTTP/1.1
@@ -508,7 +508,7 @@ def _guess_version(self, length: 'int', **kwargs: 'Any') -> 'HTTP':
               preface ahead of it, or an opaque HTTP/1 message body -- is
               genuinely undecidable from one payload, and the honest answer is
               the ``Raw`` that an escaping ``ProtocolError`` becomes under
-              :func:`~pcapkit.protocols.misc.raw.beholder`. The nine-octet frame
+              :func:`~pcapkit.corekit.decorators.beholder`. The nine-octet frame
               header is therefore never used to *identify* HTTP/2 ahead of
               HTTP/1: as identification it misfires on binary HTTP/1 bodies,
               which is precisely how garbage text was classified HTTP/2 to begin

pcapkit/protocols/application/httpv1.py

@@ -5,10 +5,11 @@
 .. module:: pcapkit.protocols.application.httpv1
 
 :mod:`pcapkit.protocols.application.httpv1` contains
-:class:`~pcapkit.protocols.application.httpv1.HTTP`
-only, which implements extractor for Hypertext Transfer
-Protocol (HTTP/1.*) [*]_, whose structure is described
-as below:
+:class:`~pcapkit.protocols.application.httpv1.HTTP`,
+which implements extractor for Hypertext Transfer
+Protocol (HTTP/1.*) [*]_, and its
+:class:`~pcapkit.protocols.application.httpv1.Type` enumeration. The
+structure of an HTTP/1.* message is described as below:
 
 .. code-block:: text
 
@@ -26,6 +27,13 @@
 
 .. [*] https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol
 
+.. versionchanged:: 1.5.0
+
+   A message rebuilds byte for byte: the whitespace of the start line and
+   header lines, and octets in them that are not UTF-8, are kept as captured
+   rather than normalised or re-encoded (:issue:`1241`, :issue:`1302`,
+   :issue:`1333`).
+
 """
 import re
 from typing import TYPE_CHECKING

pcapkit/protocols/application/httpv2.py

@@ -23,6 +23,17 @@
 
 .. [*] https://en.wikipedia.org/wiki/HTTP/2
 
+.. versionchanged:: 1.5.0
+
+   A frame's Length counts its payload alone, not the 9-octet frame header,
+   as :rfc:`9113#section-4.1` defines it, and the payload is read to that
+   Length rather than to the end of the buffer (:issue:`1121`,
+   :issue:`1243`). A frame rebuilds byte for byte: ``END_STREAM``, undefined
+   flag bits, the reserved bits, a zero Pad Length, duplicate SETTINGS
+   entries and the payload of an unpadded DATA, HEADERS or PUSH_PROMISE frame
+   are kept (:issue:`652`, :issue:`668`, :issue:`1244`, :issue:`1245`,
+   :issue:`1246`, :issue:`1247`).
+
 """
 import collections
 from typing import TYPE_CHECKING, cast

pcapkit/protocols/application/ngap.py

@@ -90,7 +90,7 @@
 * **NGAP over a fragmented SCTP association is not reassembled.** A DATA chunk
   is decoded on its own, so an ``NGAP-PDU`` split across chunks by SCTP
   fragmentation fails to decode rather than being reassembled first.
-* **Private IEs (``PrivateMessage``) carry no schema.** Their contents are
+* **Private IEs** (``PrivateMessage``) **carry no schema.** Their contents are
   vendor defined, so the generic conversion reports whatever ASN.1 shape the
   encoding declares and cannot name the fields.
 
@@ -187,7 +187,7 @@ def load_pycrate() -> 'Optional[Any]':
 # dependency this module needs to decode NGAP at all. They therefore live under
 # pcapkit.const.ngap, generated by pcapkit.vendor.ngap.procedure_code and
 # pcapkit.vendor.ngap.protocol_ie, and are imported here as Enum_ProcedureCode
-# and Enum_ProtocolIE; this module no longer re-exports them under their
+# and Enum_ProtocolIE; this module does not re-export them under their
 # original names (#1744). Only the vendor crawler that generates the const module
 # needs pycrate; neither this module nor the generated one does at import.
 #

pcapkit/protocols/application/ospf.py

@@ -34,6 +34,25 @@
 
 .. [*] https://en.wikipedia.org/wiki/Open_Shortest_Path_First
 
+.. versionadded:: 1.5.0
+
+   Moved here from ``pcapkit.protocols.link.ospf``, with its data model and
+   header schema. :class:`~pcapkit.protocols.application.ospf.OSPF` derives
+   from :class:`~pcapkit.protocols.application.application.Application`, so
+   its layer is ``'Application'`` (:issue:`719`).
+
+.. versionremoved:: 1.5.0
+
+   ``pcapkit.protocols.link.ospf``, ``pcapkit.protocols.data.link.ospf`` and
+   ``pcapkit.protocols.schema.link.ospf``, with no alias (:issue:`719`).
+
+.. versionchanged:: 1.5.0
+
+   A rebuild keeps the parsed Packet Length and the octets past it, where
+   :meth:`~pcapkit.protocols.application.ospf.OSPF.make` recomputed the
+   length from the payload and dropped those octets; it takes them as
+   ``packet_length`` and ``trailer`` (:issue:`1249`, :issue:`1455`).
+
 """
 import ipaddress
 import re

pcapkit/protocols/application/rarp.py

@@ -27,6 +27,19 @@
 
 .. [*] http://en.wikipedia.org/wiki/Address_Resolution_Protocol
 
+.. versionadded:: 1.5.0
+
+   Moved here from ``pcapkit.protocols.link.rarp``.
+   :class:`~pcapkit.protocols.application.rarp.RARP` and
+   :class:`~pcapkit.protocols.application.rarp.DRARP` derive from
+   :class:`~pcapkit.protocols.application.application.Application` as well as
+   :class:`~pcapkit.protocols.link.arp.ARP`, so their layer is
+   ``'Application'`` (:issue:`719`).
+
+.. versionremoved:: 1.5.0
+
+   ``pcapkit.protocols.link.rarp``, with no alias (:issue:`719`).
+
 """
 from typing import TYPE_CHECKING