ci(vendor): test the regenerated tree before committing it (#1853)

.github/workflows/cron-vendor.yml

@@ -229,7 +229,7 @@ jobs:
               fi
               echo '```'
               echo
-              echo 'Const files from targets that exited zero are still isorted, diffed, committed and pushed below -- only the failing target(s) above were left as they were.'
+              echo 'Const files from targets that exited zero are still isorted, tested, diffed, committed and pushed below -- only the failing target(s) above were left as they were.'
             } >> "$GITHUB_STEP_SUMMARY"
 
             if [ -n "$failed_targets" ]; then
@@ -251,6 +251,51 @@ jobs:
           # still skips `pcapkit/const/__init__.py` itself.
           isort -l100 -ppcapkit $(find pcapkit/const -mindepth 2 -name '*.py')
 
+      # #1853: neither gate above sees the regenerated tree -- `unit-tests` runs
+      # on the schedule alone, and `verdict` accepts the Unit Tests result for
+      # the commit from before the crawl -- so 1.5.0b13 was committed with a test
+      # failing on it. The owner's ruling: test the regenerated tree here, before
+      # anything is committed, under #872's rule above. A module the tests fail
+      # with is put back as HEAD has it and the rest go on to be committed, as
+      # for a sub-vendor that raised. util/vendor_test_gate.py, whose docstring
+      # gives the rule, decides which modules those are.
+      #
+      # `continue-on-error`, so the tests never fail the job. The script exits
+      # zero once it has decided; if this step fails or times out instead, the
+      # tree may hold modules no run passed with, and the next step discards
+      # every regenerated module. The test extra is installed here because only
+      # this step needs it. The script's budget bounds its runs together, and
+      # `timeout-minutes` sits above it as the backstop. Run serially, pytest
+      # peaked at 0.7 GB resident over tests/const, measured locally, so the
+      # memory cap leaves headroom on this runner; files are capped at 2 GiB, as
+      # in experimental.yml.
+      - name: Test the regenerated tree
+        id: vendor-tests
+        continue-on-error: true
+        timeout-minutes: 50
+        run: |
+          set -x
+
+          python -m pip install -e '.[test]'
+          python util/vendor_test_gate.py --budget-minutes 40 \
+              --max-memory-mib 4096 --max-file-size-mib 2048 -- \
+              python -m pytest -q -p no:cacheprovider tests/const tests/vendor
+
+      # What the step above left is untested if it did not finish, so HEAD's
+      # copy of every module goes back, new modules are removed, and nothing
+      # from this crawl is committed. Not a failure either, per #872.
+      - name: Discard untested const changes
+        if: steps.vendor-tests.outcome != 'success'
+        run: |
+          git checkout HEAD -- pcapkit/const
+          git clean -fdq -- pcapkit/const
+          echo "::warning title=Vendor test gate did not finish::the regenerated tree could not be tested, so every regenerated module was discarded; see the step log"
+          {
+            echo '### :warning: Vendor test gate did not finish'
+            echo
+            echo 'The step that tests the regenerated tree failed or timed out, so nothing from this crawl is committed.'
+          } >> "$GITHUB_STEP_SUMMARY"
+
       - name: Verify Changed files
         uses: tj-actions/verify-changed-files@v17
         id: verify-changed-files

pyproject.toml

@@ -304,9 +304,13 @@ test = [
     # Enough of the ``vendor`` extra for a test to import a vendor crawler.
     # Without these, ``tests/vendor/test_ipx_socket_unit.py`` reports its tests as
     # skipped rather than failing, so the byte-for-byte regeneration check that is the
-    # only guard on the hand-maintained IPX socket registry never runs on any CI
-    # path: every ``unit-tests.yml`` job's install line builds on ``.[test]``,
-    # and ``cron-vendor.yml`` installs ``.[all,vendor]`` but never invokes pytest.
+    # only guard on the hand-maintained IPX socket registry runs in no
+    # ``unit-tests.yml`` job: every one's install line builds on ``.[test]``.
+    # ``cron-vendor.yml`` does run it, but needs nothing from here to: its job
+    # installs ``.[all,vendor]`` before ``.[test]``, and runs ``tests/vendor``
+    # only when a crawl changed a constant module, by which point
+    # ``pcapkit-vendor`` has rewritten the IPX constant file from that same table
+    # (#1853).
     #
     # Both are needed, not just ``requests``. ``pcapkit.vendor.default`` imports
     # ``requests`` at module scope, and ``pcapkit.vendor.__init__`` then imports

tests/_dependency_gates.py

@@ -117,9 +117,11 @@
 
 #: The workflow holding every blocking job that runs :program:`pytest`.
 #: :file:`experimental.yml` runs the unit tier as well, on legs nothing waits on
-#: (#1732), and is not read here. Other workflows install ``.[all]``
-#: somewhere -- a docs build, a conda recipe, a vendor crawl, the lint pass,
-#: and the release packaging workflow -- and none of them invokes the suite;
+#: (#1732), and :file:`cron-vendor.yml` runs ``tests/const`` and ``tests/vendor``
+#: over a freshly crawled tree, where a failure discards modules rather than
+#: failing the job (#1853); neither is read here. Other workflows install
+#: ``.[all]`` somewhere -- a docs build, a conda recipe, the lint pass, and the
+#: release packaging workflow -- and none of them invokes the suite;
 #: :file:`python-compatibility.yml` installs a bare ``.`` and only compiles and
 #: imports, and :file:`codeql-analysis.yml` installs nothing explicitly at all
 #: (CodeQL's own autobuild step). That is exactly why this module keys on *jobs that
@@ -1429,13 +1431,16 @@ def _engine_matrix_variants(name: 'str', section: 'str') -> 'Optional[tuple[Job,
 def pytest_jobs(workflow: 'Optional[pathlib.Path]' = None) -> 'tuple[Job, ...]':
     """The jobs of ``workflow`` that run :program:`pytest`.
 
-    Keyed on running the suite, not on holding an install line: five other
-    workflows install ``.[all]`` somewhere, three of them as ``.[all,dev]`` --
-    which does carry ``pypcapfile``, through the ``dev`` extra #910 added when
-    it narrowed ``all`` to core addons only -- and never invoke
-    :program:`pytest`, so a guard that looked at install lines anywhere would
-    pass vacuously. Within this workflow the ``changelog`` job is excluded by
-    the same rule; it installs nothing and runs a generator.
+    Keyed on running the suite, not on holding an install line: other
+    workflows install ``.[all]`` somewhere, some as ``.[all,dev]`` -- which
+    does carry ``pypcapfile``, through the ``dev`` extra #910 added when it
+    narrowed ``all`` to core addons only -- and never invoke :program:`pytest`,
+    so a guard that looked at install lines anywhere would pass vacuously. The
+    exception, :file:`cron-vendor.yml`, runs only ``tests/const`` and
+    ``tests/vendor``, over a freshly crawled tree, and a failure there discards
+    modules rather than failing the job (#1853), so it is not read here either.
+    Within this workflow the ``changelog`` job is excluded by the same rule; it
+    installs nothing and runs a generator.
 
     The ``workflow`` argument exists so the guard can be pointed at a doctored
     copy and shown to fail; see

tests/project/test_isort_clean.py

@@ -112,8 +112,8 @@
 relying on ``bash`` >= 4.0 would have been exactly the mistake the Makefile's
 line first made. This module does not cover that fix, though: it only
 exercises ``make isort``, and :file:`cron-vendor.yml`'s ``vendor-update`` job
-never runs this test suite at all -- it commits and pushes on its own, with
-no pytest step in between.
+never runs it -- the tests that job runs before it commits are
+:file:`tests/const` and :file:`tests/vendor` alone (#1853).
 
 Until #766 this module gated on isort with a ``try: import isort`` inside
 :meth:`~TestIsortIsCleanOnThePackage.setUpClass`, raising