Skip to content

fix: honor fail-on-flaky with xdist workers - #387

Open
harbinresearcher wants to merge 1 commit into
pytest-dev:masterfrom
harbinresearcher:codex/fail-on-flaky-xdist
Open

harbinresearcher wants to merge 1 commit into
pytest-dev:masterfrom
harbinresearcher:codex/fail-on-flaky-xdist

Conversation

@harbinresearcher

Copy link
Copy Markdown

Closes #386.

With --reruns 1 --fail-on-flaky, a test that fails once and then passes returns exit code 7 in serial execution, but returns 0 under pytest-xdist. The controller's session.items is empty, so the existing exit-status check cannot see the worker's repeated execution.

Record rerun reports on the controller through the existing XDistHooks instance and honor that state in pytest_sessionfinish. Keep the existing nonzero-exit guard and serial item check. No socket protocol, retry-selection, or public API changes.

Extend the existing exit-status tests across serial, one-worker and two-worker execution, including both CLI/marker reruns, tests that pass immediately, permanent failures, and a test skipped on its second attempt. Add the required changelog fragment.

Validation:

  • Focused matrix: original 6 failed/15 passed; patched 21 passed.
  • Full Windows Python 3.12.14 and Linux Python 3.14.8, pytest 9.1.1/xdist 3.8.0: baseline 258 passed; patched 273 passed.
  • Linux Python 3.10.22/pytest 8.2.1/xdist 3.8.0: baseline 243 passed, 1 failed, 14 skipped; patched 258 passed, the same 1 failed, 14 skipped. JUnit failure identities match. The existing test_only_rerun_ini_override expects a contiguous FRF progress line, while pytest 8.2.1 inserts a percentage before the last F; its outcome counts are correct in both runs.
  • Linux Python 3.14.8 without xdist: baseline 248 passed, 10 skipped; patched 249 passed, 24 skipped. New parallel variants correctly skip when the optional plugin is absent.
  • Required pinned pre-commit checks pass. Configured Sphinx HTML build succeeds with one warning.
  • Isolated sdist/wheel builds pass. The wheel was installed into a separate target; its import path was verified and all 21 exit-status cases pass using that installed copy.
  • towncrier check --compare-with origin/master and git diff --check pass.
  • Independent automated review found no actionable issue and separately verified two workers without the terminal plugin, teardown recovery and crash recovery (exit 7), and flaky success alongside a persistent failure (exit 1). This is not human maintainer review.

Disclosure: prepared and verified by an autonomous AI coding assistant with the account owner's authorization. macOS, PyPy, free-threaded Python and the entire CI version matrix were not run locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fail-on-flaky returns success after a rerun with xdist

1 participant