Skip to content

fix: stop counting a mutant whose subprocess never ran a test as killed - #45

Open
tiagoabsantos wants to merge 2 commits into
pestphp:5.xfrom
tiagoabsantos:fix/mutant-subprocess-paratest-options-and-php-binary
Open

tiagoabsantos wants to merge 2 commits into
pestphp:5.xfrom
tiagoabsantos:fix/mutant-subprocess-paratest-options-and-php-binary

Conversation

@tiagoabsantos

Copy link
Copy Markdown

Fixes pestphp/pest#1937.

--mutate can report a 100% score for mutants whose subprocess never ran a single test. MutationTest::hasFinished() reads any unsuccessful exit as a kill, so a subprocess that dies before reaching the test suite is indistinguishable from a legitimately killed mutant. Two inherited arguments make that happen.

ParaTest-only options are forwarded to the mutant (all platforms)

A mutant runs one PHPUnit process, never ParaTest, but the original arguments are passed through as they were. --parallel is already dropped by ParallelOption::remove(); --processes and --passthru-php are not, and PHPUnit rejects them:

INFO  Unknown option "--processes". Most similar options are --covers, --colors, --group, --no-progress, --no-results.

Exit code 2 in a fraction of a second, counted as killed. This is the Linux CI case in the issue, and it applies to every ParaTest-only option, so MutantArguments strips them all rather than only the two that were reported. It also takes over the coverage stripping that was inline, so the rules for what a mutant may not receive live in one place.

The options are removed from the mutant's arguments only, not from the configuration, so the original run keeps --processes and --passthru-php for its own parallel execution — dropping --passthru-php there would take pcov with it and leave the run without coverage.

--processes 2 spends two arguments, so the value is dropped along with the option; a stray 2 would otherwise be read as a test path.

The Pest script is used as the executable (Windows)

new Process([...$filteredArguments, ...]) puts $originalArguments[0], the Pest script, in the executable position. That works wherever the shebang is honoured. On Windows cmd.exe reads vendor/bin/pest as the command vendor:

'vendor' is not recognized as an internal or external command, operable program or batch file.

Exit code 1, counted as killed. The mutant now runs through PHP_BINARY, the same way the original run reached the same script.

Verification

Against tests/.tests/Untested, whose snapshot expects 3 untested out of 4, on Windows 11 with PHP 8.4.25:

before after
serial Mutations: 4 tested, Score: 100.00%, 0.17s Mutations: 3 untested, 1 tested, Score: 25.00%, 2.36s
--parallel --processes=2 --passthru-php=... Mutations: 4 tested, Score: 100.00% Mutations: 3 untested, 1 tested, Score: 25.00%

Both new tests in tests/Unit/MutationTestTest.php fail on the current 5.x and pass here. They run a real subprocess against a stand-in for the Pest script that records its own $argv, so they cover the executable and the forwarded arguments the way a mutant actually sees them, on Linux as well as Windows.

pint --test, phpstan, and pest --type-coverage --min=100 are clean. The 7 failures left in pest on my Windows machine are all present on 5.x before this branch: they are the path and line-ending issues already covered by #36 and #44.

Not included

The issue also suggests not counting a run in which no test was executed as tested, as a backstop for any other environment problem in the subprocess. That needs a signal beyond the exit code — exit code 2 is also what a legitimately killed mutant produces when it makes a test error — so it is a separate change, and #42 is already exploring one shape of it. This PR fixes the two causes that are unambiguous.

A mutant's subprocess inherited two things it cannot use, and both made it
exit before running a single test. MutationTest::hasFinished() reads any
unsuccessful exit as a kill, so those mutants were reported as tested and
the score read 100%.

ParaTest-only options such as --processes and --passthru-php were forwarded
verbatim. The mutant runs one PHPUnit process, which aborts with
'Unknown option "--processes"' and exit code 2. They are now stripped in
MutantArguments, alongside the coverage options that were already dropped,
so the original run keeps them for its own parallel execution.

The subprocess also used the Pest script as its executable. That works
wherever the shebang is honoured, but on Windows cmd.exe reads
'vendor/bin/pest' as the command 'vendor' and fails with exit code 1. The
mutant now runs through PHP_BINARY, the same way the original run did.

Closes pestphp/pest#1937
@mechgw

mechgw commented Oct 16, 2026

Copy link
Copy Markdown

I can confirm the Windows part independently (Windows 11, PHP 8.4.21, Pest 5.3.2, pest-plugin-mutate 5.0.2). Without running the mutant subprocess through PHP_BINARY, the subprocess never reaches the test suite, and every mutant is reported as tested.

On the repository's own tests/.tests/Untested fixture, with the FileFinder change of #36 applied so that the files are found:

  • the mutant subprocess started as the Pest script itself gives Mutations: 4 tested, Score: 100.00%;
  • the subprocess started through PHP_BINARY gives the expected Mutations: 3 untested, 1 tested, Score: 25.00%.

So #36 and this PR are both needed for correct results on Windows.

-p is Pest's parallel flag, so the subprocess started another
parallel run. --shard was not in the ParaTest list. A coverage
option split across two arguments left its path as a test path.
@tiagoabsantos

Copy link
Copy Markdown
Author

@mechgw Thanks for running this on Windows.

FileFinder on 5.x still treats a drive-letter path as relative. The check is str_starts_with($path, DIRECTORY_SEPARATOR), so C:\project\app becomes getcwd()\C:\project\app and the finder skips it. This PR does not change that. A correct score on Windows with no --path needs both this PR and #36.

I also pushed a follow-up on the argument filter. -p was still forwarded, and Pest reads it as --parallel, so the mutant started another parallel run. --shard was missing from the ParaTest list. A coverage option written as two arguments, such as --coverage-php /tmp/coverage.php, left the path behind, and PHPUnit would read it as a test path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants