[TestRepair] BigCodeBench/541: `@patch('pkgutil.iter_modules')` is a silent no-op (from-import binding) -> test is vacuous; a stub with no core logic scores PASS

[IgorVoytyuk]

Summary

The unit test for task BigCodeBench/541 (task_func(package_name), in data/BigCodeBench.jsonl.gz) does not exercise the behaviour it is supposed to verify. Its key case, test_package_module_addition, tries to inject fake submodules with @patch('pkgutil.iter_modules'), but the task code binds that name with from pkgutil import iter_modules. Patching the module attribute pkgutil.iter_modules does not rebind the local name already imported into the solution's namespace, so the mock is a silent no-op.

Two concrete consequences:

1. False positive. A stub that omits the entire core of the task — the loop that appends discovered module paths to sys.path and records them — passes all 5 tests.
2. The assertion is inverted. If the exact same mock is made effective, the official canonical solution fails test_package_module_addition: the case named "…module_addition" asserts assertFalse(len(modules_added) > 0), i.e. that no module was added.

The test currently passes the canonical solution only by coincidence: with the mock inert, iter_modules runs for real against the mocked __path__ = ['mocked_path'], finds nothing, so the loop body (the thing under test) never runs and modules_added == [] satisfies assertFalse(len > 0).

Root cause

# code_prompt / canonical solution
from pkgutil import iter_modules          # <- local name bound at import time
...
for _, module_name, _ in iter_modules(package.__path__):   # uses the local name

# test
@patch('pkgutil.iter_modules')            # <- patches the attribute, not the local name
def test_package_module_addition(self, mock_iter_modules, mock_import_module):
    ...
    mock_iter_modules.return_value = [(None,'module1',True),(None,'module2',True)]
    modules_added = task_func('numpy')
    self.assertFalse(len(modules_added) > 0)   # asserts the OPPOSITE of "addition"

@patch('importlib.import_module') works (the code calls importlib.import_module, an attribute lookup), but @patch('pkgutil.iter_modules') misses.

Reproduction

Standalone, no benchmark harness needed (only numpy installed). Full script attached below.

[1] canonical, as shipped:
    CANONICAL: ran=5 failures=0 errors=0 -> PASS
[2] wrong stub (core sys.path logic deleted) -- should FAIL, but:
    STUB     : ran=5 failures=0 errors=0 -> PASS
[3] canonical with the mock made effective (import pkgutil):
    CANON+eff: ran=5 failures=1 errors=0 -> FAIL
   fired: test_package_module_addition :: AssertionError: True is not false

The STUB in [2] is literally:

import importlib
def task_func(package_name):
    try:
        importlib.import_module(package_name)
    except ImportError:
        raise ImportError("not installed")
    return []

It never touches sys.path, never enumerates submodules — the entire task — yet scores PASS.

Suggested fix

Two independent changes; the test needs the first regardless:

1. Patch the name where it is looked up (or drive the code so the mock actually applies), and assert the behaviour the case is named for:

   @patch('__main__.iter_modules')   # or the module the task is exec'd into
   @patch('__main__.importlib.import_module')
   def test_package_module_addition(self, mock_import_module, mock_iter_modules):
       ...
       modules_added = task_func('numpy')
       self.assertEqual(sorted(modules_added), ['module1', 'module2'])

2. (Optional) change the task code to import pkgutil + pkgutil.iter_modules(...) so attribute patching is possible for anyone mocking it.

Scope note

The same "@patch('mod.name') while the solution uses from mod import name" pattern also appears in BigCodeBench/186 (@patch('geopy.distance.geodesic')) and BigCodeBench/407 (@patch('openpyxl.load_workbook')). There the ineffective patch is harmless — the surrounding cases still create real objects and assert on real values, so no false positive results — but the mocks in those cases are dead code. 541 is the one where the inert patch changes the outcome.

Found by static analysis of the v0.1.4 task set (1140 tasks), then confirmed by execution.

Full standalone reproduction script (bcb_541_repro.py)

#!/usr/bin/env python3
"""
Reproduction for BigCodeBench/541: the unit test is vacuous.

Root cause: the test decorates a case with @patch('pkgutil.iter_modules'),
but the solution binds the name via `from pkgutil import iter_modules`.
Patching the attribute `pkgutil.iter_modules` does NOT rebind the local
name already imported into the solution's namespace, so the mock is a
silent no-op. The loop body that is the actual point of the task
(appending discovered module paths to sys.path) therefore never runs
under the "addition" test, and the assertion checks the OPPOSITE of the
test's stated purpose.

Consequences demonstrated below:
  1. A stub that omits the entire sys.path/added_modules core logic
     passes all 5 tests (false positive).
  2. If the very same mock is made EFFECTIVE (module-style import), the
     official canonical solution FAILS `test_package_module_addition`,
     because that test asserts `assertFalse(len(modules_added) > 0)` --
     i.e. it asserts that NO module was added, the opposite of its name.

Run:  python bcb_541_repro.py   (requires numpy installed)
"""
import unittest, io

# --- Official canonical solution, as shipped (from-import binding) ------------
CANON = '''
import os, sys, importlib
from pkgutil import iter_modules
def task_func(package_name):
    added_modules = []
    try:
        package = importlib.import_module(package_name)
    except ImportError:
        raise ImportError("not installed")
    for _, module_name, _ in iter_modules(package.__path__):
        module_path = os.path.join(package.__path__[0], module_name)
        if module_path not in sys.path:
            sys.path.append(module_path)
            added_modules.append(module_name)
    return added_modules
'''

# --- Wrong stub: the entire core logic (the loop) is removed ------------------
STUB = '''
import importlib
def task_func(package_name):
    try:
        importlib.import_module(package_name)
    except ImportError:
        raise ImportError("not installed")
    return []
'''

# --- Same canonical logic but module-style import => the @patch is EFFECTIVE --
CANON_MODULE_IMPORT = '''
import os, sys, importlib, pkgutil
def task_func(package_name):
    added_modules = []
    try:
        package = importlib.import_module(package_name)
    except ImportError:
        raise ImportError("not installed")
    for _, module_name, _ in pkgutil.iter_modules(package.__path__):
        module_path = os.path.join(package.__path__[0], module_name)
        if module_path not in sys.path:
            sys.path.append(module_path)
            added_modules.append(module_name)
    return added_modules
'''

# --- The task's own test suite (verbatim) ------------------------------------
TEST = '''
import unittest
from unittest.mock import patch, MagicMock
import sys
class TestCases(unittest.TestCase):
    @patch('importlib.import_module')
    @patch('pkgutil.iter_modules')
    def test_package_module_addition(self, mock_iter_modules, mock_import_module):
        package_mock = MagicMock()
        package_mock.__path__ = ['mocked_path']
        mock_import_module.return_value = package_mock
        mock_iter_modules.return_value = [
            (None, 'module1', True),
            (None, 'module2', True)
        ]
        modules_added = task_func('numpy')
        self.assertFalse(len(modules_added) > 0)
    def test_nonexistent_package(self):
        with self.assertRaises(ImportError):
            task_func('nonexistentpkg')
    def test_empty_package(self):
        try:
            modules_added = task_func('empty_package')
            self.assertEqual(len(modules_added), 0)
        except ImportError:
            self.assertTrue(True, "Package not found, which is expected in this test.")
    def test_module_path_in_sys_path(self):
        modules_added = task_func('numpy')
        for module in modules_added:
            self.assertTrue(any(module in path for path in sys.path))
    def test_no_duplicates_in_sys_path(self):
        modules_added = task_func('numpy')
        for module in modules_added:
            self.assertEqual(sum(module in path for path in sys.path), 1)
'''


def run(label, code):
    ns = {}
    exec(compile(code + "\n" + TEST, "<t>", "exec"), ns)
    suite = unittest.TestLoader().loadTestsFromTestCase(ns['TestCases'])
    r = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
    verdict = "PASS" if r.wasSuccessful() else "FAIL"
    print(f"{label}: ran={r.testsRun} failures={len(r.failures)} errors={len(r.errors)} -> {verdict}")
    for t, tr in (r.failures + r.errors):
        print("   fired:", t.id().split('.')[-1], "::", tr.strip().splitlines()[-1])


if __name__ == "__main__":
    print("[1] canonical, as shipped:")
    run("    CANONICAL", CANON)
    print("[2] wrong stub (core sys.path logic deleted) -- should FAIL, but:")
    run("    STUB     ", STUB)
    print("[3] canonical with the mock made effective (import pkgutil):")
    run("    CANON+eff", CANON_MODULE_IMPORT)