2.11.1: tests/test_self.py::TestRunTC failures

[sandrotosi]

Bug description

Hello,
when preparing 2.11.1 for Debian unstable, we noticed several tests in tests/test_self.py::TestRunTC failing:

=========================== short test summary info ============================
FAILED tests/test_self.py::TestRunTC::test_pkginfo - AssertionError: expected...
FAILED tests/test_self.py::TestRunTC::test_do_not_import_files_from_local_directory_with_pythonpath
FAILED tests/test_self.py::TestRunTC::test_can_list_directories_without_dunder_init
FAILED tests/test_self.py::TestRunTC::test_fail_on_exit_code[args3-6] - Asser...
FAILED tests/test_self.py::TestRunTC::test_fail_on_exit_code[args4-6] - Asser...
FAILED tests/test_self.py::TestRunTC::test_fail_on_exit_code[args7-6] - Asser...
= 6 failed, 1400 passed, 61 skipped, 201 deselected, 11 warnings in 111.86s (0:01:51) =

and in details:

=================================== FAILURES ===================================
____________________________ TestRunTC.test_pkginfo ____________________________

self = <test_self.TestRunTC object at 0x7f0916b4ba90>

    def test_pkginfo(self) -> None:
        """Make pylint check itself."""
>       self._runtest(["pylint.__pkginfo__"], reporter=TextReporter(StringIO()), code=0)

tests/test_self.py:205: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_self.TestRunTC object at 0x7f0916b4ba90>
args = ['pylint.__pkginfo__']
reporter = <pylint.reporters.text.TextReporter object at 0x7f0916b4b070>
out = <_io.StringIO object at 0x7f08f8a0d8b0>, code = 0

    def _runtest(
        self,
        args: List[str],
        reporter: Any = None,
        out: Optional[StringIO] = None,
        code: int = None,
    ) -> None:
        if out is None:
            out = StringIO()
        pylint_code = self._run_pylint(args, reporter=reporter, out=out)
        if reporter:
            output = reporter.out.getvalue()
        elif hasattr(out, "getvalue"):
            output = out.getvalue()
        else:
            output = None
        msg = f"expected output status {code}, got {pylint_code}"
        if output is not None:
            msg = f"{msg}. Below pylint output: \n{output}"
>       assert pylint_code == code, msg
E       AssertionError: expected output status 0, got 16. Below pylint output: 
E         ************* Module pylint.__pkginfo__
E         pylint/__pkginfo__.py:1:0: C0114: Missing module docstring (missing-module-docstring)
E         pylint/__pkginfo__.py:8:0: C0103: Argument name "v" doesn't conform to snake_case naming style (invalid-name)
E         pylint/__pkginfo__.py:16:8: C0103: Variable name "n" doesn't conform to snake_case naming style (invalid-name)
E         pylint/__pkginfo__.py:21:16: C0103: Variable name "c" doesn't conform to snake_case naming style (invalid-name)
E         
E         -----------------------------------
E         Your code has been rated at 8.18/10
E         
E         
E       assert 16 == 0

tests/test_self.py:159: AssertionError
___ TestRunTC.test_do_not_import_files_from_local_directory_with_pythonpath ____

tmpdir = local('/tmp/pytest-of-pbuilder/pytest-0/test_do_not_import_files_from_1')

    @staticmethod
    def test_do_not_import_files_from_local_directory_with_pythonpath(
        tmpdir: LocalPath,
    ) -> None:
        p_astroid = tmpdir / "astroid.py"
        p_astroid.write("'Docstring'\nimport completely_unknown\n")
        p_hmac = tmpdir / "hmac.py"
        p_hmac.write("'Docstring'\nimport completely_unknown\n")
    
        # Appending a colon to PYTHONPATH should not break path stripping
        # https://github.com/PyCQA/pylint/issues/3636
        with tmpdir.as_cwd():
            orig_pythonpath = os.environ.get("PYTHONPATH")
            os.environ["PYTHONPATH"] = f"{(orig_pythonpath or '').strip(':')}:"
>           subprocess.check_output(
                [
                    sys.executable,
                    "-m",
                    "pylint",
                    "astroid.py",
                    "--disable=import-error,unused-import",
                ],
                cwd=str(tmpdir),
            )

tests/test_self.py:1003: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.9/subprocess.py:424: in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

input = None, capture_output = False, timeout = None, check = True
popenargs = (['/usr/bin/python3.9', '-m', 'pylint', 'astroid.py', '--disable=import-error,unused-import'],)
kwargs = {'cwd': '/tmp/pytest-of-pbuilder/pytest-0/test_do_not_import_files_from_1', 'stdout': -1}
process = <Popen: returncode: 1 args: ['/usr/bin/python3.9', '-m', 'pylint', 'astroid....>
stdout = b'', stderr = None, retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['/usr/bin/python3.9', '-m', 'pylint', 'astroid.py', '--disable=import-error,unused-import']' returned non-zero exit status 1.

/usr/lib/python3.9/subprocess.py:528: CalledProcessError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/build/pylint-2.11.1/.pybuild/cpython3_3.9/build/pylint/__main__.py", line 9, in <module>
    pylint.run_pylint()
  File "/build/pylint-2.11.1/.pybuild/cpython3_3.9/build/pylint/__init__.py", line 21, in run_pylint
    from pylint.lint import Run as PylintRun
  File "/build/pylint-2.11.1/.pybuild/cpython3_3.9/build/pylint/lint/__init__.py", line 76, in <module>
    from pylint.lint.parallel import check_parallel
  File "/build/pylint-2.11.1/.pybuild/cpython3_3.9/build/pylint/lint/parallel.py", line 8, in <module>
    from pylint import reporters
  File "/build/pylint-2.11.1/.pybuild/cpython3_3.9/build/pylint/reporters/__init__.py", line 26, in <module>
    from pylint import utils
  File "/build/pylint-2.11.1/.pybuild/cpython3_3.9/build/pylint/utils/__init__.py", line 46, in <module>
    from pylint.utils.ast_walker import ASTWalker
  File "/build/pylint-2.11.1/.pybuild/cpython3_3.9/build/pylint/utils/ast_walker.py", line 7, in <module>
    from astroid import nodes
  File "/tmp/pytest-of-pbuilder/pytest-0/test_do_not_import_files_from_1/astroid.py", line 2, in <module>
    import completely_unknown
ModuleNotFoundError: No module named 'completely_unknown'
___________ TestRunTC.test_can_list_directories_without_dunder_init ____________

tmpdir = local('/tmp/pytest-of-pbuilder/pytest-0/test_can_list_directories_with0')

    @staticmethod
    def test_can_list_directories_without_dunder_init(tmpdir: LocalPath) -> None:
        test_directory = tmpdir / "test_directory"
        test_directory.mkdir()
        spam_module = test_directory / "spam.py"
        spam_module.write("'Empty'")
    
>       subprocess.check_output(
            [
                sys.executable,
                "-m",
                "pylint",
                "--disable=missing-docstring, missing-final-newline",
                "test_directory",
            ],
            cwd=str(tmpdir),
            stderr=subprocess.PIPE,
        )

tests/test_self.py:1092: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.9/subprocess.py:424: in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

input = None, capture_output = False, timeout = None, check = True
popenargs = (['/usr/bin/python3.9', '-m', 'pylint', '--disable=missing-docstring, missing-final-newline', 'test_directory'],)
kwargs = {'cwd': '/tmp/pytest-of-pbuilder/pytest-0/test_can_list_directories_with0', 'stderr': -1, 'stdout': -1}
process = <Popen: returncode: 1 args: ['/usr/bin/python3.9', '-m', 'pylint', '--disabl...>
stdout = b''
stderr = b'Problem importing module refactoring: Unable to find module for /build/pylint-2.11.1/.pybuild/cpython3_3.9/build/pyl...porter_by_name\n    raise exceptions.InvalidReporterError(name) from e\npylint.exceptions.InvalidReporterError: text\n'
retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['/usr/bin/python3.9', '-m', 'pylint', '--disable=missing-docstring, missing-final-newline', 'test_directory']' returned non-zero exit status 1.

/usr/lib/python3.9/subprocess.py:528: CalledProcessError
__________________ TestRunTC.test_fail_on_exit_code[args3-6] ___________________

self = <test_self.TestRunTC object at 0x7f08facadc40>
args = ['--fail-on=import-error'], expected = 6

    @pytest.mark.parametrize(
        "args, expected",
        [
            ([], 0),
            (["--enable=C"], 0),
            (["--fail-on=superfluous-parens"], 0),
            (["--fail-on=import-error"], 6),
            (["--fail-on=unused-import"], 6),
            (["--fail-on=unused-import", "--enable=C"], 22),
            (["--fail-on=missing-function-docstring"], 22),
            (["--fail-on=useless-suppression"], 6),
            (["--fail-on=useless-suppression", "--enable=C"], 22),
        ],
    )
    def test_fail_on_exit_code(self, args, expected):
        path = join(HERE, "regrtest_data", "fail_on.py")
        # We set fail-under to be something very low so that even with the warnings
        # and errors that are generated they don't affect the exit code.
>       self._runtest([path, "--fail-under=-10"] + args, code=expected)

tests/test_self.py:1158: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_self.TestRunTC object at 0x7f08facadc40>
args = ['/build/pylint-2.11.1/.pybuild/cpython3_3.9/build/tests/regrtest_data/fail_on.py', '--fail-under=-10', '--fail-on=import-error']
reporter = None, out = <_io.StringIO object at 0x7f08f7c9f790>, code = 6

    def _runtest(
        self,
        args: List[str],
        reporter: Any = None,
        out: Optional[StringIO] = None,
        code: int = None,
    ) -> None:
        if out is None:
            out = StringIO()
        pylint_code = self._run_pylint(args, reporter=reporter, out=out)
        if reporter:
            output = reporter.out.getvalue()
        elif hasattr(out, "getvalue"):
            output = out.getvalue()
        else:
            output = None
        msg = f"expected output status {code}, got {pylint_code}"
        if output is not None:
            msg = f"{msg}. Below pylint output: \n{output}"
>       assert pylint_code == code, msg
E       AssertionError: expected output status 6, got 22. Below pylint output: 
E         ************* Module fail_on
E         tests/regrtest_data/fail_on.py:4:0: E0401: Unable to import 'nonexistent' (import-error)
E         tests/regrtest_data/fail_on.py:8:0: C0116: Missing function or method docstring (missing-function-docstring)
E         tests/regrtest_data/fail_on.py:4:0: W0611: Unused import nonexistent (unused-import)
E         
E         ------------------------------------
E         Your code has been rated at -1.67/10
E         
E         
E       assert 22 == 6

tests/test_self.py:159: AssertionError
__________________ TestRunTC.test_fail_on_exit_code[args4-6] ___________________

self = <test_self.TestRunTC object at 0x7f08fbce0df0>
args = ['--fail-on=unused-import'], expected = 6

    @pytest.mark.parametrize(
        "args, expected",
        [
            ([], 0),
            (["--enable=C"], 0),
            (["--fail-on=superfluous-parens"], 0),
            (["--fail-on=import-error"], 6),
            (["--fail-on=unused-import"], 6),
            (["--fail-on=unused-import", "--enable=C"], 22),
            (["--fail-on=missing-function-docstring"], 22),
            (["--fail-on=useless-suppression"], 6),
            (["--fail-on=useless-suppression", "--enable=C"], 22),
        ],
    )
    def test_fail_on_exit_code(self, args, expected):
        path = join(HERE, "regrtest_data", "fail_on.py")
        # We set fail-under to be something very low so that even with the warnings
        # and errors that are generated they don't affect the exit code.
>       self._runtest([path, "--fail-under=-10"] + args, code=expected)

tests/test_self.py:1158: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_self.TestRunTC object at 0x7f08fbce0df0>
args = ['/build/pylint-2.11.1/.pybuild/cpython3_3.9/build/tests/regrtest_data/fail_on.py', '--fail-under=-10', '--fail-on=unused-import']
reporter = None, out = <_io.StringIO object at 0x7f08f7c9f280>, code = 6

    def _runtest(
        self,
        args: List[str],
        reporter: Any = None,
        out: Optional[StringIO] = None,
        code: int = None,
    ) -> None:
        if out is None:
            out = StringIO()
        pylint_code = self._run_pylint(args, reporter=reporter, out=out)
        if reporter:
            output = reporter.out.getvalue()
        elif hasattr(out, "getvalue"):
            output = out.getvalue()
        else:
            output = None
        msg = f"expected output status {code}, got {pylint_code}"
        if output is not None:
            msg = f"{msg}. Below pylint output: \n{output}"
>       assert pylint_code == code, msg
E       AssertionError: expected output status 6, got 22. Below pylint output: 
E         ************* Module fail_on
E         tests/regrtest_data/fail_on.py:4:0: E0401: Unable to import 'nonexistent' (import-error)
E         tests/regrtest_data/fail_on.py:8:0: C0116: Missing function or method docstring (missing-function-docstring)
E         tests/regrtest_data/fail_on.py:4:0: W0611: Unused import nonexistent (unused-import)
E         
E         ------------------------------------
E         Your code has been rated at -1.67/10
E         
E         
E       assert 22 == 6

tests/test_self.py:159: AssertionError
__________________ TestRunTC.test_fail_on_exit_code[args7-6] ___________________

self = <test_self.TestRunTC object at 0x7f08f973be20>
args = ['--fail-on=useless-suppression'], expected = 6

    @pytest.mark.parametrize(
        "args, expected",
        [
            ([], 0),
            (["--enable=C"], 0),
            (["--fail-on=superfluous-parens"], 0),
            (["--fail-on=import-error"], 6),
            (["--fail-on=unused-import"], 6),
            (["--fail-on=unused-import", "--enable=C"], 22),
            (["--fail-on=missing-function-docstring"], 22),
            (["--fail-on=useless-suppression"], 6),
            (["--fail-on=useless-suppression", "--enable=C"], 22),
        ],
    )
    def test_fail_on_exit_code(self, args, expected):
        path = join(HERE, "regrtest_data", "fail_on.py")
        # We set fail-under to be something very low so that even with the warnings
        # and errors that are generated they don't affect the exit code.
>       self._runtest([path, "--fail-under=-10"] + args, code=expected)

tests/test_self.py:1158: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_self.TestRunTC object at 0x7f08f973be20>
args = ['/build/pylint-2.11.1/.pybuild/cpython3_3.9/build/tests/regrtest_data/fail_on.py', '--fail-under=-10', '--fail-on=useless-suppression']
reporter = None, out = <_io.StringIO object at 0x7f08f821d670>, code = 6

    def _runtest(
        self,
        args: List[str],
        reporter: Any = None,
        out: Optional[StringIO] = None,
        code: int = None,
    ) -> None:
        if out is None:
            out = StringIO()
        pylint_code = self._run_pylint(args, reporter=reporter, out=out)
        if reporter:
            output = reporter.out.getvalue()
        elif hasattr(out, "getvalue"):
            output = out.getvalue()
        else:
            output = None
        msg = f"expected output status {code}, got {pylint_code}"
        if output is not None:
            msg = f"{msg}. Below pylint output: \n{output}"
>       assert pylint_code == code, msg
E       AssertionError: expected output status 6, got 22. Below pylint output: 
E         ************* Module fail_on
E         tests/regrtest_data/fail_on.py:4:0: E0401: Unable to import 'nonexistent' (import-error)
E         tests/regrtest_data/fail_on.py:8:0: C0116: Missing function or method docstring (missing-function-docstring)
E         tests/regrtest_data/fail_on.py:4:0: W0611: Unused import nonexistent (unused-import)
E         tests/regrtest_data/fail_on.py:5:0: I0021: Useless suppression of 'broad-except' (useless-suppression)
E         
E         ------------------------------------
E         Your code has been rated at -1.67/10
E         
E         
E       assert 22 == 6

tests/test_self.py:159: AssertionError

it is worth noticing that we're using android/2.8.4 instead of the requirements_test.txt pinned 2.8.0 version (because in debian we can only have a single version)

Command used

python3.9 -m pytest tests

Expected behavior

test not failing

Pylint version

pylint 2.11.1
astroid 2.8.4
python 3.9.7

OS / Environment

Debian unstable

[Pierre-Sassoulas]

Thank you for the report. We had to upgrade pylint's test when we upgraded astroid to 2.8.4, it's probably safe to assume astroid 2.8.4 is actually better than 2.8.0 and that pylint's tests are "wrong". We're going to release 2.12 soon (based on astroid 2.8.4 or 2.8.5) and that would be my preferred version to put in debian. What is the deadline to make it in debian unstable ?

[sandrotosi]

sounds good, there's no deadline: debian is not in the process of releasing a new stable release soon so this was just to keep up with development; no rush

[Pierre-Sassoulas]

Closing because it's a known issue and when we release 2.12 the tests will be launched and work.