Surprisingly, pytest doesn't collect warnings when importing the package for plugins

[A5rocks]

• a detailed description of the bug or problem you are having

When importing a package for its plugin, pytest lets any warnings fall to the ground. This means that subsequent imports in tests do not get any warnings, meaning it's possible to miss deprecation warnings. We ran into this in python-trio/trio#3053. I'm not certain this is a bug, but it's certainly surprising and I don't remember seeing anything about this while we were adding our custom plugin!

• output of pip list from the virtual environment you are using

Package   Version
--------- -------
colorama  0.4.6
example   0.0.1
iniconfig 2.0.0
packaging 24.1
pip       24.0
pluggy    1.5.0
pytest    8.3.2

• pytest and operating system versions

As shown above, pytest is 8.3.2. I'm reproducing this on Windows 11, but it happened in CI for us for every base actions runner OS + Alpine Linux.

• minimal example if possible

pyproject.toml:

[build-system]
requires = ["flit_core>=3.2,<4"]
build-backend = "flit_core.buildapi"

[project]
name = "example"
dynamic = ["version", "description"]

test.py:

import example

def test_version():
    assert example.__version__ == "0.0.1"

src/example/__init__.py:

"""example"""
__version__ = "0.0.1"

import warnings

warnings.warn(DeprecationWarning("example"))

With that in place, run pip install . pytest and then these two commands have conflicting results:

(.venv) PS C:\Users\A5rocks\Documents\pytest-repro> pytest test.py
================================================= test session starts =================================================
platform win32 -- Python 3.12.4, pytest-8.3.2, pluggy-1.5.0
rootdir: C:\Users\A5rocks\Documents\pytest-repro
configfile: pyproject.toml
collected 1 item

test.py .                                                                                                        [100%]

================================================== warnings summary ===================================================
.venv\Lib\site-packages\example\__init__.py:6
  C:\Users\A5rocks\Documents\pytest-repro\.venv\Lib\site-packages\example\__init__.py:6: DeprecationWarning: example
    warnings.warn(DeprecationWarning("example"))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================ 1 passed, 1 warning in 0.05s =============================================
(.venv) PS C:\Users\A5rocks\Documents\pytest-repro> pytest test.py -p example
================================================= test session starts =================================================
platform win32 -- Python 3.12.4, pytest-8.3.2, pluggy-1.5.0
rootdir: C:\Users\A5rocks\Documents\pytest-repro
configfile: pyproject.toml
collected 1 item

test.py .                                                                                                        [100%]

================================================== 1 passed in 0.04s ==================================================

[dongfangtianyu]

Hi, I want try to explain the difference in outcomes between the two execution methods:

The warning is captured during the collection and execution of test cases after pytest is initialized and ready.

If you execute

pytest test.py

pytest will import example during the collection phase, and the DeprecationWarning will be captured.

If you execute

pytest test.py -p example

pytest will first import the example plugin during the initialization phase. At this point, the DeprecationWarning is raised, but according to Python's default rules, the warning is ignored (reference: https://docs.python.org/3/library/exceptions.html#DeprecationWarning).
Later, pytest will import example again during the collection phase, but this time, the DeprecationWarning won't be raised again, so it cannot be captured.

---

The root cause is that Python ignores the DeprecationWarning, not pytest.

If you want to display warnings regardless, you can:

• Change DeprecationWarning to a type that is not ignored by Python, such as UserWarning, PytestDeprecationWarning, etc.
• Use code or environment variables to request Python not to ignore DeprecationWarning.

If you want to ensure that warnings are not displayed under any circumstances, you can:

• Modify pytest.ini to instruct pytest to ignore DeprecationWarning.

[A5rocks]

I'm aware of why this currently doesn't work, but it isn't immediately obvious and has caused (at least) one case where tests didn't match reality.

Specifically, shouldn't it be possible for pytest to capture warnings from plugin imports and apply its warning filter on them? That isn't status quo, yes, but I think that would be a less surprising place for pytest to be. (Though maybe trio's case of using a plugin instead of conftest.py for privacy reasons is unique? And so this isn't worth it?)

[dongfangtianyu]

Right,

Shouldn't Python, pytest, and plugins all capture warnings from plugin imports and apply their warning filter on them?

If Python, as the upstream, and the plugins, as the downstream, didn't do this, it is debatable for pytest to do so on its own.

From the perspective of a plugin maintainer, I believe that changing the upstream software to alter the default behavior of the further upstream should be handled with caution

[RonnyPfannschmidt]

pytest should enact its warning filters before starting to import from entrypoitns/given plugins of possible