4.21.1 test regressions, probably resulting from tox.ini being removed from sdist

[mgorny]

Issue

When running tests from 4.21.1 sdist, the following test failures occur. They may be caused by the fact that (compared to 4.21.0), it no longer contains tox.ini.

$ pytest
========================================================= test session starts =========================================================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
rootdir: /tmp/tox-4.21.1
configfile: pyproject.toml
testpaths: tests
plugins: flaky-3.8.1, anyio-4.6.0, time-machine-2.15.0, xdist-3.6.1, mock-3.14.0, cov-5.0.0, devpi-server-6.13.0
collected 1799 items                                                                                                                  

tests/config/cli/test_cli_env_var.py ......                                                                                     [  0%]
tests/config/cli/test_cli_ini.py .............                                                                                  [  1%]
tests/config/cli/test_parse.py .F..                                                                                             [  1%]
tests/config/cli/test_parser.py ............................................................................................... [  6%]
............................................................................................................................... [ 13%]
............................................................................................................................... [ 20%]
............................................................................................................................... [ 27%]
.........................................................................................................                       [ 33%]
tests/config/loader/ini/replace/test_replace_env_var.py ..................                                                      [ 34%]
tests/config/loader/ini/replace/test_replace_os_pathsep.py .                                                                    [ 34%]
tests/config/loader/ini/replace/test_replace_os_sep.py .....                                                                    [ 34%]
tests/config/loader/ini/replace/test_replace_posargs.py ........................                                                [ 36%]
tests/config/loader/ini/replace/test_replace_tox_env.py .........................................                               [ 38%]
tests/config/loader/ini/replace/test_replace_tty.py ..                                                                          [ 38%]
tests/config/loader/ini/test_factor.py ........................                                                                 [ 39%]
tests/config/loader/ini/test_ini_loader.py ..............                                                                       [ 40%]
tests/config/loader/test_loader.py FF..........                                                                                 [ 41%]
tests/config/loader/test_memory_loader.py ..................................                                                    [ 43%]
tests/config/loader/test_replace.py ...........................                                                                 [ 44%]
tests/config/loader/test_section.py ........                                                                                    [ 45%]
tests/config/loader/test_str_convert.py ....................................................................................... [ 50%]
..............................................................................                                                  [ 54%]
tests/config/loader/test_toml_loader.py ........................                                                                [ 55%]
tests/config/source/test_discover.py ....                                                                                       [ 55%]
tests/config/source/test_legacy_toml.py .                                                                                       [ 56%]
tests/config/source/test_setup_cfg.py ..                                                                                        [ 56%]
tests/config/source/test_source_ini.py ....                                                                                     [ 56%]
tests/config/source/test_toml_pyproject.py ............................                                                         [ 57%]
tests/config/source/test_toml_tox.py ........                                                                                   [ 58%]
tests/config/test_main.py .....................                                                                                 [ 59%]
tests/config/test_of_types.py ..                                                                                                [ 59%]
tests/config/test_set_env.py ................                                                                                   [ 60%]
tests/config/test_sets.py ....................                                                                                  [ 61%]
tests/config/test_types.py ........                                                                                             [ 62%]
tests/execute/local_subprocess/test_execute_util.py ....                                                                        [ 62%]
tests/execute/local_subprocess/test_local_subprocess.py ...........................                                             [ 63%]
tests/execute/test_request.py ..                                                                                                [ 63%]
tests/execute/test_stream.py ..                                                                                                 [ 64%]
tests/journal/test_main_journal.py ....                                                                                         [ 64%]
tests/plugin/test_inline.py .                                                                                                   [ 64%]
tests/plugin/test_plugin.py ..........                                                                                          [ 64%]
tests/plugin/test_plugin_custom_config_set.py ......                                                                            [ 65%]
tests/pytest_/test_init.py ..............                                                                                       [ 65%]
tests/session/cmd/run/test_common.py ..........                                                                                 [ 66%]
tests/session/cmd/test_depends.py ...                                                                                           [ 66%]
tests/session/cmd/test_devenv.py ..                                                                                             [ 66%]
tests/session/cmd/test_exec_.py ......                                                                                          [ 67%]
tests/session/cmd/test_legacy.py ...................                                                                            [ 68%]
tests/session/cmd/test_list_envs.py .......                                                                                     [ 68%]
tests/session/cmd/test_parallel.py ...............                                                                              [ 69%]
tests/session/cmd/test_quickstart.py ....                                                                                       [ 69%]
tests/session/cmd/test_sequential.py ................................                                                           [ 71%]
tests/session/cmd/test_show_config.py .......................                                                                   [ 72%]
tests/session/cmd/test_state.py ..                                                                                              [ 72%]
tests/session/test_env_select.py .........................................................                                      [ 75%]
tests/session/test_session_common.py ........                                                                                   [ 76%]
tests/test_call_modes.py ..                                                                                                     [ 76%]
tests/test_provision.py ............                                                                                            [ 77%]
tests/test_report.py ..............                                                                                             [ 77%]
tests/test_run.py ......                                                                                                        [ 78%]
tests/test_version.py ...                                                                                                       [ 78%]
tests/tox_env/python/pip/req/test_file.py ..................................................................................... [ 83%]
.................                                                                                                               [ 84%]
tests/tox_env/python/pip/test_pip_install.py ............................................                                       [ 86%]
tests/tox_env/python/pip/test_req_file.py ........                                                                              [ 87%]
tests/tox_env/python/test_python_api.py ..........................................................................              [ 91%]
tests/tox_env/python/test_python_runner.py ...................                                                                  [ 92%]
tests/tox_env/python/virtual_env/package/test_package_cmd_builder.py ...........                                                [ 92%]
tests/tox_env/python/virtual_env/package/test_package_pyproject.py .................................                            [ 94%]
tests/tox_env/python/virtual_env/package/test_python_package_util.py .......                                                    [ 95%]
tests/tox_env/python/virtual_env/test_virtualenv_api.py ..........                                                              [ 95%]
tests/tox_env/test_api.py ...                                                                                                   [ 95%]
tests/tox_env/test_info.py .                                                                                                    [ 95%]
tests/tox_env/test_register.py ..                                                                                               [ 95%]
tests/tox_env/test_tox_env_api.py ............                                                                                  [ 96%]
tests/tox_env/test_tox_env_runner.py .                                                                                          [ 96%]
tests/util/test_ci.py .....................                                                                                     [ 97%]
tests/util/test_cpu.py ..                                                                                                       [ 97%]
tests/util/test_graph.py .......                                                                                                [ 98%]
tests/util/test_path.py .                                                                                                       [ 98%]
tests/util/test_spinner.py ..................                                                                                   [ 99%]
tests/session/cmd/test_devenv.py s                                                                                              [ 99%]
tests/session/cmd/test_sequential.py ss                                                                                         [ 99%]
tests/test_provision.py sssssss                                                                                                 [ 99%]
tests/tox_env/python/virtual_env/test_setuptools.py s                                                                           [100%]

============================================================== FAILURES ===============================================================
___________________________________________________ test_verbosity_guess_miss_match ___________________________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7f4e9fe16f00>

    def test_verbosity_guess_miss_match(capsys: CaptureFixture) -> None:
        result = get_options("-rv")
        assert result.parsed.verbosity == 3
    
        assert logging.getLogger().level == logging.INFO
    
        for name in ("distlib.util", "filelock"):
            logger = logging.getLogger(name)
            assert logger.disabled
        logging.error("E")
        logging.warning("W")
        logging.info("I")
        logging.debug("D")
    
        out, _err = capsys.readouterr()
>       assert out == "ROOT: E\nROOT: W\nROOT: I\n"
E       AssertionError: assert 'ROOT: No tox... W\nROOT: I\n' == 'ROOT: E\nROOT: W\nROOT: I\n'
E         
E         + ROOT: No tox.ini or setup.cfg or pyproject.toml or tox.toml found, assuming empty tox.ini at /tmp/tox-4.21.1
E           ROOT: E
E           ROOT: W
E           ROOT: I

_err       = ''
capsys     = <_pytest.capture.CaptureFixture object at 0x7f4e9fe16f00>
logger     = <Logger filelock (INFO)>
name       = 'filelock'
out        = 'ROOT: No tox.ini or setup.cfg or pyproject.toml or tox.toml found, assuming empty tox.ini at /tmp/tox-4.21.1\nROOT: E\nROOT: W\nROOT: I\n'
result     = Options(parsed=Parsed(colored='no', verbose=3, quiet=0, exit_and_dump_after=0, config_file=None, work_dir=None, root_d...acy': <function legacy at 0x7f4ea08518a0>, 'le': <function legacy at 0x7f4ea08518a0>}, log_handler=<ToxHandler (INFO)>)

tests/config/cli/test_parse.py:38: AssertionError
_____________________________________________________ test_override_incorrect[-x] _____________________________________________________

flag = '-x', capsys = <_pytest.capture.CaptureFixture object at 0x7f4e9fb520c0>

    @pytest.mark.parametrize("flag", ["-x", "--override"])
    def test_override_incorrect(flag: str, capsys: CaptureFixture) -> None:
        with pytest.raises(SystemExit):
            get_options(flag, "magic")
        out, err = capsys.readouterr()
>       assert not out
E       AssertionError: assert not 'ROOT: No tox.ini or setup.cfg or pyproject.toml or tox.toml found, assuming empty tox.ini at /tmp/tox-4.21.1\n'

capsys     = <_pytest.capture.CaptureFixture object at 0x7f4e9fb520c0>
err        = 'usage: tox legacy [-h] [--colored {yes,no}] [--exit-and-dump-after seconds] [-c file] [--workdir dir] [--root dir] [-...[-r]\n                  [-x OVERRIDE]\ntox legacy: error: argument -x/--override: override magic has no = sign in it\n'
flag       = '-x'
out        = 'ROOT: No tox.ini or setup.cfg or pyproject.toml or tox.toml found, assuming empty tox.ini at /tmp/tox-4.21.1\n'

tests/config/loader/test_loader.py:19: AssertionError
_________________________________________________ test_override_incorrect[--override] _________________________________________________

flag = '--override', capsys = <_pytest.capture.CaptureFixture object at 0x7f4e9fd03140>

    @pytest.mark.parametrize("flag", ["-x", "--override"])
    def test_override_incorrect(flag: str, capsys: CaptureFixture) -> None:
        with pytest.raises(SystemExit):
            get_options(flag, "magic")
        out, err = capsys.readouterr()
>       assert not out
E       AssertionError: assert not 'ROOT: No tox.ini or setup.cfg or pyproject.toml or tox.toml found, assuming empty tox.ini at /tmp/tox-4.21.1\n'

capsys     = <_pytest.capture.CaptureFixture object at 0x7f4e9fd03140>
err        = 'usage: tox legacy [-h] [--colored {yes,no}] [--exit-and-dump-after seconds] [-c file] [--workdir dir] [--root dir] [-...[-r]\n                  [-x OVERRIDE]\ntox legacy: error: argument -x/--override: override magic has no = sign in it\n'
flag       = '--override'
out        = 'ROOT: No tox.ini or setup.cfg or pyproject.toml or tox.toml found, assuming empty tox.ini at /tmp/tox-4.21.1\n'

tests/config/loader/test_loader.py:19: AssertionError
========================================================== warnings summary ===========================================================
tests/tox_env/python/virtual_env/package/test_package_cmd_builder.py::test_tox_install_pkg_sdist
  /tmp/tox-4.21.1/.venv/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/cmd_builder.py:110: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
    tar.extractall(path=str(work_dir))  # noqa: S202

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================= short test summary info =======================================================
SKIPPED [1] tests/session/cmd/test_devenv.py:18: integration tests not run (no --run-integration flag)
SKIPPED [1] tests/session/cmd/test_sequential.py:59: integration tests not run (no --run-integration flag)
SKIPPED [1] tests/session/cmd/test_sequential.py:159: integration tests not run (no --run-integration flag)
SKIPPED [1] tests/test_provision.py:130: integration tests not run (no --run-integration flag)
SKIPPED [1] tests/test_provision.py:163: integration tests not run (no --run-integration flag)
SKIPPED [2] tests/test_provision.py:197: integration tests not run (no --run-integration flag)
SKIPPED [1] tests/test_provision.py:220: integration tests not run (no --run-integration flag)
SKIPPED [2] tests/test_provision.py:229: integration tests not run (no --run-integration flag)
SKIPPED [1] tests/tox_env/python/virtual_env/test_setuptools.py:19: integration tests not run (no --run-integration flag)
FAILED tests/config/cli/test_parse.py::test_verbosity_guess_miss_match - AssertionError: assert 'ROOT: No tox... W\nROOT: I\n' == 'ROOT: E\nROOT: W\nROOT: I\n'
FAILED tests/config/loader/test_loader.py::test_override_incorrect[-x] - AssertionError: assert not 'ROOT: No tox.ini or setup.cfg or pyproject.toml or tox.toml found, assuming empty tox.ini at /tmp/tox-...
FAILED tests/config/loader/test_loader.py::test_override_incorrect[--override] - AssertionError: assert not 'ROOT: No tox.ini or setup.cfg or pyproject.toml or tox.toml found, assuming empty tox.ini at /tmp/tox-...
================================== 3 failed, 1785 passed, 11 skipped, 1 warning in 99.92s (0:01:39) ===================================

Environment

Provide at least:

• OS: Gentoo Linux amd64

Output of pip list of the host Python, where tox is installed

Package               Version
--------------------- -----------
anyio                 4.6.0
argon2-cffi           23.1.0
argon2-cffi-bindings  21.2.0
attrs                 24.2.0
build                 1.2.2
cachetools            5.5.0
certifi               2024.8.30
cffi                  1.17.1
chardet               5.2.0
charset-normalizer    3.3.2
check-manifest        0.49
colorama              0.4.6
covdefaults           2.3.0
coverage              7.6.1
defusedxml            0.7.1
detect-test-pollution 1.2.0
devpi-client          7.1.0
devpi-common          4.0.4
devpi-process         1.0.2
devpi-server          6.13.0
diff_cover            9.2.0
distlib               0.3.8
execnet               2.1.1
filelock              3.16.1
flaky                 3.8.1
h11                   0.14.0
hatch-vcs             0.4.0
hatchling             1.25.0
httpcore              1.0.6
httpx                 0.27.2
hupper                1.12.1
idna                  3.10
iniconfig             2.0.0
itsdangerous          2.2.0
Jinja2                3.1.4
lazy                  1.6
MarkupSafe            2.1.5
packaging             24.1
packaging-legacy      23.0.post0
passlib               1.7.4
PasteDeploy           3.1.0
pathspec              0.12.1
pip                   24.2
pkginfo               1.11.1
plaster               1.1.2
plaster-pastedeploy   1.0.1
platformdirs          4.3.6
pluggy                1.5.0
psutil                6.0.0
py                    1.11.0
pycparser             2.22
Pygments              2.18.0
pyproject-api         1.8.0
pyproject_hooks       1.2.0
pyramid               2.0.2
pytest                8.3.3
pytest-cov            5.0.0
pytest-mock           3.14.0
pytest-xdist          3.6.1
python-dateutil       2.9.0.post0
re-assert             1.1.0
regex                 2024.9.11
repoze.lru            0.7
requests              2.32.3
ruamel.yaml           0.18.6
ruamel.yaml.clib      0.2.8
setuptools            75.1.0
setuptools-scm        8.1.0
six                   1.16.0
sniffio               1.3.1
strictyaml            1.7.3
time-machine          2.15.0
tox                   4.21.1
translationstring     1.4
trove-classifiers     2024.9.12
urllib3               2.2.3
venusian              3.1.0
virtualenv            20.26.6
waitress              3.0.0
WebOb                 1.8.8
wheel                 0.44.0
zope.deprecation      5.0
zope.interface        7.0.3

Output of running tox

n/a

Minimal example

tar -xf tox-4.21.1.tar.gz
cd tox-4.21.1
pip install .[testing]
pytest

[gaborbernat]

PR Welcome 💪

[mgorny]

Thanks!