package = wheel not respected for individual env if prior env uses package = editable #3034

f3flight · 2023-06-15T22:47:04Z

Issue

Disclaimer - I'm using tox-pdm plugin but I don't think it matters as it doesn't use pdm to install the main package - it only handles dependencies.

Some parts of the config:

[tox]
envlist = prepare,py310,quality,cli,ci

[testenv]
package = editable

I have a few envs, and one of them overrides this with "package = wheel":

[testenv:cli]
depends = prepare
package = wheel

When I run tox, if there's another env before that (I use "depends" as you can see) which has no override and therefore uses editable package, then this env called "cli" also gets editable package installed, despite the override.

If I run the env separately with "tox -e cli" then I get expected behavior (not an editable wheel but a normal wheel), but if I run all envs I get incorrect behavior, cli env gets editable wheel installed.

Environment

Provide at least:

OS: Linux
pip list of the host Python where tox is installed:

Package                           Version
--------------------------------- ---------
... list is pretty long so showing what's important:
tox                               4.4.8
tox-pdm                           0.6.1
...

Output of running tox

Provide the output of tox -rvv:

$ grep editable tox.out
Backend: Wrote response {'return': {'get_requires_for_build_sdist': True, 'prepare_metadata_for_build_wheel': True, 'get_requires_for_build_wheel': True, 'build_editable': True, 'get_requires_for_build_editable': True, 'prepare_metadata_for_build_editable': True}} to /tmp/pep517__optional_hooks-c58s8t1i.json
.pkg: 12313 W get_requires_for_build_editable> python /home/user/.shiv/tools.pyz_7b47253112c59d7d9564833f351bcff8e94f39b278af5838f13df13f179ac62c/site-packages/pyproject_api/_backend.py True setuptools.build_meta [tox/tox_env/api.py:428]
Backend: run command get_requires_for_build_editable with args {'config_settings': None}
Backend: Wrote response {'return': ['wheel']} to /tmp/pep517_get_requires_for_build_editable-s9qogo0m.json
.pkg: 12683 W install_requires_for_build_editable> python -I -m pip install wheel [tox/tox_env/api.py:428]
.pkg: 14194 W build_editable> python /home/user/.shiv/tools.pyz_7b47253112c59d7d9564833f351bcff8e94f39b278af5838f13df13f179ac62c/site-packages/pyproject_api/_backend.py True setuptools.build_meta [tox/tox_env/api.py:428]
Backend: run command build_editable with args {'wheel_directory': '/home/user/Repos/python-native-build-test-cli/.tox/.pkg/dist', 'config_settings': {'--build-option': []}, 'metadata_directory': None}
running editable_wheel
adding '__editable__.python_native_build_test_cli-0.0.1.pth'
creating '/home/user/Repos/python-native-build-test-cli/.tox/.pkg/dist/.tmp-wj05idmr/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl' and adding '/tmp/tmp9o0vhv1opython_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl' to it
Backend: Wrote response {'return': 'python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl'} to /tmp/pep517_build_editable-9ophnhgz.json
.pkg: 14274 D package .tmp/package/5/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl links to .pkg/dist/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl (/home/user/Repos/python-native-build-test-cli/.tox) [tox/util/file_view.py:36]
py310: 68116 W install_package> python -I -m pip install --force-reinstall --no-deps /home/user/Repos/python-native-build-test-cli/.tox/.tmp/package/5/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl [tox/tox_env/api.py:428]
Processing ./.tox/.tmp/package/5/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl
py310: 68875 I exit 0 (0.76 seconds) /home/user/Repos/python-native-build-test-cli> python -I -m pip install --force-reinstall --no-deps /home/user/Repos/python-native-build-test-cli/.tox/.tmp/package/5/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl pid=15449 [tox/execute/api.py:275]
.pkg: 78990 D package .tmp/package/6/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl links to .pkg/dist/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl (/home/user/Repos/python-native-build-test-cli/.tox) [tox/util/file_view.py:36]
quality: 131484 W install_package> python -I -m pip install --force-reinstall --no-deps /home/user/Repos/python-native-build-test-cli/.tox/.tmp/package/6/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl [tox/tox_env/api.py:428]
Processing ./.tox/.tmp/package/6/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl
quality: 132270 I exit 0 (0.78 seconds) /home/user/Repos/python-native-build-test-cli> python -I -m pip install --force-reinstall --no-deps /home/user/Repos/python-native-build-test-cli/.tox/.tmp/package/6/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl pid=16296 [tox/execute/api.py:275]
.pkg: 143165 D package .tmp/package/7/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl links to .pkg/dist/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl (/home/user/Repos/python-native-build-test-cli/.tox) [tox/util/file_view.py:36]
cli: 192020 W install_package> python -I -m pip install --force-reinstall --no-deps /home/user/Repos/python-native-build-test-cli/.tox/.tmp/package/7/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl [tox/tox_env/api.py:428]
Processing ./.tox/.tmp/package/7/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl
cli: 192801 I exit 0 (0.78 seconds) /home/user/Repos/python-native-build-test-cli> python -I -m pip install --force-reinstall --no-deps /home/user/Repos/python-native-build-test-cli/.tox/.tmp/package/7/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl pid=17257 [tox/execute/api.py:275]
.pkg: 216109 D delete package /home/user/Repos/python-native-build-test-cli/.tox/.tmp/package/5/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl [tox/tox_env/python/virtual_env/package/pyproject.py:179]
.pkg: 216109 D delete package /home/user/Repos/python-native-build-test-cli/.tox/.tmp/package/7/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl [tox/tox_env/python/virtual_env/package/pyproject.py:179]
.pkg: 216109 D delete package /home/user/Repos/python-native-build-test-cli/.tox/.tmp/package/6/python_native_build_test_cli-0.0.1-0.editable-py3-none-any.whl [tox/tox_env/python/virtual_env/package/pyproject.py:179]

The text was updated successfully, but these errors were encountered:

gaborbernat · 2023-06-16T17:09:24Z

PR welcome 👍

f3flight · 2023-06-16T17:16:23Z

@gaborbernat any pointers to where I might want to add debug statements/code changes? I'm not familiar with how configs are compiled for each env and how value of "package" is used. I can try to fix but to start I'd need some pointers if possible. Is there an option to print rendered config for each env to rule out config propagation issue?

gaborbernat · 2023-06-16T17:18:36Z

No idea where the issue might be. https://tox.wiki/en/latest/cli_interface.html#tox-config-(c) should help you, tox c -e a,b -k package

gaborbernat · 2023-06-16T17:21:30Z

Looks fine, so we need a proper reproducible. Ideally a repository that can be clonned, or docker image.

f3flight · 2023-06-16T17:23:53Z

thanks, will debug further and provide a repro if possible

f3flight · 2023-06-16T17:52:36Z

[dsutiagi@dsutiagi-ld3 python-native-build-test-cli]$ ./tox c -k package
[testenv:prepare]
package = skip

[testenv:py310]
package = editable

[testenv:quality]
package = editable

[testenv:cli]
package = wheel

[testenv:ci]
package = skip

This output looks ok, but it still installs editable for cli venv, as seen here:

cli: install_package> python -I -m pip install --force-reinstall --no-deps /home/user/Repos/python-native-build-test-cli/.tox/.tmp/package/3/python_native_build_test_cli-0.0.2-0.editable-py3-none-any.whl

So the bug is somewhere else, not in config. I will try to make a simple project to reproduce and share the repo.

f3flight · 2023-06-16T18:41:36Z

ok I found the issue. It's coming from the use of cachetools to wrap wheel build call. The arguments to editable and wheel build are identical, therefore a cached result is returned.
This line - https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/python/virtual_env/package/pyproject.py#L224 is returning a cached result from editable build when of_type is different (wheel):

-> wheel = getattr(self._frontend, method)(
(Pdb) d
> /home/user/.shiv/tools.pyz_ccc1144e019e58df802e7326b8b57b4e15b3c33af30c1ed9f1f793806d99a61d/site-packages/cachetools/__init__.py(732)wrapper()
-> k = key(*args, **kwargs)
(Pdb) k
*** NameError: name 'k' is not defined
(Pdb) args
args = ()
kwargs = {'wheel_directory': PosixPath('/home/user/Repos/python-native-build-test-cli/.tox/.pkg/dist'), 'metadata_directory': None, 'config_settings': {'--build-option': []}}

Since of_type is not an argument to this function, same wheel directory = same wheel.

@gaborbernat should we add of_type to arguments of frontend build functions (even as a throwaway?) to avoid cache collision for different package types? I see cachetools has other key generation methods, such as "methodkey" and "typedkey", but we're using "hashkey" here, which I guess doesn't take function name into account (i.e. different frontend functions with same arguments = same cache result). In this particular case the key returned is a string "wheel" - both for editable and non-editable calls.

Ah I think we use a custom key fuction which is defined here, that explains why key is just "wheel" - https://github.com/tox-dev/tox/blob/main/src/tox/tox_env/python/virtual_env/package/pyproject.py#L336

gaborbernat · 2023-06-16T18:43:50Z

Indeed 🤔 you're suggestion seems reasonable so PR would be welcomed.

f3flight · 2023-06-16T19:06:17Z

@gaborbernat please have a look at the PR. Let me know if it makes sense. Also I could not find a unit test for Pep517VirtualEnvFrontend so I'm unsure if I need to write one from scratch.

gaborbernat · 2023-06-16T22:28:57Z

Yes we should write a new test 👍

…

On Fri, Jun 16, 2023, 12:06 Dmitrii Sutiagin ***@***.***> wrote: @gaborbernat <https://github.com/gaborbernat> please have a look at the PR. Let me know if it makes sense. Also I could not find a unit test for Pep517VirtualEnvFrontend so I'm unsure if I need to write one from scratch. — Reply to this email directly, view it on GitHub <#3034 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFIQPRG2HPCOZHUVKW7A5DXLSVDJANCNFSM6AAAAAAZIPMGRM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

gaborbernat added the help:wanted Issues that have been acknowledged, a solution determined and a PR might likely be accepted. label Jun 16, 2023

f3flight mentioned this issue Jun 16, 2023

Avoid cache collision between wheel and editable wheel builds #3035

Merged

5 tasks

gaborbernat closed this as completed in #3035 Jun 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

package = wheel not respected for individual env if prior env uses package = editable #3034

package = wheel not respected for individual env if prior env uses package = editable #3034

f3flight commented Jun 15, 2023

gaborbernat commented Jun 16, 2023

f3flight commented Jun 16, 2023

gaborbernat commented Jun 16, 2023

gaborbernat commented Jun 16, 2023

f3flight commented Jun 16, 2023

f3flight commented Jun 16, 2023

f3flight commented Jun 16, 2023 •

edited

Loading

gaborbernat commented Jun 16, 2023

f3flight commented Jun 16, 2023

gaborbernat commented Jun 16, 2023 via email

package = wheel not respected for individual env if prior env uses package = editable #3034

package = wheel not respected for individual env if prior env uses package = editable #3034

Comments

f3flight commented Jun 15, 2023

Issue

Environment

Output of running tox

gaborbernat commented Jun 16, 2023

f3flight commented Jun 16, 2023

gaborbernat commented Jun 16, 2023

gaborbernat commented Jun 16, 2023

f3flight commented Jun 16, 2023

f3flight commented Jun 16, 2023

f3flight commented Jun 16, 2023 • edited Loading

gaborbernat commented Jun 16, 2023

f3flight commented Jun 16, 2023

gaborbernat commented Jun 16, 2023 via email

f3flight commented Jun 16, 2023 •

edited

Loading