Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poetry selects wrong Python version if Poetry's installation differs from python3 (or python3 changes) #7158

Closed
4 tasks done
ShadowLNC opened this issue Dec 8, 2022 · 9 comments · Fixed by #7221
Closed
4 tasks done
Labels
kind/bug Something isn't working as expected status/triage This issue needs to be triaged

Comments

@ShadowLNC
Copy link

  • Poetry version: 1.2.2
  • Python version: CPython 3.9.12
  • OS version and name: MacOS (darwin) 12.6.1 Monterey (ARM architecture)
  • pyproject.toml: See below

Issue

If Poetry's internal virtual environment (used to execute Poetry) doesn't match the version resolved by the python3 command, then Poetry doesn't select the correct Python version (defined in pyproject.toml) for installation when creating a new virtual environment for a project.

There are two different ways that I've found to achieve this scenario.

Scenario 1:

  1. Install Python 3.10 (or 3.9 or 3.8, just referring to 3.10 from now on) from https://python.org (using standard installer)
  2. (Now $PATH resolves python3 to 3.10)
  3. Run curl -sSL https://install.python-poetry.org | python3 - as officially supported installation mechanism
  4. Install Python 3.11 from https://python.org
  5. (Now $PATH resolves python3 to 3.11)

Scenario 2:

  1. Install Python 3.9 (or any version other than 3.11) from https://python.org (using standard installer)
  2. Install Python 3.11 from https://python.org
  3. (Now $PATH resolves python3 to 3.11)
  4. Run curl -sSL https://install.python-poetry.org | python3.9 -

Outcome

Relaunch the terminal to ensure $PATH changes take effect.

$ poetry install --no-root
The currently activated Python version 3.9.12 is not supported by the project (~3.11).
Trying to find and use a compatible version. 
Using python3 (3.11.0)
Creating virtualenv poetrytest in /Users/scott/Documents/automation/poetrytest/.venv
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing attrs (22.1.0)

This all looks correct, but then I activate the virtual environment with source .venv/bin/activate.

Expected

The shell prompt shows (poetrytest-py3.11) and python --version shows 3.11.0. I would also expect poetry shell to work in place of source .venv/bin/activate.

Actual

The shell prompt shows (poetrytest-py3.11) and python --version shows 3.9.12 (or whatever older version I used).

$ poetry shell

Current Python version (3.9.12) is not allowed by the project (~3.11).
Please change python executable via the "env use" command.

I've had a colleague verify this behaviour on their MacOS machine as well, and another colleague indicated that it may work properly on Ubuntu (but I have not verified that first hand).

Files

poetry.toml

[virtualenvs]
in-project = true

pyproject.toml (minimal packages just to show installation)

[tool.poetry]
name = "poetrytest"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.11"
attrs = "^22.1.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

poetry.lock

[[package]]
name = "attrs"
version = "22.1.0"
description = "Classes Without Boilerplate"
category = "main"
optional = false
python-versions = ">=3.5"

[package.extras]
dev = ["cloudpickle", "coverage[toml] (>=5.0.2)", "furo", "hypothesis", "mypy (>=0.900,!=0.940)", "pre-commit", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "sphinx", "sphinx-notfound-page", "zope.interface"]
docs = ["furo", "sphinx", "sphinx-notfound-page", "zope.interface"]
tests = ["cloudpickle", "coverage[toml] (>=5.0.2)", "hypothesis", "mypy (>=0.900,!=0.940)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "zope.interface"]
tests-no-zope = ["cloudpickle", "coverage[toml] (>=5.0.2)", "hypothesis", "mypy (>=0.900,!=0.940)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins"]

[metadata]
lock-version = "1.1"
python-versions = "~3.11"
content-hash = "088f78f8dc08e7c9c2577c94e3669da3a7f2e3a6a2c4caa83caa754e5c626bef"

[metadata.files]
attrs = [
    {file = "attrs-22.1.0-py2.py3-none-any.whl", hash = "sha256:86efa402f67bf2df34f51a335487cf46b1ec130d02b8d39fd248abfd30da551c"},
    {file = "attrs-22.1.0.tar.gz", hash = "sha256:29adc2665447e5191d0e7c568fde78b21f9672d344281d0c6e1ab085429b22b6"},
]
@ShadowLNC ShadowLNC added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Dec 8, 2022
@neersighted
Copy link
Member

This is partially by design; you must set https://python-poetry.org/docs/configuration/#virtualenvsprefer-active-python-experimental if you want Poetry to resolve Python instead of using sys.interpreter. However, there are bugs related to our auto-discovery of a compatible Python; @finswimmer put some work in here recently, I'll leave it to him to determine if this is fixed in 1.3 or there is something novel here.

@ShadowLNC
Copy link
Author

Thanks for the quick reply 😄

If I add prefer-active-python = true in poetry.toml, I get this:

$ poetry install --no-root
/bin/sh: python: command not found

Command python -c "import sys; print(sys.executable)" errored with the following return code 127, and output: 

Python 2 was removed from newer MacOS, but I wouldn't expect Poetry to be looking for something other than Python 3.

I'm not sure how this relates to prefer-active-python though? Poetry tells me it's installing 3.11 (which it has discovered, and which I requested in pyproject.toml), but ends up installing 3.9 (which I think matches sys.interpreter).

@finswimmer
Copy link
Member

Hey,

unfortunately I'm not able to reproduce it. Would be helpful if you or anyone else can provide a reproducer with docker.

fin swimmer

@ShadowLNC
Copy link
Author

ShadowLNC commented Dec 10, 2022

Edited to add: Since Poetry 1.3.0 has been released, these Ubuntu results are on Poetry 1.3.0.

Hi - I think I have managed to get a set of reproduction steps in docker. Ubuntu locks the python3 symlink to 3.10, so I've changed it such that we install Poetry against 3.9, and request 3.10 in pyproject.toml.

I created poetry-repro-setup.sh:

apt-get update -y
# Need software-properties-common for apt-add-repository
apt-get install -y software-properties-common python3.10 python3.10-venv python3.11 python3.11-venv curl
add-apt-repository -y ppa:deadsnakes/ppa
apt-get install -y python3.9 python3.9-venv  # From deadsnakes

mkdir poetrytest
cd poetrytest

cat - > poetry.toml <<EOF
[virtualenvs]
in-project = true
EOF

cat - > pyproject.toml <<EOF
[tool.poetry]
name = "poetrytest"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.10"
attrs = "^22.1.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
EOF

cat - > poetry.lock <<EOF
[[package]]
name = "attrs"
version = "22.1.0"
description = "Classes Without Boilerplate"
category = "main"
optional = false
python-versions = ">=3.5"

[package.extras]
dev = ["cloudpickle", "coverage[toml] (>=5.0.2)", "furo", "hypothesis", "mypy (>=0.900,!=0.940)", "pre-commit", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "sphinx", "sphinx-notfound-page", "zope.interface"]
docs = ["furo", "sphinx", "sphinx-notfound-page", "zope.interface"]
tests = ["cloudpickle", "coverage[toml] (>=5.0.2)", "hypothesis", "mypy (>=0.900,!=0.940)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "zope.interface"]
tests-no-zope = ["cloudpickle", "coverage[toml] (>=5.0.2)", "hypothesis", "mypy (>=0.900,!=0.940)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins"]

[metadata]
lock-version = "1.1"
python-versions = "~3.10"
content-hash = "f7ca16790b2695440ca3017a7bb30137759c6612e03aec49ae380742bbc484ed"

[metadata.files]
attrs = [
    {file = "attrs-22.1.0-py2.py3-none-any.whl", hash = "sha256:86efa402f67bf2df34f51a335487cf46b1ec130d02b8d39fd248abfd30da551c"},
    {file = "attrs-22.1.0.tar.gz", hash = "sha256:29adc2665447e5191d0e7c568fde78b21f9672d344281d0c6e1ab085429b22b6"},
]
EOF

I ran the docker container as follows:

docker run --rm -it -v $(pwd)/poetry-repro-setup.sh:/poetry-repro-setup.sh ubuntu bash

Then, within the docker container:

chmod +x poetry-repro-setup.sh
/poetry-repro-setup.sh
curl -sSL https://install.python-poetry.org | python3.9 - 
export PATH="/root/.local/bin:$PATH"
cd /poetrytest

And then when I try to install the virtual environment, I get the same behaviour as MacOS:

root@84b3855a09a8:/poetrytest# poetry install --no-root
The currently activated Python version 3.9.16 is not supported by the project (~3.10).
Trying to find and use a compatible version. 
Using python3 (3.10.6)
Creating virtualenv poetrytest in /poetrytest/.venv
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing attrs (22.1.0)
root@84b3855a09a8:/poetrytest# poetry shell

Current Python version (3.9.16) is not allowed by the project (~3.10).
Please change python executable via the "env use" command.
root@84b3855a09a8:/poetrytest# source .venv/bin/activate
(poetrytest-py3.10) root@84b3855a09a8:/poetrytest# python --version
Python 3.9.16

If, instead of installing Poetry with 3.9, I install it with 3.11 (as provided in the container via above script - an older 3.11.0rc1 version from the official Ubuntu Jammy repository, instead of deadsnakes), I get different behaviour.

  • If 3.10 (pointed at by the python3 symlink) is requested by pyproject.toml and poetry.lock, it warns that 3.11 is active, finds 3.10 (Using python3 (3.10.6) is output), and installs correctly.
  • If 3.11 is requested by pyproject.toml and poetry.lock (with the content hash from my original post, 088f78f8dc08e7c9c2577c94e3669da3a7f2e3a6a2c4caa83caa754e5c626bef), it does not warn (as 3.11 is active and I am requesting 3.11), but installs 3.10 instead, with the 3.11 venv prompt.

Here's my full results (where python3 always refers to python3.10 on Ubuntu).
Edited to add: Since Poetry 1.3.0 has been released, these Ubuntu results are on Poetry 1.3.0.

Poetry Installed With Requested Python Virtualenv Prompt Installed Python
3.9 3.9 3.9 ✅ 3.9
3.9 3.10 3.10 ❌ 3.9
3.9 3.11 3.11 ✅ 3.11
3.10 3.9 3.9 ✅ 3.9
3.10 3.10 3.10 ✅ 3.10
3.10 3.11 3.11 ✅ 3.11
3.11 3.9 3.9 ✅ 3.9
3.11 3.10 3.10 ✅ 3.10
3.11 3.11 3.11 ❌ 3.10

(For completeness, the poetry.lock content hash for Python 3.9 is 1e20473c3d8816fdca442b8208dd4268efad46603021991e7219def665cf6772.)

@finswimmer
Copy link
Member

Thanks a lot for your excellent reproducer @ShadowLNC. 👍 I can reproduce it with your instructions.

Hopefully I have some time the next days to find out what's going on.

@finswimmer
Copy link
Member

I think I found the place where things go wrong.

Poetry finds python3 as a suitable executable, because it's version is 3.10.6. Now python3 is passed as an argument to virtualenv. virtualenv runs in Poetry's own venv, so python3 points to Python 3.9.

What we must do here is to always pass the full path to the executable. (This is what already happening if virtualenvs.prefer-active-python is set to true.)

@finswimmer
Copy link
Member

Fix is prepared: #7221

@rpf3
Copy link

rpf3 commented Dec 30, 2022

@finswimmer will this be released soon? I just upgraded to a new system and am getting bit by this issue using Python 3.10.6 as well.

stanislaw added a commit to mull-project/FileCheck.py that referenced this issue Jan 10, 2023
Disabling the Windows CI for now because Poetry has issues:

python-poetry/poetry#7158
stanislaw added a commit to mull-project/FileCheck.py that referenced this issue Jan 10, 2023
Disabling the Windows CI for now because Poetry has issues:

python-poetry/poetry#7158
stanislaw added a commit to mull-project/FileCheck.py that referenced this issue Jan 10, 2023
Disabling the Windows CI for now because Poetry has issues:

python-poetry/poetry#7158
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Something isn't working as expected status/triage This issue needs to be triaged
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants