Poetry 1.4.0 can't build (editable) dependencies requiring setuptools <64

[JacobHayes]

• Poetry version: Poetry (version 1.4.0)

• Python version: 3.11.2

• OS version and name: macOS 13.2.1

• pyproject.toml: https://gist.github.com/JacobHayes/110054b6c6d19ca9d6d811c8b4eca374

• I am on the latest stable Poetry version, installed using a recommended method.

• I have searched the issues of this repo and believe that this is not a duplicate.

• I have consulted the FAQ and blog for any relevant entries or release notes.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

After upgrading to poetry 1.4.0, I get errors installing local editable deps that require setuptools <64. setuptools 64 changed how editable deps are installed, breaking static analysis tools. It's limited to some perhaps esoteric cases (namespace packages where you don't want to include the namespace itself... which you usually don't).

The error is Backend operation failed: HookMissing('build_editable'), which presumably was added in setuptools 64.

In the grand scheme of setuptools versions, 64 is relatively new (Aug 2022) and think I've occasionally seen much older pins. That said, I don't particularly care if poetry supports the older setuptools versions (I'm slowly migrating away from a structure causing this issue anyway), but it would be good to have the setuptools>64 requirement for built packages documented (at least in the CHANGELOGs) and report a better error, if not some installer fallback.

---

Here's the sanitized output:

$ poetry install
Installing dependencies from lock file

Package operations: 0 installs, 16 updates, 0 removals

  • Updating my-package (7.1.0 /Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package/src -> 7.1.0 /Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package): Failed

  ChefBuildError

  Backend operation failed: HookMissing('build_editable')

  at ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/installation/chef.py:152 in _prepare
      148│
      149│                 error = ChefBuildError("\n\n".join(message_parts))
      150│
      151│             if error is not None:
    → 152│                 raise error from None
      153│
      154│             return path
      155│
      156│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with my-package (7.1.0 /Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "my-package @ file:///Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package ; python_version >= "3.7""'.

[...] # Same for other editable installs

Trying the command (which was improperly quoted with the python_version) to check whether it is a poetry or build issue succeeds:

$ pip wheel --use-pep517 'my-package @ file:///Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package ; python_version >= "3.7"'
Processing ./packages/my-package
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting hera-workflows==4.3.0
  Using cached hera_workflows-4.3.0-py3-none-any.whl (75 kB)
Collecting argo-workflows==6.3.5
  Using cached argo_workflows-6.3.5-py3-none-any.whl (1.3 MB)
Collecting pytz>=2021.3
  Using cached pytz-2022.7.1-py2.py3-none-any.whl (499 kB)
Collecting python-dateutil>=2.8.2
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting certifi>=2021.10.8
  File was already downloaded /Users/jacobhayes/src/github.com/my-org/my-repo/certifi-2022.12.7-py3-none-any.whl
Collecting urllib3>=1.26.8
  File was already downloaded /Users/jacobhayes/src/github.com/my-org/my-repo/urllib3-1.26.14-py2.py3-none-any.whl
Collecting six>=1.5
  File was already downloaded /Users/jacobhayes/src/github.com/my-org/my-repo/six-1.16.0-py2.py3-none-any.whl
Saved ./hera_workflows-4.3.0-py3-none-any.whl
Saved ./argo_workflows-6.3.5-py3-none-any.whl
Saved ./python_dateutil-2.8.2-py2.py3-none-any.whl
Saved ./pytz-2022.7.1-py2.py3-none-any.whl
Building wheels for collected packages: my-package
  Building wheel for my-package (pyproject.toml) ... done
  Created wheel for my-package: filename=my_package-7.1.0-py3-none-any.whl size=30495 sha256=4c15347e00c6f017e826918506b167142731600744cc772bd1c704347ba03ce9
  Stored in directory: /Users/jacobhayes/Library/Caches/pip/wheels/a1/2a/10/958efd559fdf9deed798ca0036f61354b4d7e1cb46e8aab7b8
Successfully built my-package

[dimbleby]

related to #7574? not sure I understand how all the parts fit together but I'd try with that fix anyway

[JacobHayes]

Ah yeah, I forgot to post, but I tried installing from that PR and got the same error.

[dimbleby]

I'm not sure how convincing the above pip command is for editable installs: pip install -e ... would be a better test of how pip treats an editable install.

I know that pip has an error message about this, perhaps you ought to run into that .

ERROR: Project <foo> has a 'pyproject.toml' and its build 
backend is missing the 'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', 
it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.

IIUC I think that the dependency on a recent setuptools is something that your project ought to be declaring, rather than a poetry issue?

[radoering]

HookMissing('build_editable') probably means that old setuptools does not support PEP 660. Not sure what to do about that.

Apart from that the proposed command has to be improved:

• To avoid the improper quoting, I assume we can strip markers from the dependency because they make no difference.
• editable is not considered in the proposed command

@JacobHayes Can you try if pip wheel --use-pep517 --editable /Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package fails as expected?

[JacobHayes]

That works fine:

$ pip wheel --use-pep517 --editable /Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package/
Obtaining file:///Users/jacobhayes/src/github.com/my-org/my-repo/packages/my-package
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: my-package
  Building wheel for my-package (pyproject.toml) ... done
  Created wheel for my-package: filename=my_package-0.30.0-py3-none-any.whl size=24771 sha256=add6ceb708edbf59043b593adb537cca5757962ea74d28f43c048993f80c9882
  Stored in directory: /private/var/folders/ss/tgmzwbx52y35qpr7rzrtv8ym0000gn/T/pip-ephem-wheel-cache-pyqynsnp/wheels/d6/09/e4/9af7ef3079cdc2eca788e4b806f2154480d17335d9b58e27d5
Successfully built my-package

I thought even before poetry 1.4.0, these editable deps were being built with --use-pep517. 🤷

---

@dimbleby

IIUC I think that the dependency on a recent setuptools is something that your project ought to be declaring, rather than a poetry issue?

I think the issue is that my sub-dependency (not the current project) requires an older version of setuptools, which is pinned in the sub-dependency's pyproject.toml

[JacobHayes]

Also, I think I tried this with #7579, but perhaps I messed up the installation or was still pointing to the wrong installation. I can test again after that is merged.

[dimbleby]

so I think pip makes this work by calling setup.py develop - which they will stop doing per pypa/pip#11457 (but perhaps not soon)

poetry is often at the vanguard of dropping support for such things (or less willing to sink effort into backwards compatibility, depending on your point of view). eg see also its insistence on pep-517 builds

hard to say as yet how common it is for projects deliberately to pin to an old version of setuptools. it feels like an unusual thing to do; on the other hand you've raised this barely a day after poetry 1.4.0 was released so perhaps there are more like you out there.

you say that you don't particularly care about support for older setuptools and I tend to think that we should encourage that position. ie I'd advocate for poetry telling people who run into this: you need to allow a newer setuptools (or fall back to the old installer, or just not use poetry).

dunno whether this is worth a new error message specifically calling out this possible cause, perhaps it depends how many duplicates of this issue show up!

Regardless, thanks for the report, it has sent me down an interesting technical rabbit-hole...

[basti8909]

We ran into the same issue that @JacobHayes mentioned since we updated to Poetry 1.4.0 with an editable dependency:

[tool.poetry.dependencies]
shared = {path = "../shared", develop = true}

Poetry install fails with:

Installing shared (0.0.3 /builds/platform/shared)
  ChefBuildError
  Backend operation failed: HookMissing('build_editable')
  at /usr/local/lib/python3.8/site-packages/poetry/installation/chef.py:152 in _prepare
      148│ 
      149│                 error = ChefBuildError("\n\n".join(message_parts))
      150│ 
      151│             if error is not None:
    → 152│                 raise error from None
      153│ 
      154│             return path
      155│ 
      156│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
Note: This error originates from the build backend, and is likely not a problem with poetry but with shared (0.0.3 /builds/platform/shared) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "shared @ file:///builds/platform/shared ; python_version >= "3.8" and python_version < "4.0""'.

The check for pep517 succeeds:

$ pip wheel --use-pep517 "shared @ file:///builds/platform/shared"
Processing /builds/platform/shared
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
...
...
...
Saved ./setuptools-67.4.0-py3-none-any.whl
Building wheels for collected packages: shared
  Building wheel for shared (pyproject.toml) ... done
  Created wheel for shared: filename=shared-0.0.3-py3-none-any.whl size=57753 sha256=b191d7f34830f93955e6c7cb3e48804755988fb5f92672d5c8188f48867bf98f
  Stored in directory: /tmp/pip-ephem-wheel-cache-1u8wosvq/wheels/22/cc/8a/beebc660a7666f45746f26a1401c4cb6ed777b35c62efb5744
Successfully built shared

For now we went back to Poetry 1.3.2

Edit
Forgot to mention: We use setuptools >67 in all projects so here it can not be related to a pinned older version in a sub-dependency.

[dimbleby]

please provide a reproducible example if you hope for anyone to look at this

[miguelvalente]

Wrong Tab

[adinhodovic]

We are generating an OpenAPI client(openapi-generator) and testing it in CI using editable mode. With 1.4, the client is not importable:

<project>/core/tests/test_client.py:15:0: E0401: Unable to import 'openapi_client.apis' (import-error)
<project>/core/tests/test_client.py:15:0: E0611: No name 'apis' in module 'openapi_client' (no-name-in-module)
<project>/core/tests/test_client.py:27:0: E0401: Unable to import 'openapi_client.models' (import-error)
etc.

Reverting to 1.3.1 worked

[dimbleby]

@miguelvalente you have replied to a comment from a different issue altogether, I see no connection with this issue

@adinhodovic: what you are describing also has no apparent connection with what this issue is about

[md384]

Can reproduce with a simple example of two projects (lib-a and lib-b),

[tool.poetry]
name = "lib-a"
version = "0.1.0"
description = ""
authors = ["John Smith <[email protected]>"]
packages = [{include = "lib_a"}]

[tool.poetry.dependencies]
python = "~3.9"
requests = "^2.21.0"

[tool.poetry.group.dev.dependencies]
pytest = "^7.2.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[tool.poetry]
name = "lib-b"
version = "0.1.0"
description = ""
authors = ["John Smith <[email protected]>"]
packages = [{include = "lib_b"}]

[tool.poetry.dependencies]
python = "~3.9"
requests = "^2.21.0"
lib-a = {path = "../libA", develop = true}

[tool.poetry.group.dev.dependencies]
pytest = "^7.2.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Throws

  • Installing lib-a (0.1.0 /Users/matt/src/pythonProject/lib/libA): Failed

  ChefBuildError

  Backend operation failed: HookMissing('build_editable')

  at ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/installation/chef.py:152 in _prepare
      148│ 
      149│                 error = ChefBuildError("\n\n".join(message_parts))
      150│ 
      151│             if error is not None:
    → 152│                 raise error from None
      153│ 
      154│             return path
      155│ 
      156│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with lib-a (0.1.0 /Users/matt/src/pythonProject/lib/libA) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "lib-a @ file:///Users/matt/src/pythonProject/lib/libA ; python_version >= "3.9" and python_version < "3.10""'.

when running poetry install on lib-b

[dimbleby]

@md384 you are using a (different) ancient backend

you want something like

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[md384]

@dimbleby yes, that was the problem for us, changing that in our pyproject.toml fixed our problems.

[yeus]

We ran into the same issue that @JacobHayes mentioned since we updated to Poetry 1.4.0 with an editable dependency:

[tool.poetry.dependencies]
shared = {path = "../shared", develop = true}

we have the same problem, also using the "develop = true" flag in on of our dependencies. Also going back to 1.3.

[schristensen-unhaze]

@md384 you are using a (different) ancient backend

you want something like

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

After upgrading to poetry 1.4.2 we had an issue where setuptools was finding multiple top level packages, and changing the build-system to the above fixed our issue. This is a huge relief because it means that we don't have to change our pyproject.toml files to follow pep 621 just yet.

[thrix]

Yeah, seems poetry init nowadays creates:

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[SandersAaronD]

This is still happening, although the circumstances for it are admittedly weird.

[franciscoalves]

This is for anyone that uses Homebrew on Mac having this issue. I've installed poetry with homebrew and I don't recommend it because over time running brew upgrades created a version mistmatch between the python formula and the poetry formula and I had python 3.11 with a poetry that used the forumla for python 3.12.
TLDR:
This is what fixed the issue for me:

1. install the python you want with homebrew.
   brew install [email protected]
2. use pip that came with that python to install pipx.
   pip3.11 install pipx
3. use pipx to install poetry
   pipx install poetry

Now poetry will be tied to that python version you chose and those errors will go away.

[rmarscher]

Sorry if this is not helpful... but I had the same error and eventually discovered it was because the python version I had built via pyenv was missing libffi. I added libffi-devel to my system and reinstalled python and then poetry was able to install the dependencies without an issue.

[dimbleby]

pretty much all comments on this are unrelated to the thing that it actually reports, which is a very specific issue about editable dependencies and pins to old versions of setuptools.

setuptools 64.0 was not that old when this was reported: but it is now a year older - the likelihood and value of anyone doing anything with this are both diminishing by the day.

suggest that the issue might as well be be closed out, it seems to be acting only as a honeytrap

[JacobHayes]

Thanks, forgot this issue was still open. Agreed on diminishing returns to OP

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.