Relative path imports are resolved relative to CWD instead of pyproject.toml #1757

swist · 2019-12-17T16:14:21Z

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: circleci/python3.7.4-stretch, osx
Poetry version:1.0.0
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/swist/89660bf70dbed6d0622bba7db381ae9c

Issue

I have 3 projects in a monorepo situation
One has a pyproject.toml in kafkaesque/pyproject.toml
It depends on another poetry package defined in config/pyproject.toml
Then there's another project that installs from kafkaesque and config into pyproject.toml like so:

config/pyproject.toml
kafkaesque/pyproject.toml
pyproject.toml

I get the following error:

[EnvCommandError]
Command ['/Users/swistak/Library/Caches/pypoetry/virtualenvs/marmot-api-SV6HPF22-py3.7/bin/pip', 'install', '--no-deps', '-U', '/Users/swistak/code/marmot/api/kafkaesque'] errored with the following return code 1, and output: 
Processing ./kafkaesque
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  Complete output from command /Users/swistak/Library/Caches/pypoetry/virtualenvs/marmot-api-SV6HPF22-py3.7/bin/python /Users/swistak/Library/Caches/pypoetry/virtualenvs/marmot-api-SV6HPF22-py3.7/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/93/78_vw68j75j3xf_z25p29_b80000gn/T/tmpmcyd92s5:
  Traceback (most recent call last):
    File "/Users/swistak/Library/Caches/pypoetry/virtualenvs/marmot-api-SV6HPF22-py3.7/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
      main()
    File "/Users/swistak/Library/Caches/pypoetry/virtualenvs/marmot-api-SV6HPF22-py3.7/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/swistak/Library/Caches/pypoetry/virtualenvs/marmot-api-SV6HPF22-py3.7/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 54, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/93/78_vw68j75j3xf_z25p29_b80000gn/T/pip-build-env-go6yfl9w/overlay/lib/python3.7/site-packages/poetry/masonry/api.py", line 25, in get_requires_for_build_wheel
      poetry = Factory().create_poetry(Path("."))
    File "/private/var/folders/93/78_vw68j75j3xf_z25p29_b80000gn/T/pip-build-env-go6yfl9w/overlay/lib/python3.7/site-packages/poetry/factory.py", line 99, in create_poetry
      package.add_dependency(name, constraint)
    File "/private/var/folders/93/78_vw68j75j3xf_z25p29_b80000gn/T/pip-build-env-go6yfl9w/overlay/lib/python3.7/site-packages/poetry/packages/package.py", line 331, in add_dependency
      develop=constraint.get("develop", True),
    File "/private/var/folders/93/78_vw68j75j3xf_z25p29_b80000gn/T/pip-build-env-go6yfl9w/overlay/lib/python3.7/site-packages/poetry/packages/directory_dependency.py", line 36, in __init__
      raise ValueError("Directory {} does not exist".format(self._path))
  ValueError: Directory ../config does not exist

Note that both kafkaesque/pyproject.toml and pyproject.toml both depend on config/pyproject.toml albeit with a different path in their respective project files

The text was updated successfully, but these errors were encountered:

finswimmer · 2019-12-17T16:39:35Z

Hello @swist ,

could be that this is similar to #1689.

What's the pip version within the venv? If it's greater >18, could you please try downgrading pip to a version <19 within the venv?

fin swimmer

swist · 2019-12-17T16:48:30Z

Hey, I have managed to carve some more time to debug this. Removing develop = false from all of the dependencies that I install via relative paths seems to have fixed it. Could it perhaps be related to the fact that poetry quite eagerly cleans setup.py when develop = false ?

I think it might have been broken all along, I'll update the initial post (just tried with 1.0.0b8 and the same happened)

swist · 2019-12-18T00:38:22Z

Another update. Even with the change outlined above I am still getting pathlib value error when I try to run poetry build

teknico · 2020-01-20T13:44:10Z

I'm hitting this too. I have two projects in one repo, the second one depends on the first, I'm using a relative path, and I get the same error in the same place.

finswimmer · 2020-02-06T20:01:23Z

Should be fixed in the latest release by #1875.

teknico · 2020-02-07T17:39:05Z

Regrettably I'm still getting the same error with v.1.0.3 (which includes #1875).

finswimmer · 2020-02-07T19:36:29Z

Hello @teknico ,

can you please show how your folder structure looks like and provide the pyproject.toml's?

fin swimmer

teknico · 2020-02-07T19:47:24Z

@finswimmer, sure, the code is in the tinydecred repo. The tinywallet project depends on the decred project's code by means of a relative path. Thanks.

finswimmer · 2020-02-08T09:06:53Z

Ok, I've got it. Will have a closer look at it the next days.

finswimmer · 2020-02-17T05:23:59Z

I tried to take a closer look at this problem. It arises if a path dependency is more then one time in the dependency tree and has different relative paths. Then something went wrong during the version solving. And that's the part in the poetry which is still hard for me to understand.

I've uploaded a small example I used for testing: https://github.com/finswimmer/poetry-issue1757

I tried to follow the code. The dependency that is passed to search_for here:

poetry/poetry/puzzle/provider.py

Line 97 in 12db4a5

def search_for(self, dependency): # type: (Dependency) -> List[Package]

has a wrong path value, while base and full_path are ok:

depency.name: packageb
dependency.base: /home/finswimmer/tmp/poetry/issue_1757/package
dependency.path: packagea/../packageb
dependency.full_path: /home/finswimmer/tmp/poetry/issue_1757/packagea/../packageb

The correct value for depency.path would be ../packageb. This is correct identified in an earlier step, when poetry finds packagea and collect it's dependencies.

So it looks like, that poetry is mixing some information when it recognize that the same package name is referenced twice and it has to decide which one it should take.

@sdispater could you please take a look at this?

sdispater · 2020-02-28T13:56:38Z

@finswimmer I'll try to take a look at this. I thought I had fixed but I missed the case where, like you mentioned, the same package is references multiple times with various relative paths.

I'll assigned myself to the issue.

Timost · 2020-04-21T12:09:33Z

I'm also hitting this issue. Not sure if this will help but I work in a monorepo context with the following structure:

.
├── acquisition
│   ├── customclient
│   │   ├── poetry.lock                                                                                                                                                                                                                                                                   
│   │   ├── pyproject.toml                                                                                                                                                                                                                                                                
│   │   ├── README.md     
│   ├── ...
├── commons
│   ├── logutils
│   │   ├── poetry.lock                                                                                                                                                                                                                                                                   
│   │   ├── pyproject.toml                                                                                                                                                                                                                                                                
│   │   ├── ...
│   ├── geoutils
│   │   ├── poetry.lock                                                                                                                                                                                                                                                                   
│   │   ├── pyproject.toml                                                                                                                                                                                                                                                                
│   │   ├── ...
├── top_project
│   ├── .....
│   ├── poetry.lock
│   ├── pyproject.toml

geoutils depends on logutils. (../logutils)
top_project depends on geoutils (../commons/geoutils) and on logutils (../commons/logutils)
customclient depends on logutils (../../commons/logutils) and top_project (../../top_project)

I know this might look messed up in terms of layout, but the path resolved for (logutils) is wrong when I do a poetry install in acquisition/customclient

yanbussieres · 2020-11-02T04:57:46Z

Python 3.9, pip 20.2.3

Is this solving every corner case ?
Please have a look at this example :
Example here

main requires dep2,
dep2 requires dep1

Trying to run poetry install from the main folder, I still have this issue :

 File "/private/var/folders/5d/yl0b7h4x00s0jt_zjqy58r040000gn/T/pip-build-env-19rc_6el/overlay/lib/python3.9/site-packages/poetry/core/packages/directory_dependency.py", line 36, in __init__
          raise ValueError("Directory {} does not exist".format(self._path))
      ValueError: Directory ../dep1 does not exist

Solution (kind of)

from lib/dep2, python 3.9, pip 20.2.3, runpoetry install
from main/ , python 3.9, Downgrading to pip 18.0, run poetry install

Or
Everything is ok with python 3.8, pip 20.2.3, poetry 1.0.10 (all installed from brew)

finswimmer · 2020-11-02T18:25:14Z

Hello @yanbussieres,

since poetry 1.1 path dependencies are no longer installed in editable mode by default. You will need to add a develop=true to your dependency definitions in the pyproject.toml, e.g. dep1 = {path = "../dep1", develop=true}.

fin swimmer

carloshenrique153 · 2021-04-12T15:35:43Z

Even though this issue is closed, this is still a bug in poetry 1.1.5.

If you have a monorepo with 3 projects where

pyproject.toml of project A has B = { path = "../B" }
pyproject.toml of project B has C = { path = "../C" }

Then running poetry update on project A causes: ValueError: Directory ../C does not exist

ElmShahar · 2021-07-20T13:31:58Z

This still seems to be an issue in 1.1.7 as well.
in pyproject.toml -
kafka-client = {path = "resources/wheel/kafka_client-0.1.0-py3-none-any.whl", develop=true}

result in requirements.txt is still absolute path:
kafka-client @ file:///LOCAL/ABSOLUTE/PATH/resources/wheel/kafka_client-0.1.0-py3-none-any.whl; python_version >= "3.9" and python_version < "4.0"
--hash=sha256:5928a097615b2f514b0523b54c0ddcdf3fd9a6eb24f05d4c272e268e2d3f0c14

CafeLungo · 2022-03-28T13:54:22Z

This is still a problem in 1.1.13 as well. I would expect the export -f requirements.txt to use a relative path, not the local absolute path.

I think in the example following from above, it would be:

./resources/wheel/kafka_client-0.1.0-py3-none-any.whl; python_version >= "3.9" and python_version < "4.0"
--hash=sha256:5928a097615b2f514b0523b54c0ddcdf3fd9a6eb24f05d4c272e268e2d3f0c14

(missing the kafka-clinet @ at the start)

github-actions · 2024-03-02T04:06:27Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

swist added the kind/bug Something isn't working as expected label Dec 17, 2019

teknico mentioned this issue Jan 20, 2020

Packaging tinydecred decred/tinydecred#35

Closed

finswimmer closed this as completed Feb 6, 2020

teknico mentioned this issue Feb 7, 2020

Multi: sort import lines with the isort tool, and add it to both projects decred/tinydecred#72

Merged

finswimmer reopened this Feb 8, 2020

finswimmer self-assigned this Feb 8, 2020

sdispater self-assigned this Feb 28, 2020

swist mentioned this issue Mar 27, 2020

Repeated builds using cache produce broken images GoogleContainerTools/kaniko#1162

Closed

sdispater mentioned this issue May 9, 2020

Fix resolution of path, url and VCS dependencies #2398

Merged

2 tasks

sdispater closed this as completed in #2398 Jun 5, 2020

severo mentioned this issue May 27, 2022

ci: 🎡 use cache with poetry huggingface/dataset-viewer#314

Merged

github-actions bot locked as resolved and limited conversation to collaborators Mar 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Relative path imports are resolved relative to CWD instead of pyproject.toml #1757

Relative path imports are resolved relative to CWD instead of pyproject.toml #1757

swist commented Dec 17, 2019 •

edited

Loading

finswimmer commented Dec 17, 2019

swist commented Dec 17, 2019 •

edited

Loading

swist commented Dec 18, 2019

teknico commented Jan 20, 2020 •

edited

Loading

finswimmer commented Feb 6, 2020

teknico commented Feb 7, 2020

finswimmer commented Feb 7, 2020

teknico commented Feb 7, 2020

finswimmer commented Feb 8, 2020

finswimmer commented Feb 17, 2020

sdispater commented Feb 28, 2020

Timost commented Apr 21, 2020

yanbussieres commented Nov 2, 2020 •

edited

Loading

finswimmer commented Nov 2, 2020

carloshenrique153 commented Apr 12, 2021

ElmShahar commented Jul 20, 2021

CafeLungo commented Mar 28, 2022 •

edited

Loading

github-actions bot commented Mar 2, 2024

Relative path imports are resolved relative to CWD instead of pyproject.toml #1757

Relative path imports are resolved relative to CWD instead of pyproject.toml #1757

Comments

swist commented Dec 17, 2019 • edited Loading

Issue

finswimmer commented Dec 17, 2019

swist commented Dec 17, 2019 • edited Loading

swist commented Dec 18, 2019

teknico commented Jan 20, 2020 • edited Loading

finswimmer commented Feb 6, 2020

teknico commented Feb 7, 2020

finswimmer commented Feb 7, 2020

teknico commented Feb 7, 2020

finswimmer commented Feb 8, 2020

finswimmer commented Feb 17, 2020

sdispater commented Feb 28, 2020

Timost commented Apr 21, 2020

yanbussieres commented Nov 2, 2020 • edited Loading

Solution (kind of)

finswimmer commented Nov 2, 2020

carloshenrique153 commented Apr 12, 2021

ElmShahar commented Jul 20, 2021

CafeLungo commented Mar 28, 2022 • edited Loading

github-actions bot commented Mar 2, 2024

swist commented Dec 17, 2019 •

edited

Loading

swist commented Dec 17, 2019 •

edited

Loading

teknico commented Jan 20, 2020 •

edited

Loading

yanbussieres commented Nov 2, 2020 •

edited

Loading

CafeLungo commented Mar 28, 2022 •

edited

Loading