Provide index to pip for installation of sdist build dependencies #3249

ghost · 2020-10-19T23:13:40Z

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: Windows 10, Version 1909
Poetry version: 1.1.3
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/ajacquah/4b6cab23073fb21c03eaa436b911f89d

Issue

Pip as invoked by poetry installs build depdencies from pypi.org instead of custom repository. And since I am running this behind a firewall with no proxy settings this fails. My understanding was that poetry would use default repository for all package actions. Can you please take a look?

To reproduce:

Create new project. $poetry new --src test-pkg
Add a private default source. This source is a mirror of pypi.org + internal packages.
From limited connectivity environment, try to add black dev dependency: poetry add black -D

Result stack trace:

Command C:\Source\test-pkg\.venv\Scripts\pip.exe install --no-deps file:///C:/Users/ajacquah/AppData/Local/pypoetry/Cache/artifacts/8b/b9/b7/d4ec53ca89b288b6be53361d96da13d397c1dfaf70aa7b48b127e6c727/black-20.8b1.tar.gz errored with the following return code 1, and output:
  Processing c:\users\ajacquah\appdata\local\pypoetry\cache\artifacts\8b\b9\b7\d4ec53ca89b288b6be53361d96da13d397c1dfaf70aa7b48b127e6c727\black-20.8b1.tar.gz
    Installing build dependencies: started
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: 'C:\Source\rails-test\.venv\Scripts\python.exe' 'C:\Source\rails-test\.venv\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\ajacquah\AppData\Local\Temp\pip-build-env-x0saifh5\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel
         cwd: None
    Complete output (7 lines):
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4CB670>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4CBAF0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4CBC40>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4CBEE0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4A3BB0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    ERROR: Could not find a version that satisfies the requirement setuptools>=41.0 (from versions: none)
    ERROR: No matching distribution found for setuptools>=41.0
    ----------------------------------------
  ERROR: Command errored out with exit status 1: 'C:\Source\test-pkg\.venv\Scripts\python.exe' 'C:\Source\test-pkg\.venv\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\ajacquah\AppData\Local\Temp\pip-build-env-x0saifh5\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.


  at c:\python38\lib\site-packages\poetry\utils\env.py:948 in _run
       944│                 output = subprocess.check_output(
       945│                     cmd, stderr=subprocess.STDOUT, **kwargs
       946│                 )
       947│         except CalledProcessError as e:
    →  948│             raise EnvCommandError(e, input=input_)
       949│
       950│         return decode(output)
       951│
       952│     def execute(self, bin, *args, **kwargs):

Thanks for your help!

The text was updated successfully, but these errors were encountered:

abn · 2020-10-19T23:59:24Z

@ajacquah can you provide the output of the following please?

poetry add black -D -vvv

# if black is already in the pyproject.toml you can do this too
poetry lock -vvv

ghost · 2020-10-20T04:55:05Z

@abn here's the output

Microsoft Windows [Version 10.0.18363.1082]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Source\test-pkg>poetry add black -D -vvv
Using virtualenv: C:\Users\ajacquah\AppData\Local\pypoetry\Cache\virtualenvs\test-pkg-8YGbIfVO-py3.8
prod-pypi: 0 packages found for black *
prod-pypi: 1 packages found for black *
Using version ^20.8b1 for black

Updating dependencies
Resolving dependencies...
   1: fact: test-pkg is 0.1.0
   1: derived: test-pkg
   1: fact: test-pkg depends on pytest (^5.2)
   1: fact: test-pkg depends on black (^20.8b1)
   1: fact: test-pkg depends on pytest (^5.2)
   1: fact: test-pkg depends on black (^20.8b1)
   1: selecting test-pkg (0.1.0)
   1: derived: black (^20.8b1)
   1: derived: pytest (^5.2)
prod-pypi: 1 packages found for black >=20.8b1,<21.0
   1: fact: black (20.8b1) depends on click (>=7.1.2)
   1: fact: black (20.8b1) depends on appdirs (*)
   1: fact: black (20.8b1) depends on toml (>=0.10.1)
   1: fact: black (20.8b1) depends on typed-ast (>=1.4.0)
   1: fact: black (20.8b1) depends on regex (>=2020.1.8)
   1: fact: black (20.8b1) depends on pathspec (>=0.6,<1)
   1: fact: black (20.8b1) depends on typing_extensions (>=3.7.4)
   1: fact: black (20.8b1) depends on mypy_extensions (>=0.4.3)
   1: selecting black (20.8b1)
   1: derived: mypy_extensions (>=0.4.3)
   1: derived: typing_extensions (>=3.7.4)
   1: derived: pathspec (>=0.6,<1)
   1: derived: regex (>=2020.1.8)
   1: derived: typed-ast (>=1.4.0)
   1: derived: toml (>=0.10.1)
   1: derived: appdirs (*)
   1: derived: click (>=7.1.2)
   1: fact: pytest (5.2.0) depends on py (>=1.5.0)
   1: fact: pytest (5.2.0) depends on packaging (*)
   1: fact: pytest (5.2.0) depends on attrs (>=17.4.0)
   1: fact: pytest (5.2.0) depends on more-itertools (>=4.0.0)
   1: fact: pytest (5.2.0) depends on atomicwrites (>=1.0)
   1: fact: pytest (5.2.0) depends on pluggy (>=0.12,<1.0)
   1: fact: pytest (5.2.0) depends on wcwidth (*)
   1: fact: pytest (5.2.0) depends on colorama (*)
   1: selecting pytest (5.2.0)
   1: derived: colorama (*)
   1: derived: wcwidth (*)
   1: derived: pluggy (>=0.12,<1.0)
   1: derived: atomicwrites (>=1.0)
   1: derived: more-itertools (>=4.0.0)
   1: derived: attrs (>=17.4.0)
   1: derived: packaging (*)
   1: derived: py (>=1.5.0)
   1: selecting mypy-extensions (0.4.3)
   1: selecting typing-extensions (3.7.4.3)
   1: selecting pathspec (0.8.0)
   1: selecting regex (2020.9.27)
   1: selecting typed-ast (1.4.1)
   1: selecting toml (0.10.1)
   1: selecting appdirs (1.4.4)
   1: selecting click (7.1.2)
   1: selecting colorama (0.4.4)
   1: selecting wcwidth (0.2.5)
   1: selecting pluggy (0.13.1)
   1: selecting atomicwrites (1.4.0)
   1: selecting more-itertools (8.5.0)
   1: selecting attrs (20.2.0)
   1: fact: packaging (20.4) depends on pyparsing (>=2.0.2)
   1: fact: packaging (20.4) depends on six (*)
   1: selecting packaging (20.4)
   1: derived: six (*)
   1: derived: pyparsing (>=2.0.2)
   1: selecting py (1.9.0)
   1: selecting six (1.15.0)
   1: selecting pyparsing (2.4.7)
   1: Version solving took 0.200 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals, 19 skipped

  • Installing pyparsing (2.4.7): Skipped for the following reason: Already installed
  • Installing six (1.15.0): Skipped for the following reason: Already installed
  • Installing appdirs (1.4.4): Skipped for the following reason: Already installed
  • Installing atomicwrites (1.4.0): Skipped for the following reason: Already installed
  • Installing attrs (20.2.0): Skipped for the following reason: Already installed
  • Installing click (7.1.2): Skipped for the following reason: Already installed
  • Installing more-itertools (8.5.0): Skipped for the following reason: Already installed
  • Installing colorama (0.4.4): Skipped for the following reason: Already installed
  • Installing packaging (20.4): Skipped for the following reason: Already installed
  • Installing pluggy (0.13.1): Skipped for the following reason: Already installed
  • Installing mypy-extensions (0.4.3): Skipped for the following reason: Already installed
  • Installing pathspec (0.8.0): Skipped for the following reason: Already installed
  • Installing regex (2020.9.27): Skipped for the following reason: Already installed
  • Installing toml (0.10.1): Skipped for the following reason: Already installed
  • Installing py (1.9.0): Skipped for the following reason: Already installed
  • Installing typed-ast (1.4.1): Skipped for the following reason: Already installed
  • Installing typing-extensions (3.7.4.3): Skipped for the following reason: Already installed
  • Installing wcwidth (0.2.5): Skipped for the following reason: Already installed
  • Installing black (20.8b1)
  • Installing pytest (5.2.0): Skipped for the following reason: Already installed

  Stack trace:

  7  c:\python38\lib\site-packages\poetry\installation\executor.py:199 in _execute_operation
      197│
      198│             try:
    → 199│                 result = self._do_execute_operation(operation)
      200│             except EnvCommandError as e:
      201│                 if e.e.returncode == -2:

  6  c:\python38\lib\site-packages\poetry\installation\executor.py:273 in _do_execute_operation
      271│             return 0
      272│
    → 273│         result = getattr(self, "_execute_{}".format(method))(operation)
      274│
      275│         if result != 0:

  5  c:\python38\lib\site-packages\poetry\installation\executor.py:408 in _execute_install
      406│
      407│     def _execute_install(self, operation):  # type: (Install) -> None
    → 408│         return self._install(operation)
      409│
      410│     def _execute_update(self, operation):  # type: (Update) -> None

  4  c:\python38\lib\site-packages\poetry\installation\executor.py:446 in _install
      444│             args.insert(2, "-U")
      445│
    → 446│         return self.run_pip(*args)
      447│
      448│     def _update(self, operation):

  3  c:\python38\lib\site-packages\poetry\installation\executor.py:297 in run_pip
      295│     def run_pip(self, *args, **kwargs):  # type: (...) -> int
      296│         try:
    → 297│             self._env.run_pip(*args, **kwargs)
      298│         except EnvCommandError as e:
      299│             output = decode(e.e.output)

  2  c:\python38\lib\site-packages\poetry\utils\env.py:916 in run_pip
       914│         pip = self.get_pip_command()
       915│         cmd = pip + list(args)
    →  916│         return self._run(cmd, **kwargs)
       917│
       918│     def _run(self, cmd, **kwargs):

  1  c:\python38\lib\site-packages\poetry\utils\env.py:1189 in _run
      1187│             self.unset_env("__PYVENV_LAUNCHER__")
      1188│
    → 1189│             return super(VirtualEnv, self)._run(cmd, **kwargs)
      1190│
      1191│     def execute(self, bin, *args, **kwargs):

  EnvCommandError

  Command C:\Users\ajacquah\AppData\Local\pypoetry\Cache\virtualenvs\test-pkg-8YGbIfVO-py3.8\Scripts\pip.exe install --no-deps file:///C:/Users/ajacquah/AppData/Local/pypoetry/Cache/artifacts/8b/b9/b7/d4ec53ca89b288b6be53361d96da13d397c1dfaf70aa7b48b127e6c727/black-20.8b1.tar.gz errored with the following return code 1, and output:
  Processing c:\users\ajacquah\appdata\local\pypoetry\cache\artifacts\8b\b9\b7\d4ec53ca89b288b6be53361d96da13d397c1dfaf70aa7b48b127e6c727\black-20.8b1.tar.gz
    Installing build dependencies: started
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\ajacquah\AppData\Local\pypoetry\Cache\virtualenvs\test-pkg-8YGbIfVO-py3.8\Scripts\python.exe' 'C:\Users\ajacquah\AppData\Local\pypoetry\Cache\virtualenvs\test-pkg-8YGbIfVO-py3.8\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\ajacquah\AppData\Local\Temp\pip-build-env-ykmktx6r\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel
         cwd: None
    Complete output (7 lines):
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000186F282E6A0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000186F282E370>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000186F282E250>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000186F282E130>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000186F282E0A0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    ERROR: Could not find a version that satisfies the requirement setuptools>=41.0 (from versions: none)
    ERROR: No matching distribution found for setuptools>=41.0
    ----------------------------------------
  ERROR: Command errored out with exit status 1: 'C:\Users\ajacquah\AppData\Local\pypoetry\Cache\virtualenvs\test-pkg-8YGbIfVO-py3.8\Scripts\python.exe' 'C:\Users\ajacquah\AppData\Local\pypoetry\Cache\virtualenvs\test-pkg-8YGbIfVO-py3.8\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\ajacquah\AppData\Local\Temp\pip-build-env-ykmktx6r\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.


  at c:\python38\lib\site-packages\poetry\utils\env.py:948 in _run
       944│                 output = subprocess.check_output(
       945│                     cmd, stderr=subprocess.STDOUT, **kwargs
       946│                 )
       947│         except CalledProcessError as e:
    →  948│             raise EnvCommandError(e, input=input_)
       949│
       950│         return decode(output)
       951│
       952│     def execute(self, bin, *args, **kwargs):


Failed to add packages, reverting the pyproject.toml file to its original content.

abn · 2020-10-20T10:29:11Z

@ajacquah the issue here is that the isolated build environment required to build balck from sdist is not able to be setup as it cannot download setuptools.

Poetry when building wheels from sdist for dependencies (not the project itself) will handover build to pip via pip install /path/to/sdist. This will in turn trigger a PEP 517 isolated build within pip. During this process, it needs to get the dependencies required for the build. This step is handled by pip and not by poetry in this case.

As for why the sdist is being used, you will have to check if proxy has the wheels available. If not, that could explain why an sdist is downloaded and used.

Eventually we would want to consume this part of the pipe too, and eventually setup the build environment ourselves. Once that happens this should, and a lot of other annoyances, go away.

ghost · 2020-10-20T14:52:05Z

@abn - Thanks for explaining, that makes sense. And glad to hear this part will be consumed by poetry in the future.

I believe black and possibly other projects only makes sdist releases. I will look into this and confirm.

But given that we have the option to set a private repository as the default one, shouldn't this repo url be set via the --index-url for pip when handing over the build? I think that would address this issue in the meantime, although I'm not sure how it would impact other parts of poetry or other package install scenarios.

abn · 2020-10-20T15:26:46Z

At initial glance, I reckon your proposal is reasonable and should not impact other aspects as only assist and path cases require the isolated build environment.

Can you maybe verify that doing so manually works in your environment?

ghost · 2020-10-20T17:04:25Z

Thanks @abn . Yes I can verify this works:

C:\Users\ajacquah>pip install black --index-url https://artifactory.com/artifactory/api/pypi/prod-pypi/simple --force-reinstall --no-cache
Looking in indexes: https://artifactory.com/artifactory/api/pypi/prod-pypi/simple
Collecting black
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/black/20.8b1/black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 1.7 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting click>=7.1.2
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/click/7.1.2/click-7.1.2-py2.py3-none-any.whl (82 kB)
     |████████████████████████████████| 82 kB 5.8 MB/s
Collecting typed-ast>=1.4.0
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/typed-ast/1.4.1/typed_ast-1.4.1-cp38-cp38-win_amd64.whl (1
58 kB)
     |████████████████████████████████| 158 kB 2.2 MB/s
Collecting typing-extensions>=3.7.4
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/typing-extensions/3.7.4.3/typing_extensions-3.7.4.3-py3-no
ne-any.whl (22 kB)
Collecting toml>=0.10.1
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/toml/0.10.1/toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting mypy-extensions>=0.4.3
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/mypy-extensions/0.4.3/mypy_extensions-0.4.3-py2.py3-none-a
ny.whl (4.5 kB)
Collecting appdirs
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/appdirs/1.4.4/appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting pathspec<1,>=0.6
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/pathspec/0.8.0/pathspec-0.8.0-py2.py3-none-any.whl (28 kB)

Collecting regex>=2020.1.8
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/regex/2020.9.27/regex-2020.9.27-cp38-cp38-win_amd64.whl (2
64 kB)
     |████████████████████████████████| 264 kB 6.4 MB/s
Building wheels for collected packages: black
  Building wheel for black (PEP 517) ... done
  Created wheel for black: filename=black-20.8b1-py3-none-any.whl size=124193 sha256=d407d2a24086842fd09d3899c5f5b39a000b7fec210843d3c01dc874cc92f5ea
  Stored in directory: C:\Users\ajacquah\AppData\Local\Temp\pip-ephem-wheel-cache-f96ou92x\wheels\fb\85\63\b3594c26bfd8e8475cf00e6c8eee29851a9ce9243a71486baa

Successfully built black
Installing collected packages: click, typed-ast, typing-extensions, toml, mypy-extensions, appdirs, pathspec, regex, black
  Attempting uninstall: click
    Found existing installation: click 7.1.2
    Uninstalling click-7.1.2:
      Successfully uninstalled click-7.1.2
  Attempting uninstall: typed-ast
    Found existing installation: typed-ast 1.4.1
    Uninstalling typed-ast-1.4.1:
      Successfully uninstalled typed-ast-1.4.1
  Attempting uninstall: typing-extensions
    Found existing installation: typing-extensions 3.7.4.3
    Uninstalling typing-extensions-3.7.4.3:
      Successfully uninstalled typing-extensions-3.7.4.3
  Attempting uninstall: toml
    Found existing installation: toml 0.10.1
    Uninstalling toml-0.10.1:
      Successfully uninstalled toml-0.10.1
  Attempting uninstall: mypy-extensions
    Found existing installation: mypy-extensions 0.4.3
    Uninstalling mypy-extensions-0.4.3:
      Successfully uninstalled mypy-extensions-0.4.3
  Attempting uninstall: appdirs
    Found existing installation: appdirs 1.4.4
    Uninstalling appdirs-1.4.4:
      Successfully uninstalled appdirs-1.4.4
  Attempting uninstall: pathspec
    Found existing installation: pathspec 0.8.0
    Uninstalling pathspec-0.8.0:
      Successfully uninstalled pathspec-0.8.0
  Attempting uninstall: regex
    Found existing installation: regex 2020.9.27
    Uninstalling regex-2020.9.27:
      Successfully uninstalled regex-2020.9.27
  Attempting uninstall: black
    Found existing installation: black 20.8b1
    Uninstalling black-20.8b1:
      Successfully uninstalled black-20.8b1
Successfully installed appdirs-1.4.4 black-20.8b1 click-7.1.2 mypy-extensions-0.4.3 pathspec-0.8.0 regex-2020.9.27 toml-0.10.1 typed-ast-1.4.1 typing-extensi
ons-3.7.4.3

C:\Users\ajacquah>

qiuwei · 2020-10-27T03:20:03Z

It seems this is already fixed in 1.1.4 release.

ghost · 2020-10-27T20:54:58Z

@qiuwei - checked and this is still an issue in 1.1.4 release.

Woklex · 2020-12-31T14:34:45Z

Faced a similar problem, but this problem is not present on version 1.0.10.

aja-intel · 2021-03-29T20:55:53Z

hello @abn and maintainers. Any further thoughts on this? I've seen a few suggestion for work-arounds on related issues but it would be great to see this working for the configured default package index.

Liz4v · 2021-08-10T21:45:14Z

I haven't found an effective workaround; plus I'm having the opposite problem: it's failing to find pypi.org dependencies in my custom secondary repository. I'm not sure if I should open a separate bug for it.

mickeykkim · 2022-10-07T09:09:05Z

Is there any update for this? I'm trying to incorporate Poetry into my company's CI workflows but this is a major issue. I can workaround it by generating requirements.txt files with poetry export and using pip to install but this is far from ideal.

neersighted · 2022-10-07T14:24:31Z

As you can see, this has been triaged. Nobody has worked on implementing anything for five reasons:

it's a rather niche issue with lots of sharp edges and implementing index-passing to pip is going to be a bit tricky to get the semantics right on
we're in the middle of a refactor of source handling and will have to figure out how to map the new semantics to pip
it's unclear if this is even correct as Poetry supports source priorities and pip does not -- unexpected packages may be pulled in
passing authentication information to pip is also fraught, both code-wise and implementation-wise
we're working on a new installer instead of using pip, that will let Poetry provide the build-system.requires directly and sidestep this entirely

If you need to set PIP_INDEX_URL, you should set in /etc/pip.conf, which represents system-wide configuration and usually will not include any options that can cause problems for Poetry (though of course, you can specify any option there; please keep it minimal!).

kolibri91 · 2023-01-04T17:29:14Z

If you need to set PIP_INDEX_URL, you should set in /etc/pip.conf, which represents system-wide configuration and usually will not include any options that can cause problems for Poetry (though of course, you can specify any option there; please keep it minimal!).

How could I handle this in closed networks where PIP_INDEX_URL could contain credentials for private repositories? I'm evaluating poetry to be used instead of pipenv. Your resolver is great and I was pretty impressed by its performance compared to pipenv but this limitation makes it almost possible to be used in closed networks.

michaelishri · 2023-02-13T00:26:54Z

The issue we're experiencing which are the same as #7472 and #7483 (but possibly different to this thread) is because of the -isolated switch introduced in v1.3.0 under #6531. When we revert back to v1.2.2, everything works fine.

The -isolated switch ignores our pip.ini file which is configured to use our internal repository (Artifactory) so tries to reach out to pypi (which we don't want). Being behind a corporate proxy, we are required to use Artifactory to fetch our dependencies.

When we try on a Poetry version newer than v.1.2.2 we get 407 Proxy Authentication Required (since it tries to fetch from the internet via our web proxy).

I can confirm uninstalling the newer version and installing v1.2.2 will workaround the issue, this will at least get your Python devs back online.

pip uninstall poetry
pip install poetry==v1.2.2

My suggestion would be to put the -isolated switch behind a configuration flag and default to using the user's configured pip config (as it did previously), I don't feel like I'm experienced enough in Python, let alone this project to be able to contribute the fix.

neersighted · 2023-02-15T00:42:47Z

Again, this is by design. The user pip configuration may do benign things like configure your intercepting/inspecting proxy's certificates, or may fundamentally break Poetry. Over the last year, as Poetry has become more of a standalone tool and less of a pip wrapper, the former has happened much more than the latter.

The solution here is to provide these credentials to pip through /etc/pip.conf, which is much more 'global' and hopefully will not need to contain any configuration for 'pip as a dev tool.'

Ultimately the tension here is between pip-as-a-dev-tool and pip-as-Python-infrastructure; our (soon to be realized) solution to this is to further divorce ourselves from usage of pip so that Poetry configuration can be the single source of truth.

damienrj · 2023-02-15T01:03:57Z

We look forward to the long term solution, and writing to the global environment in our CI does seem to resolve the issue. It might be worthwhile calling this out in the docs though since it does seem to come up at times. I could make a PR if that would be appropriate.

michaelishri · 2023-02-21T03:33:37Z

Thanks for your response @neersighted. I like that direction and understand removing the dependence on pip will give Poetry more flexibility and the ability to shape it's own future. In this case, I'm not entirely sure how to resolve this issue and am happy to keep on v1.2.2 until Poetry matures further however I'm also happy to submit a PR to add an option flag to allow the isolated option to be disabled for users that need to disable it.

Would the PR be considered if submitted?

neersighted · 2023-02-21T03:35:55Z

Such a PR would not be considered; 1.4 is imminent, the choice is deliberate and /etc/pip.conf is a better place to put this information on all but multi-user systems, which are vanishingly rare.

damienrj · 2023-02-21T03:58:40Z

Does 1.4 remove the dependency on pip?

wagnerluis1982 · 2023-02-21T06:23:02Z

@neersighted when you say /etc/pip.conf you mean the pip configuration file in a generic way?

I mean, poetry respects the pip.conf loading order accordingly to what is described at https://pip.pypa.io/en/stable/topics/configuration/?

neersighted · 2023-02-21T06:25:33Z

1.4 partially removes the dependency on a pip in a way which makes this issue obsolete.

Poetry doesn't do anything special WRT pip, we just set --isolated. You can consult pip's docs to understand what config is and is not loaded using --isolated (hint: environment variables and user-level config are ignored, which leaves two types of config: site and global).

yuval9313 · 2023-02-21T23:50:26Z

I am not sure all the comments around this issue are the same
I do understand the certs issue and why this causes an issue, but personally the issue I have is how poetry.source is not actually a source, and this comes to mind when I try to develop in close network, or, in my case, with actual private repository

I get it. This is "by design" and --isolated fixes voodoo issues, but understand me, I need sub-dep from a private repository, which unfortunately has the same name as pypi.org (I know, this is not only bad practice but stupid, but i simply cant do much about it).

Now, when I want to install this random private package that is dependent on that other random-bad-named package, it would install the one from pypi.org. completely voiding the poetry.source and the statement that poetry supports private repositories

A solution could be forwarding the sources into the pip install command.

neersighted · 2023-02-22T00:00:07Z

There is no pip install command once 1.4 is released in this scenario; now that I think about it I need to go and look if we make the source for a given package available to the PEP 517 build backend fetching process (or what semantics even make sense there); however, the point stands that the way to solve this is by replacing the pip dependency instead of trying to bridge Poetry-abstractions to pip-abstractions with a side of interference-from-configuration-of-pip-as-a-end-user-tool.

radoering · 2023-02-22T05:43:12Z

now that I think about it I need to go and look if we make the source for a given package available to the PEP 517 build backend fetching process

IIRC, we will use the sources from pyproject.toml of the root project. Relevant lines of code:

poetry/src/poetry/installation/chef.py

Line 116 in c8945eb

env = IsolatedEnv(venv, self._config)

poetry/src/poetry/installation/chef.py

Line 75 in c8945eb

pool = Factory.create_pool(self._config)

qkponton · 2023-02-23T14:11:31Z

Hi guys
Can you communicate on an expected release date for 1.4
I have the same issue in my new company where I can't update the global pip config (/etc/pip.conf yeah don't laugh)
Need to consider my options quickly about this please

qkponton · 2023-02-24T12:48:09Z

just for information even setting PIP_INDEX_URL as root doesn't fix the issue for non-root user on 1.3.2

idhyt · 2023-02-27T05:47:36Z

same issue in 1.3.2, run install command from EnvCommandError log

root@1c15287fd90a:/develop# '/develop/.venv/bin/python'  '-m'  'pip'  'install'  '--use-pep517'  '--disable-pip-version-check'  '--isolated'  '--no-input'  '--prefix'  '/develop/.venv'  '--no-deps'  '/root/.cache/pypoetry/artifacts/45/a5/fb/6083820494451489aab8af4f89f9a7ee683bac6b5213cd7f9e3119671b/tinyaes-1.0.4.tar.gz'
Processing /root/.cache/pypoetry/artifacts/45/a5/fb/6083820494451489aab8af4f89f9a7ee683bac6b5213cd7f9e3119671b/tinyaes-1.0.4.tar.gz
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 2
  ╰─> [86 lines of output]
      Collecting cython
        WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0xffffbbeca350>, 'Connection to files.pythonhosted.org timed out. (connect timeout=15)')': /packages/be/f9/198c7b9d125b29bcedaf923f5f2c29bd4b4534d4a8dd2ee5780244337af4/Cython-0.29.33-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl
        WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0xffffbbeca650>, 'Connection to files.pythonhosted.org timed out. (connect timeout=15)')': /packages/be/f9/198c7b9d125b29bcedaf923f5f2c29bd4b4534d4a8dd2ee5780244337af4/Cython-0.29.33-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl
        Downloading Cython-0.29.33-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl (1.8 MB)
                                                    0.0/1.8 MB ? eta -:--:--
      ERROR: Exception:
      Traceback (most recent call last):
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 438, in _error_catcher
          yield
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 561, in read
          data = self._fp_read(amt) if not fp_closed else b""
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 527, in _fp_read
          return self._fp.read(amt) if amt is not None else self._fp.read()
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 90, in read
          data = self.__fp.read(amt)
        File "/usr/local/lib/python3.10/http/client.py", line 465, in read
          s = self.fp.read(amt)
        File "/usr/local/lib/python3.10/socket.py", line 705, in readinto
          return self._sock.recv_into(b)
        File "/usr/local/lib/python3.10/ssl.py", line 1274, in recv_into
          return self.read(nbytes, buffer)
        File "/usr/local/lib/python3.10/ssl.py", line 1130, in read
          return self._sslobj.read(len, buffer)
      TimeoutError: The read operation timed out

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
          status = run_func(*args)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
          return func(self, options, args)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 415, in run
          requirement_set = resolver.resolve(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
          result = self._result = resolver.resolve(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
          state = resolution.resolve(requirements, max_rounds=max_rounds)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
          self._add_to_criteria(self.state.criteria, r, parent=None)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
          if not criterion.candidates:
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
          return bool(self._sequence)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
          return any(self)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
          return (c for c in iterator if id(c) not in self._incompatible_ids)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
          candidate = func()
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
          self._link_candidate_cache[link] = LinkCandidate(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 297, in __init__
          super().__init__(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 162, in __init__
          self.dist = self._prepare()
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 231, in _prepare
          dist = self._prepare_distribution()
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 308, in _prepare_distribution
          return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 491, in prepare_linked_requirement
          return self._prepare_linked_requirement(req, parallel_builds)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 536, in _prepare_linked_requirement
          local_file = unpack_url(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 166, in unpack_url
          file = get_http_url(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 107, in get_http_url
          from_path, content_type = download(link, temp_dir.path)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/network/download.py", line 147, in __call__
          for chunk in chunks:
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/cli/progress_bars.py", line 53, in _rich_progress_bar
          for chunk in iterable:
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/network/utils.py", line 63, in response_chunks
          for chunk in response.raw.stream(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 622, in stream
          data = self.read(amt=amt, decode_content=decode_content)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 560, in read
          with self._error_catcher():
        File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
          self.gen.throw(typ, value, traceback)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 443, in _error_catcher
          raise ReadTimeoutError(self._pool, None, "Read timed out.")
      pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

      [notice] A new release of pip is available: 23.0 -> 23.0.1
      [notice] To update, run: python -m pip install --upgrade pip
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

but, i run command again with -i, success

root@1c15287fd90a:/develop# '/develop/.venv/bin/python'  '-m'  'pip'  'install'  '--use-pep517'  '--disable-pip-version-check'  '--isolated'  '--no-input'  '--prefix'  '/develop/.venv'  '--no-deps'  '/root/.cache/pypoetry/artifacts/45/a5/fb/6083820494451489aab8af4f89f9a7ee683bac6b5213cd7f9e3119671b/tinyaes-1.0.4.tar.gz' -i https://pypi.tuna.tsinghua.edu.cn/simple
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Processing /root/.cache/pypoetry/artifacts/45/a5/fb/6083820494451489aab8af4f89f9a7ee683bac6b5213cd7f9e3119671b/tinyaes-1.0.4.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: tinyaes
  Building wheel for tinyaes (pyproject.toml) ... done
  Created wheel for tinyaes: filename=tinyaes-1.0.4-cp310-cp310-linux_aarch64.whl size=69242 sha256=441eed9d11bee252d488affb3e555e1d42ff066b3cd6541c730ef156353b1f63
  Stored in directory: /root/.cache/pip/wheels/61/f2/1f/f107e37fb4cfde950ba8696a16ab4bb6425f37e673e872940e
Successfully built tinyaes
Installing collected packages: tinyaes
Successfully installed tinyaes-1.0.4

YAGregor · 2023-02-27T12:25:30Z

I have the same issue but solved by change global pip config /etc/pip.conf as neersighted said

pip config set global.index-url <index-url>
cp ~/.config/pip/pip.conf /etc/

qkponton · 2023-03-01T08:17:01Z

I have the same issue but solved by change global pip config /etc/pip.conf as neersighted said
pip config set global.index-url <index-url>
cp ~/.config/pip/pip.conf /etc/

As root or with non root user?

dimbleby · 2023-03-01T19:36:40Z

poetry 1.4 is out, I guess this should be closed

YAGregor · 2023-03-02T06:27:23Z

I have the same issue but solved by change global pip config /etc/pip.conf as neersighted said
pip config set global.index-url <index-url>
cp ~/.config/pip/pip.conf /etc/
As root or with non root user?

root.this will change default pip index for every user no matter whether is it root. @qkponton

qkponton · 2023-03-02T08:34:10Z

1.4 release solved the issue for me

qkponton · 2023-03-02T08:34:56Z

I have the same issue but solved by change global pip config /etc/pip.conf as neersighted said
pip config set global.index-url <index-url>
cp ~/.config/pip/pip.conf /etc/
As root or with non root user?
root.this will change default pip index for every user no matter whether is it root. @qkponton

@YAGregor yeah the issue was with a non root user :)

michaelishri · 2023-03-02T22:11:24Z

I can confirm that this issue appears to be resolved for our case. Poetry appears to now be respecting the Artifactory instance we've defined in our pyproject.toml file.

For anyone that needs it, this is how we configured Artifactory in our pyproject.toml file.

[[tool.poetry.source]]
name = "artifactory"
url = "https://<your-artifactory-instance>/artifactory/api/pypi/<your-repository>/simple"
default = true

Interestingly, it appears that our internal certificates are not an issue, even without defining our PEM file in the config. I'm running on Windows so not sure if Poetry is using the certs in the Windows cert store or just ignoring SSL validation all together?

Secrus · 2023-04-18T11:00:40Z

Closing, as the issue seems to be fixed. If you encounter this in new versions of Poetry, please open new issue.

github-actions · 2024-02-29T15:05:37Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

ghost added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Oct 19, 2020

benjaminlee314 mentioned this issue Oct 20, 2020

Custom repository doesn't have precedence over PyPI #3238

Closed

3 tasks

finswimmer added the area/repo Meta-issues for the repository/forge itself label Oct 30, 2020

sinoroc mentioned this issue Dec 11, 2020

[[tool.poetry.source]] setting is ignored #3474

Closed

1 task

aja-intel mentioned this issue Mar 29, 2021

Poetry doesn`t use provided private repo when compiling dependencies from source #3744

Closed

3 tasks

neersighted changed the title ~~Poetry installs build dependencies from pypi.org instead of custom default repository~~ Provide index to pip for installation of sdist build dependencies Oct 5, 2022

neersighted mentioned this issue Oct 11, 2022

Allow root CA bundle to be configured #1012

Open

2 tasks

moltob mentioned this issue Dec 12, 2022

poetry install partially broken after introducing --isolated #7182

Closed

This was referenced Dec 29, 2022

1.3 Poetry Install: Isolated PIP build dependency installation fails when behind proxy #7265

Closed

How can I configure poetry to pass --no-cache-dir to pip so that it won't get an OOM signal #7250

Closed

neersighted mentioned this issue Feb 7, 2023

Poetry install fails due to the new pip --isolated change which ignores private repos set by PIP_INDEX_URL #7483

Closed

4 tasks

damienrj mentioned this issue Feb 15, 2023

Poetry 1.3.1 Pypi Certs #7513

Closed

3 tasks

qkponton mentioned this issue Feb 23, 2023

Poetry tries to remove setuptools and then complains it can't find it #4242

Closed

3 tasks

dimbleby mentioned this issue Feb 23, 2023

poetry can't be used without internet access, because it runs pip with --isolated #7551

Closed

4 tasks

dimbleby mentioned this issue Mar 7, 2023

SolveFailure with experimental installer and editable dependency #7622

Closed

4 tasks

Secrus closed this as completed Apr 18, 2023

github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024

Provide index to pip for installation of sdist build dependencies #3249

Provide index to pip for installation of sdist build dependencies #3249

Comments

ghost commented Oct 19, 2020 • edited by ghost Loading

Issue

abn commented Oct 19, 2020

ghost commented Oct 20, 2020

abn commented Oct 20, 2020 • edited Loading

ghost commented Oct 20, 2020

abn commented Oct 20, 2020

ghost commented Oct 20, 2020

qiuwei commented Oct 27, 2020

ghost commented Oct 27, 2020

Woklex commented Dec 31, 2020

aja-intel commented Mar 29, 2021

Liz4v commented Aug 10, 2021

mickeykkim commented Oct 7, 2022

neersighted commented Oct 7, 2022 • edited Loading

kolibri91 commented Jan 4, 2023

michaelishri commented Feb 13, 2023

neersighted commented Feb 15, 2023 • edited Loading

damienrj commented Feb 15, 2023

michaelishri commented Feb 21, 2023

neersighted commented Feb 21, 2023

damienrj commented Feb 21, 2023

wagnerluis1982 commented Feb 21, 2023 • edited Loading

neersighted commented Feb 21, 2023

yuval9313 commented Feb 21, 2023

neersighted commented Feb 22, 2023

radoering commented Feb 22, 2023

qkponton commented Feb 23, 2023 • edited Loading

qkponton commented Feb 24, 2023

idhyt commented Feb 27, 2023

YAGregor commented Feb 27, 2023 • edited Loading

qkponton commented Mar 1, 2023

dimbleby commented Mar 1, 2023

YAGregor commented Mar 2, 2023 • edited Loading

qkponton commented Mar 2, 2023

qkponton commented Mar 2, 2023

michaelishri commented Mar 2, 2023

Secrus commented Apr 18, 2023

github-actions bot commented Feb 29, 2024

ghost commented Oct 19, 2020 •

edited by ghost

Loading

abn commented Oct 20, 2020 •

edited

Loading

neersighted commented Oct 7, 2022 •

edited

Loading

neersighted commented Feb 15, 2023 •

edited

Loading

wagnerluis1982 commented Feb 21, 2023 •

edited

Loading

qkponton commented Feb 23, 2023 •

edited

Loading

YAGregor commented Feb 27, 2023 •

edited

Loading

YAGregor commented Mar 2, 2023 •

edited

Loading