poetry insists on intalling packages from private repository

[HenriqueAJNB]

• Poetry version: 1.4.2
• Python version: 3.9, 3.10, 3.11
• OS version and name: Windows 10
• pyproject.toml:

[tool.poetry]
name = "my-project"
version = "0.0.1"
description = ""
authors = ["..."]
readme = "README.md"

[tool.poetry.dependencies]
python = ">=3.9,<3.11"
google-cloud-bigquery = ">=1.15.0"
pandas = "1.4.3"
db-dtypes = "^1.1.1"
loguru = "^0.7.0"


[tool.poetry.group.dev.dependencies]
ipykernel = "^6.22.0"
mkdocs = "^1.4.2"
black = "^22.12.0"
pre-commit = "^3.2.2"


[tool.poetry.group.ml.dependencies]
kfp = "^1.8.20"
private-package = {version = "...", source="private-source"}
google-cloud-aiplatform = "^1.15.1"

[[tool.poetry.source]]
name = "private-source"
url = "https://region/project-id/private-package/simple/" # url for private package from Google Cloud
default = false
secondary = true

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.black]
line-length = 100

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

There is also the poetry.toml file:

[virtualenvs]
create = true
in-project = true

[repositories.private-source]
url = "https://region/project-id/private-package/simple/"

Issue

Since it's a private package from Google Cloud, I've installed the libraries for authentication:

poetry self add keyring keyrings.google-artifactregistry-auth

And I've authenticate with both gcloud auth login and gcloud auth application-default login

When I install the depencencies:

$ poetry install
Updating dependencies
Resolving dependencies... (my-private-package): Authorization error accessing https:project-id/my-private-package/simple/google-cloud-aiplatform/
Resolving dependencies... (my-private-package): Authorization error accessing https:project-id/my-private-package/simple/kfp/
Resolving dependencies... (my-private-package): Authorization error accessing https:project-id/my-private-package/simple/pre-commit/
Resolving dependencies... (my-private-package): Authorization error accessing https:project-id/my-private-package/simple/black/
Resolving dependencies... (my-private-package): Authorization error accessing https/project-id/my-private-package/simple/mkdocs/
Resolving dependencies... (my-private-package): Authorization error accessing https/project-id/my-private-package/simple/ipykernel/
Resolving dependencies... (my-private-package): Authorization error accessing https/project-id/my-private-package/simple/loguru/
Resolving dependencies... (my-private-package): Authorization error accessing https/project-id/my-private-package/simple/db-dtypes/
Resolving dependencies... (my-private-package): Authorization error accessing https/project-id/my-private-package/simple/pandas/
Resolving dependencies... (my-private-package): Authorization error accessing https/project-id/my-private-package/simple/google-cloud-bigquery/


Because my-project depends on my-private-package which doesn't match any versions, version solving failed.

The poetry install command insists searching packages at private repository.

The documentation is clear about the behaviour: https://python-poetry.org/docs/repositories/#project-configuration

Any package source not marked as secondary will take precedence over PyPI.

If you prefer to disable PyPI completely, you may choose to set one of your package sources to be the default.

If you prefer to specify a package source for a specific dependency, see Secondary Package Sources.

If you do not want any of the custom sources to take precedence over PyPI, you must declare all package sources to be secondary.

And although all private sources (there is only one) in my project are defined as secondary secondary = true and explicitly as not default with default = false (see below), poetry still used it to search for packages.

[[tool.poetry.source]]
name = "private-source"
url = "https://region/project-id/private-package/simple/" # url for private package from Google Cloud
default = false
secondary = true

[dimbleby]

this is expected: secondary sources do not take precedence over pypi, but they are still searched. Poetry is not "insisting on installing packages from private repository": if it were able to find a suitable version of my-private-package then you would see the public packages being installed from pypi.

Please read the documentation

All package sources (including secondary sources) will be searched during the package lookup process. These network requests will occur for all sources, regardless of if the package is found at one or more sources.

Please close.

[HenriqueAJNB]

But why is it searching for public packages like pandas, numpy, loguru and all others public dependencies only in private source? Shouldn't it install from PyPi if they were not found in private source?

https/project-id/my-private-package/simple/pandas. Here it is trying to read pandas from private source, and it's not even trying to get it from PyPi. This is not the expected behaviour, since the private source is set as non-default and secondary.

[dimbleby]

what makes you believe that it is searching for these packages only in the private repository and not even trying pypi? You have given no evidence to support this.

[jakobkogler]

@dimbleby Would it not make sense to limit the private source to only the packages where it is specified? Or at least provide an option to do so.

E.g. even if I mark just one single dependency in my pyproject.toml with source = "private-source", I still get hundreds of calls to the private pypi even though I know that there is nothing there except the one single package.

[dimbleby]

@jakobkogler see #6713

this issue should be closed, the reporter has simply guessed wrong about what poetry is doing

[HenriqueAJNB]

@dimbleby, wouldn't this https/project-id/my-private-package/simple/pandas/ be enough to infer that poetry is looking for the pandas package in the private repo?

Thus, the failure message Because my-project depends on my-private-package which doesn't match any versions, version solving failed. confirms that poetry is undoubtedly not looking at PyPI because it hasn't installed any other packages from it.

This is clearly indicating that panda, loguru, google-cloud-bigquery and all other PyPI dependencies don't exist on the private package. And this behaviour shouldn't be like this.

[dimbleby]

wouldn't this https/project-id/my-private-package/simple/pandas/ be enough to infer that poetry is looking for the pandas package in the private repo?

Yes, poetry is looking for pandas in the private repo. But what you are claiming is that poetry is not looking on pypi: and that message does not have that implication at all.

What Because my-project depends on my-private-package which doesn't match any versions, version solving failed. means is that poetry cannot find my-private-package.

This also implies nothing about whether poetry could or could not find pandas or where it looked for pandas: it means only what it says. poetry cannot find my-private-package.

Please close

[ralbertazzi]

Closing as decided above. Plus, Poetry 1.5 now (supplemental/explicit sources) allows to solve the initial issue.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.