Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

--extra-index-url failing when package not in --index-url (polygraphy 0.47.1) #1600

Closed
kafonek opened this issue Feb 17, 2024 · 2 comments · Fixed by #2083
Closed

--extra-index-url failing when package not in --index-url (polygraphy 0.47.1) #1600

kafonek opened this issue Feb 17, 2024 · 2 comments · Fixed by #2083
Labels
bug Something isn't working duplicate This issue or pull request already exists registry Related to package indexes and registries

Comments

@kafonek
Copy link

kafonek commented Feb 17, 2024

polygraphy==0.47.1 is present on the nvidia package index but not regular PyPI. On uv 0.1.3, I can uv pip install polygraphy==0.47.1 --index-url https://pypi.ngc.nvidia.com but not uv pip install polygraphy==0.47.1 --extra-index-url https://pypi.ngc.nvidia.com.

I realize this is very similar to #1377 but the discussion in that ticket seemed to be more about priority of indices to install from than an absence of checking the extra index url.

Verbose output below -

❯ uv pip install polygraphy==0.47.1 --extra-index-url https://pypi.ngc.nvidia.com -v
 uv::requirements::from_source source=polygraphy==0.47.1
    0.006585s DEBUG uv_interpreter::virtual_env Found a virtualenv named .venv at: /home/kafonek/scratch/.venv
    0.006640s DEBUG uv_interpreter::interpreter Detecting markers for: /home/kafonek/scratch/.venv/bin/python
    0.032748s DEBUG uv::commands::pip_install Using Python 3.10.12 environment at /home/kafonek/scratch/.venv/bin/python
 uv_client::flat_index::from_entries
 uv_resolver::resolver::solve
      0.039637s   0ms DEBUG uv_resolver::resolver Solving with target Python version 3.10.12
   uv_resolver::resolver::choose_version package=root
   uv_resolver::resolver::get_dependencies package=root, version=0a0.dev0
        0.039680s   0ms DEBUG uv_resolver::resolver Adding direct dependency: polygraphy==0.47.1
   uv_resolver::resolver::choose_version package=polygraphy
     uv_resolver::resolver::package_wait package_name=polygraphy
 uv_resolver::resolver::process_request request=Versions polygraphy
   uv_client::registry_client::simple_api package=polygraphy
     uv_client::cached_client::get_cacheable
       uv_client::cached_client::read_and_parse_cache file=/home/kafonek/.cache/uv/simple-v0/pypi/polygraphy.rkyv
 uv_resolver::resolver::process_request request=Prefetch polygraphy ==0.47.1
          0.045662s   5ms  WARN uv_client::cached_client Broken cache entry at /home/kafonek/.cache/uv/simple-v0/pypi/polygraphy.rkyv, removing: failed to open file `/home/kafonek/.cache/uv/simple-v0/pypi/polygraphy.rkyv`
          0.045932s   6ms DEBUG uv_client::cached_client No cache entry for: https://pypi.org/simple/polygraphy/
       uv_client::cached_client::fresh_request url="https://pypi.org/simple/polygraphy/"
       uv_client::cached_client::new_cache file=/home/kafonek/.cache/uv/simple-v0/pypi/polygraphy.rkyv
       uv_client::registry_client::parse_simple_api package=polygraphy
 uv_resolver::version_map::from_metadata
        0.093293s  53ms DEBUG uv_resolver::resolver Searching for a compatible version of polygraphy (==0.47.1)
      0.093341s  53ms DEBUG uv_resolver::resolver No compatible version found for: polygraphy
  × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of polygraphy==0.47.1 and you require polygraphy==0.47.1, we can conclude that the requirements are unsatisfiable.
@zanieb zanieb added bug Something isn't working registry Related to package indexes and registries labels Feb 17, 2024
@exs-avianello
Copy link

I think that this is the same as observed in #1451

@kafonek
Copy link
Author

kafonek commented Feb 17, 2024

Ah, I glanced at your ticket and ducked out after seeing the timeout. I think you're right that it's the same thing. At least this is another example to test with.

@kafonek kafonek closed this as completed Feb 17, 2024
@kafonek kafonek mentioned this issue Feb 17, 2024
Closed
@zanieb zanieb added the duplicate This issue or pull request already exists label Feb 18, 2024
@zanieb zanieb closed this as not planned Won't fix, can't repro, duplicate, stale Feb 18, 2024
@BurntSushi BurntSushi mentioned this issue Feb 29, 2024
Open
BurntSushi added a commit that referenced this issue Feb 29, 2024
@BurntSushi
distribution-types: flip the ordering of index preference
31faf28 
Previously, we would prioritize `--index-url` over all
`--extra-index-url` values. But now, we prioritize all
`--extra-index-url` values over `--index-url`. That is, `--index-url`
has gone from the "primary" index to the "fallback" index. In most
setups, `--index-url` is left as its default value, which is PyPI.

The ordering of `--extra-index-url` with respect to one another remains
the same. That is, in `--extra-index-url foo --extra-index-url bar`,
`foo` will be tried before `bar`.

Finally, note that this specifically does not match `pip`'s behavior.
`pip` will attempt to look at versions of a package from all indexes in
which in occurs. `uv` will stop looking for versions of a package once
it finds it in an index. That is, for any given package, `uv` will only
utilize versions of it from a single index.

Ref #171, Fixes #1377, Fixes #1451, Fixes #1600
BurntSushi added a commit that referenced this issue Feb 29, 2024
@BurntSushi
distribution-types: flip the ordering of index preference
f622625 
Previously, we would prioritize `--index-url` over all
`--extra-index-url` values. But now, we prioritize all
`--extra-index-url` values over `--index-url`. That is, `--index-url`
has gone from the "primary" index to the "fallback" index. In most
setups, `--index-url` is left as its default value, which is PyPI.

The ordering of `--extra-index-url` with respect to one another remains
the same. That is, in `--extra-index-url foo --extra-index-url bar`,
`foo` will be tried before `bar`.

Finally, note that this specifically does not match `pip`'s behavior.
`pip` will attempt to look at versions of a package from all indexes in
which in occurs. `uv` will stop looking for versions of a package once
it finds it in an index. That is, for any given package, `uv` will only
utilize versions of it from a single index.

Ref #171, Fixes #1377, Fixes #1451, Fixes #1600
@BurntSushi BurntSushi mentioned this issue Feb 29, 2024
Merged
BurntSushi added a commit that referenced this issue Feb 29, 2024
@BurntSushi
distribution-types: flip the ordering of index preference
dd82e71 
Previously, we would prioritize `--index-url` over all
`--extra-index-url` values. But now, we prioritize all
`--extra-index-url` values over `--index-url`. That is, `--index-url`
has gone from the "primary" index to the "fallback" index. In most
setups, `--index-url` is left as its default value, which is PyPI.

The ordering of `--extra-index-url` with respect to one another remains
the same. That is, in `--extra-index-url foo --extra-index-url bar`,
`foo` will be tried before `bar`.

Finally, note that this specifically does not match `pip`'s behavior.
`pip` will attempt to look at versions of a package from all indexes in
which in occurs. `uv` will stop looking for versions of a package once
it finds it in an index. That is, for any given package, `uv` will only
utilize versions of it from a single index.

Ref #171, Fixes #1377, Fixes #1451, Fixes #1600
BurntSushi added a commit that referenced this issue Feb 29, 2024
@BurntSushi
tweak the order of index priority (#2083)
5e35134 
Previously, `uv` would always prioritize the index given by
`--index-url`. It would then try any indexes after that given by zero
or more `--extra-index-url` flags. This differed from `pip` in that any
priority was given at all, where `pip` doesn't guarantee any priority
ordering of indexes.

We could go in the direction of mimicing `pip`'s behavior here, but it
at present has issues with dependency confusion attacks where packages
may get installed from indexes you don't control. More specifically,
there is an issue of different trust levels. See discussion in #171 and
[PEP-0708] for more on the security impact.

In contrast, `uv` will only select versions for a package from a single
index. That is, even if `foo` is in indexes `a` and `b`, it will
only consider the versions from the index that it checks first. This
probably helps with respect to dependency confusion attacks, but also
means that `uv` doesn't quite cover all of the same use cases as `pip`.

In this PR, we retain the notion of prioritizing indexes, but
tweak it so that PyPI is preferred last as opposed to first. Or
more precisely, the `--index-url` flag specifies a fallback index,
not the primary index, and is deprioritized beneath every index
specified by `--extra-index-url`. The ordering among indexes given by
`--extra-index-url` remains the same: earlier indexes are prioritized
over later indexes.

While this tweak likely won't hit all use cases, I believe it will
resolve some of the most common pain points without exacerbating
dependency confusion problems.

Ref #171, Fixes #1377, Fixes #1451, Fixes #1600

[PEP-0708]: https://peps.python.org/pep-0708/
This was referenced Mar 4, 2024
Closed
Closed
@charliermarsh charliermarsh mentioned this issue Mar 19, 2024
Closed
@zanieb zanieb mentioned this issue Mar 28, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working duplicate This issue or pull request already exists registry Related to package indexes and registries
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

tweak the order of index priority astral-sh/uv
3 participants
@zanieb @kafonek @exs-avianello