Skip to content

Commit

Permalink
Update the resolution concept documentation (#5813)
Browse files Browse the repository at this point in the history
  • Loading branch information
@zanieb
zanieb authored Aug 6, 2024
1 parent e651e67 commit 83d6f59
Showing 1 changed file with 159 additions and 78 deletions.
237 changes: 159 additions & 78 deletions docs/concepts/resolution.md
View file
Original file line number Diff line number Diff line change
@@ -1,50 +1,126 @@
# Resolution

Dependency resolution is the process of taking your requirements and converting them to a list of
package versions that fulfil your requirements and the requirements of all included packages.
Resolution is the process of taking a list of requirements and converting them to a list of package
versions that fulfill the requirements. Resolution requires recursively searching for compatible
versions of packages, ensuring that the requested requirements are fulfilled and that the
requirements of the requested packages are compatible.

## Overview
## Dependencies

Imagine you have the following dependency tree:
Most projects and packages have dependencies. Dependencies are other packages that are needed in
order for the current package to work. A package defines its dependencies as _requirements_, roughly
a combination of a package name and acceptable versions. The dependencies defined by the current
project are called _direct dependencies_. The requirements added by each dependency of the current
project are called _indirect_ or _transitive dependencies_.

- Your project depends on `foo>=1,<3` and `bar>=1,<3`.
- `foo` has two versions, 1.0.0 and 2.0.0. `foo` 2.0.0 depends on `lib==2.0.0`, `foo` 1.0.0 has no
dependencies.
- `bar` has two versions, 1.0.0 and 2.0.0. `bar` 2.0.0 depends on `lib==1.0.0`, `bar` 1.0.0 has no
dependencies.
!!! note

See the [dependency specifiers
page](https://packaging.python.org/en/latest/specifications/dependency-specifiers/)
in the Python Packaging documentation for details about dependencies.

## Basic examples

To help demonstrate the resolution process, consider the following dependencies:

<!-- prettier-ignore -->
- The project depends on `foo` and `bar`.
- `foo` has one version, 1.0.0:
- `foo 1.0.0` depends on `lib>=1.0.0`.
- `bar` has one version, 1.0.0:
- `bar 1.0.0` depends on `lib>=2.0.0`.
- `lib` has two versions, 1.0.0 and 2.0.0. Both versions have no dependencies.

In this example, the resolver must find a set of package versions which satisfies the project
requirements. Since there is only one version of both `foo` and `bar`, those will be used. The
resolution must also include the transitive dependencies, so a version of `lib` must be chosen.
`foo 1.0.0` allows all of the available versions of `lib`, but `bar 1.0.0` requires `lib>=2.0.0` so
`lib 2.0.0` must be used.

In some resolutions, there is more than one solution. Consider the following dependencies:

<!-- prettier-ignore -->
- The project depends on `foo` and `bar`.
- `foo` has two versions, 1.0.0 and 2.0.0:
- `foo 1.0.0` has no dependencies.
- `foo 2.0.0` depends on `lib==2.0.0`.
- `bar` has two versions, 1.0.0 and 2.0.0:
- `bar 1.0.0` has no dependencies.
- `bar 2.0.0` depends on `lib==1.0.0`
- `lib` has two versions, 1.0.0 and 2.0.0. Both versions have no dependencies.

We can't install both `foo` 2.0.0 and `bar` 2.0.0 because they conflict on the version of `lib`, so
the resolver will pick either `foo` 1.0.0 or `bar` 1.0.0. Both are valid solutions, at the resolvers
choice.
In this example, some version of both `foo` and `bar` must be picked, however, determining which
version requires considering the dependencies of each version of `foo` and `bar`. `foo 2.0.0` and
`bar 2.0.0` cannot be installed together because they conflict on their required version of `lib`,
so the resolver must select either `foo 1.0.0` or `bar 1.0.0`. Both are valid solutions, and
different resolution algorithms may give either result.

## Platform markers

Markers allow attaching an expression to requirements that indicate when the dependency should be
used. For example `bar; python_version<"3.9"` can be used to only require `bar` on Python 3.8 and
older.

Markers are used to adjust a package's dependencies dependending on the current environment or
platform. For example, markers can be used to change dependencies based on the operating system, the
CPU architecture, the Python version, the Python implementation, and more.

!!! note

See the [environment
markers](https://packaging.python.org/en/latest/specifications/dependency-specifiers/#environment-markers)
section in the Python Packaging documentation for more details about markers.

## Platform-specific and universal resolution
Markers are important for resolution because their values change the required dependencies.
Typically, Python package resolvers use the markers of the _current_ platform to determine which
dependencies to use since the package is often being _installed_ on the current platform. However,
for _locking_ dependencies this is problematic — the lockfile would only work for developers using
the same platform the lockfile was created on. To solve this problem, platform-independent, or
"universal" resolvers exist.

uv supports two modes of resolution: Platform-specific and universal (platform-independent).
uv supports both [platform-specific](#platform-specific-resolution) and
[universal](#universal-resolution) resolution.

Like `pip` and `pip-tools`, `uv pip compile` produces a resolution that's only known to be
compatible with the current operating system, architecture, Python version and Python interpreter.
`uv pip compile --universal` and the [project](../guides/projects.md) interface on the other hand
will solve to a host-agnostic universal resolution that can be used across platforms.
## Universal resolution

For universal resolution, you need to configure the minimum required python version. For
`uv pip compile --universal`, you can pass `--python-version`, otherwise the current Python version
uv's lockfile (`uv.lock`) is created with a universal resolution and is portable across platforms.
This ensures that dependencies are locked for everyone working on the project, regardless of
operating system, architecture, and Python version. The uv lockfile is created and modified by
[project](../concepts/projects.md) commands such as `uv lock`, `uv sync`, and `uv add`.

universal resolution is also available in uv's pip interface, i.e.,
[`uv pip compile`](../pip/compile.md), with the `--universal` flag. The resulting requirements file
will contain markers to indicate which platform each dependency is relevant for.

During universal resolution, a package may be listed multiple times with different versions or URLs
if different versions are needed for different platforms — the markers determine which version will
be used. A universal resolution is often more constrained than a platform-specific resolution, since
we need to take the requirements for all markers into account.

During universal resolution, a minimum Python version must be specified. Project commands read the
minimum required version from `project.requires-python` in the `pyproject.toml`. When using the pip
interface, provide a value with the `--python-version` option, otherwise the current Python version
will be treated as a lower bound. For example, `--universal --python-version 3.9` writes a universal
resolution for Python 3.9 and later. Project commands such as `uv sync` or `uv lock` read
`project.requires-python` from your `pyproject.toml`.
resolution for Python 3.9 and later.

Setting the minimum Python version is important because all package versions we select have to be
compatible with the python range. For example, a universal resolution of `numpy<2` with
compatible with the Python version range. For example, a universal resolution of `numpy<2` with
`--python-version 3.8` resolves to `numpy==1.24.4`, while `--python-version 3.9` resolves to
`numpy==1.26.4`, as `numpy` releases after 1.26.4 require at least Python 3.9. Note that we only
consider the lower bound of any Python requirement.
`numpy==1.26.4`, as `numpy` releases after 1.26.4 require at Python 3.9+. Note that we only consider
the lower bound of any Python requirement, upper bounds are always ignored.

## Platform-specific resolution

In platform-specific mode, the `uv pip` interface also supports resolving for specific alternate
platforms and Python versions with `--python-platform` and `--python-version`. For example, if
you're running Python 3.12 on macOS, but want to resolve for Linux with Python 3.10, you can run
`uv pip compile --python-platform linux --python-version 3.10 requirements.in` to produce a
`manylinux2014`-compatible resolution. In this mode, `--python-version` is the exact python version
to use, not a lower bound.
By default, uv's pip interface, i.e., [`uv pip compile`](../pip/compile.md), produces a resolution
that is platform-specific, like `pip-tools`. There is no way to use platform-specific resolution in
the uv's project interface.

uv also supports resolving for specific, alternate platforms and Python versions with the
`--python-platform` and `--python-version` options. For example, if using Python 3.12 on macOS,
`uv pip compile --python-platform linux --python-version 3.10 requirements.in` can be used to
produce a resolution for Python 3.10 on Linux instead. Unlike universal resolution, during
platform-specific resolution, the provided `--python-version` is the exact python version to use,
not a lower bound.

!!! note

Expand All @@ -55,34 +131,28 @@ to use, not a lower bound.
compatible with any machine running on the target `--python-platform`, which should be sufficient for
most use cases, but may lose fidelity for complex package and platform combinations.

In universal mode, a package may be listed multiple times with different versions or URLs. In this
case, uv determined that we need different versions to be compatible different platforms, and the
markers decides on which platform we use which version. A universal resolution is often more
constrained than a platform-specific resolution, since we need to take the requirements for all
markers into account.
## Dependency preferences

If an output file is used with `uv pip` or `uv.lock` exist with the project commands, we try to
resolve to the versions present there, considering them preferences in the resolution. The same
applies to version already installed to the active virtual environments. You can override this with
`--upgrade`.
If resolution output file exists, i.e. a uv lockfile (`uv.lock`) or a requirements output file
(`requirements.txt`), uv will _prefer_ the dependency versions listed there. Similarly, if
installing a package into a virtual environment, uv will prefer the already installed version if
present. This means that locked or installed versions will not change unless an incompatible version
is requested or an upgrade is explicitly requested with `--upgrade`.

## Resolution strategy

By default, uv tries to use the latest version of each package. For example,
`uv pip install flask>=2.0.0` will install the latest version of Flask (at time of writing:
`3.0.0`). If you have `flask>=2.0.0` as a dependency of your library, you will only test `flask`
3.0.0 this way, but not if you are actually still compatible with `flask` 2.0.0.
`uv pip install flask>=2.0.0` will install the latest version of Flask, e.g., 3.0.0. If
`flask>=2.0.0` is a dependency of the project, only `flask` 3.0.0 will be used. This is important,
for example, because running tests will not check that the the project is actually compatible with
its stated lower bound of `flask` 2.0.0.

With `--resolution lowest`, uv will install the lowest possible version for all dependencies, both
direct and indirect (transitive). Alternatively, `--resolution lowest-direct` will opt for the
lowest compatible versions for all direct dependencies, while using the latest compatible versions
for all other dependencies. uv will always use the latest versions for build dependencies.
direct and indirect (transitive). Alternatively, `--resolution lowest-direct` will use the lowest
compatible versions for all direct dependencies, while using the latest compatible versions for all
other dependencies. uv will always use the latest versions for build dependencies.

For libraries, we recommend separately running tests with `--resolution lowest` or
`--resolution lowest-direct` in continuous integration to ensure compatibility with the declared
lower bounds.

As an example, given the following `requirements.in` file:
For example, given the following `requirements.in` file:

```text title="requirements.in"
flask>=2.0.0
Expand Down Expand Up @@ -128,6 +198,10 @@ werkzeug==2.0.0
# via flask
```

When publishing libraries, it is recommended to separately run tests with `--resolution lowest` or
`--resolution lowest-direct` in continuous integration to ensure compatibility with the declared
lower bounds.

## Pre-release handling

By default, uv will accept pre-release versions during dependency resolution in two cases:
Expand All @@ -136,46 +210,53 @@ By default, uv will accept pre-release versions during dependency resolution in
(e.g., `flask>=2.0.0rc1`).
1. If _all_ published versions of a package are pre-releases.

If dependency resolution fails due to a transitive pre-release, uv will prompt the user to re-run
with `--prerelease allow`, to allow pre-releases for all dependencies.
If dependency resolution fails due to a transitive pre-release, uv will prompt use of
`--prerelease allow` to allow pre-releases for all dependencies.

Alternatively, you can add the transitive dependency to your `requirements.in` file with a
pre-release specifier (e.g., `flask>=2.0.0rc1`) to opt in to pre-release support for that specific
dependency.
Alternatively, the transitive dependency can be added as a [constraint](#dependency-constraints) or
direct dependency (i.e. in `requirements.in` or `pyproject.toml`) with a pre-release version
specifier (e.g., `flask>=2.0.0rc1`) to opt-in to pre-release support for that specific dependency.

Pre-releases are
[notoriously difficult](https://pubgrub-rs-guide.netlify.app/limitations/prerelease_versions) to
model, and are a frequent source of bugs in other packaging tools. uv's pre-release handling is
_intentionally_ limited and _intentionally_ requires user opt-in for pre-releases, to ensure
correctness.
_intentionally_ limited and requires user opt-in for pre-releases to ensure correctness.

For more details, see
[Pre-release compatibility](../pip/compatibility.md#pre-release-compatibility).

## Dependency constraints

For more, see [Pre-release compatibility](../pip/compatibility.md#pre-release-compatibility).
uv supports constraints files (`--constraint constraints.txt`), like pip, which narrow the set of
acceptable versions for the given packages. Constraint files are like a regular requirements files,
but they do not add packages to the requirements — they only take affect if the package is requested
in a direct or transitive dependency. Constraints are often useful for reducing the range of
available versions for a transitive dependency without adding a direct requirement on the package.

## Constraints
## Dependency overrides

Like `pip`, uv supports constraints files (`--constraint constraints.txt`), which allows users to
narrow the set of acceptable versions for a given package. A constraint files is like a regular
requirements files, but it doesn't add packages, it only constrains their version range when they
are depended on by a regular requirement.
Sometimes, the requirements defined by a dependency are too strict, and a working version of a
package is not allowed (often, causing resolution to fail). uv allows overriding requirements
defined by dependencies to unblock resolution.

## Overrides
For example, if a transitive dependency declares the requirement `pydantic>=1.0,<2.0`, but _works_
with `pydantic>=2.0`, the user can override the declared dependency with `pydantic>=1.0,<3` to allow
the resolver to installer a newer version of `pydantic`.

Sometimes, the requirements in one of your (transitive) dependencies are too strict, and you want to
install a version of a package that you know to work, but wouldn't be allowed regularly. Overrides
allow you to lie to the resolver, replacing all other requirements for that package with the
override. They break the usual rules of version resolving and should only be used as last resort
measure.
Constraints and direct dependency declarations will _not_ result in the same behavior as an
override, as they can only add _additional_ constraints to the allowed version instead of
_replacing_ existing constraints. As with constraints, overrides do not _add_ a dependency on the
package and only take affect if the package is requested in a direct or transitive dependency.

For example, if a transitive dependency declares `pydantic>=1.0,<2.0`, but the user knows that the
package is compatible with `pydantic>=2.0`, the user can override the declared dependency with
`pydantic>=2.0,<3` to allow the resolver to continue.
In a `pyproject.toml`, use `tool.uv.override-dependencies` to define a list of overrides. In the
pip-compatible interface, the `--override` option can be used to pass files with the same format as
constraints files.

Overrides are passed to `uv pip` as `--override` with an overrides file with the same syntax as
requirements or constraints files. In `pyproject.toml`, you can set `tool.uv.override-dependencies`
to a list of requirements. If you provide multiple overrides for the same package, we apply them
simultaneously, while markers are applied as usual.
If multiple overrides are provided for the same package, they must be differentiated with
[markers](#platform-markers). If a package has a dependency with a marker, it is replaced
unconditionally when using overrides — it does not matter if the marker evaluates to true or false.

## Time-restricted reproducible resolutions
## Reproducible resolutions

uv supports an `--exclude-newer` option to limit resolution to distributions published before a
specific date, allowing reproduction of installations regardless of new package releases. The date
Expand Down

0 comments on commit 83d6f59

Please sign in to comment.