Package left un-upgraded with pip install --upgrade

[aphedges]

Description

When upgrading both datasets and fsspec in an environment, only datasets was upgraded to the latest version even though there was nothing constraining fsspec from being updated. fsspec is a dependency of datasets (fsspec[http]>=2021.05.0 is specified in its setup.py), and from looking at the resolver's debug output, it seems that pip is preferring to use the pre-installed version of fsspec instead of the newer one.

The documentation for --upgrade says, "Upgrade all specified packages to the newest available version." Unless I'm misunderstanding, all packages explicitly listed on the command line should have been updated if possible.

Expected behavior

fsspec should be upgraded to 2021.10.1.

pip version

21.3.1

Python version

3.7.12

OS

macOS 10.15.7

How to Reproduce

1. Create a file named pip_fsspec_test.sh with the following contents:

#!/usr/bin/env bash

set -euxo pipefail

# Prepare environment
python --version
pip install -U pip setuptools wheel
pip list

# Install outdated packages
pip install datasets==1.13.3 fsspec==2021.10.0
pip list --outdated

# Attempt to upgrade both packages
pip install -U 'datasets<=1.14.0' 'fsspec<=2021.10.1'
pip list --outdated

2. Run bash pip_fsspec_test.sh &> pip_fsspec_log.txt in a new environment to create pip_fsspec_log.txt.
3. Change the penultimate line of the script to PIP_RESOLVER_DEBUG=1 pip -vvv install -U 'datasets<=1.14.0' 'fsspec<=2021.10.1'.
4. Run bash pip_fsspec_test.sh &> pip_fsspec_verbose_log.txt in a new environment to create pip_fsspec_verbose_log.txt.

Output

See files under "How to Reproduce".

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

That said, looking at the report more closely, we're preferring the installed version of a package if it's specified with an extra, as a dependency of another package -- even when that dependency is explicitly specified on the CLI. That's... suboptimal.

Does --upgrade-strategy=eager do what you want?

[uranusjr]

Yeah it seems like we need the split("[") treatment in

pip/src/pip/_internal/resolution/resolvelib/provider.py

Lines 174 to 189 in b160e83

	def _eligible_for_upgrade(name: str) -> bool:
	"""Are upgrades allowed for this project?
	This checks the upgrade strategy, and whether the project was one
	that the user specified in the command line, in order to decide
	whether we should upgrade if there's a newer version available.
	(Note that we don't need access to the `--upgrade` flag, because
	an upgrade strategy of "to-satisfy-only" means that `--upgrade`
	was not specified).
	"""
	if self._upgrade_strategy == "eager":
	return True
	elif self._upgrade_strategy == "only-if-needed":
	return name in self._user_requested
	return False

It's quite interesting this bug took so long to surface.

[aphedges]

@pradyunsg, I can confirm that your workaround (--upgrade-strategy=eager) works.

@uranusjr, thanks for creating a fix! I can confirm your PR also solves the problem for me.