Deprecate pip install --editable calling setup.py develop

[sbidoul]

There is now a standardized mechanism for an installer like pip to request an editable install of a project.

pip is transitioning to using this standard only instead of invoking the deprecated setup.py develop command.

What's being deprecated?

TL;DR: the execution of setup.py develop when pip does an editable install. The -e / --editable pip install option is not deprecated.

There is a standardized mechanism (called PEP 660) for an installers to request an editable install of a project. The benefit of this modern mechanism is that it allows projects packaged using other tools (formally called "build backends") such as Hatch and Poetry to support editable installs.

Since pip 21.3, pip has switched to using this mechanism where possible. Now the pip project seeks to deprecate and eventually remove the old and setuptools-specific fallback to running setup.py develop when a pyproject.toml is not present or the PEP 660 mechanism is unsupported for some reason.

Some historical context if you're interested

• Historically, there was no standard way for installers like pip to request an editable install of a project. Thus, the -e / --editable option was implemented as a wrapper over the setup.py develop command. Using this method, only setuptools projects could use editable installs. PEP 660 defines an interface which build backends can implement in order to support editable installs.

• If you are still interested and want more details, they are available on @ichard26's blog post on the changes in pip 24.2. You should read this issue first though!

How will this affect my project?

TL;DR: if your project does not support or function properly using the modern mechanism for editable installs, pip install -e is liable to stop working starting with pip 25.0 (Q1 2025).

If you have received a deprecation warning about a legacy editable install, pip is using the legacy setup.py develop method for performing an editable installation of your project. In practice, this usually means one of two things:

• The pyproject.toml file doesn’t exist at all, thus pip will opt-out of the modern mechanism and use setup.py develop
• The declared setuptools version (in the [build-system].requires field) is too old and doesn’t support PEP 660, i.e. anything older than setuptools 64.0.0

The current plan is that support for legacy editable installs will be entirely removed in pip 25.0 (Q1 2025). If your package doesn't not support or function correctly under an editable install using the modern mechanism, pip install -e will break starting with pip 25.0.

What should I do?

There are a couple of choices:

• Add a pyproject.toml to your project, making sure the [build-system] section requires setuptools >= 64, as well as other libraries required to build your project (the equivalent of setuptools' setup_requires parameter). A basic example is included below:

  [build-system]
  # XXX: If your project needs other packages to build properly, add them to this list.
  requires = ["setuptools >= 64"]
  build-backend = "setuptools.build_meta"

• Alternatively, enable the --use-pep517 pip option, possibly with --no-build-isolation. The --use-pip517 flag will force pip to use the modern mechanism for editable installs. --no-build-isolation may be needed if your project has build-time requirements beyond setuptools and wheel. By passing this flag, you are responsible for making sure your environment already has the required dependencies to build your package. Once the legacy mechanism is removed, --use-pep517 will have no effect and will essentially be enabled by default in this context.

Tip

If the resulting installation does not behave correctly, you may want to try the --config-setting editable_mode=compat pip option. Please consult the setuptools documentation for more information.

Tip

Which solution you choose to use depends on the project, but option one is recommended as there are other benefits to adding a pyproject.toml file. For example, you can declare your project metadata in pyproject.toml. If your project already works fine using the modern mechanism, option two is an acceptable way to silence the deprecation warning in the meanwhile.

[pradyunsg]

This is complicated by the fact that numpy and everything that uses numpy.distutils as a build-dependency cannot be built with setuptools >= 60. setuptools v64.0.0 was the first version to add a PEP 660 implementation.

https://numpy.org/doc/stable/reference/distutils_status_migration.html
https://setuptools.pypa.io/en/latest/history.html#v64-0-0

[pradyunsg]

Adding a cross-reference to the idea of dropping support for this, starting Python 3.12+ (circa #8102 (comment)).

This also makes sense since we'd stop install setuptools by default in ensurepip starting with that Python version, and get-pip.py as well, I imagine.

[edmorley]

Is it worth adding a deprecation message for the setup.py develop fallback in the next Pip release?

[edmorley]

Is here the correct place for the deprecation warning?

pip/src/pip/_internal/req/req_install.py

Lines 829 to 842 in 2a0acb5

	if self.editable and not self.is_wheel:
	install_editable_legacy(
	global_options=global_options if global_options is not None else [],
	prefix=prefix,
	home=home,
	use_user_site=use_user_site,
	name=self.req.name,
	setup_py_path=self.setup_py_path,
	isolated=self.isolated,
	build_env=self.build_env,
	unpacked_source_directory=self.unpacked_source_directory,
	)
	self.install_succeeded = True
	return

[uranusjr]

I think so, either this or inside install_editable_legacy

[sbidoul]

Is it worth adding a deprecation message for the setup.py develop fallback in the next Pip release?

@edmorley I personally think so yes.

A question I have is if the deprecation message should mention that --config-setting editable_mode=compat exists for maximum compatibility.

[edmorley]

I've opened #12830 to add the deprecation warning for the setup.py develop fallback.

[sneakers-the-rat]

Nice work yall ♥

[uranusjr]

I’ll do the edit.

[timmc-edx]

Does this mean that pip install -e on older packages will start failing for pip 25+? (Where "older" here means "hasn't had any of the above changes/workarounds applied".) I'm thinking of cases where packages can be upgraded, but people nevertheless need to install older versions (for compatibility, bisection-style debugging, etc.)

[ichard26]

@timmc-edx The key phrase is that things will break if your package is already broken when the modern editable install mechanism is used. The breakage possible with pip 25 is that the automatic fallback that keeps those packages working will be removed.

if your project does not support or function properly using the modern mechanism for editable installs, pip install -e is liable to stop working starting with pip 25.0 (Q1 2025).

I guess a valid third "solution" is simply to do nothing. If the package functions correctly under the modern editable install mechanism (you can verify that by forcing the modern mechanism via --use-pep517), then great. When the legacy mechanism is entirely removed in pip 25, the warning will go away and the modern mechanism will be used instead, transparently. Is this something we should recommend? No, I don't believe so. Projects should really start using the modern packaging standards, including pyproject.toml, as they do offer a better experience (especially if you want to use alternative tools, like uv)¹. The main reason to do nothing would be with legacy, read-only projects or archives as you've mentioned.

In essence, "it depends" is the answer. If you can, the pip team strongly recommends migrating to the supported and standardized solutions, but doing nothing can be a valid path forward. Also, older versions of pip can be used when needed.

Footnotes

1. Of course, the transition is inevitably a bit painful and there are issues with gaps in the standards, but the standards are here to stay. ↩

[potiuk]

Comment from a side. I am really sorry for it to be a long message, I just wanted to provide a bit more context. So for those who would like to get the gist of it here is TL;DR;.

TL;DR; while I would love to see the legacy support removed in 25.0. I think it would be worth to think about one more escalation step that will allow package developers to use "user pressure" to make sure their dependencies are also migrated to modern standards - namely to change current warnings into error and allow the user to "opt-in" to use the old mechanism for some time.

I am not sure - maybe it is already planned - after reading very carefully the entirely thread I think we are going to go straight from "warning" to "remove support" - but I could have missed it or misunderstand it, so wanted to raise it here.

I think this would be the "right escalation path" - for example change to opt-in in 24.3 and complete removal in 25.0 (or maybe 25.1 or 25.2 or even do it with 25.0 (opt-in) followed by 26.0 (removal) scenario.

Similar approaches were taken in the past in pip (--use-legacy-resolver) and it's been a preetty successful (even if at times painful) process.

Why I think it would be good to follow this path?

Because there is only as much package developer can do as to convert their own package to use modern standards (for example we did it in Airflow with apache/airflow#36537 for example. But when you have a lot of dependencies, convincing them and their dependencies and their dependencies to modernize is a long process and unless there is a really strong incentive, you have very limited impact on that.

I think in many cases 'supply chain' issues are "people's/maintainers" bound not "tooling bound" and the more we enable the right people to communicate and teach each other - the better. For example push-down the efforts to modernize things where higher-layer dependencies will fill incentive and need to reach out to the 'lower layer" dependencies in dependency tree, the more effective such transition will be.

The problem with creating thie incentive is that currently the "warnings" are mostly for "end-users". While maintainers can fix their own project setup, they are rarely aware that any of their dependencies (including transitive ones) has such problems. Mainly because in most cases the pre-packaged .whl files are available, there is no need to build the packages when you install even 100s of your dependencies - especially if you are using modern distros. And you will not even be aware how many and which of your dependencies need to be reached out to (and maybe even helped with) to fix it. Because often times users who experience that (for example using some old distros or requiring for some reasons older versions of those packages) just plainly ignore the warnings and you are not even aware of it as package maintainer.

So, what will it change if we turn the warnings into errors and allow the user to "opt-in" to remove them.

IMHO - a lot. What will happen in most cases, the users's installation wil break, they will see the error message that will suggest them to raise an issue to the package that is problematic, or switch to the "opt-in" - but that one could clearly say - "this will stop working on that date or version of pip". In many cases those users will not really reach out to the problematic packages, but they will open issue with the package they tried to install in the first place (say "apache-airflow" - we have 700+ dependencies in total and it's very, very likely some of them will fail to install, people will open an issue for Airflow. And this is precisely what we want to have. We want to have a notification that some of our users have problems with some of our dependencies (and we want to have those engaged users who want to get the problem solved), so that we can do some actions there - as "airflow" maintainers. And we will have quite a few options then:

• We can help the user telling them to use the "opt-in" flag. This is what we will not have if "opt-in" legacy behaviour is not available. With the right warning, this will have clear expiry date and will allow us to follow up with the "dependent" packages.

• We can tell them to use older pip version. This is something we would like to avoid, because this one has no "clear expiry date" (use it for as long as you want ??? or what?).

• We can also reach out (or ask the user to do so) to reach out to the "offending" dependency. Having clear expiry date and having an "end-user" who complains is as strong of a incentive and "pressure" you can put on the dependency

• We can also offer the dependency to help them to migrate and tell them how, open a PR with it, etc. etc. This is especially if we see that they need help. This is something I am currently doing as a new project (together with @sethmlarson and Alpha-Omega) where we will eventually analyse and if needs be reach out to ALL dependencies of Airflow in order to improve their processes and security status. This is rather new and experimental project which we call "Airflow Beach Cleaning" where we attempt to see if improving the whole "supply chain" security of a significant/important part of the ecosystem is possible this way (I am going to talk about it in two days at the Airlfow Summit Keynote in San Francisco - together with Michael Winser from Alpha-Omega - https://airflowsummit.org/sessions/2024/security-united-collaborative-effort-on-securing-airflow-ecosystem-with-alpha-omega-psf-asf/. I think this might be one of the most efficient way where we can "trickle-down" the modernizing efforts - by packages having an incentive and even helping their dependencies, and their dependencies helping their dependencies and so on... I hope in 6-12 months we will have result of the experiment and we will be able to share some learnings and blueprints.

• Finally - the last option we will have - if the "offending" packages will be unresponsive/unwilling to even get help - this will give us (higher-level package maintainers) - time to search for alternatives. By giving the user the option to use the "opt-in" flag and a clear "expiry date" - we will have a pressure on us, to solve the problem and either replace or vendor-in the dependencies that will be problematic for our users.

So, summarising - I think it would be great to consider one step escalation process (error out with opt-in to enable old behaviour back) as this will give a chance for this kind of "trickle-down" effort to be less disruptive.

[edmorley]

@potiuk What you are saying makes sense for wider-reaching changes like the historic switch from the legacy resolver, however, I don't think it's as applicable here. Since:

1. This deprecation is only for packages installed in editable mode (which is typically only the top level package, not the hundreds of transitive dependencies case mentioned above)
2. As far as I understand it, the ~removal will most likely be to instead to fall back to the isolated build environment mode, which will automatically install newer setuptools (via the fallback build backend, used for packages that don't have a build backend declared in pyproject.toml, since they haven't migrated from setup.py yet) which already supports PEP-660, and therefore for most cases I would imagine these legacy packages would continue to just work - even if they haven't been updated to use pyproject.toml.

In general, I would like us to enable build isolation by default soon (whilst still allowing a way to opt out, like uv does) - I think in fact that change doesn't need to wait for this one to be complete. I've just added a comment about that here:
#9175 (comment)

[ichard26]

I can't edit the post, but just as a friendly reminder to all, the setup.py file itself is not deprecated. Even in 2025, you'll be able to use it to configure setuptools. What's changing that is that pip is not going to special case for setup.py while performing an editable install (i.e. pip won't be calling the CLI interface provided by setup.py such as setup.py develop). Going forward, pip is going to exclusively use the standardized mechanism for editable installs (PEP 660) which pip and setuptools already support and use.

What's tricky here is that pip defaults to using the legacy setup.py develop mechanism when the pyproject.toml is not present. If your package is pure Python and follows the src/ layout, the modern mechanism for editable installs will likely "just work" without any major issues. However, pip is issuing a deprecation warning any time the legacy mechanism is used in an effort to warn you that your project may be affected.

You can keep your setup.py file if you wish. We strongly recommend (at the bare minimum) declaring your package's build backend in pyproject.toml as described in the issue description, but it is not mandatory. If no build backend is declared, pip will assume you're using setuptools, and as long as the modern editable install mechanism works fine with your package, pip install -e will continue to work. So, old or legacy projects should have a high likelihood of working, but please, add a pyproject.toml if you can because it makes all of our lives easier :)

[potiuk]

@potiuk What you are saying makes sense for wider-reaching changes like the historic switch from the legacy resolver, however, I don't think it's as applicable here. Since:

What you wrote makes perfect sense. Thanks for explaining in detail. Yeah - what you describe makes perfect sense and my concerns are addressed. Going with straight removal should be fine.