Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement protections for externally managed Python environments (PEP 668) #11381

Closed
1 task done
pradyunsg opened this issue Aug 15, 2022 · 7 comments · Fixed by #11663
Closed
1 task done

Implement protections for externally managed Python environments (PEP 668) #11381

pradyunsg opened this issue Aug 15, 2022 · 7 comments · Fixed by #11663
Labels
PEP implementation Involves some PEP type: feature request Request for a new feature

Comments

@pradyunsg
Copy link
Member

What's the problem this feature will solve?

See https://peps.python.org/pep-0668/#motivation for the details.

Describe the solution you'd like

Implement the protections, as described in https://peps.python.org/pep-0668/#specification

Alternative Solutions

N/A

Additional context

See https://peps.python.org/pep-0668/. If there's broader issues here, add to the discussion on https://discuss.python.org/t/10302/

Code of Conduct

@astrojuanlu
Copy link
Contributor

Does this mean that, say, pip won't try to touch conda or apt installed packages? Or are there more changes needed in apt and conda?

@pfmoore
Copy link
Member

pfmoore commented Jan 18, 2023

It means that conda and apt can configure the system environment to tell pip not to touch it. See the linked PEP for details, but basically:

  1. It's at the level of environment, not package. If conda says their environment's site-packages is externally managed, you won't be able to use pip to install anything there.
  2. The system integrator needs to opt into it.

@pradyunsg
Copy link
Member Author

conda/conda#12245 is relevant.

@jezdez
Copy link
Member

jezdez commented Jan 20, 2023

Edit: moved my comment about conda to the issue @pradyunsg created there.

@hroncok
Copy link
Contributor

hroncok commented Jan 21, 2023

Would it make sense to move this discussion to the discussion page for the PEP itself rather than this implementation issue in pip?

@pradyunsg
Copy link
Member Author

I think it would. :)

@stefanor
Copy link
Contributor

stefanor commented Jan 21, 2023

Thank you for implementing this. The implementation seems to do the right thing in my testing :)

I assume this will break people's workflows, and so they'll delete the EXTERNALLY-MANAGED files. But at least they know what they're getting into.

And we can drop our most invasive pip patch in Debian, once this is released :)

The next Python upload to Debian unstable will declare this.
But I suspect we won't manage to ship a pip that supports it in time for the next Debian stable release. So, only early-adopters who install their own pip will see it, initially.

inmantaci pushed a commit to inmanta/inmanta-core that referenced this issue Jan 31, 2023
Bumps [pip](https://github.com/pypa/pip) from 22.3.1 to 23.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p>
<blockquote>
<h1>23.0 (2023-01-30)</h1>
<h2>Features</h2>
<ul>
<li>Change the hashes in the installation report to be a mapping. Emit the
<code>archive_info.hashes</code> dictionary in <code>direct_url.json</code>. (<code>[#11312](pypa/pip#11312) &lt;https://github.com/pypa/pip/issues/11312&gt;</code>_)</li>
<li>Implement logic to read the <code>EXTERNALLY-MANAGED</code> file as specified in PEP 668.
This allows a downstream Python distributor to prevent users from using pip to
modify the externally managed environment. (<code>[#11381](pypa/pip#11381) &lt;https://github.com/pypa/pip/issues/11381&gt;</code>_)</li>
<li>Enable the use of <code>keyring</code> found on <code>PATH</code>. This allows <code>keyring</code>
installed using <code>pipx</code> to be used by <code>pip</code>. (<code>[#11589](pypa/pip#11589) &lt;https://github.com/pypa/pip/issues/11589&gt;</code>_)</li>
<li>The inspect and installation report formats are now declared stabled, and their version
has been bumped from <code>0</code> to <code>1</code>. (<code>[#11757](pypa/pip#11757) &lt;https://github.com/pypa/pip/issues/11757&gt;</code>_)</li>
</ul>
<h2>Bug Fixes</h2>
<ul>
<li>Wheel cache behavior is restored to match previous versions, allowing the
cache to find existing entries. (<code>[#11527](pypa/pip#11527) &lt;https://github.com/pypa/pip/issues/11527&gt;</code>_)</li>
<li>Use the &quot;venv&quot; scheme if available to obtain prefixed lib paths. (<code>[#11598](pypa/pip#11598) &lt;https://github.com/pypa/pip/issues/11598&gt;</code>_)</li>
<li>Deprecated a historical ambiguity in how <code>egg</code> fragments in URL-style
requirements are formatted and handled. <code>egg</code> fragments that do not look
like PEP 508 names now produce a deprecation warning. (<code>[#11617](pypa/pip#11617) &lt;https://github.com/pypa/pip/issues/11617&gt;</code>_)</li>
<li>Fix scripts path in isolated build environment on Debian. (<code>[#11623](pypa/pip#11623) &lt;https://github.com/pypa/pip/issues/11623&gt;</code>_)</li>
<li>Make <code>pip show</code> show the editable location if package is editable (<code>[#11638](pypa/pip#11638) &lt;https://github.com/pypa/pip/issues/11638&gt;</code>_)</li>
<li>Stop checking that <code>wheel</code> is present when <code>build-system.requires</code>
is provided without <code>build-system.build-backend</code> as <code>setuptools</code>
(which we still check for) will inject it anyway. (<code>[#11673](pypa/pip#11673) &lt;https://github.com/pypa/pip/issues/11673&gt;</code>_)</li>
<li>Fix an issue when an already existing in-memory distribution would cause
exceptions in <code>pip install</code> (<code>[#11704](pypa/pip#11704) &lt;https://github.com/pypa/pip/issues/11704&gt;</code>_)</li>
</ul>
<h2>Vendored Libraries</h2>
<ul>
<li>Upgrade certifi to 2022.12.7</li>
<li>Upgrade chardet to 5.1.0</li>
<li>Upgrade colorama to 0.4.6</li>
<li>Upgrade distro to 1.8.0</li>
<li>Remove pep517 from vendored packages</li>
<li>Upgrade platformdirs to 2.6.2</li>
<li>Add pyproject-hooks 1.0.0</li>
<li>Upgrade requests to 2.28.2</li>
<li>Upgrade rich to 12.6.0</li>
<li>Upgrade urllib3 to 1.26.14</li>
</ul>
<h2>Improved Documentation</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/pypa/pip/commit/368c7b4c557e673b05b0f8cffc967d3e333eee19"><code>368c7b4</code></a> Bump for release</li>
<li><a href="https://github.com/pypa/pip/commit/aa94ccadb45d6ee44defea8a82bd5b647ccba799"><code>aa94cca</code></a> Update AUTHORS.txt</li>
<li><a href="https://github.com/pypa/pip/commit/60ce5c0943c303e48f0aed8bce650f725dcd222d"><code>60ce5c0</code></a> Fix the kind of news fragment</li>
<li><a href="https://github.com/pypa/pip/commit/e3e7bc34eb486622ebbb6412afc98ee57fcbff4a"><code>e3e7bc3</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11766">#11766</a> from uranusjr/upgrade-pre-commit-isort</li>
<li><a href="https://github.com/pypa/pip/commit/b653b129c56b29ad565886c1f423de89639d20f3"><code>b653b12</code></a> Bump pre-commit isort to 5.12.0</li>
<li><a href="https://github.com/pypa/pip/commit/a2a4feb588edc7233ae262d76b2c7291d6857a31"><code>a2a4feb</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11761">#11761</a> from sbidoul/direct-url-hashes-part-3-sbi</li>
<li><a href="https://github.com/pypa/pip/commit/ec7eb6f179866151f148c7695fc773e66b8c3adc"><code>ec7eb6f</code></a> Add version history to inspect and install report docs</li>
<li><a href="https://github.com/pypa/pip/commit/169511e68eb64efff5705305f72b0c53d7bff580"><code>169511e</code></a> Update direct URL hashes examples</li>
<li><a href="https://github.com/pypa/pip/commit/efedf09c4967dcbe3105e3746aaca7bfb55d605f"><code>efedf09</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11759">#11759</a> from pradyunsg/fix-keyring-auth</li>
<li><a href="https://github.com/pypa/pip/commit/60a45984404460192067f3990e0258deeeafa636"><code>60a4598</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11758">#11758</a> from pradyunsg/vendoring-update</li>
<li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/22.3.1...23.0">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=22.3.1&new-version=23.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

</details>
blattersturm added a commit to citizenfx/fivem that referenced this issue Feb 3, 2023
Apparently this behavior changed (pypa/pip#11663), and as seen in a
comment on pypa/pip#11381 this is *intended* to annoy people and 'break
workflows'.

Pesky open-source maintainers. Sigh.

The 'motivation' here is fun, too:

> This may pose a critical problem for the integrity of distros, which
> often have package-management tools that are themselves written in
> Python.

Who'd have thought that the 'everything is installed in a global place'
issue Linux distros have can't 'just' be solved by package managers as
suddenly the package managers can break themselves because, like, who
manages the package manager?

Hopefully the `--user` flag will work here and won't somehow not apply
as something in the build process uses a second user or something.
mergify bot pushed a commit to aws/jsii that referenced this issue Feb 8, 2023
…s/@jsii/python-runtime (#3950)

Updates the requirements on [pip](https://github.com/pypa/pip) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p>
<blockquote>
<h1>23.0 (2023-01-30)</h1>
<h2>Features</h2>
<ul>
<li>Change the hashes in the installation report to be a mapping. Emit the
<code>archive_info.hashes</code> dictionary in <code>direct_url.json</code>. (<code>[#11312](pypa/pip#11312) &lt;https://github.com/pypa/pip/issues/11312&gt;</code>_)</li>
<li>Implement logic to read the <code>EXTERNALLY-MANAGED</code> file as specified in PEP 668.
This allows a downstream Python distributor to prevent users from using pip to
modify the externally managed environment. (<code>[#11381](pypa/pip#11381) &lt;https://github.com/pypa/pip/issues/11381&gt;</code>_)</li>
<li>Enable the use of <code>keyring</code> found on <code>PATH</code>. This allows <code>keyring</code>
installed using <code>pipx</code> to be used by <code>pip</code>. (<code>[#11589](pypa/pip#11589) &lt;https://github.com/pypa/pip/issues/11589&gt;</code>_)</li>
<li>The inspect and installation report formats are now declared stabled, and their version
has been bumped from <code>0</code> to <code>1</code>. (<code>[#11757](pypa/pip#11757) &lt;https://github.com/pypa/pip/issues/11757&gt;</code>_)</li>
</ul>
<h2>Bug Fixes</h2>
<ul>
<li>Wheel cache behavior is restored to match previous versions, allowing the
cache to find existing entries. (<code>[#11527](pypa/pip#11527) &lt;https://github.com/pypa/pip/issues/11527&gt;</code>_)</li>
<li>Use the &quot;venv&quot; scheme if available to obtain prefixed lib paths. (<code>[#11598](pypa/pip#11598) &lt;https://github.com/pypa/pip/issues/11598&gt;</code>_)</li>
<li>Deprecated a historical ambiguity in how <code>egg</code> fragments in URL-style
requirements are formatted and handled. <code>egg</code> fragments that do not look
like PEP 508 names now produce a deprecation warning. (<code>[#11617](pypa/pip#11617) &lt;https://github.com/pypa/pip/issues/11617&gt;</code>_)</li>
<li>Fix scripts path in isolated build environment on Debian. (<code>[#11623](pypa/pip#11623) &lt;https://github.com/pypa/pip/issues/11623&gt;</code>_)</li>
<li>Make <code>pip show</code> show the editable location if package is editable (<code>[#11638](pypa/pip#11638) &lt;https://github.com/pypa/pip/issues/11638&gt;</code>_)</li>
<li>Stop checking that <code>wheel</code> is present when <code>build-system.requires</code>
is provided without <code>build-system.build-backend</code> as <code>setuptools</code>
(which we still check for) will inject it anyway. (<code>[#11673](pypa/pip#11673) &lt;https://github.com/pypa/pip/issues/11673&gt;</code>_)</li>
<li>Fix an issue when an already existing in-memory distribution would cause
exceptions in <code>pip install</code> (<code>[#11704](pypa/pip#11704) &lt;https://github.com/pypa/pip/issues/11704&gt;</code>_)</li>
</ul>
<h2>Vendored Libraries</h2>
<ul>
<li>Upgrade certifi to 2022.12.7</li>
<li>Upgrade chardet to 5.1.0</li>
<li>Upgrade colorama to 0.4.6</li>
<li>Upgrade distro to 1.8.0</li>
<li>Remove pep517 from vendored packages</li>
<li>Upgrade platformdirs to 2.6.2</li>
<li>Add pyproject-hooks 1.0.0</li>
<li>Upgrade requests to 2.28.2</li>
<li>Upgrade rich to 12.6.0</li>
<li>Upgrade urllib3 to 1.26.14</li>
</ul>
<h2>Improved Documentation</h2>

</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/pypa/pip/commit/368c7b4c557e673b05b0f8cffc967d3e333eee19"><code>368c7b4</code></a> Bump for release</li>
<li><a href="https://github.com/pypa/pip/commit/aa94ccadb45d6ee44defea8a82bd5b647ccba799"><code>aa94cca</code></a> Update AUTHORS.txt</li>
<li><a href="https://github.com/pypa/pip/commit/60ce5c0943c303e48f0aed8bce650f725dcd222d"><code>60ce5c0</code></a> Fix the kind of news fragment</li>
<li><a href="https://github.com/pypa/pip/commit/e3e7bc34eb486622ebbb6412afc98ee57fcbff4a"><code>e3e7bc3</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11766">#11766</a> from uranusjr/upgrade-pre-commit-isort</li>
<li><a href="https://github.com/pypa/pip/commit/b653b129c56b29ad565886c1f423de89639d20f3"><code>b653b12</code></a> Bump pre-commit isort to 5.12.0</li>
<li><a href="https://github.com/pypa/pip/commit/a2a4feb588edc7233ae262d76b2c7291d6857a31"><code>a2a4feb</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11761">#11761</a> from sbidoul/direct-url-hashes-part-3-sbi</li>
<li><a href="https://github.com/pypa/pip/commit/ec7eb6f179866151f148c7695fc773e66b8c3adc"><code>ec7eb6f</code></a> Add version history to inspect and install report docs</li>
<li><a href="https://github.com/pypa/pip/commit/169511e68eb64efff5705305f72b0c53d7bff580"><code>169511e</code></a> Update direct URL hashes examples</li>
<li><a href="https://github.com/pypa/pip/commit/efedf09c4967dcbe3105e3746aaca7bfb55d605f"><code>efedf09</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11759">#11759</a> from pradyunsg/fix-keyring-auth</li>
<li><a href="https://github.com/pypa/pip/commit/60a45984404460192067f3990e0258deeeafa636"><code>60a4598</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11758">#11758</a> from pradyunsg/vendoring-update</li>
<li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/22.3...23.0">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
PEP implementation Involves some PEP type: feature request Request for a new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants