Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP 610 support (git packages) can break pip freeze #8139

Closed
bmartinn opened this issue Apr 25, 2020 · 5 comments · Fixed by #8144
Closed

PEP 610 support (git packages) can break pip freeze #8139

bmartinn opened this issue Apr 25, 2020 · 5 comments · Fixed by #8144
Labels
C: freeze 'pip freeze' related kind: crash For situations where pip crashes !release blocker Hold a release until this is resolved type: bug A confirmed bug or unintended behavior
Milestone

Comments

@bmartinn
Copy link

bmartinn commented Apr 25, 2020

First of all, great job guys on adding PEP 610 support, this is really awesome !

But, if you install a git package with sudo, it will cause any non root pip freeze to crash

Environment

  • pip version: 20.1b1
  • Python version: 3.6
  • OS: Ubuntu

If you do

sudo pip install git+https://github.com/...

Then when you run pip freeze (notice without sudo)

You get the following exception:

ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/freeze.py", line 98, in run
    for line in freeze(**freeze_kwargs):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/operations/freeze.py", line 68, in freeze
    req = FrozenRequirement.from_dist(dist)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/operations/freeze.py", line 256, in from_dist
    direct_url = dist_get_direct_url(dist)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/utils/direct_url_helpers.py", line 118, in dist_get_direct_url
    return DirectUrl.from_json(dist.get_metadata(DIRECT_URL_METADATA_NAME))
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1420, in get_metadata
    value = self._get(path)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1616, in _get
    with open(path, 'rb') as stream:
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/site-packages/trains-0.14.1.dist-info/direct_url.json'

As you can see the direct_url.json file is missing the r flag for group / others , which raises the exception.

-rw------- 1 root root 135 Apr 25 23:22 /usr/local/lib/python3.6/site-packages/trains-0.14.1.dist-info/direct_url.json

I tested running chmod 644 on this file, and after that pip freeze works like a charm.

For reference, the top_level.txt file that is next to the direct_url.json has the correct permission

@triage-new-issues triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label Apr 25, 2020
@bmartinn bmartinn changed the title PEP 610 support (git packages) PEP 610 support (git packages) can break pip freeze Apr 25, 2020
@uranusjr uranusjr added C: freeze 'pip freeze' related kind: crash For situations where pip crashes type: bug A confirmed bug or unintended behavior labels Apr 25, 2020
@triage-new-issues triage-new-issues bot removed the S: needs triage Issues/PRs that need to be triaged label Apr 25, 2020
@uranusjr uranusjr added this to the 20.1 milestone Apr 25, 2020
@uranusjr
Copy link
Member

For completeness, does the RECORD file has the same permission? Most files in.dist-info are extracted from the wheel, and get their permission fixed during the process. But these two are written from scratch, and should be in a similar situation.

(Note for implementer: the permission fix for files extracted from wheel is in pip._internal.utils.unpacking)

@bmartinn
Copy link
Author

bmartinn commented Apr 25, 2020

@uranusjr see below full listing of the directory

ll /usr/local/lib/python3.6/site-packages/trains-0.14.1.dist-info/                                                                                  total 72K
-rw------- 1 root root 135 Apr 26 02:05 direct_url.json
-rw-r--r-- 1 root root  69 Apr 26 02:05 entry_points.txt
-rw------- 1 root root   4 Apr 26 02:05 INSTALLER
-rw-r--r-- 1 root root 12K Apr 26 02:05 LICENSE
-rw-r--r-- 1 root root 11K Apr 26 02:05 METADATA
-rw------- 1 root root 26K Apr 26 02:05 RECORD
-rw-r--r-- 1 root root   7 Apr 26 02:05 top_level.txt
-rw-r--r-- 1 root root 110 Apr 26 02:05 WHEEL

@deveshks
Copy link
Contributor

The permissions shown as per above comment also hold true when performing a pip install without sudo using a git VCS URL, but the pip freeze doesn't crash in this case

$ pip --version
pip 20.1.dev1 from /Users/devesh/pip/src/pip (python 3.8)
$ python --version
Python 3.8.2

$ pip install git+https://github.com/pypa/twine#egg=master
Collecting master
  Cloning https://github.com/pypa/twine to /private/var/folders/xg/blp845_s0xn093dyrtgy936h0000gp/T/pip-install-d7wn0lz6/master
  Running command git clone -q https://github.com/pypa/twine /private/var/folders/xg/blp845_s0xn093dyrtgy936h0000gp/T/pip-install-d7wn0lz6/master
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  WARNING: Generating metadata for package master produced metadata for project name twine. Fix your #egg=master fragments.
Requirement already satisfied: requests>=2.20 in ./.env/lib/python3.8/site-packages (from twine) (2.23.0)
Requirement already satisfied: requests-toolbelt!=0.9.0,>=0.8.0 in ./.env/lib/python3.8/site-packages (from twine) (0.9.1)
Requirement already satisfied: pkginfo>=1.4.2 in ./.env/lib/python3.8/site-packages (from twine) (1.5.0.1)
Requirement already satisfied: keyring>=15.1 in ./.env/lib/python3.8/site-packages (from twine) (21.2.0)
Requirement already satisfied: setuptools>=0.7.0 in ./.env/lib/python3.8/site-packages (from twine) (41.2.0)
Requirement already satisfied: readme-renderer>=21.0 in ./.env/lib/python3.8/site-packages (from twine) (26.0)
Requirement already satisfied: tqdm>=4.14 in ./.env/lib/python3.8/site-packages (from twine) (4.45.0)
Requirement already satisfied: idna<3,>=2.5 in ./.env/lib/python3.8/site-packages (from requests>=2.20->twine) (2.9)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./.env/lib/python3.8/site-packages (from requests>=2.20->twine) (1.25.9)
Requirement already satisfied: chardet<4,>=3.0.2 in ./.env/lib/python3.8/site-packages (from requests>=2.20->twine) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./.env/lib/python3.8/site-packages (from requests>=2.20->twine) (2020.4.5.1)
Requirement already satisfied: docutils>=0.13.1 in ./.env/lib/python3.8/site-packages (from readme-renderer>=21.0->twine) (0.16)
Requirement already satisfied: Pygments>=2.5.1 in ./.env/lib/python3.8/site-packages (from readme-renderer>=21.0->twine) (2.6.1)
Requirement already satisfied: six in ./.env/lib/python3.8/site-packages (from readme-renderer>=21.0->twine) (1.14.0)
Requirement already satisfied: bleach>=2.1.0 in ./.env/lib/python3.8/site-packages (from readme-renderer>=21.0->twine) (3.1.4)
Requirement already satisfied: webencodings in ./.env/lib/python3.8/site-packages (from bleach>=2.1.0->readme-renderer>=21.0->twine) (0.5.1)
Building wheels for collected packages: twine, twine
  Building wheel for twine (PEP 517) ... done
  Created wheel for twine: filename=twine-3.1.2.dev52+g2cec216-py3-none-any.whl size=36679 sha256=72ec0b185e4b51d5e5e95860a68b81daf25d3cbe83667a79a8649a0eec04622a
  Stored in directory: /private/var/folders/xg/blp845_s0xn093dyrtgy936h0000gp/T/pip-ephem-wheel-cache-t4yq_ai4/wheels/73/dd/fb/3039711df8645ab0ccc3e7d5f834289bf9ee7569c222c1fbc4
  Building wheel for twine (PEP 517) ... done
  Created wheel for twine: filename=twine-3.1.2.dev52+g2cec216-py3-none-any.whl size=36679 sha256=90d3e0d5596caa33d2cc171f611f1c1430450a66085ec72d5fcda503a09c9806
  Stored in directory: /private/var/folders/xg/blp845_s0xn093dyrtgy936h0000gp/T/pip-ephem-wheel-cache-t4yq_ai4/wheels/d4/ae/3f/06dd543cc8064e720fa3f9bcc69f3c644ea7c931864ee2c7b3
Successfully built twine twine
Installing collected packages: twine
Successfully installed twine-3.1.2.dev52+g2cec216

$ ls -l .env/lib/python3.8/site-packages/twine-3.1.2.dev52+g2cec216.dist-info/
total 104
-rw-------  1 devesh  staff      4 Apr 26 10:51 INSTALLER
-rw-r--r--  1 devesh  staff   9695 Apr 26 10:51 LICENSE
-rw-r--r--  1 devesh  staff  16240 Apr 26 10:51 METADATA
-rw-------  1 devesh  staff   2811 Apr 26 10:51 RECORD
-rw-r--r--  1 devesh  staff     92 Apr 26 10:51 WHEEL
-rw-------  1 devesh  staff    125 Apr 26 10:51 direct_url.json
-rw-r--r--  1 devesh  staff    186 Apr 26 10:51 entry_points.txt
-rw-r--r--  1 devesh  staff      6 Apr 26 10:51 top_level.txt


$ pip freeze
...
twine @ git+https://github.com/pypa/twine@2cec2169faeb9a2906b2288776be91d660561a06
...

@sbidoul
Copy link
Member

sbidoul commented Apr 26, 2020

The problem has been introduced in #7929 when we started using adjacent_tmp_file to create RECORD, direct_url.json and INSTALLER. Files created with tempfile functions have restricted permissions to avoid security issues when such files are created in world-readable directories such as /tmp.

That is something that needs fixing, possibly in 20.1.

@hroncok
Copy link
Contributor

hroncok commented Apr 27, 2020

Reproducer without git:

$ python3.8 -m venv __venv__
$ . __venv__/bin/activate
(__venv__) $ pip install --pre -U pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/ec/05/82d3fababbf462d876883ebc36f030f4fa057a563a80f5a26ee63679d9ea/pip-20.1b1-py2.py3-none-any.whl (1.5MB)
     |████████████████████████████████| 1.5MB 1.6MB/s 
Installing collected packages: pip
  Found existing installation: pip 19.3.1
    Uninstalling pip-19.3.1:
      Successfully uninstalled pip-19.3.1
Successfully installed pip-20.1b1
(__venv__) $ ll __venv__/lib/python3.8/site-packages/pip-20.1b1.dist-info/
.rw-rw-r--@  125 ... 27 apr 17:35 entry_points.txt
.rw-rw-r--@    4 ... 27 apr 17:35 INSTALLER
.rw-rw-r--@ 1,1k ... 27 apr 17:35 LICENSE.txt
.rw-rw-r--@ 3,6k ... 27 apr 17:35 METADATA
.rw-rw-r--@  57k ... 27 apr 17:35 RECORD
.rw-rw-r--@    4 ... 27 apr 17:35 top_level.txt
.rw-rw-r--@  110 ... 27 apr 17:35 WHEEL
(__venv__) $ pip install -U 'pip<20.1'
Collecting pip<20.1
  Using cached pip-20.0.2-py2.py3-none-any.whl (1.4 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.1b1
    Uninstalling pip-20.1b1:
      Successfully uninstalled pip-20.1b1
Successfully installed pip-20.0.2
(__venv__) $ ll __venv__/lib/python3.8/site-packages/pip-20.0.2.dist-info/
.rw-rw-r--@  125 ... 27 apr 17:36 entry_points.txt
.rw-------@    4 ... 27 apr 17:36 INSTALLER
.rw-rw-r--@ 1,1k ... 27 apr 17:36 LICENSE.txt
.rw-rw-r--@ 3,4k ... 27 apr 17:36 METADATA
.rw-------@  54k ... 27 apr 17:36 RECORD
.rw-rw-r--@    4 ... 27 apr 17:36 top_level.txt
.rw-rw-r--@  110 ... 27 apr 17:36 WHEEL

bors bot referenced this issue in duckinator/emanate May 13, 2020
118: Update pip to 20.1 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.0.2** to **20.1**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.1
   ```
   =================

Process
-------

- Document that pip 21.0 will drop support for Python 2.7.

Features
--------

- Add ``pip cache dir`` to show the cache directory. (`7350 &lt;https://github.com/pypa/pip/issues/7350&gt;`_)

Bug Fixes
---------

- Abort pip cache commands early when cache is disabled. (`8124 &lt;https://github.com/pypa/pip/issues/8124&gt;`_)
- Correctly set permissions on metadata files during wheel installation,
  to permit non-privileged users to read from system site-packages. (`8139 &lt;https://github.com/pypa/pip/issues/8139&gt;`_)
   ```
   
  
  
   ### 20.1b1
   ```
   ===================

Deprecations and Removals
-------------------------

- Remove emails from AUTHORS.txt to prevent usage for spamming, and only populate names in AUTHORS.txt at time of release (`5979 &lt;https://github.com/pypa/pip/issues/5979&gt;`_)
- Remove deprecated ``--skip-requirements-regex`` option. (`7297 &lt;https://github.com/pypa/pip/issues/7297&gt;`_)
- Building of local directories is now done in place, instead of a temporary
  location containing a copy of the directory tree. (`7555 &lt;https://github.com/pypa/pip/issues/7555&gt;`_)
- Remove unused ``tests/scripts/test_all_pip.py`` test script and the ``tests/scripts`` folder. (`7680 &lt;https://github.com/pypa/pip/issues/7680&gt;`_)

Features
--------

- pip now implements PEP 610, so ``pip freeze`` has better fidelity
  in presence of distributions installed from Direct URL requirements. (`609 &lt;https://github.com/pypa/pip/issues/609&gt;`_)
- Add ``pip cache`` command for inspecting/managing pip&#39;s wheel cache. (`6391 &lt;https://github.com/pypa/pip/issues/6391&gt;`_)
- Raise error if ``--user`` and ``--target`` are used together in ``pip install`` (`7249 &lt;https://github.com/pypa/pip/issues/7249&gt;`_)
- Significantly improve performance when ``--find-links`` points to a very large HTML page. (`7729 &lt;https://github.com/pypa/pip/issues/7729&gt;`_)
- Indicate when wheel building is skipped, due to lack of the ``wheel`` package. (`7768 &lt;https://github.com/pypa/pip/issues/7768&gt;`_)
- Change default behaviour to always cache responses from trusted-host source. (`7847 &lt;https://github.com/pypa/pip/issues/7847&gt;`_)
- An alpha version of a new resolver is available via ``--unstable-feature=resolver``. (`988 &lt;https://github.com/pypa/pip/issues/988&gt;`_)

Bug Fixes
---------

- Correctly freeze a VCS editable package when it is nested inside another VCS repository. (`3988 &lt;https://github.com/pypa/pip/issues/3988&gt;`_)
- Correctly handle ``%2F`` in URL parameters to avoid accidentally unescape them
  into ``/``. (`6446 &lt;https://github.com/pypa/pip/issues/6446&gt;`_)
- Reject VCS URLs with an empty revision. (`7402 &lt;https://github.com/pypa/pip/issues/7402&gt;`_)
- Warn when an invalid URL is passed with ``--index-url`` (`7430 &lt;https://github.com/pypa/pip/issues/7430&gt;`_)
- Use better mechanism for handling temporary files, when recording metadata
  about installed files (RECORD) and the installer (INSTALLER). (`7699 &lt;https://github.com/pypa/pip/issues/7699&gt;`_)
- Correctly detect global site-packages availability of virtual environments
  created by PyPA’s virtualenv&gt;=20.0. (`7718 &lt;https://github.com/pypa/pip/issues/7718&gt;`_)
- Remove current directory from ``sys.path`` when invoked as ``python -m pip &lt;command&gt;`` (`7731 &lt;https://github.com/pypa/pip/issues/7731&gt;`_)
- Stop failing uninstallation, when trying to remove non-existent files. (`7856 &lt;https://github.com/pypa/pip/issues/7856&gt;`_)
- Prevent an infinite recursion with ``pip wheel`` when ``$TMPDIR`` is within the source directory. (`7872 &lt;https://github.com/pypa/pip/issues/7872&gt;`_)
- Significantly speedup ``pip list --outdated`` by parallelizing index interaction. (`7962 &lt;https://github.com/pypa/pip/issues/7962&gt;`_)
- Improve Windows compatibility when detecting writability in folder. (`8013 &lt;https://github.com/pypa/pip/issues/8013&gt;`_)

Vendored Libraries
------------------

- Update semi-supported debundling script to reflect that appdirs is vendored.
- Add ResolveLib as a vendored dependency.
- Upgrade certifi to 2020.04.05.1
- Upgrade contextlib2 to 0.6.0.post1
- Upgrade distro to 1.5.0.
- Upgrade idna to 2.9.
- Upgrade msgpack to 1.0.0.
- Upgrade packaging to 20.3.
- Upgrade pep517 to 0.8.2.
- Upgrade pyparsing to 2.4.7.
- Remove pytoml as a vendored dependency.
- Upgrade requests to 2.23.0.
- Add toml as a vendored dependency.
- Upgrade urllib3 to 1.25.8.

Improved Documentation
----------------------

- Emphasize that VCS URLs using git, git+git and git+http are insecure due to
  lack of authentication and encryption (`1983 &lt;https://github.com/pypa/pip/issues/1983&gt;`_)
- Clarify the usage of --no-binary command. (`3191 &lt;https://github.com/pypa/pip/issues/3191&gt;`_)
- Clarify the usage of freeze command in the example of Using pip in your program (`7008 &lt;https://github.com/pypa/pip/issues/7008&gt;`_)
- Add a &quot;Copyright&quot; page. (`7767 &lt;https://github.com/pypa/pip/issues/7767&gt;`_)
- Added example of defining multiple values for options which support them (`7803 &lt;https://github.com/pypa/pip/issues/7803&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <[email protected]>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 15, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
C: freeze 'pip freeze' related kind: crash For situations where pip crashes !release blocker Hold a release until this is resolved type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants