Implement explicit lockfiles integration #23

jaimergp · 2024-05-09T07:27:17Z

Closes #21

conda list --explicit extensions
conda install | create extensions for the above output
Tests
Docs

jaimergp · 2024-05-22T11:05:53Z

This is how the output of conda list --explicit would work, btw:

# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: osx-arm64
@EXPLICIT
https://conda.anaconda.org/conda-forge/osx-arm64/bzip2-1.0.8-h93a5062_5.conda
https://conda.anaconda.org/conda-forge/osx-arm64/c-ares-1.23.0-h93a5062_0.conda
https://conda.anaconda.org/conda-forge/osx-arm64/ca-certificates-2024.2.2-hf0a4a13_0.conda
https://conda.anaconda.org/conda-forge/osx-arm64/icu-73.2-hc8870d7_0.conda
https://conda.anaconda.org/conda-forge/osx-arm64/jpeg-9e-h1a8c8d9_3.conda
https://conda.anaconda.org/conda-forge/osx-arm64/libcxx-16.0.6-h4653b0c_0.conda
https://conda.anaconda.org/conda-forge/osx-arm64/libdeflate-1.17-h1a8c8d9_0.conda
https://conda.anaconda.org/conda-forge/osx-arm64/libedit-3.1.20191231-hc8eb9b7_2.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/libev-4.33-h93a5062_2.conda
https://conda.anaconda.org/conda-forge/osx-arm64/libffi-3.4.2-h3422bc3_5.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/libiconv-1.17-h0d3ecfb_1.conda
https://conda.anaconda.org/conda-forge/osx-arm64/libwebp-base-1.3.2-hb547adb_0.conda
https://conda.anaconda.org/conda-forge/osx-arm64/libzlib-1.2.13-h53f4e23_5.conda
https://conda.anaconda.org/conda-forge/osx-arm64/llvm-openmp-17.0.6-hcd81f8e_0.conda
https://conda.anaconda.org/conda-forge/osx-arm64/lzo-2.10-h642e427_1000.tar.bz2
# Following lines added by conda-pypi
# pypi: gitpython==3.1.32 --python-version 3.9.9 --implementation cp
# pypi: platformdirs==4.2.2 --python-version 3.9.9 --implementation cp
# pypi: sphinx==4.2.0 --python-version 3.9.9 --implementation cp
# pypi: python-dateutil==2.8.1 --python-version 3.9.9 --implementation cp
# pypi: pathspec==0.8.1 --python-version 3.9.9 --implementation cp
# pypi: charset-normalizer==2.0.7 --python-version 3.9.9 --implementation cp
# pypi: tornado==6.1 --python-version 3.9.9 --implementation cp --abi cp39 --platform macosx_11_0_arm64
# pypi: gitdb==4.0.10 --python-version 3.9.9 --implementation cp
# pypi: black==21.6b0 --python-version 3.9.9 --implementation cp
# pypi: regex==2021.4.4 --python-version 3.9.9 --implementation cp --abi cp39 --platform macosx_11_0_arm64
# pypi: tzdata==2023.3 --python-version 3.9.9 --implementation cp
# pypi: pillow==8.3.2 --python-version 3.9.9 --implementation cp --abi cp39 --platform macosx_11_0_arm64

We can't print the URL for PyPI wheels because it wasn't kept around 😬

jaimergp · 2024-05-22T11:48:19Z

I'm not personally a fan of using the CLI flags here but these are the problems I faced trying to obtain something else:

There's no record of the origin URL for an installed PyPI package (see pip should record the original index url or find links url of the package pypa/pip#10736). This might change with PEP-710, but it's still a draft.
The environment markers (`name=version; python_version='2.7' etc) do not limit which wheels will be obtained, but which lines are accepted for inclusion in the request list given the current Python interpreter.

I wish there was a way to encode the constraints in a single string field (like a URI or something), but right now this is the only way, I think. We can document this as the official syntax... which happens to be compatible with a subset of the pip CLI 😬 Note we are adding more flags to the command.

jezdez

A few notes:

I'm not sure how I feel you expanding this format without a spec and a CEP since that's definitely a core functionality in conda
There has to be other ways than implementing custom argv parsing and custom Python packaging standards parsing (big maintenance risk based on my experience)
Inline docs and high-level reasoning for some of the design decisions are sorely needed before merging this

jezdez · 2024-05-28T07:15:40Z

conda_pypi/main.py

+            if path.name == "WHEEL" and path.parent.suffix == ".dist-info":
+                for line in path.read_text().splitlines():
+                    line = line.strip()
+                    if ":" not in line:
+                        continue
+                    key, value = line.split(":", 1)
+                    if key == "Tag":
+                        wheel.setdefault(key, []).append(value.strip())
+                    else:
+                        wheel[key] = value.strip()


Isn't there a standard way to get this info without manual parsing?

I dug in the docs and it uses RFC email headers, so email.parser.HeaderParser from stdlib can be used. It requires a bit of post-processing, quite similar to what we see in conda.common.pkg_formats.python. That module would better be here in this project now. That whole thing is mostly there to support conda list listing pypi dependencies after all. We can refactor it out and finally remove the pip_interop_enabled thingy.

conda_pypi/main.py

conda_pypi/cli/install.py

conda_pypi/__init__.py

jezdez · 2024-05-28T07:20:15Z

conda_pypi/cli/install.py

+if TYPE_CHECKING:
+    from typing import Iterable
+
+log = getLogger(f"conda.{__name__}")


Interesting, is there a reason to reuse the conda logger instead of an own?

I've been doing this in all my plugins to reuse conda's configuration for consistency. CLS does this too, for example.

conda_pypi/main.py

jezdez · 2024-05-28T07:40:25Z

conda_pypi/main.py

+        "--report",
+        json_output.name,
+        "--target",
+        json_output.name + ".dir",


We need a writable path here. I'm just reusing the tmpfile. I don't know if the path is actually created, though. This is a dry run after all.

conda_pypi/main.py

jezdez · 2024-05-28T07:42:25Z

conda_pypi/main.py

+
+
+def compute_record_sum(record_path, algo):
+    record = Path(record_path).read_text()


OSError again

jezdez · 2024-05-28T07:42:55Z

conda_pypi/main.py

+            # we only want to check the metadata and wheel parts of dist-info; everything else
+            # is not deterministic or useful
+            continue
+        if path.parts[0] == ".." and any(part in path.parts for part in ("bin", "lib", "Scripts")):


Suggested change

if path.parts[0] == ".." and any(part in path.parts for part in ("bin", "lib", "Scripts")):

elif path.parts[0] == ".." and any(part in path.parts for part in ("bin", "lib", "Scripts")):

jezdez

A few notes:

I'm not sure how I feel you expanding this format without a spec and a CEP since that's definitely a core functionality in conda
There has to be other ways than implementing custom argv parsing and custom Python packaging standards parsing (big maintenance risk based on my experience)
Inline docs and high-level reasoning for some of the design decisions are sorely needed before merging this

jaimergp · 2024-05-28T07:56:03Z

Thanks for the thorough review!

I'm not sure how I feel you expanding this format without a spec and a CEP since that's definitely a core functionality in conda

I do want the CEP. For the original format too. There's no standard to refer to other than the implementation. It should be straightforward.

There has to be other ways than implementing custom argv parsing and custom Python packaging standards parsing (big maintenance risk based on my experience)

conda has its own parsers but I only learnt that after writing this prototype. I'm sure we can find some parsers somewhere in packaging and friends too. Right now I'm mostly exploring how this idea would shape up, but I agree we need to polish this custom code better. Even the structure of the code could use some improvement.

Inline docs and high-level reasoning for some of the design decisions are sorely needed before merging this

Yep, just wanted to get some external confirmation that this idea is worth pursuing before writing details on something that might not fly. I take that this review is also a 👍 to proceed further?

…s when feasible

jaimergp · 2024-06-03T19:19:53Z

I do want the CEP. For the original format too.

There you go: conda/ceps#79

jaimergp · 2024-06-05T09:43:52Z

Hey @jezdez, I think I addressed most of your comments now. The PR is getting a bit too big so we can address further changes in new PRs. WDYT? After this PR I feel comfortable moving things to conda-incubator.

jezdez · 2024-06-05T10:18:37Z

@jaimergp thank you for doing the rounds on the standardization, this'll be a net benefit!

jaimergp added 7 commits May 9, 2024 09:25

add list extensions

48c7279

Merge branch 'main' of github.com:jaimergp/conda-pip into lockfiles

0d4ec77

add more details to the explicit list

bc1d8c4

implement install from '# pypi:' lockfiles

3dd6657

fix call

6875927

add test

6364f7f

order can vary across runs 🤷

563238a

jaimergp added 7 commits May 22, 2024 17:16

add disclaimer

faa1899

add version to disclaimer

608f27e

add md5 of the RECORD file

abb77ed

test with --md5

c15ef18

pre-commit

d3de85e

Verify checksums of RECORD files upon install

9534226

fix test

545deb2

jaimergp force-pushed the lockfiles branch from 3fff49c to 545deb2 Compare May 24, 2024 08:11

jaimergp added 2 commits May 24, 2024 10:11

fix test

8b1cb98

fix windows entry points

9fd169c

jezdez reviewed May 28, 2024

View reviewed changes

jaimergp added 6 commits May 29, 2024 11:58

fix test

70c24d6

fix context cmd_line value fetching

43451ab

use 0.1.0 as t0

69e1661

refactor pypi_lines with a helper object and rely on 3rd party parser…

0c49317

…s when feasible

mroe refactors

b0cc8cf

fix test

674cec6

more docs!

7cc2bbc

jaimergp mentioned this pull request Jun 5, 2024

Migrate to conda-incubator? #25

Closed

jezdez approved these changes Jun 5, 2024

View reviewed changes

jaimergp merged commit ab90dbc into main Jun 5, 2024
12 checks passed

jaimergp deleted the lockfiles branch June 5, 2024 10:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement explicit lockfiles integration #23

Implement explicit lockfiles integration #23

jaimergp commented May 9, 2024 •

edited

Loading

jaimergp commented May 22, 2024

jaimergp commented May 22, 2024 •

edited

Loading

jezdez left a comment

jezdez May 28, 2024

jaimergp May 29, 2024

jezdez May 28, 2024

jaimergp May 28, 2024

jezdez May 28, 2024

jaimergp May 28, 2024

jezdez May 28, 2024

jezdez May 28, 2024

jezdez left a comment

jaimergp commented May 28, 2024

jaimergp commented Jun 3, 2024

jaimergp commented Jun 5, 2024

jezdez commented Jun 5, 2024



		def compute_record_sum(record_path, algo):
		record = Path(record_path).read_text()

	if path.parts[0] == ".." and any(part in path.parts for part in ("bin", "lib", "Scripts")):
	elif path.parts[0] == ".." and any(part in path.parts for part in ("bin", "lib", "Scripts")):

Implement explicit lockfiles integration #23

Implement explicit lockfiles integration #23

Conversation

jaimergp commented May 9, 2024 • edited Loading

jaimergp commented May 22, 2024

jaimergp commented May 22, 2024 • edited Loading

jezdez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jezdez left a comment

Choose a reason for hiding this comment

jaimergp commented May 28, 2024

jaimergp commented Jun 3, 2024

jaimergp commented Jun 5, 2024

jezdez commented Jun 5, 2024

jaimergp commented May 9, 2024 •

edited

Loading

jaimergp commented May 22, 2024 •

edited

Loading