CEP for specifying the location of site-packages in repodata #90

jjhelmus · 2024-09-13T16:29:05Z

Description

This CEP proposed to introduce an optional python_site_packages_path field in repodata that python packages can use to explicitly specify the location of the site-packages directory. This allows for noarch: python packages to be linked into the correct location when this path does not match the on used by standard builds of CPython. This includes alternative implementation of the Python programming langauge (PyPy, GraalPy) and free-threading builds of CPython.

The motivation and request for this CEP came from conda/conda#14053, which discusses noarch: python support in conda in environments with the experimental free-threading configuration of CPython 3.13.

cep-999.md

dholth · 2024-09-16T14:44:14Z

https://github.com/conda/ceps/pull/86/files is working in the same space, to allow ABI3 packages

dholth

I tend to be in favor of putting the site-packages path in a separate file in python's info/, for example in link.json or paths.json where it would be possible to increment the version, and make it non-installable by incompatible condas.

cep-999.md

dholth · 2024-09-20T15:55:16Z

cep-999.md

+      "subdir": osx-arm64,
+      "version": "3.13.0rc1",
+      "sha256": "fa0ae22c13450fe6c30c754ee5efbd7fe7e7533b878d7be96e74de56211d19df",
+      "python_site_packages_path": "lib/python3.13t/site-packages"


Suggested change

"python_site_packages_path": "lib/python3.13t/site-packages"

"python_site_packages": "lib/python3.13t/site-packages"

The field specified in the CEP is python_site_packages_path with _path

Yes, suggestion is to rename without _path

I'm open to either name. Does anyone have a strong opinion one way or another?

Co-authored-by: Daniel Holth <[email protected]>

jjhelmus · 2024-09-20T19:47:06Z

I have put together a draft PR for conda-build that implements support for specifying this field in the build section of a recipe, see conda/conda-build#5502.

With this I created sample packages that specify this metadata field for testing with downstream tools. They are available on anaconda.org in the jjhelmus/label/sp_path channel.

There are three noarch packages with the following paths specified

python 3.99.99 : lib/python3.99t/site-packages - A "free-threading" like path
python 3.99.90 : sample_path/from_the_file/index_json - Something that stands out
python 3.99.50 : ../../../../../invalid_path : Should be rejected by tooling as being invalid

jjhelmus · 2024-09-20T20:28:23Z

conda/conda#14256 contains an rough draft of the change needed for conda to make use of this field.

isuruf · 2024-09-29T23:45:23Z

Should we fix the name for python in the shebang of entry-points in this CEP or a separate one? Currently there's a python3.13t->python3.13 symlink

jezdez · 2024-10-04T11:01:24Z

Should we fix the name for python in the shebang of entry-points in this CEP or a separate one? Currently there's a python3.13t->python3.13 symlink

I'd suggest keeping it here TBH, since a whole CEP for a simple change like this may be overkill.

jezdez

Left a few comments, mostly just tweaks for readability

cep-999.md

jezdez · 2024-10-04T11:55:04Z

cep-999.md

+def is_valid(target_prefix: str, python_site_packages_path: str) -> bool:
+    target_prefix = os.path.realpath(target_prefix)
+    full_path = os.path.realpath(os.path.join(target_prefix, python_site_packages_path))
+    test_prefix = os.path.commonpath((target_prefix, full_path))
+    return test_prefix == target_prefix


Suggested change

def is_valid(target_prefix: str, python_site_packages_path: str) -> bool:

target_prefix = os.path.realpath(target_prefix)

full_path = os.path.realpath(os.path.join(target_prefix, python_site_packages_path))

test_prefix = os.path.commonpath((target_prefix, full_path))

return test_prefix == target_prefix

from pathlib import Path

def is_valid(target_prefix: str, python_site_packages_path: str) -> bool:

target_prefix = Path(target_prefix).resolve()

full_path = (target_prefix / python_site_packages_path).resolve()

return target_prefix in full_path.parents

This style of check would require resolving target_prefix as well as it needs to take into account symbolic links and various re-directs on Windows file systems.
The function as written in the CEP closely matches the filter used by by tarfile module in the CPython standard library for validating paths, https://github.com/python/cpython/blob/v3.13.0/Lib/tarfile.py#L816-L818

isuruf · 2024-10-04T15:34:31Z

A bit of context about the python executable name.

On Unix, CPython (upstream and conda) has python3.13 for freethreading and default builds. For freethreading python3.13 is a symlink to python 3.13t. On Unix, PyPy does the same

On Windows, CPython upstream has python.exe for default build and python3.13t.exe for freethreading builds as there's only one installer. On conda-forge, python3.13.exe and python.exe is a copy of python3.13t.exe for freethreading builds.
On PyPy, they have python3.10.exe copy for python.exe

For noarch: packages, the entry-points right now uses python3.13 for unix and python.exe for windows. Do we want to keep using these or have a way for specifying the executable name? Note that it's straightforward to change the name in Unix as it's about changing the shebang, but it's hard on windows due to the need to create an executable.

I'm okay with keeping the current status quo and requiring python<major>.<minor> on Unix and python.exe on Windows for all python flavours.

jjhelmus · 2024-10-04T16:39:25Z

At least on macOS, Python virtual environment created using venv from the standard library provide executables named python, python3 and python3.x with the second two being symlinks to the first. A t executable is not provided for freethreading environments which is surprising to me.

By default the path of the executable used to run pip is used when creating console entry-points in these environments. The pip, pip3 and pip3.x command all use the path to python for the executable so this path will be used unless pip is started using python3{.X} -m pip.

Based on this I would be in favor of conda using python for the shebang in entry-points.

jjhelmus · 2024-10-04T18:50:13Z

I tend to be in favor of putting the site-packages path in a separate file in python's info/, for example in link.json or paths.json where it would be possible to increment the version, and make it non-installable by incompatible condas.

The benefit of including the site-packages path in repodata is the ability to correct it with hotfixing after the package is built. This can be used to add this field to existing packages, like the PyPy and GraalPy packages in conda-forge, or correct mistakes in future packages. If functionality is not needed then the field could be moved to link.json or paths.json.

Another benefit of including this is repodata is discoverability. Repodata can be examined without downloading and extracting the package itself.

Incrementing the version would prevent the installation with incompatible version of condas but the user experience is not great. Specifically the environment would solve and the user would not be aware of a problem until the packages are downloaded, extracted and staged for linking.

Co-authored-by: Jannis Leidel <[email protected]>

baszalmstra · 2024-10-05T06:27:26Z

Another advantage of storing this information in the repodata is that that information is always readily available during installation, even for already installed records (even if the cache is cleared).

baszalmstra · 2024-10-05T06:31:23Z

So what should the behavior be if more than one package in installed with this field set?

Good question! I assumed it would be ignored except for the package that provides python. In rattler we currently assume that is the package called “python”. I wonder how conda does this.

jjhelmus · 2024-10-08T14:54:13Z

So what should the behavior be if more than one package in installed with this field set?

I assume failure of some kind?

This field is only allowed on the conda package named python, if packages with other names include this field it is ignored as mentioned in the CEP. There can only be one python package installed in an environment so there will never be a case where there is more that one package installed with this field.

Another way of thinking about this change is that instead of using the package with name python to determine the major and minor version that are used in the site-packages path, the python package can also use this field to declare the path. The sample conda PR shows how this could be implemented.

jezdez · 2024-10-08T15:22:01Z

dholth · 2024-10-08T15:39:03Z

When conda doesn't understand this field, does that Python package work?

jjhelmus · 2024-10-08T15:51:04Z

When conda doesn't understand this field, does that Python package work?

Yes, the current GraalPy, PyPy and free-threading CPython python packages in conda-forge include work arounds like symlinks or sitecustomize.py files that so that noarch: python packages work even though conda installs these packages into the incorrect path. The CEP mentions these work arounds the recommends that they be maintained for some time:

Because existing releases of conda, mamba and pixi do not support this field, python packages should continue to provide work around to allow files to be installed into an incorrect site-packages directory until such time as this proposal has been implemented and available for a sufficient amount of time.

cep-999.md

Co-authored-by: jaimergp <[email protected]>

wolfv · 2024-10-13T19:37:16Z

I just had one shower-thought: what if we would store a map for this, and could that help with noarch: ruby / noarch: php etc. packages?

It's been a long standing issue that we cannot support other languages with noarch packages as easily as Python for which each package manager needs to add some specific code.

If we had a noarch_map: { ... } field of some sort that would map directories, such as:

"noarch_map": {
   "site-packages": "lib/python3.10/site-packages"
}

And for Ruby it could read something like:

"noarch_map": {
   "gems": "lib/ruby5.3/gemfiles/gems"
}

This would still leave open the question on pre-compiling py / rb / ... files but it would solve one of the noarch-ruby problems and would not introduce more python special handling.

The noarch_map could be consulted for any noarch package (generic or not) that contains a folder of the sort.

I think this might be worth it to revisit if we can afford it.

I was trying to find some old references / issues, could only find one so far.

conda convert all conda#702 (comment)

isuruf · 2024-10-14T15:27:29Z

I don't think we should introduce any other noarch types. noarch: generic is enough for most use-cases and we should just use a version independent path for other languages. For python, it's historic debt.

mcg1969 · 2024-10-14T15:52:40Z

But is that possible?

Certainly, one interpretation of noarch-Python is that it's historic debt. But eliminating it would require patching the default path search behavior of python; and there is version-dependent byte compilation as well.

In contrast, for R we are fortunate that the search path is platform and version-independent, but the byte compilation issue remains, which means that currently we have to rebuild many noarch packages with each major R release.

I think it would be good to enumerate the additional languages being considered and the potential obstacles to the use of noarch-generic. If there are none, great, but at least for me I'm speaking from ignorance of these alternatives.

wolfv · 2024-10-14T16:00:10Z

@isuruf we wouldn't have to introduce any new noarch types. Just move the files in the right places and the noarch-map would take care of it :)
I am just advocating to produce less technical debt for python specifics.

isuruf · 2024-10-14T16:05:27Z

Certainly, one interpretation of noarch-Python is that it's historic debt. But eliminating it would require patching the default path search behavior of python; and there is version-dependent byte compilation as well.

Right, it would require a .pth file to change the search behaviour. noarch: python should be used only for byte compilation, the moving of files shouldn't be necessary IMO, but unrelated to this CEP.

@wolfv, yes, but it's adding another layer of unnecessary behaviour when all we need to do is to make sure the packages end up in a version-independent path in the first place. You are producing more general technical debt in order to produce less technical debt for python specifics.

baszalmstra · 2024-10-20T12:22:57Z

@jezdez What is the status of this CEP? Did it pass or are there still discussions to be had?

jezdez · 2024-10-20T12:48:59Z

@jezdez What is the status of this CEP? Did it pass or are there still discussions to be had?

I'll update the PR in the coming week, currently out of office.

jjhelmus · 2024-10-21T17:30:43Z

If we had a noarch_map: { ... } field of some sort that would map directories, such as:

Why make this a nested field when any individual package should only ever define one entry. A python package can declare the location of the site-packages directory, it would not declare a R or Ruby location. In addition "Flat is better than nested".

wolfv · 2024-10-21T17:47:22Z

One could imagine multiple entries, such as:

noarch_map: {
  "r_bin": "lib/r/5.0/bin",
  "r_lib": "lib/r/lib",
}

and noarch packages that would have r_bin and r_lib at the root would get mapped.

But yeah, idk, maybe not useful enough :)

jjhelmus · 2024-10-21T18:00:08Z

conda/rattler#909 has the beginnings of an implementation of this CEP in rattler.

wolfv · 2024-10-21T18:04:07Z

Awesome, that's great! :)

jezdez · 2024-10-21T22:09:06Z

The vote is closed, and we have the following result:

Total voters: 17 (valid: 13 = 76.47%)

Yes votes (13 / 100.00%):

@xhochy (Uwe Korn)
@mariusvniekerk (Marius van Niekerk)
@chenghlee (Cheng H. Lee)
@ocefpaf (Filipe Fernandes)
@marcelotrevisani (Marcelo Duarte Trevisani)
@msarahan (Michael Sarahan)
@jezdez (Jannis Leidel)
@wolfv (Wolf Vollprecht)
@jaimergp (Jaime Rodríguez-Guerra)
@baszalmstra (Bas Zalmstra)
@beckermr (Matthew R. Becker)
@Hind-M (Hind Montassif)
@trallard (Tania Allard)

No votes (0 / 0.00%)):

Abstain votes (0 / 0.00%):

Not voted (4):

@CJ-Wright (Christopher J. 'CJ' Wright)
@mbargull (Marcel Bargull)
@jakirkham (John Kirkham)
@kkraus14 (Keith Kraus)

Invalid votes (0):

Thus we reached quorum and enough YES votes to mark this as accepted. 🎉

…ng noarch: python packages, CEP-17 (#909) ### Description Use the `python_site_packages_path` field from repodata to set the directory when `noarch: python` packages get installed as specified in [CEP-17](conda/ceps#90). This the start of an implementation. Wanted to get feedback on the approach before updating other crates and tests. --------- Co-authored-by: Bas Zalmstra <[email protected]>

baszalmstra · 2024-11-05T18:44:11Z

This has been implemented in rattler (and thus in the next pixi release) and rattler-build !

jjhelmus added 2 commits September 13, 2024 11:24

CEP for specifying the location of site-packages in repodata

2218dc0

update discussion in table

f069e1b

jjhelmus mentioned this pull request Sep 13, 2024

noarch: python packages are not importable when installed with the experimental free-threading Python 3.13 build conda/conda#14053

Open

2 tasks

isuruf approved these changes Sep 13, 2024

View reviewed changes

baszalmstra approved these changes Sep 13, 2024

View reviewed changes

cep-999.md Outdated Show resolved Hide resolved

cep-999.md Outdated Show resolved Hide resolved

cep-999.md Outdated Show resolved Hide resolved

fix typo of pixi

500bb1c

jezdez mentioned this pull request Sep 16, 2024

Python packages listed as from the pypi channel when installed in an environment with the experimental free-threading Python 3.13 build conda/conda#14153

Open

2 tasks

dholth reviewed Sep 20, 2024

View reviewed changes

intentionally-left-nil mentioned this pull request Sep 20, 2024

Add python_site_packages_path for 3.13 GIL/free threading support conda-forge/python-feedstock#715

Open

jjhelmus and others added 2 commits September 20, 2024 11:30

Update cep-999.md

68c385b

Co-authored-by: Daniel Holth <[email protected]>

Update cep-999.md

4e58baa

Co-authored-by: Daniel Holth <[email protected]>

jjhelmus mentioned this pull request Sep 20, 2024

python_site_packages_path can be specified in the build section conda/conda-build#5502

Merged

3 tasks

jezdez reviewed Oct 4, 2024

View reviewed changes

jjhelmus and others added 9 commits October 4, 2024 13:57

Update cep-999.md

b4c2152

Co-authored-by: Jannis Leidel <[email protected]>

Update cep-999.md

1a71ed3

Co-authored-by: Jannis Leidel <[email protected]>

Update cep-999.md

8584756

Co-authored-by: Jannis Leidel <[email protected]>

Update cep-999.md

6ed4ce7

Co-authored-by: Jannis Leidel <[email protected]>

Update cep-999.md

013e8f0

Co-authored-by: Jannis Leidel <[email protected]>

Update cep-999.md

23648a0

Co-authored-by: Jannis Leidel <[email protected]>

Update cep-999.md

f6c84ae

Co-authored-by: Jannis Leidel <[email protected]>

Update cep-999.md

4b744c8

Co-authored-by: Jannis Leidel <[email protected]>

Update cep-999.md

4c14da6

Co-authored-by: Jannis Leidel <[email protected]>

jezdez added the vote Voting following governance policy label Oct 9, 2024

jaimergp reviewed Oct 10, 2024

View reviewed changes

cep-999.md Outdated Show resolved Hide resolved

Update cep-999.md

12f1825

Co-authored-by: jaimergp <[email protected]>

jjhelmus added 4 commits October 21, 2024 12:21

Use ascii quotes

9ee2913

fix typo

6c3ea4d

move to cep-17.md

a975481

add CEP 16 and 17 to README table

8097f1a

jjhelmus mentioned this pull request Oct 21, 2024

feat: use python_site_packages_path field when available for installing noarch: python packages, CEP-17 conda/rattler#909

Merged

jezdez merged commit 982868f into conda:main Oct 22, 2024

Hind-M mentioned this pull request Oct 22, 2024

Implement CEP #90 mamba-org/mamba#3558

Open

	"python_site_packages_path": "lib/python3.13t/site-packages"
	"python_site_packages": "lib/python3.13t/site-packages"

CEP for specifying the location of site-packages in repodata #90

CEP for specifying the location of site-packages in repodata #90

Conversation

jjhelmus commented Sep 13, 2024

Description

dholth commented Sep 16, 2024

dholth left a comment

Choose a reason for hiding this comment

dholth Sep 20, 2024

Choose a reason for hiding this comment

jjhelmus Sep 20, 2024

Choose a reason for hiding this comment

dholth Sep 20, 2024

Choose a reason for hiding this comment

jjhelmus Sep 20, 2024

Choose a reason for hiding this comment

jjhelmus commented Sep 20, 2024

jjhelmus commented Sep 20, 2024

isuruf commented Sep 29, 2024

jezdez commented Oct 4, 2024

jezdez left a comment

Choose a reason for hiding this comment

jezdez Oct 4, 2024

Choose a reason for hiding this comment

jjhelmus Oct 8, 2024

Choose a reason for hiding this comment

isuruf commented Oct 4, 2024

jjhelmus commented Oct 4, 2024

jjhelmus commented Oct 4, 2024

baszalmstra commented Oct 5, 2024

baszalmstra commented Oct 5, 2024

jjhelmus commented Oct 8, 2024

jezdez commented Oct 8, 2024 • edited Loading

dholth commented Oct 8, 2024

jjhelmus commented Oct 8, 2024 • edited Loading

wolfv commented Oct 13, 2024 • edited Loading

isuruf commented Oct 14, 2024

mcg1969 commented Oct 14, 2024

wolfv commented Oct 14, 2024

isuruf commented Oct 14, 2024

baszalmstra commented Oct 20, 2024

jezdez commented Oct 20, 2024

jjhelmus commented Oct 21, 2024

wolfv commented Oct 21, 2024

jjhelmus commented Oct 21, 2024

wolfv commented Oct 21, 2024

jezdez commented Oct 21, 2024

baszalmstra commented Nov 5, 2024

jezdez commented Oct 8, 2024 •

edited

Loading

jjhelmus commented Oct 8, 2024 •

edited

Loading

wolfv commented Oct 13, 2024 •

edited

Loading