Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Meta-ticket: ./configure --enable-system-site-packages (optionally use system Python packages) #29023

Open
mkoeppe opened this issue Jan 15, 2020 · 38 comments
Open

Meta-ticket: ./configure --enable-system-site-packages (optionally use system Python packages) #29023

mkoeppe opened this issue Jan 15, 2020 · 38 comments
Labels
c: packages: standard p: major / 3 t: enhancement

Comments

@mkoeppe
Copy link
Member

mkoeppe commented Jan 15, 2020

This is a followup on #27824: spkg-configure.m4 for python3.

In the next step, we make it possible for users to enable use of system Python packages (site packages).

It is crucial to distinguish two categories of Python packages.

A. Python-implemented applications not running in the same process as sagelib

For some specific packages that provide Python-implemented applications and do not run in the same process as sagelib, there are separate tickets that may use spkg-configure.m4

'''B. Packages that run in the same process as sagelib.``

Use of these packages would be enabled by a new configure option --enable-system-site-packages.

This will set up the venv using build/bin/sage-venv --system-site-packages

Approach 1:

Approach 2:

  • Separate the installation process of a Python package into several phases (makefile targets):

    i) Downloading the spkg tarball (in most cases the tarball will already be an sdist, i.e., with embedded metadata)

    ii) (When patches are needed or the upstream tarball is not a real sdist:) Unpack, patch, make a new sdist

    iii) Copy the sdist to $SAGE_SPKG_WHEELS or a new directory next to it

    iv) Build a wheel and copy to $SAGE_SPKG_WHEELS

    v) Install the wheel

Questions to be resolved:

  • #32492 comment:17 -- installed packages are only considered by pip if they can also be found in an index. As we use --no-index, the workaround in sage-pip-install: Try installing without --no-deps first #32492 provides the directory $SAGE_SPKG_WHEELS as the index.

    To be checked if the specific installed version of the package needs to be found in the index. If presence of any version (= spkg version) will cause pip to consider the installed version as a candidate for resolution, then we can just leave the decision whether to use the installed package or the spkg to the resolver and the pip upgrade-strategy.

    If a package needs any special preparation for building a wheel such as setting of environment variables, we need to:

    a) either invoke phase iv) explicitly (so wheels of such packages may be built but remain unused);

    b) or try to move the environment settings to the general build environment;

    c) or patch the Python package's build script instead, treating it like phase ii).

  • If we need a specific version (with patches that have not been upstreamed), should this be reflected in dependency information (install-requires.txt)?

  • Investigate pip's and other tools behavior when installing an upgraded/downgraded version of a package in the venv when the package is installed in the system site packages

Related/follow-up tickets:

Tickets/issues for individual packages (moved here from #27330, will be obsolete with this ticket):

  • numpy
  • scipy
  • cypari (depends on pari)

References:

https://packaging.python.org/discussions/install-requires-vs-requirements/

PEP 508 -- Dependency specification for Python Software Packages | Python.org​
https://www.python.org/dev/peps/pep-0508/#pep440

PEP 440 -- Version Identification and Dependency Specification | Python.org
​https://www.python.org/dev/peps/pep-0440/#local-version-segments

https://python-poetry.org/docs/pyproject/#dependencies-and-dev-dependencies

CC: @tobihan @sagetrac-tmonteil @orlitzky @timokau @kiwifb @isuruf @mezzarobba @embray @dimpase @thierry-FreeBSD @dkwo

Component: packages: standard

Issue created by migration from https://trac.sagemath.org/ticket/29023
@Volker-Weissmann
Copy link
Contributor

comment:1

I think there should be an option to disable the use of system packages. Sage's installation is already compilcated enough and I don't want a faulty package to mess up sage.

@jhpalmieri
Copy link
Member

comment:2

You can already do ./configure --with-system-python3=no to avoid using the system's Python. Are you asking for something which allows the use of the system Python but not the system Python packages? Is it likely that there will be a good system Python installation which has faulty Python packages?

@Volker-Weissmann
Copy link
Contributor

comment:3

I just found out about the ./configure --help option. --with-system-python3 is actually exactly what I meant.

@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe
Copy link
Member Author

mkoeppe commented Apr 5, 2020

comment:5

Replying to @Volker-Weissmann:

I think there should be an option to disable the use of system packages. Sage's installation is already compilcated enough and I don't want a faulty package to mess up sage.

Thanks for the input. I've updated the ticket description.

@mkoeppe

This comment has been minimized.

Sign in to view
@tobihan
Copy link

tobihan commented Apr 17, 2020

comment:7

I think once python system packages are detected, the spkg-configure mechanism becomes interesting for packagers. Looking forward to this change.

@mkoeppe
Copy link
Member Author

mkoeppe commented Apr 17, 2020

comment:8

I don't think python packages would be handled through spkg-configure.
I think it's better to turn sagelib into a pip-installable package and provide standard prereq information -- see #29041 (at ./bootstrap time, generate src/requirements.txt, src/constraints.txt, src/setup.cfg [install_requires] from build/pkgs)

@thierry-FreeBSD
Copy link

comment:9

I tried to create an experimental spkg-configure.m4 for cvxopt in #29665, and it seems OK for me. Any comments?

@Volker-Weissmann
Copy link
Contributor

comment:10

If you take look at sage-env you see the line

PYTHONUSERBASE="$DOT_SAGE/local"

but I think it should be

PYTHONUSERBASE="$SAGE_ROOT/local"

, because (at least on my machine) the folder $DOT_SAGE/local does not exist.

@mkoeppe
Copy link
Member Author

mkoeppe commented May 8, 2020

comment:11

The way to enable system site packages is to use --system-site-packages when creating the venv.

Using PYTHONUSERBASE is not a suitable way forward.

@mkoeppe
Copy link
Member Author

mkoeppe commented May 8, 2020

comment:12

(And neither is an ad-hoc modification of PYTHONPATH proposed in #29665.)

@mkoeppe
Copy link
Member Author

mkoeppe commented May 8, 2020

comment:13

Replying to @Volker-Weissmann:

the folder $DOT_SAGE/local does not exist.

Users create this folder if they want to install user packages, for example using sage -pip install --user ...

@Volker-Weissmann
Copy link
Contributor

comment:14

Ok. Thank you. I thought it was a bug. But this means that if you run

sudo pip uninstall cython
pip install cython

cou can't build sage.

@mkoeppe
Copy link
Member Author

mkoeppe commented May 8, 2020

comment:15

Well, isolating the Sage user installation scheme from the Python user installation scheme was done deliberately in the past to avoid possible breakage by incompatible installations.

This decision may have to be revisited at some point, but I don't think it's necessary for this ticket.

Sage's install scripts will continue to make sure that the Sage venv has all necessary packages - whether user of system site packages is enabled or not.

@mkoeppe
Copy link
Member Author

mkoeppe commented May 8, 2020

comment:16

Just a quick note that I plan to work on this for Sage 9.3, not earlier.

@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe mkoeppe changed the title In a python 3 build, use system Python packages Meta-ticket: In a python 3 build, use system Python packages May 8, 2020
@mkoeppe

This comment has been minimized.

Sign in to view
@Volker-Weissmann
Copy link
Contributor

comment:19

Replying to @mkoeppe:

Well, isolating the Sage user installation scheme from the Python user installation scheme was done deliberately in the past to avoid possible breakage by incompatible installations.

It's not always easy to know what the programmers thought. What is the best source to learn things like this? Reading the tickets?

@mkoeppe
Copy link
Member Author

mkoeppe commented May 8, 2020

comment:20

That, and asking questions on the sage-devel list, I suppose.

@mkoeppe
Copy link
Member Author

mkoeppe commented May 8, 2020

comment:21

In the case of PYTHONUSERBASE, comments in sage-env point to some relevant tickets.

@Volker-Weissmann
Copy link
Contributor

comment:22

Thank you.

@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe
Copy link
Member Author

mkoeppe commented Jun 5, 2020

Changed dependencies from #27824 to none

@mkoeppe
Copy link
Member Author

mkoeppe commented Jun 5, 2020

comment:24

Before it makes sense to work on this, we should update all Python packages (#29141); in particular, setuptools, pip, ... (#29803)

@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe mkoeppe changed the title Meta-ticket: In a python 3 build, use system Python packages Meta-ticket: In a python 3 build, optionally use system Python packages Oct 22, 2020
@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe mkoeppe changed the title Meta-ticket: In a python 3 build, optionally use system Python packages Meta-ticket: ./configure --enable-system-site-packages (optionally use system Python packages) Nov 11, 2020
@mkoeppe
Copy link
Member Author

mkoeppe commented Feb 13, 2021

comment:34

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

@mkoeppe mkoeppe modified the milestones: sage-9.3, sage-9.4 Feb 13, 2021
@mkoeppe mkoeppe modified the milestones: sage-9.4, sage-9.5 Jul 19, 2021
@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe mkoeppe modified the milestones: sage-9.5, sage-9.6 Dec 14, 2021
@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe mkoeppe modified the milestones: sage-9.6, sage-9.7 Mar 5, 2022
@mkoeppe mkoeppe modified the milestones: sage-9.7, sage-9.8 Aug 31, 2022
@mkoeppe mkoeppe modified the milestones: sage-9.8, sage-9.9 Jan 7, 2023
@embray embray mentioned this issue Jan 29, 2023
Open
This was referenced Jan 17, 2020
Closed
Open
Closed
@sheerluck
Copy link
Contributor

I wish https://github.com/sagemath/sage/actions had CI Linux system-site-packages workflow next to CI Linux

@mkoeppe mkoeppe removed this from the sage-10.0 milestone Mar 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
c: packages: standard p: major / 3 t: enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@tobihan @jhpalmieri @sheerluck @thierry-FreeBSD @mkoeppe @Volker-Weissmann