spkg-configure.m4 for python3 #27824

dimpase · 2019-05-13T15:52:41Z

Do not install python3 if a sufficiently new version is available in $PATH. This could be a system python3, or the python3 from within a venv.

As suggested in #29032, we make $SAGE_LOCAL a venv (https://docs.python.org/3/library/venv.html) over the system python3 -- if a suitable system python3 is found -- by using the venv.EnvBuilder API.

This is done by the new script build/bin/sage-venv, which we use both during configure (to make sure that we select a system python3 for which venvs work correctly) and during make.

The venv does not include the system site-packages: We continue to install all Python packages into our venv using the existing build infrastructure. We keep the task of using system site-packages for the follow-up ticket #29023.

(Note, we use venv (new since Python 3.3), not virtualenv. So this change is limited to Python 3 builds of Sage.)

CC: @jdemeyer @embray @vbraun @kiwifb @jhpalmieri @videlec

Component: build: configure

Author: Matthias Koeppe, Erik Bray

Branch: 1c845a4

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/27824

The text was updated successfully, but these errors were encountered:

embray · 2019-06-14T14:55:02Z

comment:3

As the Sage-8.8 release milestone is pending, we should delete the sage-8.8 milestone for tickets that are not actively being worked on or that still require significant work to move forward. If you feel that this ticket should be included in the next Sage release at the soonest please set its milestone to the next release milestone (sage-8.9).

embray · 2019-08-14T11:48:42Z

comment:4

For Python packages when installing using the system Python we definitely need a well thought out approach, likely with different options for handling questions such as where to install additional Python packages.

$SAGE_LOCAL already acts as a sort-of virtualenv, but we might still consider using virtualenv/venv in $SAGE_LOCAL (with the --system-site-packages option) as a way of extending sys.path while still using as many system Python packages as possible.

mkoeppe · 2020-01-14T22:27:03Z

comment:5

See #29013 - "In a python3 build, install all Python packages into a venv"

mkoeppe · 2020-01-15T02:05:52Z

Dependencies: #29013

mkoeppe · 2020-01-15T02:05:52Z

comment:6

Narrowed the ticket to py3.

mkoeppe · 2020-01-15T04:42:13Z

Branch: public/27824-python3-spkg-configure

mkoeppe · 2020-01-15T04:45:23Z

comment:8

Here's a version that works. The venv is created in build/make/Makefile target (in #29013, I created the venv in the spkg-postinst script of the python3 spkg.)

New commits:

`1363425`	`In a python3 build, install all Python packages into a venv`
`a394268`	`sage-pip-install: Use PYTHON=sage-python23`
`a84815b`	`spkg-configure.m4 for python3`

mkoeppe · 2020-01-15T04:45:23Z

Commit: a84815b

mkoeppe · 2020-01-15T04:48:24Z

Author: Matthias Koeppe

dimpase · 2020-01-15T10:32:08Z

comment:10

how about checking for Py3 deps: sqlite libpng bzip2 xz libffi ?
(I skipped in this list ones that are 2nd order, so no need to check them)

dimpase · 2020-01-15T11:56:27Z

comment:11

On Gentoo Python 3 does not have sqlite module by default. I had to do
$ pip install pysqlite3 --user, and still elliptic_curves spkg installation errors out.

sagetrac-git · 2020-01-15T14:37:08Z

Changed commit from a84815b to 0d0495f

sagetrac-git · 2020-01-15T14:37:08Z

Branch pushed to git repo; I updated commit sha1. New commits:

`0d0495f`	`build/pkgs/python3/spkg-configure.m4: Add SAGE_SPKG_DEPCHECK for sqlite libpng bzip2 xz libffi`

sagetrac-git · 2020-01-15T14:48:35Z

Branch pushed to git repo; I updated commit sha1. New commits:

`98dff9d`	`Remove use of --system-site-packages`

sagetrac-git · 2020-01-15T14:48:35Z

Changed commit from 0d0495f to 98dff9d

mkoeppe · 2020-01-15T14:50:07Z

comment:14

I've reduced the scope of the ticket, excluding the use of system site packages.

sagetrac-git · 2020-01-15T16:45:38Z

Changed commit from 98dff9d to 6180132

sagetrac-git · 2020-01-15T16:45:38Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6180132`	`build/pkgs/python3/spkg-configure.m4: Find @PYTHON_FOR_VENV@ that has sqlite3 module`

dimpase · 2020-01-15T18:01:08Z

comment:16

I tried this on gentoo (after a usual struggle to get sqlite3 module built in Python)
and it broke at docbuild, with

RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to work

even though libSingular has built just fine (it's some magic with from sage.env import SINGULAR_SO that is broken, so SINGULAR_SO is None)

mkoeppe · 2020-03-22T09:23:43Z

comment:93

Breaks python 2 builds

sagetrac-git · 2020-03-22T16:02:45Z

Branch pushed to git repo; I updated commit sha1. New commits:

`65a9f02`	`build/make/deps: Do not set PYTHON = python3_venv in a python2 build`

sagetrac-git · 2020-03-22T16:02:45Z

Changed commit from c570d6b to 65a9f02

mkoeppe · 2020-03-22T17:42:45Z

comment:96

Ready for review

mkoeppe · 2020-03-22T19:48:34Z

comment:97

New tests at https://github.com/mkoeppe/sage/actions/runs/60877259; see also #29367

sagetrac-git · 2020-03-23T00:47:23Z

Changed commit from 65a9f02 to 1c845a4

sagetrac-git · 2020-03-23T00:47:23Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1c845a4`	`build/pkgs/python3/spkg-configure.m4: Do not use venv for python3 if we do a python2 build`

mkoeppe · 2020-03-23T00:48:18Z

comment:99

Tests at https://github.com/mkoeppe/sage/actions/runs/61077298

dimpase · 2020-03-23T14:04:45Z

comment:100

Let's move this, looks good to me.

dimpase · 2020-03-23T14:04:45Z

Reviewer: Dima Pasechnik

mkoeppe · 2020-03-23T16:59:28Z

comment:101

Thanks!

vbraun · 2020-03-29T00:24:15Z

Changed branch from public/27824-python3-spkg-configure to 1c845a4

mezzarobba · 2020-05-18T14:10:34Z

Changed commit from 1c845a4 to none

mezzarobba · 2020-05-18T14:10:34Z

comment:103

Hello everyone,

When trying to upgrade to 9.1.rc5, I'm seeing an issue that I suspect could be a side effect of this ticket.

For some reason my sage/local/bin/python3.7 was a symlink to the system python binary. Sage decided it wanted to install its own python-3.7.3.p1, but that failed with a permission error on the symlink. After deleting the symlink, python builds and installs correctly. (I let the logs be overwritten, so I won't be able to provide much details — sorry.)

The overall Sage build still fails with an error about six not being found while building setuptools that I haven't yet investigated.

dimpase · 2020-05-18T17:09:10Z

comment:104

if you switch to py3 from py2, you probably need to run ./configure --with-python=3

it is not clear what version you were on, but the changes to the build system were very substantial, so ymmv.

mkoeppe · 2020-05-18T17:36:15Z

comment:105

Yes, this is a known limitation. I have created #29708 (Fix switching between --with-system-python3 and --without-system-python3) for it. We shall fix it in 9.2.

Remove crypt from the list of modules that we try to import when checking whether a given system python will work for our purposes, since: - It doesn't work on Windows (so it prevents us from running sagemath on Windows) - Searching the entire repo doesn't find any uses of crypt - crypt is deprecated anyways (PEP 594 has its removal scheduled for Python 3.13) The check for crypt was added at the same time as the other modules being checked, but neither the commit message for 6fcd7cd nor the discussion in Trac sagemath#27824 mentions why crypt specifically was added (Most of the relevant discussion in that ticket is about the distutils check).

dimpase added this to the sage-8.8 milestone May 13, 2019

dimpase added c: build: configure labels May 13, 2019

This comment has been minimized.

Sign in to view

embray removed this from the sage-8.8 milestone Jun 14, 2019

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~spkg-configure.m4 for pythons~~ spkg-configure.m4 for python3 Jan 15, 2020

mkoeppe added this to the sage-9.2 milestone Jan 15, 2020

This comment has been minimized.

Sign in to view

mkoeppe added s: needs work and removed s: needs review labels Mar 22, 2020

mkoeppe added s: needs review and removed s: needs work labels Mar 22, 2020

dimpase added s: positive review and removed s: needs review labels Mar 23, 2020

vbraun removed the s: positive review label Mar 29, 2020

vbraun closed this as completed in 7dc2587 Mar 29, 2020

This was referenced Jan 29, 2023

Meta-ticket: spkg-configure: Try to use as many system packages as possible #27330

Open

spkg-configure.m4 for python3 with SAGE_LOCAL as a venv for system pythons #29032

Closed

Support minimal system Python version 3.6 (for venv) #29033

Closed

tornaria mentioned this issue Feb 9, 2024

Prepare python3 spkg-configure.m4 for Python 3.12 (setuptools instead of distutils) #36983

Merged

5 tasks

spkg-configure.m4 for python3 #27824

spkg-configure.m4 for python3 #27824

Comments

dimpase commented May 13, 2019

This comment has been minimized.

This comment has been minimized.

embray commented Jun 14, 2019

embray commented Aug 14, 2019

mkoeppe commented Jan 14, 2020

This comment has been minimized.

mkoeppe commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

dimpase commented Jan 15, 2020

dimpase commented Jan 15, 2020

sagetrac-git mannequin commented Jan 15, 2020

sagetrac-git mannequin commented Jan 15, 2020

sagetrac-git mannequin commented Jan 15, 2020

sagetrac-git mannequin commented Jan 15, 2020

mkoeppe commented Jan 15, 2020

This comment has been minimized.

sagetrac-git mannequin commented Jan 15, 2020

sagetrac-git mannequin commented Jan 15, 2020

dimpase commented Jan 15, 2020

mkoeppe commented Mar 22, 2020

sagetrac-git mannequin commented Mar 22, 2020

sagetrac-git mannequin commented Mar 22, 2020

mkoeppe commented Mar 22, 2020

mkoeppe commented Mar 22, 2020

sagetrac-git mannequin commented Mar 23, 2020

sagetrac-git mannequin commented Mar 23, 2020

mkoeppe commented Mar 23, 2020

dimpase commented Mar 23, 2020

dimpase commented Mar 23, 2020

mkoeppe commented Mar 23, 2020

vbraun commented Mar 29, 2020

mezzarobba commented May 18, 2020

mezzarobba commented May 18, 2020

dimpase commented May 18, 2020

mkoeppe commented May 18, 2020