numpy incompatibility when using system python and numpy 1.22 is installed in the system #33473

tornaria · 2022-03-07T01:45:45Z

This retroactively affects sage-9.5, meaning sage-9.5 used to build ok in my system but now that system numpy has been upgraded to 1.22 it breaks. This is using the standard tarball, unpatched, as long as system python is used and numpy 1.22 happens to be installed in the system when sage is build.

NOTE: this is NOT trying to use system numpy. Sage builds and installs its own numpy (1.21) but headers for the system numpy (1.22) get incorrectly mixed up when building sagelib.

The symptom is that any sage module that uses cimport numpy will fail to load, making sage almost unusable, e.g.:

sage: import sage.rings.polynomial.real_roots
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
...
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Background:

numpy 1.21 and numpy 1.22 are ABI incompatible (ndarray changed size)
building sagelib needs to know location of include files:
- for python: since I'm using system python, this is a system include dir, here /usr/include/python3.10
- for numpy: since numpy is bundled with sage, this is a dir in the venv, here /home/tornaria/src/sage/sage-9.5/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/numpy/core/include
my system numpy (1.22) has its own headers installed in /usr/lib/python3.10/site-packages/numpy/core/include but there's also a symlink /usr/include/python3.10/numpy -> ../../lib/python3.10/site-packages/numpy/core/include/numpy.
in the CC command line the python include location (in the system) is before the numpy include location (in the venv)
since the system python contains the system numpy, all numpy headers will be included from system and not from the venv, triggering the incompatibility.

NOTE: the symlink /usr/include/python3.10/numpy is not shipped by numpy itself, but it seems some distros add it. I'm using void linux, which has it, but at least debian also has it (see https://packages.debian.org/bullseye/amd64/python3-numpy/filelist)

The solution should be changing the order of the include locations so that the numpy location comes before the python location.

CC: @vbraun

Component: build

Author: Gonzalo Tornaría

Branch: f50e4cb

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/33473

The text was updated successfully, but these errors were encountered:

tornaria · 2022-03-07T01:45:59Z

comment:1

It seems scipy may also be affected by this:

$ grep -l 'usr.*numpy.*\.h' logs/pkgs/*
logs/pkgs/sagelib-9.6.beta3.log
logs/pkgs/scipy-1.7.3.log

Indeed

$ grep 'usr.*numpy.*\.h' logs/pkgs/scipy-1.7.3.log 
  In file included from /usr/include/python3.10/numpy/ndarraytypes.h:1960,
                   from /usr/include/python3.10/numpy/ndarrayobject.h:12,
  /usr/include/python3.10/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
  In file included from /usr/include/python3.10/numpy/ndarraytypes.h:1960,
                   from /usr/include/python3.10/numpy/ndarrayobject.h:12,
                   from /usr/include/python3.10/numpy/arrayobject.h:5,
  /usr/include/python3.10/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]

we see that scipy also uses some incorrect headers (the warning is unrelated).

tornaria · 2022-03-07T02:13:04Z

comment:2

For sagelib, the includes are determined by

sage: sage.env.sage_include_directories()
['/home/tornaria/src/sage/sage-9.6.beta3/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages',
 '/usr/include/python3.10',
 '/home/tornaria/src/sage/sage-9.6.beta3/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/numpy/core/include']

I think modifying this function so that the numpy include location comes before the system python include location should fix the issue.

As for scipy, it seems the issue happens in module scipy.spatial as in:

sage: import scipy.spatial
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
...
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

mkoeppe · 2022-03-07T02:14:07Z

comment:3

What does python3.10 -m sysconfig | grep 'C.*FLAGS' show?

mkoeppe · 2022-03-07T02:16:26Z

comment:4

Replying to @tornaria:

For sagelib, the includes are determined by
sage: sage.env.sage_include_directories()
['/home/tornaria/src/sage/sage-9.6.beta3/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages',
 '/usr/include/python3.10',
 '/home/tornaria/src/sage/sage-9.6.beta3/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/numpy/core/include']
I think modifying this function so that the numpy include location comes before the system python include location should fix the issue.

Yes, this sounds like a good fix

tornaria · 2022-03-07T02:25:05Z

comment:5

Here's an attempt to a solution. I based it on 9.5 and I wonder if this justifies an eventual 9.5.1.

New commits:

`f50e4cb`	`numpy: prefer venv headers before system headers`

tornaria · 2022-03-07T02:25:05Z

Commit: f50e4cb

tornaria · 2022-03-07T02:25:05Z

Branch: u/tornaria/33473-numpy-includes

mkoeppe · 2022-03-07T18:19:26Z

Author: Gonzalo Tornaría

mkoeppe · 2022-03-07T18:24:58Z

comment:10

I've updated the description - Sage uses numpy 1.21, not 1.19

mkoeppe · 2022-03-07T18:28:42Z

Reviewer: Matthias Koeppe

mkoeppe · 2022-03-07T18:35:10Z

Replying to @tornaria:

NOTE: the symlink /usr/include/python3.10/numpy is not shipped by numpy itself, but it seems some distros add it. I'm using void linux, which has it, but at least debian also has it (see https://packages.debian.org/bullseye/amd64/python3-numpy/filelist)

Arguably, this is a distribution bug. But we definitely have to work around it, given that it is widespread in Debian. Right now it is not acute because no released version of Debian/Ubuntu ships numpy 1.22.x. But it will become a bigger problem when we upgrade to numpy 1.22 and then we face the issue on all versions of Debian/Ubuntu.

mkoeppe · 2022-03-07T18:52:55Z

comment:13

Replying to @tornaria:

Here's an attempt to a solution. I based it on 9.5 and I wonder if this justifies an eventual 9.5.1.

I don't think anyone is in the mood for making bugfix releases. But it looks like we might be ready for 9.6

tornaria · 2022-03-09T16:58:16Z

comment:15

Thanks.

I've reported this as a possible bug in the numpy package in void linux (void-linux/void-packages#36062), to try to get an opinion on the distro side.

We could try to get the scipy patch upstreamed.

BTW, I think eventually distutils.sysconfig.get_python_inc() should be changed to sysconfig.get_config_var("INCLUDEPY") since distutils is deprecated but sysconfig doesn't have a function get_python_inc().

tornaria · 2022-03-09T17:19:40Z

comment:16

I reverted the description by accident. Indeed sage 9.5 uses numpy 1.21 and the binary incompatibility arises in the upgrade to 1.22.

mkoeppe · 2022-03-09T17:30:13Z

comment:17

Replying to @tornaria:

We could try to get the scipy patch upstreamed.

+1

mkoeppe · 2022-03-09T17:31:15Z

comment:18

Replying to @tornaria:

BTW, I think eventually distutils.sysconfig.get_python_inc() should be changed to sysconfig.get_config_var("INCLUDEPY") since distutils is deprecated but sysconfig doesn't have a function get_python_inc().

Is it even needed or does setuptools already put this include on the command line?

tornaria · 2022-03-09T17:54:52Z

comment:19

Replying to @mkoeppe:

Replying to @tornaria:

BTW, I think eventually distutils.sysconfig.get_python_inc() should be changed to sysconfig.get_config_var("INCLUDEPY") since distutils is deprecated but sysconfig doesn't have a function get_python_inc().

Is it even needed or does setuptools already put this include on the command line?

No idea. If so, then why is sage.env.sage_include_directories() using distutils.sysconfig.get_python_inc at all?

Can we just

--- a/src/sage/env.py
+++ b/src/sage/env.py
@@ -377,8 +377,7 @@ def sage_include_directories(use_sources=False):
 
     TOP = SAGE_SRC if use_sources else SAGE_LIB
 
-    dirs = [TOP,
-            distutils.sysconfig.get_python_inc()]
+    dirs = [TOP]
     try:
         import numpy
         dirs.insert(1, numpy.get_include())

?

mkoeppe · 2022-03-09T18:05:05Z

comment:20

worth a try

vbraun · 2022-03-09T23:38:40Z

Changed branch from u/tornaria/33473-numpy-includes to f50e4cb

tornaria · 2022-03-10T16:44:51Z

Changed commit from f50e4cb to none

tornaria · 2022-03-10T16:44:51Z

comment:22

For the record:

numpy maintainer in void linux acknowledged this as a package bug and it's already fixed (numpy headers symlink breaks using numpy in a venv void-linux/void-packages#36062)
debian already had a bug for this filed in october 2021, maintainer is not sure this is a bug, I've added an explanation of how this is expected to break sage 9.5 at some point (atm debian unstable has numpy-1.21, but 1.22 is already in experimental and that's the cue to break sage 9.5) https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=998084#17
indeed it seems setuptools adds the python include location to the compiler command line, in fact with the following comment in setuptools/_distutils/command/build_ext.py:

        # Put the Python "system" include dir at the end, so that
        # any local include dirs take precedence.

tobiasdiez · 2022-03-22T22:17:47Z

comment:23

This now fails on the github action:

sage -t --random-seed=174296915894383156674390339433925467897 sage/env.py
**********************************************************************
File "sage/env.py", line 359, in sage.env.sage_include_directories
Failed example:
    sage.env.sage_include_directories()
Expected:
    ['.../site-packages',
     '.../site-packages/numpy/core/include',
     '.../include/python...']
Got:
    ['/__w/sagetrac-mirror/sagetrac-mirror/src',
     '/sage/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/numpy/core/include',
     '/usr/include/python3.8']

See https://github.com/sagemath/sage/runs/5522519846

mkoeppe · 2022-03-22T22:56:52Z

comment:24

Already fixed in #33532

tobiasdiez · 2022-03-22T22:58:41Z

comment:25

Replying to @mkoeppe:

Already fixed in #33532

Ah good!

mkoeppe · 2022-07-02T22:31:30Z

comment:26

Replying to @tornaria:

Can we just

--- a/src/sage/env.py
+++ b/src/sage/env.py
@@ -377,8 +377,7 @@ def sage_include_directories(use_sources=False):
 
     TOP = SAGE_SRC if use_sources else SAGE_LIB
 
-    dirs = [TOP,
-            distutils.sysconfig.get_python_inc()]
+    dirs = [TOP]
     try:
         import numpy
         dirs.insert(1, numpy.get_include())

?

Let's do this in #33137

tornaria added this to the sage-9.6 milestone Mar 7, 2022

tornaria added p: blocker / 1 labels Mar 7, 2022

tornaria added the s: needs review label Mar 7, 2022

mkoeppe added the c: build label Mar 7, 2022

This comment has been minimized.

Sign in to view

mkoeppe added s: positive review and removed s: needs review labels Mar 7, 2022

This comment has been minimized.

Sign in to view

vbraun removed the s: positive review label Mar 9, 2022

vbraun closed this as completed in eb21746 Mar 9, 2022

tornaria mentioned this issue Mar 21, 2022

Doctest failure when SAGE_LIB doesn't match .../site-packages #33532

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

numpy incompatibility when using system python and numpy 1.22 is installed in the system #33473

numpy incompatibility when using system python and numpy 1.22 is installed in the system #33473

tornaria commented Mar 7, 2022

tornaria commented Mar 7, 2022

tornaria commented Mar 7, 2022

mkoeppe commented Mar 7, 2022

mkoeppe commented Mar 7, 2022

tornaria commented Mar 7, 2022

tornaria commented Mar 7, 2022

tornaria commented Mar 7, 2022

mkoeppe commented Mar 7, 2022

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Mar 7, 2022

mkoeppe commented Mar 7, 2022

mkoeppe commented Mar 7, 2022

mkoeppe commented Mar 7, 2022

tornaria commented Mar 9, 2022

This comment has been minimized.

This comment has been minimized.

tornaria commented Mar 9, 2022

mkoeppe commented Mar 9, 2022

mkoeppe commented Mar 9, 2022

tornaria commented Mar 9, 2022

mkoeppe commented Mar 9, 2022

vbraun commented Mar 9, 2022

tornaria commented Mar 10, 2022

tornaria commented Mar 10, 2022

tobiasdiez commented Mar 22, 2022

mkoeppe commented Mar 22, 2022

tobiasdiez commented Mar 22, 2022

mkoeppe commented Jul 2, 2022

numpy incompatibility when using system python and numpy 1.22 is installed in the system #33473

numpy incompatibility when using system python and numpy 1.22 is installed in the system #33473

Comments

tornaria commented Mar 7, 2022

tornaria commented Mar 7, 2022

tornaria commented Mar 7, 2022

mkoeppe commented Mar 7, 2022

mkoeppe commented Mar 7, 2022

tornaria commented Mar 7, 2022

tornaria commented Mar 7, 2022

tornaria commented Mar 7, 2022

mkoeppe commented Mar 7, 2022

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Mar 7, 2022

mkoeppe commented Mar 7, 2022

mkoeppe commented Mar 7, 2022

mkoeppe commented Mar 7, 2022

tornaria commented Mar 9, 2022

This comment has been minimized.

This comment has been minimized.

tornaria commented Mar 9, 2022

mkoeppe commented Mar 9, 2022

mkoeppe commented Mar 9, 2022

tornaria commented Mar 9, 2022

mkoeppe commented Mar 9, 2022

vbraun commented Mar 9, 2022

tornaria commented Mar 10, 2022

tornaria commented Mar 10, 2022

tobiasdiez commented Mar 22, 2022

mkoeppe commented Mar 22, 2022

tobiasdiez commented Mar 22, 2022

mkoeppe commented Jul 2, 2022