Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 1 - some packages without OptionalExtensions) #29706

mkoeppe · 2020-05-18T00:46:51Z

This simplifies src/module_list.py a lot. This is preparation for splitting sagelib into separate distutils packages.

Follow-up tickets take care of the remaining packages; in the end, in #29701, src/module_list.py will no longer be used, and in another follow-up ticket, it will disappear completely:

Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 2 - OptionalExtensions) #29720: Move Extension options ... (part 2 - OptionalExtensions)
Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 3: Get rid of uname_specific) #29785: Move Extension options ... (part 3: Get rid of uname_specific)
Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 4: sage.rings) #29786: Move Extension options ... (part 4: sage.rings)
Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 5: sage.graphs) #29790: Move Extension options ... (part 5: sage.graphs)
Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 6: last) #29791: Move Extension options ... (part 6: last)
coxeter3: Add spkg-configure.m4, use coxeter/ prefix for includes, move OptionalExtension options to .pyx file directives #29721: takes care of sage.libs.coxeter3
Replace use of module_list and OptionalExtension by extending find_python_sources #29701: takes care of sage.graphs.graph_decompositions

CC: @kiwifb @dimpase @jhpalmieri @tscrim @kliem @roed314 @videlec @vbraun

Component: refactoring

Keywords: sd109

Author: Matthias Koeppe

Branch/Commit: 1bfe30d

Reviewer: Jonathan Kliem, John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/29706

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-05-18T03:28:41Z

Branch: u/mkoeppe/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files

sagetrac-git · 2020-05-18T03:41:11Z

Commit: f31b6a0

sagetrac-git · 2020-05-18T03:41:11Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f31b6a0`	`src/sage/geometry: Move Extension options from src/module_list.py to distutils directives`

sagetrac-git · 2020-05-18T05:09:17Z

Changed commit from f31b6a0 to 346bd88

sagetrac-git · 2020-05-18T05:09:17Z

Branch pushed to git repo; I updated commit sha1. New commits:

`0c0ef43`	`src/sage/libs/ntl: Move Extension options from src/module_list.py to distutils directives`
`a4d04d3`	`src/sage/modular: Move Extension options from src/module_list.py to distutils directives`
`17ca1d7`	`src/sage/env.py (cython_aliases): Add aliases for libraries use in module_list.py`
`346bd88`	`src/sage/modules: Move Extension options from src/module_list.py to distutils directives`

mkoeppe · 2020-05-18T05:42:18Z

Author: Matthias Koeppe

sagetrac-git · 2020-05-18T05:54:42Z

Changed commit from 346bd88 to 70901ea

sagetrac-git · 2020-05-18T05:54:42Z

Branch pushed to git repo; I updated commit sha1. New commits:

`70901ea`	`src/sage/env.py (cython_aliases): Update doctest`

dimpase · 2020-05-18T09:53:53Z

comment:10

Aren't all these extra_compile_args=['-std=c++11'] (and the corresponding cython #-directives obsolete (as we now set this globally for the C++ compiler)?

mkoeppe · 2020-05-18T15:10:32Z

comment:11

I would guess so but would prefer to do that on a different ticket

sagetrac-git · 2020-05-21T05:07:36Z

Changed commit from 70901ea to 10f7542

sagetrac-git · 2020-05-21T05:07:36Z

Branch pushed to git repo; I updated commit sha1. New commits:

`eed920e`	`src/sage/tests: Move Extension options from src/module_list.py to distutils directives`
`0d25d1d`	`src/sage/structure: Move Extension options from src/module_list.py to distutils directives`
`808f46a`	`src/sage/stats: Move Extension options from src/module_list.py to distutils directives`
`10f7542`	`src/sage/schemes: Move Extension options from src/module_list.py to distutils directives`

kliem · 2020-05-27T18:17:33Z

comment:14

Isn't src/sage/geometry/triangulation/base.pyx itself redundant here?

-# distutils: sources = sage/geometry/triangulation/base.pyx sage/geometry/triangulation/functions.cc sage/geometry/triangulation/data.cc sage/geometry/triangulation/triangulations.cc
+# distutils: sources = sage/geometry/triangulation/functions.cc sage/geometry/triangulation/data.cc sage/geometry/triangulation/triangulations.cc

Same thing in

/src/sage/schemes/hyperelliptic_curves/hypellfrob.pyx
sage/stats/distributions/discrete_gaussian_integer.pyx)

I think you missed this one. At least I don't find the corresponding file in the diffs.

-    Extension('sage.modular.pollack_stevens.dist',
-              sources = ['sage/modular/pollack_stevens/dist.pyx'],
-              libraries = ["gmp", "zn_poly"],
-              extra_compile_args = ["-D_XPG6"]),

I don't understand (this is probably reasonable, but I don't have any knowledge about that)

    # TODO: Remove Cygwin hack by installing a suitable cblas.pc
    if os.path.exists('/usr/lib/libblas.dll.a'):
        aliases["CBLAS_LIBS"] = ['gslcblas']

Everything else looks great to me. However, I didn't test anything yet.

kliem · 2020-05-27T18:18:13Z

Changed keywords from none to sd109

mkoeppe · 2020-05-29T18:05:37Z

comment:16

Replying to @kliem:

Isn't src/sage/geometry/triangulation/base.pyx itself redundant here?
-# distutils: sources = sage/geometry/triangulation/base.pyx sage/geometry/triangulation/functions.cc sage/geometry/triangulation/data.cc sage/geometry/triangulation/triangulations.cc
+# distutils: sources = sage/geometry/triangulation/functions.cc sage/geometry/triangulation/data.cc sage/geometry/triangulation/triangulations.cc
Same thing in

/src/sage/schemes/hyperelliptic_curves/hypellfrob.pyx

sage/stats/distributions/discrete_gaussian_integer.pyx)

I think you missed this one. At least I don't find the corresponding file in the diffs.
-    Extension('sage.modular.pollack_stevens.dist',
-              sources = ['sage/modular/pollack_stevens/dist.pyx'],
-              libraries = ["gmp", "zn_poly"],
-              extra_compile_args = ["-D_XPG6"]),

Quite possible on both of these. If you have time to test and make these fixes, that would be great!

I don't understand (this is probably reasonable, but I don't have any knowledge about that)
    # TODO: Remove Cygwin hack by installing a suitable cblas.pc
    if os.path.exists('/usr/lib/libblas.dll.a'):
        aliases["CBLAS_LIBS"] = ['gslcblas']

I don't understand it either, I only it moved it there from its origin. It's quite likely that it is outdated - this should be addressed on a separate ticket.

kliem · 2020-05-30T07:48:44Z

comment:17

Ok. I'm testing it now both locally and with github actions.

kliem · 2020-05-30T18:54:38Z

comment:18

It all worked on my end (sage -t --all --long)

Testing it on github reavealed some strange error with debian buster standard.

https://github.com/kliem/sage-test-27122/runs/722743115

  [dochtml]     File "/sage/local/lib/python3.7/site-packages/sage/matrix/__init__.py", line 2, in <module>
  [dochtml]       import sage.matrix.args
  [dochtml]     File "sage/matrix/args.pyx", line 23, in init sage.matrix.args (build/cythonized/sage/matrix/args.c:21224)
  [dochtml]     File "/sage/local/lib/python3.7/site-packages/sage/matrix/matrix_space.py", line 44, in <module>
  [dochtml]       from . import matrix_modn_sparse
  [dochtml]     File "sage/matrix/matrix_integer_sparse.pxd", line 5, in init sage.matrix.matrix_modn_sparse (build/cythonized/sage/matrix/matrix_modn_sparse.cpp:16301)
  [dochtml]   ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /sage/local/lib/python3.7/site-packages/sage/matrix/matrix_integer_sparse.cpython-

How come numerical, rings etc. wheren't touched?

dimpase · 2020-05-30T20:46:01Z

comment:19

I also see this kind of strange error on GH Actions on buster. Perhaps the buster image used is faulty?

kliem · 2020-05-31T05:48:20Z

comment:20

Also happens on ubuntu bionic and linuxmint 19.

Overall the testing experience is awful at the moment and it appears to have nothing to do with this ticket: https://github.com/kliem/sage-test-27122/actions/runs/119764262

Except for ubuntu focal and eoan and debian bullseye and sid (all standard not minimal) nothing passes within 6 hours and many failures.

mkoeppe · 2020-06-06T19:12:34Z

Changed branch from c536daa to u/mkoeppe/c536daa32dcc69adbc7b37d9a170e22fd733bca8

kliem · 2020-06-06T19:18:14Z

Commit: b1b3787

kliem · 2020-06-06T19:18:14Z

comment:42

That fix looks plausible to me.

Last 10 new commits:

`17ca1d7`	`src/sage/env.py (cython_aliases): Add aliases for libraries use in module_list.py`
`346bd88`	`src/sage/modules: Move Extension options from src/module_list.py to distutils directives`
`70901ea`	`src/sage/env.py (cython_aliases): Update doctest`
`eed920e`	`src/sage/tests: Move Extension options from src/module_list.py to distutils directives`
`0d25d1d`	`src/sage/structure: Move Extension options from src/module_list.py to distutils directives`
`808f46a`	`src/sage/stats: Move Extension options from src/module_list.py to distutils directives`
`10f7542`	`src/sage/schemes: Move Extension options from src/module_list.py to distutils directives`
`5867c05`	`src/sage/modular/pollack_stevens/dist.pyx: Add missing distutils directives`
`c536daa`	`Remove self-listing in distutils sources directive`
`b1b3787`	`sage.env.cython_aliases: Fix for systems without zlib pc`

mkoeppe · 2020-06-06T19:25:33Z

comment:45

Thanks!

vbraun · 2020-06-21T19:53:22Z

comment:46

Still getting on OSX:

[sagelib-9.2.beta1] ************************************************************************
[sagelib-9.2.beta1] Traceback (most recent call last):
[sagelib-9.2.beta1]   File "setup.py", line 72, in <module>
[sagelib-9.2.beta1]     from module_list import ext_modules, library_order
[sagelib-9.2.beta1]   File "/Users/buildbot-sage/slave/sage_git/build/src/module_list.py", line 80, in <module>
[sagelib-9.2.beta1]     aliases = cython_aliases()
[sagelib-9.2.beta1]   File "/Users/buildbot-sage/slave/sage_git/build/src/sage/env.py", line 409, in cython_aliases
[sagelib-9.2.beta1]     aliases[var + "LIBEXTRA"] = list(filter(lambda s: not s.startswith(('-l','-L')), pkgconfig.libs(lib).split()))
[sagelib-9.2.beta1]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.7/site-packages/pkgconfig/pkgconfig.py", line 165, in libs
[sagelib-9.2.beta1]     _raise_if_not_exists(package)
[sagelib-9.2.beta1]   File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.7/site-packages/pkgconfig/pkgconfig.py", line 103, in _raise_if_not_exists
[sagelib-9.2.beta1]     raise PackageNotFoundError(package)
[sagelib-9.2.beta1] pkgconfig.pkgconfig.PackageNotFoundError: zlib not found
[sagelib-9.2.beta1] ************************************************************************
[sagelib-9.2.beta1] Error building the Sage library

sagetrac-git · 2020-06-21T21:13:54Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1bfe30d`	`sage.env.cython_aliases: Another fix for macOS without zlib pc`

sagetrac-git · 2020-06-21T21:13:54Z

Changed commit from b1b3787 to 1bfe30d

mkoeppe · 2020-06-21T21:14:19Z

comment:48

Sorry, here's another fix.

jhpalmieri · 2020-06-22T05:26:10Z

comment:49

Any suggestions for how to reproduce the error? I've tried using the previous version of the branch, and either building with lots of system packages or with very few, but I haven't hit this build error.

mkoeppe · 2020-06-22T05:28:03Z

comment:50

Basically, one needs a macOS without homebrew (which installs .pc files for system libraries). Or do the following to simulate that:

  (cd /usr/local/Homebrew/Library/Homebrew/os/mac/ && mv pkgconfig pkgconfig-moved)

mkoeppe · 2020-06-22T05:30:25Z

comment:51

I created #29929 (tox.ini: Add a macos environment without homebrew, conda) so we can test such a barebones macOS environment on GH Actions

kliem · 2020-06-22T09:04:24Z

comment:52

Have you tried the current branch with such a system? I am completely clueless, whether the last commit resolves the problem.

mkoeppe · 2020-06-22T21:34:40Z

comment:53

Replying to @kliem:

Have you tried the current branch with such a system? I am completely clueless, whether the last commit resolves the problem.

Yes, a build from scratch with the manipulation mentioned in comment 50 just finished successfully on my machine.

jhpalmieri · 2020-06-22T22:04:25Z

comment:54

I have not yet been able to reproduce the error, so I can't tell if the most recent change fixes anything. Renaming /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig didn't cause breakage, nor did renaming /usr/local/lib/pkgconfig. I'm trying on a different machine by just removing homebrew altogether.

jhpalmieri · 2020-06-22T22:57:43Z

comment:55

I can confirm on a machine without Homebrew, I hit the failure that Volker mentioned, and the most recent change fixes it. I'm going to switch to a positive review.

mkoeppe · 2020-06-22T23:31:48Z

Changed reviewer from Jonathan Kliem to Jonathan Kliem, John Palmieri

mkoeppe · 2020-06-22T23:31:48Z

comment:56

Thank you!

vbraun · 2020-06-24T23:38:17Z

Changed branch from u/mkoeppe/c536daa32dcc69adbc7b37d9a170e22fd733bca8 to 1bfe30d

mkoeppe added this to the sage-9.2 milestone May 18, 2020

mkoeppe added t: refactoring labels May 18, 2020

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Move Extension options from src/module_list.py to "distutils:" directives in the individual files~~ Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 1) May 18, 2020

mkoeppe added the s: needs review label May 18, 2020

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 1)~~ Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 1 - packages without OptionalExtensions) May 21, 2020

vbraun reopened this Jun 6, 2020

mkoeppe added the s: needs review label Jun 6, 2020

kliem added s: positive review and removed s: needs review labels Jun 6, 2020

vbraun added s: needs work and removed s: positive review labels Jun 21, 2020

mkoeppe added s: needs review and removed s: needs work labels Jun 21, 2020

jhpalmieri added s: positive review and removed s: needs review labels Jun 22, 2020

vbraun removed the s: positive review label Jun 24, 2020

vbraun closed this as completed in 54b9dba Jun 24, 2020

Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 1 - some packages without OptionalExtensions) #29706

Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 1 - some packages without OptionalExtensions) #29706

Comments

mkoeppe commented May 18, 2020

This comment has been minimized.

mkoeppe commented May 18, 2020

sagetrac-git mannequin commented May 18, 2020

sagetrac-git mannequin commented May 18, 2020

sagetrac-git mannequin commented May 18, 2020

sagetrac-git mannequin commented May 18, 2020

mkoeppe commented May 18, 2020

This comment has been minimized.

This comment has been minimized.

sagetrac-git mannequin commented May 18, 2020

sagetrac-git mannequin commented May 18, 2020

This comment has been minimized.

dimpase commented May 18, 2020

mkoeppe commented May 18, 2020

sagetrac-git mannequin commented May 21, 2020

sagetrac-git mannequin commented May 21, 2020

kliem commented May 27, 2020

kliem commented May 27, 2020

mkoeppe commented May 29, 2020

kliem commented May 30, 2020

kliem commented May 30, 2020

dimpase commented May 30, 2020

kliem commented May 31, 2020

mkoeppe commented Jun 6, 2020

kliem commented Jun 6, 2020

kliem commented Jun 6, 2020

mkoeppe commented Jun 6, 2020

vbraun commented Jun 21, 2020

sagetrac-git mannequin commented Jun 21, 2020

sagetrac-git mannequin commented Jun 21, 2020

mkoeppe commented Jun 21, 2020

jhpalmieri commented Jun 22, 2020

mkoeppe commented Jun 22, 2020

mkoeppe commented Jun 22, 2020

kliem commented Jun 22, 2020

mkoeppe commented Jun 22, 2020

jhpalmieri commented Jun 22, 2020

jhpalmieri commented Jun 22, 2020

mkoeppe commented Jun 22, 2020

mkoeppe commented Jun 22, 2020

vbraun commented Jun 24, 2020