Fix sagelib sdist (src/setup.py sdist) #21516

mkoeppe · 2016-09-17T04:36:36Z

This ticket adds some targets to src/Makefile.in: sdist and sdistcheck.
The latter, after building an sdist (using distutils), unpacks it into a subdirectory, and builds and installs (into SAGE_LOCAL) from there.

(cd src && make sdist) gives the following warnings. They need fixing.

warning: sdist: standard file not found: should have one of README, README.txt

(this one is #21565)

reading manifest template 'MANIFEST.in'
warning: sdist: MANIFEST.in, line 6: 'recursive-include' expects <dir> <pattern1> <pattern2> ...

warning: no previously-included files matching '*.h' found under directory 'sage/ext/interpreters'
warning: no previously-included files found matching 'sage/libs/pari/gen.h'
warning: no previously-included files found matching 'sage/modular/arithgroup/farey_symbol.h'
warning: no previously-included files found matching 'sage/rings/real_mpfi.h'
warning: no previously-included files found matching 'sage/symbolic/pynac.h'
warning: no directories found matching 'doc/common/static'
warning: no files found matching 'doc/en/bordeaux_2008/birch.png'
warning: no files found matching 'doc/en/bordeaux_2008/modpcurve.png'
no previously-included directories found matching 'doc/output'

There is something more seriously wrong with the sdist. The sage that is built from there (using (cd src && make sdistcheck)) crashes as follows.

/Users/mkoeppe/s/sage/sage-rebasing/src/sage/ext/interpreters/wrapper_rdf.pxd in init sage.plot.plot3d.parametric_surface (build/cythonized/sage/plot/plot3d/parametric_surface.c:11409)()
      1 # Automatically generated by sage_setup/autogen/interpreters.pyc.  Do not edit!
      2 
      3 from cpython cimport PyObject
      4 
      5 from sage.ext.fast_callable cimport Wrapper
      6 
----> 7 cdef class Wrapper_rdf(Wrapper):
        global cdef = undefined
        global Wrapper_rdf = undefined
        global Wrapper = undefined
      8     cdef int _n_args
      9     cdef double* _args
     10     cdef int _n_constants
     11     cdef double* _constants
     12     cdef object _list_py_constants
     13     cdef int _n_py_constants
     14     cdef PyObject** _py_constants
     15     cdef int _n_stack
     16     cdef double* _stack
     17     cdef int _n_code
     18     cdef int* _code
     19     cdef object _domain
     20     cdef bint call_c(self,
     21                      double* args,
     22                      double* result) except 0

ImportError: No module named wrapper_rdf

This needs fixing.

The branch is on top of #21480.

References:

Other technologies:

See also: #29845 - PEP 517 buildapi for sage_setup

Depends on #13190

CC: @jdemeyer @vbraun @embray @nexttime @kiwifb @dimpase @orlitzky @fchapoton

Component: build

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/21516

The text was updated successfully, but these errors were encountered:

mkoeppe · 2016-09-17T04:41:29Z

Branch: u/mkoeppe/sagelib_sdist

mkoeppe · 2016-09-17T04:44:54Z

Dependencies: #21480

mkoeppe · 2016-09-17T04:44:54Z

Commit: c7964a0

mkoeppe · 2016-09-17T04:44:54Z

Last 10 new commits:

`bd670af`	`Ignore generated files`
`751bd0f`	`Reword TODO item`
`3a8cc0e`	`Fix typo in comment`
`0dd9c50`	`Respect environment variable MAKE`
`17f90d8`	`beautification`
`e5f9065`	`More comments`
`7791cd9`	`Remove --buildbase code`
`74169e7`	`Pass SAGE_SRC to generate_py_source.mk`
`0394333`	`Add new file to MANIFEST.in`
`c7964a0`	`sdist* Makefile targets`

kiwifb · 2016-09-17T04:56:22Z

comment:3

Obviously you are missing an include that would point where to find sage/ext/fast_callable.*. I will have a deeper look.

kiwifb · 2016-09-17T05:06:55Z

comment:4

So this fails after you built and you try to run ./sage? There may be some points in sage that sets where to find the hierarchy of include files under SAGE_SRC when, once you have installed sage properly you should point to SAGE_LIB (usually SAGE_LOCAL/lib/python2.7/site-package).

If that's the case that may recoup with some concerns I expressed in one of the earlier tickets. sage's internal should be fixed before you do something like that to the build system. Off course I could be wrong :)

mkoeppe · 2016-09-17T05:10:59Z

comment:5

Replying to @kiwifb:

So this fails after you built and you try to run ./sage?

Yes.

There may be some points in sage that sets where to find the hierarchy of include files under SAGE_SRC when, once you have installed sage properly you should point to SAGE_LIB (usually SAGE_LOCAL/lib/python2.7/site-package).

I don't understand, could you explain more?

kiwifb · 2016-09-17T05:19:58Z

comment:6

Is SAGE_SRC pointing to anything or is it blank? From the little bit I see, I am guessing some piece of code is autogenerated at runtime to be compiled and executed. And to work, it needs to have an include (-I$SOMEPATH) that points to the top of the sage python install (SAGE_LIB) or as it is usually wrongly done in sage, where the source is in SAGE_SRC. You would find this in sage/misc/cython.py.

Have to prepare dinner for my family. Be back in about 2-3 hours.

sagetrac-git · 2016-09-17T06:03:25Z

Changed commit from c7964a0 to 4fe9088

sagetrac-git · 2016-09-17T06:03:25Z

Branch pushed to git repo; I updated commit sha1. New commits:

`4fe9088`	`New target: sdistcheck-compare-trees`

mkoeppe · 2016-09-17T06:07:48Z

comment:8

I added another debugging helper make sdistcheck-compare-trees, which shows the differences between the source tree and the sdist tree.

There are various differences (for example, all README files are missing); I won't have time to take a closer look before tomorrow.

mkoeppe · 2016-09-30T22:00:55Z

Changed dependencies from #21480 to #21480, #21565

sagetrac-git · 2016-10-23T01:31:48Z

Changed commit from 4fe9088 to 286adb0

sagetrac-git · 2016-10-23T01:31:48Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`f93a6a2`	`sdist* Makefile targets`
`286adb0`	`New target: sdistcheck-compare-trees`

sagetrac-git · 2016-10-23T05:39:09Z

Changed commit from 286adb0 to 6a314d7

sagetrac-git · 2016-10-23T05:39:09Z

Branch pushed to git repo; I updated commit sha1. New commits:

`4c6efbd`	`reconfigure if src/Makefile.in changed`
`be24c34`	`New target 'make sagelib-sdistcheck'`
`376bf1a`	`Refactor sdistcheck targets and fix dependencies`
`6a314d7`	`More work on sdistcheck-compare-trees`

mkoeppe · 2016-10-23T06:05:02Z

comment:12

This ticket needs some help from the distutils experts

embray · 2016-11-07T12:59:24Z

comment:13

What can I help with?

mkoeppe · 2016-11-10T06:48:36Z

comment:14

Replying to @embray:

What can I help with?

I'd need help in figuring out why some files are missing in the sdist and what is the idiomatic way (or the most appropriate way for Sage) of adding them.

embray · 2016-11-10T11:48:26Z

comment:15

I think this calls for a cleaning up of the MANIFEST.in. I'll play around with it.

embray · 2016-11-10T12:59:07Z

comment:16

I think that #21682 might help with this issue as well. As I've repeatedly suggested, it's best to include Cythonized sources in the sdist (taking the onus off the user to create the files with the correct version of Cython--especially important since we currently use a patched Cython for Sage...). This would also ensure that generated modules like the pari interfaces are included (on possible problem with this is if we want to be able to target different versions of pari--that's a problem to be pushed down the line though...)

mkoeppe · 2020-02-06T12:26:56Z

comment:17

@embray This is the ticket that I mentioned in #26964.

Shlokatadistance · 2020-03-30T05:45:20Z

comment:19

Can you tell me how did that error go from no previously-included files to. no directories found, the time I recreated this error, this happened after I had quit the command of ./sage

Shlokatadistance · 2020-04-04T12:49:01Z

comment:20

Also I had obtained such an error after a direct installation of SageMath as an application, did you do it the same way? or was it through the tar file and unpacking?

dimpase · 2020-05-16T12:33:48Z

comment:22

Is this still relevant? Has the world converged onto distributing via pip?

mkoeppe · 2020-05-16T15:36:01Z

comment:23

Yes, still need to fix this.

sagetrac-git · 2020-05-17T05:39:14Z

Changed commit from 6a314d7 to 086fc86

sagetrac-git · 2020-05-17T05:39:14Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`8ce279c`	`sdist* Makefile targets`
`3abffd8`	`New target: sdistcheck-compare-trees`
`9c7ecbe`	`New target 'make sagelib-sdistcheck'`
`4d8b72c`	`Refactor sdistcheck targets and fix dependencies`
`086fc86`	`More work on sdistcheck-compare-trees`

sagetrac-git · 2020-05-17T06:04:53Z

Changed commit from 086fc86 to 51c4da6

sagetrac-git · 2020-05-17T06:04:53Z

Branch pushed to git repo; I updated commit sha1. New commits:

`51c4da6`	`src/MANIFEST.in: prune .tox`

mkoeppe · 2020-06-11T01:56:10Z

Changed dependencies from #21480, #21565 to #13190

mkoeppe · 2020-06-25T06:35:43Z

comment:29

Development continues on #29950 (Build sagelib using the installed sage_setup, add spkg-src). This ticket can be closed.

dimpase · 2020-07-16T19:07:17Z

Reviewer: Dima Pasechnik

slel · 2020-10-11T16:57:56Z

Changed branch from u/mkoeppe/sagelib_sdist to none

slel · 2020-10-11T16:57:56Z

Changed author from Matthias Koeppe to none

slel · 2020-10-11T16:57:56Z

Changed commit from 51c4da6 to none

mkoeppe added this to the sage-7.4 milestone Sep 17, 2016

mkoeppe added c: build labels Sep 17, 2016

This comment has been minimized.

Sign in to view

mkoeppe modified the milestones: sage-7.4, sage-7.5 Oct 23, 2016

mkoeppe modified the milestones: sage-7.5, sage-9.1 Feb 6, 2020

mkoeppe modified the milestones: sage-9.1, sage-9.2 Apr 9, 2020

This comment has been minimized.

Sign in to view

mkoeppe removed this from the sage-9.2 milestone Jun 25, 2020

mkoeppe added the s: needs review label Jun 25, 2020

dimpase added s: positive review and removed s: needs review labels Jul 16, 2020

slel removed the s: positive review label Oct 11, 2020

slel closed this as completed Oct 11, 2020

slel added r: duplicate and removed p: major / 3 labels Oct 11, 2020

Fix sagelib sdist (src/setup.py sdist) #21516

Fix sagelib sdist (src/setup.py sdist) #21516

Comments

mkoeppe commented Sep 17, 2016

mkoeppe commented Sep 17, 2016

mkoeppe commented Sep 17, 2016

mkoeppe commented Sep 17, 2016

mkoeppe commented Sep 17, 2016

This comment has been minimized.

kiwifb commented Sep 17, 2016

kiwifb commented Sep 17, 2016

mkoeppe commented Sep 17, 2016

kiwifb commented Sep 17, 2016

sagetrac-git mannequin commented Sep 17, 2016

sagetrac-git mannequin commented Sep 17, 2016

mkoeppe commented Sep 17, 2016

mkoeppe commented Sep 30, 2016

This comment has been minimized.

sagetrac-git mannequin commented Oct 23, 2016

sagetrac-git mannequin commented Oct 23, 2016

sagetrac-git mannequin commented Oct 23, 2016

sagetrac-git mannequin commented Oct 23, 2016

mkoeppe commented Oct 23, 2016

embray commented Nov 7, 2016

mkoeppe commented Nov 10, 2016

embray commented Nov 10, 2016

embray commented Nov 10, 2016

mkoeppe commented Feb 6, 2020

Shlokatadistance mannequin commented Mar 30, 2020

Shlokatadistance mannequin commented Apr 4, 2020

dimpase commented May 16, 2020

mkoeppe commented May 16, 2020

sagetrac-git mannequin commented May 17, 2020

sagetrac-git mannequin commented May 17, 2020

sagetrac-git mannequin commented May 17, 2020

sagetrac-git mannequin commented May 17, 2020

This comment has been minimized.

mkoeppe commented Jun 11, 2020

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Jun 25, 2020

dimpase commented Jul 16, 2020

slel commented Oct 11, 2020

slel commented Oct 11, 2020

slel commented Oct 11, 2020