Meta-ticket: Clean up src/setup.py to bring it to standard setuptools behavior

[mkoeppe]

Sub-tasks:

• Make sagelib setup.py self-contained and independent of SAGE_ROOT #21480: Make sagelib setup.py self-contained and independent of SAGE_ROOT
• Don't use make for autogenerated modules in sagelib #21613: Make setup.py not depend on make
• Cleaning up stale installed files in setup() #21604: Cleaning up stale installed files in setup()
  • See also src/setup.py: Disentangle cleaning of stale installed files in build directory and in install directory #21654: Disentangle cleaning of stale installed files in build directory and in install directory
• Use custom build_ext to compile Cython code #21600: Use custom build_ext to compile Cython code

In particular,

• src/setup.py --help should return immediately.
• src/setup.py clean should not build anything but just clean (and everything that is now done by make clean in the src directory should be done by it; see make sagelib a script package #29411)

Also the following should be done:

• Move all code from src/setup.py, src/fpickle_setup.py to sage_setup #29702: Move all code from src/setup.py, src/fpickle_setup.py to sage_setup
• Fix sagelib sdist (src/setup.py sdist) #21516: Fix sagelib sdist (src/setup.py sdist)
• Make src/setup.py respect --build-base and --inplace, independent of SAGE_CYTHONIZED #21535: Make src/setup.py respect --build-base and --inplace, independent of SAGE_CYTHONIZED
• Make sure src/setup.py respects --prefix, --install-base, --root, --user #21573: Make sure src/setup.py respects --install-base and --root
• Testsuite for src/setup.py #21678 Testsuite for src/setup.py
• sage_setup: Remove import-time dependency (setup_requires) on pkgconfig, numpy #30580 sage_setup: Remove import-time dependency (setup_requires) on pkgconfig, numpy

Also see:

• Use package_data instead of data_files in setup.py #20108: Use package_data instead of data_files in setup.py
• Add a separate "build_cython" command to setup.py #21682: Add a separate "cythonize" command to setup.py
• build the sage library in place #12659: build the sage library in place
• make sagelib use setuptools instead of distutils #13190: make sage use setuptools instead of distutils
• Replace use of module_list and OptionalExtension by extending find_python_sources #29701: Meta-ticket: Replace use of OptionalExtension by namespace packages

CC: @jdemeyer @vbraun @embray @nexttime @kiwifb @williamstein @mezzarobba @dimpase @jhpalmieri

Component: build

Issue created by migration from https://trac.sagemath.org/ticket/21508

[jdemeyer]

Dependencies: #20596

[embray]

comment:8

and everything that is now done by make clean should be done by it

By which you mean the makefile in src/ right?

[mkoeppe]

comment:9

Yes -- I've clarified it in the description.

[jdemeyer]

Changed dependencies from #20596 to #21600

[jdemeyer]

Changed dependencies from #21600 to #21604, #21600

[jdemeyer]

Changed dependencies from #21604, #21600 to #21480, #21604, #21600

[embray]

comment:13

Definitely keep in touch with me as a point of contact on any work you do on this--I know distutils + setuptools from top to bottom.

[mkoeppe]

comment:15

Let me just say that I consider #21613 not an important step towards this task ("... standard distutils behavior"). The present task ticket (#21508) is concerned with getting the behavior of setup.py standard (how does it respond to standard commands and options), not necessarily its implementation. The fact that setup.py with #21480 invokes make is an implementation detail, which is not very relevant for the goal of #21507.

[embray]

comment:16

Except it is pretty non-standard for it to have non-trivial side-effects (e.g. running a subprocess that generates files) unconditionally. Behavior-wise that should only happen when it's actually important for that to happen, such as during the build command.

[mkoeppe]

comment:17

Replying to @embray:

Except it is pretty non-standard for it to have non-trivial side-effects (e.g. running a subprocess that generates files) unconditionally. Behavior-wise that should only happen when it's actually important for that to happen, such as during the build command.

I agree that it should happen during the build command.

[embray]

comment:18

I will work on #21613, and you'll see it's not that big a deal.

[jdemeyer]

comment:19

Replying to @embray:

I will work on #21613, and you'll see it's not that big a deal.

Beware of conflicts with other tickets, in particular #21600 and its dependencies.

[embray]

comment:20

Yep. I don't think there will be significant conflicts, but I can make those tickets dependencies of #21613 and ensure any work I do there integrates well with the others before suggesting it to be merged.

[mkoeppe]

Changed dependencies from #21480, #21604, #21600 to none

[mkoeppe]

comment:35

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.