Filter out some distutils deprecation warnings with python 3.10 #33135

tornaria · 2022-01-09T19:51:10Z

As reported in #30766 comment:111

CC: @antonio-rojas

Component: misc

Author: Gonzalo Tornaría

Branch/Commit: 79234bb

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/33135

The text was updated successfully, but these errors were encountered:

tornaria · 2022-01-09T19:52:59Z

Branch: u/tornaria/deprecation_distutils

tornaria · 2022-01-09T19:52:59Z

New commits:

`836249c`	`trac 33135: add filter to ignore deprecation warnings`

tornaria · 2022-01-09T19:52:59Z

Author: Gonzalo Tornaría

tornaria · 2022-01-09T19:52:59Z

Commit: 836249c

kiwifb · 2022-01-09T20:03:47Z

comment:2

I see the warning a lot during docbuilding coming from matplotlib (building on sage-on-gentoo with python 3.10 right now)

[quadratic] /usr/lib/python3.10/site-packages/matplotlib/__init__.py:88: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives

for example. Should it be added to the warnings to catch?

tornaria · 2022-01-09T20:04:12Z

comment:3

I wonder if filters should strive be more specific (which would be itself a more precise form of documenting the filter). For example, the pythran warning in the context of this patch could have

message='`np.(bool|float|complex)` is a deprecated alias'

which catches the deprecation warnings and is explicit about what is being ignored.

WRT the current patch, there are two uses of distutils in sage source code:

distutils.sysconfig.get_python_inc() is deprecated, it has no replacement in sysconfig
distutils.errors.CCompilerError is not present in setuptools.errors

So I have no clue how to remove these two uses of distutil.

The third one is in numpy, not yet fixed upstream (see numpy/numpy#20225)

tornaria · 2022-01-09T20:08:25Z

comment:4

Replying to @kiwifb:

I see the warning a lot during docbuilding coming from matplotlib (building on sage-on-gentoo with python 3.10 right now)
[quadratic] /usr/lib/python3.10/site-packages/matplotlib/__init__.py:88: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
for example. Should it be added to the warnings to catch?

I don't get that one. Are you using matplotlib 3.5?

kiwifb · 2022-01-09T20:10:13Z

comment:5

Right, I am still at 3.4.3 in Gentoo (on the stable channel), switching to 3.5 is easy if that fixes it.

mkoeppe · 2022-01-09T20:21:21Z

Reviewer: Matthias Koeppe

kiwifb · 2022-01-09T20:24:59Z

comment:7

For distutils.sysconfig.get_python_inc() we could use sysconfig.get_config_var('INCLUDEPY') which should give the same value.

antonio-rojas · 2022-01-09T21:16:41Z

comment:8

I'm still getting lots of warnings coming from cython

sage -t --long --random-seed=173302895909105717773601440401632228334 /usr/lib/python3.10/site-packages/sage/env.py
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/env.py", line 440, in sage.env.?
Failed example:
    cython('''
    #distutils: extra_compile_args = OPENMP_CFLAGS
    #distutils: extra_link_args = OPENMP_CFLAGS
    from cython.parallel import prange

    cdef int i
    cdef int n = 30
    cdef int sum = 0

    for i in prange(n, num_threads=4, nogil=True):
        sum += i

    print(sum)
    ''')
Expected:
    435
Got:
    doctest:warning
      File "/usr/bin/sage-runtests", line 155, in <module>
        err = DC.run()
      File "/usr/lib/python3.10/site-packages/sage/doctest/control.py", line 1256, in run
        self.run_doctests()
      File "/usr/lib/python3.10/site-packages/sage/doctest/control.py", line 957, in run_doctests
        self.dispatcher.dispatch()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2004, in dispatch
        self.parallel_dispatch()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1899, in parallel_dispatch
        w.start()  # This might take some time
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2173, in start
        super(DocTestWorker, self).start()
      File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
        self._popen = self._Popen(self)
      File "/usr/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
        return _default_context.get_context().Process._Popen(process_obj)
      File "/usr/lib/python3.10/multiprocessing/context.py", line 277, in _Popen
        return Popen(process_obj)
      File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
        self._launch(process_obj)
      File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 71, in _launch
        code = process_obj._bootstrap(parent_sentinel=child_r)
      File "/usr/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
        self.run()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2145, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2475, in __call__
        doctests, extras = self._run(runner, options, results)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2527, in _run
        result = runner.run(test)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 861, in run
        return self._run(test, compileflags, out)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.env.?[5]>", line 1, in <module>
        cython('''
      File "/usr/lib/python3.10/site-packages/sage/misc/cython.py", line 661, in cython_compile
        return cython_import_all(tmpfile, get_globals(), **kwds)
      File "/usr/lib/python3.10/site-packages/sage/misc/cython.py", line 551, in cython_import_all
        m = cython_import(filename, **kwds)
      File "/usr/lib/python3.10/site-packages/sage/misc/cython.py", line 526, in cython_import
        name, build_dir = cython(filename, **kwds)
      File "/usr/lib/python3.10/site-packages/sage/misc/cython.py", line 288, in cython
        from Cython.Build import cythonize
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/Cython/Build/__init__.py", line 1, in <module>
        from .Dependencies import cythonize
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 17, in <module>
        from distutils.extension import Extension
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/distutils/__init__.py", line 19, in <module>
        warnings.warn(_DEPRECATION_MESSAGE,
      File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg
        sw(msg.message, msg.category, msg.filename, msg.lineno,
    :
    DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
    doctest:warning
      File "/usr/bin/sage-runtests", line 155, in <module>
        err = DC.run()
      File "/usr/lib/python3.10/site-packages/sage/doctest/control.py", line 1256, in run
        self.run_doctests()
      File "/usr/lib/python3.10/site-packages/sage/doctest/control.py", line 957, in run_doctests
        self.dispatcher.dispatch()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2004, in dispatch
        self.parallel_dispatch()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1899, in parallel_dispatch
        w.start()  # This might take some time
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2173, in start
        super(DocTestWorker, self).start()
      File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
        self._popen = self._Popen(self)
      File "/usr/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
        return _default_context.get_context().Process._Popen(process_obj)
      File "/usr/lib/python3.10/multiprocessing/context.py", line 277, in _Popen
        return Popen(process_obj)
      File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
        self._launch(process_obj)
      File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 71, in _launch
        code = process_obj._bootstrap(parent_sentinel=child_r)
      File "/usr/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
        self.run()
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2145, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2475, in __call__
        doctests, extras = self._run(runner, options, results)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 2527, in _run
        result = runner.run(test)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 861, in run
        return self._run(test, compileflags, out)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.env.?[5]>", line 1, in <module>
        cython('''
      File "/usr/lib/python3.10/site-packages/sage/misc/cython.py", line 661, in cython_compile
        return cython_import_all(tmpfile, get_globals(), **kwds)
      File "/usr/lib/python3.10/site-packages/sage/misc/cython.py", line 551, in cython_import_all
        m = cython_import(filename, **kwds)
      File "/usr/lib/python3.10/site-packages/sage/misc/cython.py", line 526, in cython_import
        name, build_dir = cython(filename, **kwds)
      File "/usr/lib/python3.10/site-packages/sage/misc/cython.py", line 288, in cython
        from Cython.Build import cythonize
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/Cython/Build/__init__.py", line 2, in <module>
        from .Distutils import build_ext
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/Cython/Build/Distutils.py", line 1, in <module>
        from Cython.Distutils.build_ext import build_ext
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/Cython/Distutils/__init__.py", line 1, in <module>
        from Cython.Distutils.build_ext import build_ext
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/site-packages/Cython/Distutils/build_ext.py", line 11, in <module>
        from distutils.command.build_ext import build_ext as _build_ext
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/distutils/command/build_ext.py", line 13, in <module>
        from distutils.sysconfig import customize_compiler, get_python_version
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/usr/lib/python3.10/distutils/sysconfig.py", line 58, in <module>
        warnings.warn(
      File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg
        sw(msg.message, msg.category, msg.filename, msg.lineno,
    :
    DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
    435
**********************************************************************

antonio-rojas · 2022-01-09T21:18:16Z

comment:9

Replying to @tornaria:

distutils.errors.CCompilerError is not present in setuptools.errors

I don't see any code in sagelib or cython that would emit this. Is this still needed at all?

tornaria · 2022-01-09T22:28:52Z

comment:10

I don't get any warnings from cython myself (I'm using the one installed by sage):

$ sage-9.5.rc0_1 
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.5.rc0, Release Date: 2022-01-09                 │
│ Using Python 3.10.1. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: cython('''
....: #distutils: extra_compile_args = OPENMP_CFLAGS
....: #distutils: extra_link_args = OPENMP_CFLAGS
....: from cython.parallel import prange
....: 
....: cdef int i
....: cdef int n = 30
....: cdef int sum = 0
....: 
....: for i in prange(n, num_threads=4, nogil=True):
....:     sum += i
....: 
....: print(sum)
....: ''')
435
sage:

Can you try again adding Cython to the module regex as in module='Cython|numpy|sage.env|sage.features'.

tornaria · 2022-01-09T22:39:49Z

comment:11

I moved the issue of removing distutils from the sage library to #33137 so this ticket can be merged quickly for 9.5.

antonio-rojas · 2022-01-10T18:14:47Z

comment:13

Adding Cython to the regex removes only half the warning. I had to add the distutils module too.

antonio-rojas · 2022-01-11T19:26:43Z

Changed branch from u/tornaria/deprecation_distutils to u/arojas/deprecation_distutils

sheerluck · 2022-01-12T15:06:13Z

Changed commit from 836249c to 79234bb

sheerluck · 2022-01-12T15:06:13Z

comment:16

https://github.com/sheerluck/sage/runs/4788938218 shows that with u/arojas/deprecation_distutils branch all 3.10-related-warnings disappeared and GH Actions for 3.9 and 3.10 give same result:

sage -t --random-seed=32 src/sage/interfaces/giac.py  # 6 doctests failed
sage -t --random-seed=32 src/sage/tests/cmdline.py  # 1 doctest failed

New commits:

`79234bb`	`Filter distutils warnings form more modules`

vbraun · 2022-01-12T17:17:47Z

Changed branch from u/arojas/deprecation_distutils to 79234bb

tornaria added this to the sage-9.5 milestone Jan 9, 2022

tornaria added p: blocker / 1 labels Jan 9, 2022

tornaria added the s: needs review label Jan 9, 2022

mkoeppe added s: positive review and removed s: needs review labels Jan 9, 2022

antonio-rojas added s: needs work and removed s: positive review labels Jan 9, 2022

mkoeppe added s: positive review and removed s: needs work labels Jan 11, 2022

mkoeppe changed the title ~~Fix deprecation warnings with python 3.10~~ Filter out some distutils deprecation warnings with python 3.10 Jan 11, 2022

vbraun removed the s: positive review label Jan 12, 2022

vbraun closed this as completed in 2e4b8be Jan 12, 2022

tornaria mentioned this issue Jan 7, 2023

sage.env.sage_include_directories: Don't use distutils and SAGE_LIB #33137

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filter out some distutils deprecation warnings with python 3.10 #33135

Filter out some distutils deprecation warnings with python 3.10 #33135

tornaria commented Jan 9, 2022

tornaria commented Jan 9, 2022

tornaria commented Jan 9, 2022

tornaria commented Jan 9, 2022

tornaria commented Jan 9, 2022

kiwifb commented Jan 9, 2022

tornaria commented Jan 9, 2022

tornaria commented Jan 9, 2022

kiwifb commented Jan 9, 2022

mkoeppe commented Jan 9, 2022

kiwifb commented Jan 9, 2022

antonio-rojas commented Jan 9, 2022

antonio-rojas commented Jan 9, 2022

tornaria commented Jan 9, 2022

tornaria commented Jan 9, 2022

antonio-rojas commented Jan 10, 2022

antonio-rojas commented Jan 11, 2022

sheerluck commented Jan 12, 2022

sheerluck commented Jan 12, 2022

vbraun commented Jan 12, 2022

Filter out some distutils deprecation warnings with python 3.10 #33135

Filter out some distutils deprecation warnings with python 3.10 #33135

Comments

tornaria commented Jan 9, 2022

tornaria commented Jan 9, 2022

tornaria commented Jan 9, 2022

tornaria commented Jan 9, 2022

tornaria commented Jan 9, 2022

kiwifb commented Jan 9, 2022

tornaria commented Jan 9, 2022

tornaria commented Jan 9, 2022

kiwifb commented Jan 9, 2022

mkoeppe commented Jan 9, 2022

kiwifb commented Jan 9, 2022

antonio-rojas commented Jan 9, 2022

antonio-rojas commented Jan 9, 2022

tornaria commented Jan 9, 2022

tornaria commented Jan 9, 2022

antonio-rojas commented Jan 10, 2022

antonio-rojas commented Jan 11, 2022

sheerluck commented Jan 12, 2022

sheerluck commented Jan 12, 2022

vbraun commented Jan 12, 2022