cantera fails to build with Python 3.11: ValueError: invalid mode: 'rU'

[mefuller]

Reported in full at https://bugzilla.redhat.com/show_bug.cgi?id=2094258

Problem description

cantera fails to build with Python 3.11.0b3.

Steps to reproduce

ValueError: invalid mode: 'rU':
File "/builddir/build/BUILD/cantera-2.6.0/SConstruct", line 2020:
SConscript("build/python/SConscript")
File "/usr/lib/python3.11/site-packages/SCons/Script/SConscript.py", line 660:
return method(*args, **kw)
File "/usr/lib/python3.11/site-packages/SCons/Script/SConscript.py", line 597:
return _SConscript(self.fs, *files, **subst_kw)
File "/usr/lib/python3.11/site-packages/SCons/Script/SConscript.py", line 285:
exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
File "/builddir/build/BUILD/cantera-2.6.0/build/python/SConscript", line 98:
setup_cfg = localenv.SubstFile("setup.cfg", "setup.cfg.in")
File "/usr/lib/python3.11/site-packages/SCons/Util.py", line 742:
return self.method(*nargs, **kwargs)
File "/builddir/build/BUILD/cantera-2.6.0/site_scons/site_tools/subst.py", line 154:
return env.SubstGeneric(target,
File "/usr/lib/python3.11/site-packages/SCons/Environment.py", line 238:
return super().call(target, source, *args, **kw)
File "/usr/lib/python3.11/site-packages/SCons/Util.py", line 742:
return self.method(*nargs, **kwargs)
File "/usr/lib/python3.11/site-packages/SCons/Builder.py", line 662:
return self._execute(env, target, source, OverrideWarner(kw), ekw)
File "/usr/lib/python3.11/site-packages/SCons/Builder.py", line 568:
tlist, slist = self._create_nodes(env, target, source)
File "/usr/lib/python3.11/site-packages/SCons/Builder.py", line 530:
target, source = self.emitter(target=tlist, source=slist, env=env)
File "/builddir/build/BUILD/cantera-2.6.0/site_scons/site_tools/subst.py", line 124:
keys = _subst_keys(path, pattern)
File "/builddir/build/BUILD/cantera-2.6.0/site_scons/site_tools/subst.py", line 49:
f = open(source, "rU")
error: Bad exit status from /var/tmp/rpm-tmp.S5BgPF (%build)

open(), io.open(), codecs.open() and fileinput.FileInput no longer accept 'U' (“universal newline”) in the file mode. This flag was deprecated since Python 3.3. In Python 3, the “universal newline” is used by default when a file is open in text mode. The newline parameter of open() controls how universal newlines works. (Contributed by Victor Stinner in bpo-37330.)

https://bugs.python.org/issue?@action=redirect&bpo=37330
https://docs.python.org/3.11/whatsnew/3.11.html

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.11/fedora-rawhide-x86_64/04495077-cantera/

For all our attempts to build cantera with Python 3.11, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.11/package/cantera/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.11:
https://copr.fedorainfracloud.org/coprs/g/python/python3.11/

Let us know here if you have any questions.

Python 3.11 is planned to be included in Fedora 37. To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.11.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

System information

• Cantera version: 2.6.0 release
• OS: Fedora Linux 37/rawhide
• Python/MATLAB/other software versions: Python 3.11b

[mefuller]

I fixed the above error in (draft) PR #1326, but then we get another error:

/usr/bin/python3 -c "import Cython.Build; Cython.Build.cythonize('build/python/cantera/_cantera.pyx')"
Compiling build/python/cantera/_cantera.pyx because it changed.
[1/1] Cythonizing build/python/cantera/_cantera.pyx
g++ -o build/temp-py/_cantera.os -c -std=c++11 -pthread -O3 -Wno-inline -g -fPIC -DNDEBUG -Iinclude -I/usr/include/python3.11 -I/usr/lib64/python3.11/site-packages/numpy/core/include -I/usr/include/eigen3 build/python/cantera/_cantera.cpp
In file included from /usr/lib64/python3.11/site-packages/numpy/core/include/numpy/ndarraytypes.h:1960,
                 from /usr/lib64/python3.11/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /usr/lib64/python3.11/site-packages/numpy/core/include/numpy/arrayobject.h:5,
                 from build/python/cantera/_cantera.cpp:749:
/usr/lib64/python3.11/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
In file included from /usr/include/python3.11/Python.h:38,
                 from build/python/cantera/_cantera.cpp:20:
/usr/include/python3.11/pyport.h: In instantiation of 'type {anonymous}::_Py_CAST_impl(const expr_type&) [with type = _object*; expr_type = int]':
build/python/cantera/_cantera.cpp:193144:9:   required from here
/usr/include/python3.11/pyport.h:47:24: error: invalid 'static_cast' from type 'int' to type '_object*'
   47 |                 return static_cast<type>(const_cast<expr_type &>(expr));
      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
scons: *** [build/temp-py/_cantera.os] Error 1
scons: building terminated because of errors.

[ischoegl]

For a recent discussion on Python 3.11, see #1287 ... I recall that there were some issues, and things were tabled. If you fix this, 3.11 should likely be added to CI (as long as the GH version is sufficiently recent). Tagging @bryanwweber on this ...

Here are some build logs with failures ... as things ran farther on Ubuntu/macOS/Windows, I am wondering whether there are any additional fedora-specific issues at play.

[bryanwweber]

The error in this message appears to be to do with Cython, which will have to be updated for Python 3.11. They may or may not have backported any required fixes to a 0.* release, or maybe it's only on main (their 3.0.0a*), or even maybe it's not been fixed yet 🤷‍♂️

[mefuller]

@ischoegl thanks - I wasn't aware of the previous discussion, but I feel pretty well caught up now.
@bryanwweber Fedora has a python 3.11 rebuild repo for testing: https://copr.fedorainfracloud.org/coprs/g/python/python3.11/, so the preceding is with a cython rebuilt against python 3.11

I'm building with the 2.6.0 release source code with only the modification in #1326 applied

We're anticipating running with 3.11 on Fedora 37, to be released in <=6 months, so my personal hope is to be able to build cantera 2.6 with python 3.11 by about 4 months from now.

[bryanwweber]

so the preceding is with a cython rebuilt against python 3.11

It's not just that Cython needs to be rebuilt, it sometimes needs to be updated in terms of the C++ code it generates to match changes in the Python C-API. If that hasn't been done yet, there's nothing we can do until they do. I'm not sure that's the problem, but since the error is due to an include file from Python itself, that's my interpretation of the error.

[hrnciar]

The error in this message appears to be to do with Cython, which will have to be updated for Python 3.11. They may or may not have backported any required fixes to a 0.* release, or maybe it's only on main (their 3.0.0a*), or even maybe it's not been fixed yet man_shrugging

Cython was updated for Python 3.11 on the main branch (3.0.0a*) and necessary commits were backported to 0.x branch. In Fedora we have Cython 0.29.30 which should be compatible with Python 3.11. If Canteras is broken by Cython it might be possible that there is some bug that needs to be fixed on Cython's side. Could you please open an issue there?

[hroncok]

Possibly this is python/cpython#93442

[mefuller]

Possibly this is python/cpython#93442

It is exactly that.
The latter issue is fixed by python/cpython#93442

@bryanwweber @ischoegl The original issue regarding deprecated 'rU' is the only issue in the Cantera source code currently preventing builds under Python 3.11
PR #1326 would close this issue, despite not being a comprehensive update for Python 3.11