Allow Cython module components to be compiled independently

[speth]

Changes proposed in this pull request

• Separate includes (both C++ and Cython) needed by each .pyx file so they can be compiled independently
• Combine the separately compiled submodules into a single shared library
• Use the solution from this StackOverflow post to import the resulting submodules from the same shared library

If applicable, provide an example illustrating new features this pull request is introducing

This change substantially speeds up compilation of the Cython module when compiling with multiple cores, or when making changes that only affect a subset of the .pyx files.

Timings for the current main branch:

# Compile everything
$ time scons build -j4 &>/dev/null

real    2m35.281s
user    5m58.835s
sys     0m18.177s

# Re-cythonize and compile just the Python module
$ rm -rf build/temp-py build/python
$ time scons build -j4 &>/dev/null

real    1m14.313s
user    1m10.681s
sys     0m5.905s

Timings for this PR:

# Compile everything
$ rm -rf build/temp-py build/python
$ time scons build -j4

real    2m5.328s
user    6m42.189s
sys     0m20.347s

# Re-cythonize and compile just the Python module
$ time scons build -j4

real    0m42.615s
user    1m54.346s
sys     0m7.753s

# With a lot of cores
$ time scons build -j16

real    0m26.209s
user    1m55.752s
sys     0m8.489s

# With just 2 cores
$ time scons build -j2
real    1m6.462s
user    1m54.324s
sys     0m7.826s

Note that the total CPU time increases by a fair amount, due to the duplicated work done in compiling some common code for each .pyx file. However, even with 2 cores, the wall-clock time decreases.

In addition, this provides some of the structure envisioned by Cantera/enhancements#153, in that the classes and methods are accessible using names qualified with the corresponding submodule, such as:

from cantera.reactor import IdealGasReactor
from cantera.onedim import FreeFlame

although they are also available directly in the cantera module as well. While I don't want to pursue this as part of the current PR, I think this restructuring also provides most of the necessary machinery to support building only a subset of the C++ source and the corresponding portions of the Python module, such as compiling everything except the reactor/flame models.

Checklist

• The pull request includes a clear description of this code change
• Commit messages have short titles and reference relevant issues
• Build passes (scons build & scons test) and unit tests address code coverage
• Style & formatting of contributed code follows contributing guidelines
• The pull request is ready for review

[codecov]

Codecov Report

Merging #1334 (24b5fc6) into main (0925fc0) will decrease coverage by 0.03%.
The diff coverage is 84.92%.

@@            Coverage Diff             @@
##             main    #1334      +/-   ##
==========================================
- Coverage   68.04%   68.00%   -0.04%     
==========================================
  Files         314      318       +4     
  Lines       42007    42007              
  Branches    16880    16880              
==========================================
- Hits        28582    28568      -14     
  Misses      11167    11167              
- Partials     2258     2272      +14     

Impacted Files	Coverage Δ
include/cantera/cython/utils_utils.h	63.15% <63.15%> (ø)
include/cantera/cython/kinetics_utils.h	82.85% <82.85%> (ø)
include/cantera/cython/thermo_utils.h	100.00% <100.00%> (ø)
include/cantera/cython/transport_utils.h	100.00% <100.00%> (ø)
include/cantera/cython/wrappers.h	100.00% <100.00%> (+14.84%)	⬆️
include/cantera/cython/funcWrapper.h	72.72% <0.00%> (-12.73%)	⬇️

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

[bryanwweber]

Just a few minor changes here @speth for consistency/cleanup. Thanks for taking this on!

[ischoegl]

🎉 Looks good to me, but it may be good to time the merge together with a resolution of #1258? Then merge conflicts in other PR’s can be resolved in a single pass.

[speth]

I was thinking about tackling #1258 next, but I don't know that there's anything that particularly helps with the merge conflicts in branches/PRs that already exist. Having to resolve merge conflicts in more files at once doesn't really make things simpler. I think the most useful thing to do is merge this now (assuming there are no further comments) so that no new branches/PRs will be created that don't already incorporate these changes.

[ischoegl]

@speth … good to hear that #1258 is next, so your suggestion is good. @bryanwweber … do you have anything else?

[bryanwweber]

@speth I added the license header to all the files in the Cython folder, I'm not sure if it was intentional to avoid adding it to the .pxd files, but some of them had it and some of them didn't, so I figured adding it everywhere was the best course. Assuming CI passes, this LGTM

[ischoegl]

Looks good to me as well. @bryanwweber, please approve if you’re fine with everything

[speth]

I thought I had added the license header everywhere. Some of that must have gotten lost when I was fixing the merge conflicts to rebase past #1327 being merged.

[bryanwweber]

Thanks @speth! In it goes!