RFC: Consider using CMake / meson? (3320)

[GalacticEmperor1]

Issue №3320 opened by h-vetinari at 2022-07-17 23:14:30

Hello all

I'm packaging pygame for conda-forge in conda-forge/staged-recipes#19665, as it's needed as a dependency for gym.

I'm having quite a hard time to get the DLL/header detection to get working across all platforms, and I'm somewhat surprised by the amount of handmade config scripting here, when all of these things have been pretty comprehensively solved by tools like CMake.

Considering the sheer amount of stuff under /buildconfig, I'm guessing this would be a non-trivial undertaking (and perhaps someone already tried and discarded that idea - I couldn't find something in the tracker from a quick search), so this is more of an RFC.

For the case of conda-forge (and other packagers / distros) in general, it's frowned upon to keep vendoring and rebuilding libs that are already available in the ecosystem - otherwise things just don't scale (or scale even less 😅). I managed to patch things together on linux & unix (sidenote: the LOCALPATH stuff from config_unix.py only works on linux, despite the filename implying osx as well), but windows eludes me so far - pointers would be appreciated (ideally in conda-forge/staged-recipes#19665, to keep this issue here focussed)! 🙃

Through the nested python scripts, the configuration errors on OSX/windows are also not propagated correctly (i.e. they don't fail the build). That too would be solved by a tool like CMake.

---

Comments

# # Starbuck5 commented at 2022-07-17 23:44:53

Thanks for trying to port pygame to conda! See # 1953

Yes, the buildconfig code is whack, but I don't think CMake is a solution.

We don't interact with compilers directly, we do it through distutils, because we're a python project, so I don't see how CMake would help out at all. I'm not aware of any python projects using this for their C extension compilation.

As far as I've seen, the buildconfig system is pretty simple in what it wants
On Mac and Linux, it wants the C deps to be installed globally somewhere (pretty standard for unix)
On Windows, it wants the C deps to be located in a folder called prebuilt-x64, or prebuilt-x86, depending on arch. (because there is no standard for windows)

I have no idea what conda does when pulling in dependency packages, or where they go.

---

# # h-vetinari commented at 2022-07-18 06:12:19

We don't interact with compilers directly, we do it through distutils, because we're a python project, so I don't see how CMake would help out at all. I'm not aware of any python projects using this for their C extension compilation.

Actually many projects do, and especially for detecting libraries and headers (which comes built-in for many common libraries, and can be easily set up for more niche ones). And more importantly, it's possible to do this in a mostly cross platform way, e.g. set a single variable to set up the prefix everywhere, resp. that config errors don't get eaten somewhere in python+templating+...

I'm not sure if you're aware, but distutils is deprecated and going away with python 3.12. Of course, setuptools will take over most of that, but it's also a chance to bring a very bespoke setup towards more modern (and maintainable) tools.

Perhaps you might also be interested in meson - it's more pythonic and has less of a weird DSL than CMake. It's also what SciPy moved to for the upcoming 1.9 release, and numpy will follow. Here's a blog post that explains the rationale.

---

# # h-vetinari commented at 2022-07-18 06:13:54

Here's a blog post that explains the rationale.

And of course, the SciPy repo contains a fully worked out setup these days.

PS. Thanks for the quick response!

---

# # ankith26 commented at 2022-07-22 03:39:05

Hi!

I personally think this is a sound proposal.

The current buildconfig code, while having served pygame well for many years, is now showing its age. Maintenance of this codebase is difficult (I mean TBF, porting to CMake or meson does not magically solve this fully either). Packaging things in C is generally a pain, and tools like CMake and meson have their own set of quirks. We are aware of the distutils deprecation, and we need an alternate working solution by mid-2023
One option is ofc, moving to standard setuptools which has its own distutils under the hood. This makes full sense for a small C-extension codebase, but this is not quite practical for large and complex projects like numpy and SciPy. As to which category pygame fits, it probably is somewhere in between, but is surely leaning towards the latter side.

For easier maintenance, it makes sense to follow what other leading C-extension projects do, because knowledge can be shared easily. If projects like numpy and SciPy have plans of a meson port and are working towards it, it's definitely not something pygame can outright reject implementing.
OFC, there are many differences between these projects. pygame only needs to worry with compiling pure C and Cython code (we don't have C++ or fortran code ATM). But pygame aims to support more platforms/configs than say, numpy. For instance we still have things like python 3.6 and manylinux1 and we don't have any plans ATM to be dropping either. We also have people working towards better android support and even a wasm port. So the meson/cmake port will probably take a lot of efforts, but hopefully it should be helpful in the long term

Currently, I am leaning on waiting a bit more to see how things progress on other projects going this way, and later making an informed decision after consensus with other active contributors.

---

# # eli-schwartz commented at 2022-07-22 03:55:21

For instance we still have things like python 3.6 and manylinux1 and we don't have any plans ATM to be dropping either.

Hmm, the opposite was said in pygame/pygame#3166 pullrequestreview-963104323

---

# # ankith26 commented at 2022-07-22 04:12:07

Well I don't take the final call on this 😄

That comment was more of a suggestion not a decision, although now I see it makes sense to hold onto 3.6 support for a bit longer, considering Ubuntu 18.04 is still in standard support (and this ships with python 3.6)

Relevant PR on this topic: pygame/pygame#3167

---

# # eli-schwartz commented at 2022-07-22 04:17:19

Yeah I noticed that PR too. The status seems to be best described as "confusing".

(I don't get it. It's end of life, and anyone using end of life pythons on distros whose big thing is LTS, otherwise known as "frozen in amber for stability purposes", is unlikely to update pygame either. For sure, Ubuntu won't update python3-pygame.)

---

# # illume commented at 2022-08-20 13:02:58

Currently, I am leaning on waiting a bit more to see how things progress on other projects going this way

Same. Will wait. buildconfig is fine for now, but can obviously use code quality improvements. Suggestions welcome in separate issues/PRs.

---

# # illume commented at 2022-08-20 17:52:42

I added an issue for buildconfig modernization. Probably the first step to replacing it is cleaning it up so the linter and type checker are at least running on it. pygame/pygame#3415 Part of that work I expect we can upstream some of those conda patches (with modifications probably). Less patches for the conda package will mean easier maintenance going forward.