gh-90815: Add mimalloc memory allocator

[DinoV]

• Issue: Add mimalloc memory allocator #90815

This adds mimalloc as an optional (but preferred when available) allocator to CPython. This is a bit of a mashup of the work from #109914 and the work of @colesbury to use mimalloc for no-gil and various updates to bring it up to current CPython.

The configuration logic added by @tiran is re-used and we keep pymalloc support unlike in the version from @colesbury. mimalloc is updated to 2.12 and along with a few changes @colesbury made to it.

This has run into some issues with subinterpreter support in that the allocator's are now stored in thread state and are per-thread. Sub interpreters in some scenarios will create a thread state on one thread and run that on another thread. Most of these are documented in the code base as being known issues. I've modified these so that we will find the right thread based upon the current thread ID and switch to it rather than getting the head thread. This seems pretty reasonable but looks pretty weird when we need to do it at interpreter shutdown.

[ericsnowcurrently]

FTR, @tiran's gh-31164 is a similar change and has a bunch of discussion.

CC @daanx

[DinoV]

@ericsnowcurrently Yep, and much of the autoconf stuff is taken from there... I just finally filled in the description with a little more background on where things are coming from :)

[ericsnowcurrently]

This has run into some issues with subinterpreter support in that the allocator's are now stored in thread state and are per-thread.

I expect that most of this PR is a relatively vanilla use of mimalloc in CPython, like @tiran's PR is. Would it be much trouble to keep the whole PR vanilla and have a follow-up PR that applies the customizations to mimalloc (e.g. per-thread allocator state)? That distinction would help, possibly a lot, when reviewing.

[DinoV]

Okay, the default is now pymalloc, for now PYTHONMALLOC=mimalloc ./python will pick up mimalloc. When we start to get nogil integration with it we'll need to make it the default/required in that config.

[thesamesam]

Is it possible for mimalloc?

That's not possible because we will have changes to mimalloc.

Is there any chance of those changes getting upstreamed? There's advantages for cpython then too -- way less work to rebase the internal copy and no real risk of bugs being cpython introduced.

Any thoughts? I think this got missed earlier in the review flurry. I've not seen any substantive discussion about why bundling is a good idea, which feels like it's worth proper consideration, given it's short-term easier but long-term expensive.

I understand upstreaming it initially is going to take some time, but it is going to be worthwhile for cpython upstream maintenance long-term too, as you can avoid:

• having cpython being stuck on outdated versions in perpetuity, or
• figuring out how to rebase (and include the local changes)
• backporting fixes for various platforms/new compilers/etc to increasingly old versions of mimalloc

This also, of course, has the advantage of forcing one to document what the changes are which I think should be done either way.

[vstinner]

See also issue #111499: PYTHONMALLOCSTATS=1 fails with fatal error at Python exit. Using mimalloc, Python is not affected :-)

[vstinner]

@DinoV: I pushed a few cleanup changes. I fixed the outdated doc which still said that mimalloc is the default.

[vstinner]

I cleaned the commit message to only keep most important information.

The configuration logic added by @tiran is re-used

I added @tiran as a co-author in the commit message.

[ericsnowcurrently]

Is there any chance of those changes getting upstreamed? There's advantages for cpython then too -- way less work to rebase the internal copy and no real risk of bugs being cpython introduced.

Any thoughts?

It's worth further discussion.

CC @daanx

[vstinner]

Congrats @DinoV, I merged your PR :-) Do you want to propose a follow-up PR to enable it by default when --disable-gil is used?

[ericsnowcurrently]

Looks like this broke the WASM buildbot: https://buildbot.python.org/all/#/builders/1046/builds/3371.

[brettcannon]

Looks like this broke the WASM buildbot: https://buildbot.python.org/all/#/builders/1046/builds/3371.

Yep, looks like there's an implicit function definition:

In file included from ../../Objects/obmalloc.c:15:
In file included from ../../Objects/mimalloc/static.c:37:
In file included from ../../Objects/mimalloc/prim/prim.c:19:
../../Objects/mimalloc/prim/wasi/prim.c:44:15: error: implicit declaration of function 'sbrk' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    void* p = sbrk(size);
              ^
1 error generated.
make: *** [Makefile:2723: Objects/obmalloc.o] Error 1

[vstinner]

../../Objects/mimalloc/prim/wasi/prim.c:44:15: error: implicit declaration of function 'sbrk' is invalid in C99 [-Werror,-Wimplicit-function-declaration]

I wrote PR #111524 to fix WASI build.

[vstinner]

I wrote 4 follow-up PRs to fix different issues:

• gh-90815: Fix _PyMem_MimallocEnabled() compiler warning #111522
• gh-90815: Fix mimalloc build on WASI #111524
• gh-90815: Fix mimalloc atomic.h on Windows arm64 #111527
• gh-90815: Define WITH_MIMALLOC macro in PC/pyconfig.h #111528

While mimalloc C code is built on Windows, Python cannot currently use mimalloc on Windows: see PR #111528.