GH-35176: [C++] Add support for disabling threading for emscripten

[joemarshall]

As previously discussed in #35176 this is a patch that adds an option ARROW_DISABLE_THREADING. When it is turned on, arrow threadpool and serial executors don't spawn threads, and instead run tasks in the main thread when futures are waited for.

It doesn't mess with threading in projects included as dependencies, e.g. multithreaded malloc implementations because if you're building for a non threaded environment, you can't use those anyway.

Basically where this is at is that it runs the test suite okay, and I think should work well enough to be a backend for pandas on emscripten/pyodide, so subject to reviews etc. it is worth merging (and would be jolly convenient for me if it was).

What this means is:

1. It is possible to use arrow in non-threaded emscripten/webassembly environments (with some build patches specific to emscripten which I'll put in once this is in)
2. Most of arrow just works, albeit slower in parts.

Things that don't work and probably won't:

1. Server stuff that relies on threads. Not a massive problem I think because environments with threading restrictions are currently typically also restricted from making servers anyway (i.e. they are web browsers)
2. Anything that relies on actually doing two things at once (for obvious reasons)

Things that don't work yet and could be fixed in future:

1. use of asynchronous file/network APIs in emscripten which would mean I/O could work efficiently in one thread.
2. asofjoin - right now the implementation relies on std::thread - it needs refactoring to work with threadpool like everything else in arrow, but I'm not sure I am expert enough in the codebase to do it well.

• Closes: [C++] Add support for disablging threading for emscripten #35176

[pitrou]

Hmm, I've started reviewing this PR but it adds a lot of code in various delicate internals (Future, ThreadPool, TaskGroup...) that makes me uneasy. Do we value emscripten/webassembly compat so much that it warrants the cost in maintainability? Would there be another way to support emscripten/webassembly?

[joemarshall]

I think I've done the current review comments - so as i understand it I can run the test from tasks.yml by commenting in here like this?

@github-actions crossbow submit test-ubuntu-22.04-cpp-no-threading

[joemarshall]

@github-actions crossbow submit test-ubuntu-22.04-cpp-no-threading

[github-actions]

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/4934990365

[pitrou]

@github-actions crossbow submit test-ubuntu-22.04-cpp-no-threading

[github-actions]

Command '['git', '-C', '/tmp/tmplwp1r9ca/arrow', 'fetch', 'origin', 'pull/35471/head:main']' returned non-zero exit status 128.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/4935043997

[kou]

@joemarshall Could you create a feature branch for this instead of using the main branch?
@github-actions crossbow submit test-ubuntu-22.04-cpp-no-threading doesn't work with the main branch.

[joemarshall]

@kou A feature branch in my repo? Or does someone need to make a feature branch in the main repo which I can target the pull request against?

[kou]

A feature branch in my repo?

Yes.

[joemarshall]

This is now in #35672 which is the same but from a differently named branch because that means that the CI stuff (crossbow submit commands) actually work, whereas they don't work if the name of the source branch is main.