allow threading backend to be replaced by caller

[stevengj]

This PR adds a new function blosc_set_threads_callback that allows the caller to register a callback in order to change the threading backend at runtime — instead of spawning its own threads and passing work to them with a mutex, Blosc will pass an array of work (an array of thread_context) to the callback, which can execute the work in serial or in parallel using whatever mechanism it wants.

Motivation: this allows Blosc threading to be composable with caller multithreading, instead of having Blosc and the caller fight over the same CPUs. In particular, this enables Blosc threading to be composable with:

• Julia's new partr threading scheduler.
• Callers using Intels's Threading Building Blocks (TBB).
• Callers using Cilk, OpenMP, etcetera.

The design and implementation is very similar to the new pluggable threading backend for FFTW (FFTW/fftw3#175), which worked out very well for us. Aside from a trivial refactoring of the t_blosc function, the actual new code seems pretty minimal — only a couple of dozen lines, with a practical overhead of a single if statement in the parallel_blosc function for people not using this feature.

Marking it as a WIP since I haven't tested it yet other than checking that it compiles, but I wanted to get some early feedback. (Moved from Blosc/c-blosc#276)

[FrancescAlted]

Thanks for this cool contribution! So I have had a look into this and it looks like it is quite unobstrusive with the exisitng API; also, all tests are passing, which makes me confident about it. I'd like to see some benchmarks on how this works in real life (i.e. in combination with Julia's partr), but in principle I am +1 on merging this. Just tell me when this would be ready.

[stevengj]

Thanks @FrancescAlted!

I just updated the test suite so that Travis runs all the tests with both native threads and a dummy (serial) callback backend (controlled by a BLOSC_TEST_CALLBACK=yes environment variable in the test suite).

It looks like the callback code is working, so I will now work on updating the Julia Blosc.jl package to use blosc2 and let you know what the scaling looks like when I enable a partr backend.

[stevengj]

I just tried it with the Julia 1.3 partr backend by hacking Blosc.jl to install a callback, and got roughly the same speedup on my laptop as when using native threads (about a factor of 2 for 2 threads, for both compression and decompression, on an array of 10⁶ 64-bit integers randomly chosen in [1,10], which compresses about 90%).

So, it looks good to me to merge whenever you are ready.

[FrancescAlted]

Thanks @stevengj !