Task-pooling based parallelization & incremental output

[Profpatsch]

When scheduling longer tasks (like audio file conversion), all cores should be used by the script.
Right now Turtle only contains parallel, which will schedule as many parallel processes as fork allows and only return the results after every element has been processed.

I’d like to use something like async-pool, here’s a first try:

  finished <- liftIO $ Pool.withTaskGroup 4 $ \tg ->
    Pool.mapTasks tg $ map action infiles
  select finished >>= printf s

It uses exactly four cores now, but the output is still deferred until the very end; I’m not sure if it’s possible to integrate that into the streaming abstraction?

[Gabriella439]

To be precise, parallel doesn't exactly block until every element has been processed. For example, if the first element completes first, then parallel will emit it immediately and not wait for the remaining elements to complete. The only restriction is that parallel returns the elements in order, which means that parallel cannot return the second element before the first element.

The reason I'm explaining that is to point out that if you want something that produces results earlier than that then you need to drop the requirement to preserve the original order. Usually the way you do this is to have each action store the result in a buffer and then you stream the results from the buffer.

Going back to your original question: you can get the task-based approach to stream incrementally using the same trick. Have each task store its result in a buffer and then stream the results from the buffer.

[Profpatsch]

Ah, I understand how parallel can only produce one output per item; one is kind of trapped in IO anyway, since withAsync enforces it.

What would you use for the buffer? A TChan from stm?
Then if one would like pooling, it would have to be implemented with the Async.Pool withAsync anyway.

[Gabriella439]

Yeah, it could be any buffer