MapReduce framework in python coprocessor

[killme2008]

What problem does the new feature solve?

As we discussed before, if we can support MapReduce processing in python coprocessor, it would be beneficial to process extensive time-series data in parallel and distributed.

For example, we have a mapper and a reducer function written in python, try to processing data distributed in multi regions:

  Table data    Mapper                         

     region1       --->       data part1 -  >
     region2       --->       data part2   ->       reducer     ---> Result
        ...                                   ...
     region N      --->       data part N. ->

The user submits the functions, and it will split into multi tasks, every task running the mapper function with a region of the table data in parallel and distributed. The reducer function collects all data partitions generated by the mapper function ,calculates and returns the final result.

What does the feature do?

Described above.

Implementation challenges

Looks like we need the following:

• A job tracker to track all tasks, may be any frontend nodes.
• Programing API in python to write the mapper and reducer functions.
• The executor to execute the tasks, may be the python coprocessor engine.

[e1ijah1]

I'm interested in this issue, maybe we can break it down into tracking issues and start from the python api and job tracker? @killme2008

[killme2008]

@e1ijah1 Yep. I'm so sorry I didn't get back to you sooner. If you are interested, you can create a discussion or RFC that we can discuss in detail.

I haven't yet carefully thought over the API and operational flow.

[evenyag]

DataFrame API should be more convenient to use than mapreduce.

[discord9]

DataFrame API should be more convenient to use than mapreduce.

Yes, I guess we can just parallelize in PyDataFrame.collect()?

[e1ijah1]

Perhaps the DataFrame API is a better choice, as it enables optimizations such as columnar storage and vectorized computation, making it more efficient than MapReduce.

[tisonkun]

It seems we're working on GreptimeFlow for a standalone mapreduce execution engine.

This issue can be still valuable but it requires a more in details design. Move to open-ended discussion and we can come back if the design (draft) is present.