JOIN Vertex to join data from more than one Vertices

[vigith]

Numaflow should have the ability to JOIN during the following scenarios.

JOIN by Key by Window

JOIN by Reduce and using Watermark to close the book. The output will be Keys with Value as an iterator. Please note that the JOIN could be from more than Reduce vertices or consists of one or more Map and Reduce vertices.

JOIN on Map/Sink

A pure non-join write to Sink or Map without really joining by any key.

Multiple Sources

This could be based on the previous two use-cases.

Cyclic Write

We can write back to itself or a previous vertex, but Watermark cannot retrogress, so this can only happen on a pipeline with only Map, and the data written back will implicitly be delayed.

Tasks

• A design doc which identifies all the changes needed and challenges
• Controller changes to be able to provision join vertex topologies
• Watermark changes to support multi-bucket [Map/reduce]
• Cycles for Join Vertex #934
• Document JOIN Vertex #935
• UI support for Join Vertex #936
• blog for JOIN feature #946

Tasks

Beta Give feedback

No tasks being tracked yet.

[juliev0]

@yhl25 Per discussion today, should this be reassigned to me or no?

[vigith]

@juliev0 let's start with JOIN on Map/Sink