feat: Reduce UI Support #500
Conversation
…given pipeline Signed-off-by: DevilAeron <[email protected]>
…given pipeline Signed-off-by: DevilAeron <[email protected]>
Signed-off-by: DevilAeron <[email protected]>
Signed-off-by: DevilAeron <[email protected]>
Signed-off-by: DevilAeron <[email protected]>
Signed-off-by: DevilAeron <[email protected]>
Signed-off-by: DevilAeron <[email protected]>
Signed-off-by: DevilAeron <[email protected]>
Signed-off-by: veds-g <[email protected]>
veds-g
force-pushed
the
pipeline-watermark
branch
3 times, most recently
from
7ec4297
to
e3f7627
Compare
Signed-off-by: veds-g <[email protected]>
veds-g
force-pushed
the
pipeline-watermark
branch
from
e3f7627
to
a7ea0a5
Compare
There was a problem hiding this comment.
I am debating whether we should move PodWatermark under the UI section because it is not required otherwise. It can be derived from other functions. Personally, I do not want to change/add anything to the watermark code without a thorough review since it is very integral to data processing correctness.
@whynowy @chromevoid @ashwinidulams WDYT?
Signed-off-by: veds-g <[email protected]>
veds-g
force-pushed
the
pipeline-watermark
branch
from
848e256
to
5cecfb7
Compare
Signed-off-by: veds-g <[email protected]>
Signed-off-by: veds-g <[email protected]>
veds-g
force-pushed
the
pipeline-watermark
branch
from
c9e57d0
to
926e803
Compare
"edges" is a misnomer, what we are doing now is back-pressure on buffers. What that means is, for a reduce vertex a logical edge is backed by multiple buffers. @veds-g is showing backpressure per partition of an edge. |
Signed-off-by: veds-g <[email protected]>
veds-g
force-pushed
the
pipeline-watermark
branch
from
a3f35c0
to
b62fda8
Compare
| @@ -40,13 +44,17 @@ func GetVertexWatermarkFetchers(ctx context.Context, pipeline *v1alpha1.Pipeline | |||
| } | |||
|
|
|||
| for _, vertex := range pipeline.Spec.Vertices { | |||
| // key for fetcher map ~ vertexName/replicas | |||
| replicas := int64(1) | |||
There was a problem hiding this comment.
Shouldn't it be 0? For a sink, there's only 1 partition on its right, then we put the key as vertexName/replicaIndex, which is vertexName/0.
Signed-off-by: veds-g <[email protected]>
Fixes #470
Added pod level data for vertices - show metrics, watermarks, processing rates for individual pods in a vertex
Added edge level data - show back-pressure for each edge in case of multiple edges between vertices