Kafka perf improvements

[slinkydeveloper]

This is the result of investigating the Kafka ingress performance.

Before:

After:

Red line is Bifrost append thpt, green line is PP Invoke command thpt. The kafka load tool generates as much load as possible (around 25k/s records on my machine).

In both situations the initial slow section seems to be caused by the load tool, which takes a good amount of my cpu. After it finishes producing, restate takes all the CPU.

In the after case, the kafka container OOMs before finishing (probably caused by the high load generated by the consumer), so I cut the section afterwards.

I ran this test using Rust SDK and a virtual object as target, and the following subscription:

http localhost:9070/subscriptions source="kafka://my-cluster/test-topic" sink="service://Greeter/greet" \
  options["fetch.queue.backoff.ms"]="500" \
  options["queued.max.messages.kbytes"]="131072" \
  options["reconnect.backoff.max.ms"]="1000"

The Kafka topic has 24 partitions (same number of Restate's partitions). The thpt improvement in this PR is greatly affected by the Kafka topic partition number, meaning higher number of Kafka partitions equals to higher throughput.

Tuning the knobs though has irrelevant impact most of the times (at least on my machine). I tried to tune fetch.wait.max.ms too, just increases CPU usage.

[slinkydeveloper]

In the last commit I've added a metric to monitor the consumer group too, blue line here:

[slinkydeveloper]

The last commit I've added optimizes for reading as fast as possible from Kafka, reaching a much higher throughput (but potentially overloading Bifrost?!)

This test was run with Kafka partitions = 3

Not sure if we should do this.

[slinkydeveloper]

This send down the hood awaits on the bifrost append.

[slinkydeveloper]

After some thinking, perhaps for the time being, it just makes sense to remove this commit with the buffering. In any case this code will dramatically change when #1651 happens.

[slinkydeveloper]

Removed it.

[slinkydeveloper]

What's your take on this @AhmedSoliman ?

[AhmedSoliman]

I suggest you use Appender or BackgroundAppender when writing to bifrost, it's an easy change and will get you even better performance IMHO. This is adjacent to what we'll do regarding ingress -> pp communication in the future. The downside with all current options (your PR doesn't introduce the issue) is that we don't have proper back-pressure or control over faireness from partition processor so it's easy to cause overwhelm the partition processor if kafka's ingestion rate is faster than PP's processing rate.

[slinkydeveloper]

For now I won't do this change, we should revisit this though once we sort out how the kafka ingress behaves in the distributed setup.

[AhmedSoliman]

I suggest you use Appender or BackgroundAppender when writing to bifrost, it's an easy change and will get you even better performance IMHO. This is adjacent to what we'll do regarding ingress -> pp communication in the future. The downside with all current options (your PR doesn't introduce the issue) is that we don't have proper back-pressure or control over faireness from partition processor so it's easy to cause overwhelm the partition processor if kafka's ingestion rate is faster than PP's processing rate.

[AhmedSoliman]

Nice improvements indeed. 🚢