Add streaming to Remote Read endpoint

[gouthamve]

Remote read is quite inefficient right now and could be made much better through streaming. This has been discussed in the dev-summit: https://docs.google.com/document/d/1-C5PycocOZEVIPrmM1hn8fBelShqtqiAmFptoG4yK70/edit#heading=h.5d2mc4i0bnnp

[bwplotka]

I can take it if no one started working on this.

Design doc: https://docs.google.com/document/d/1JqrU3NjM9HoGLSTPYOvR217f5HBKBiJTqikEB9UiJL0/edit?usp=sharing

Dev-mailing topic: https://groups.google.com/forum/#!topic/prometheus-developers/79b6j_6F_Vc

[harshal-shah]

According to #4655 it seems this has been resolved in prometheus 2.5.0

[bwplotka]

Well, the memory was reduced (not sure how largely), but there still is a need for streamed remote read, potentially with chunk encoding to reduce used memory on query to minium.

[rshivane]

I would like to add that, as a vendor implementing the server-side of remote read API, the lack of streaming adds similar memory pressure on the marshaling side.

@bwplotka I notice that your already have a PR for streaming support in #4591. Whenever this is approved for merge, we would be happy to implement it on our storage and validate.

[bwplotka]

Yes. This is working code for streaming with raw samples, but we have some internal discussion how it should be done. I closed it to reduce confusion for now: #4591

TL;DR: It would be nice to have remote read stream with Prometheus chunk encoded samples (instead of row samples), to avoid unnecessary decoding & encoding. Does it
The plan is to:

1. Change the doc to propose this idea (right now it proposes something in between)
2. Setup a call to gather feedback on this idea.

Did not have much time to jump into it yet, but I am happy to hear that you are ok to help with this!

[rshivane]

It would be nice to have remote read stream with Prometheus chunk encoded samples (instead of row samples), to avoid unnecessary decoding & encoding.

Is that essentially "Alternative 1" in the design doc ?

[bwplotka]

Exactly, but need to reorder ;p

[bwplotka]

Just FYI, since we got concensus on this, expect more finalized design doc (same link: https://docs.google.com/document/d/1JqrU3NjM9HoGLSTPYOvR217f5HBKBiJTqikEB9UiJL0/edit#) and initial benchmarks soon. I scheduled some time for this work in June.

[bwplotka]

Back to business!

Server side changes: #5703
Caller usage example (Thanos): thanos-io/thanos#1268

Living document: https://docs.google.com/document/d/1JqrU3NjM9HoGLSTPYOvR217f5HBKBiJTqikEB9UiJL0/edit#

[bwplotka]

First MVP should be safe to use. I build 2 docker images if someone wants to try:

• bplotka/prometheus:v2.11.0-rc.0-rr-streaming with remote read streaming server support (remote-read: streamed chunked server side; Extended protobuf; Added chunked, checksumed reader  #5703)

If you use sidecar, I updated caller side as well:

• improbable/thanos:v0.5.0-rr-streamed2 with thanos sidecar remote read streaming support.

No config changes, just move docker images to that Prometheus and Thanos sidecar tags and enjoy!

2 big requests against Prometheus + sidecar in upgraded version look like this memory and CPU wise:

Versus old one with non-streaming sampled read (3GB of mem!):

[bwplotka]

Remote read now supports streaming.

[bwplotka]

Follow up tasks:

• Docs: Add docs for remote read streaming #5925
• Chunk iterator instead of reencoding: Add ChunksIterator method to Series interface. #5882
• Prometheus client code: Allow Prometheus to use ReadRequest_STREAMED_XOR_CHUNKS when calling remote read API. #5926