Thanos receive replication waits for unnecessary requests to finish

[brancz]

Thanos, Prometheus and Golang version used:

master-2020-05-05-e5804d80

Object Storage Provider:

s3

What happened:

When rolling out a new version of Thanos receive, the latency of ingestion requests massively spike so I tried to investigate why and I could nail it down to the fact that the replication strategy always waits for all requests to other instances to finish, even if quorum requests have already succeeded.

What you expected to happen:

Only wait for quorum success of replication requests.

How to reproduce it (as minimally and precisely as possible):

3x replication thanos receive setup with one instance being unavailable.

Full logs to relevant components:

n/a

Anything else we need to know:

My env is on Kubernetes, but this is irrelevant to the issue described.

@bwplotka @krasi-georgiev @metalmatze @squat @kakkoyun

[brancz]

This shouldn't actually be too difficult, it just needs a refactoring of the parallelizeRequests function to not wait for all to return:

thanos/pkg/receive/handler.go

Line 333 in e5804d8

func (h *Handler) parallelizeRequests(ctx context.Context, tenant string, replicas map[string]replica, wreqs map[string]*prompb.WriteRequest) error {

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[brancz]

Closed by #2621 and #2679