Complete writing a batch even if some spans refused

[joe-elliott]

Describe the bug
Currently if an ingester refuses traces b/c a trace is too large it will stop writing the current batch and return an error. The end result is that the first part of the batch was written but everything after the "trace too large" error is dropped. Let's continue writing the batch even if we receive a trace too large error.

Loop to write traces:

tempo/modules/ingester/instance.go

Line 167 in 909b487

for j := range req.Traces {

Trace too large returned here:

tempo/modules/ingester/instance.go

Line 210 in 909b487

return status.Errorf(codes.FailedPrecondition, (newTraceTooLargeError(id, i.instanceID, maxBytes, reqSize).Error()))

[97amarnathk]

@joe-elliott can I pick this up ?

[joe-elliott]

Absolutely! The team is going to be pretty barebones for the next 2 weeks due to the holidays so if you don't get an immediate review don't worry. I promise I will get to all PRs when I get back.

Thanks for giving it a shot.

[97amarnathk]

@joe-elliott 1 small implementation doubt.

Since this is now a bulk operation how should the ingester convey back the traces which failed ?

[mapno]

Hi @97amarnathk!

Since this is now a bulk operation how should the ingester convey back the traces which failed ?

The error in non-retryable, so we're good returning a list of the traces that failed (i.e. similar to what we already do, but for multiple traces). I don't think we need a more comprehensive approach than that.

[97amarnathk]

Got it.
I have implemented multierror handling for this here: #1962

Do go through it whenever possible

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.

[bilbof]

keepalive

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.