Fix distributor http status codes

[joe-elliott]

Describe the bug
As reported by the community if the ingester returns trace too large then the returned http error is 500. This error suggests to the otel collector to retry however it should return 400 which suggests to not retry. Currently the GRPC status codes correctly suggest retrying or not.

Review the status codes for various errors and return 400s where appropriate.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.

[joe-elliott]

@mghildiy is going to take a crack at this.

cc @ie-pham b/c you're also looking at return codes (but for proto/grpc)

[mghildiy]

So currently error code returned is 'FailedPrecondition

tempo/modules/ingester/instance.go

Line 189 in c0127c7

return status.Errorf(codes.FailedPrecondition, (newTraceTooLargeError(id, i.instanceID, maxBytes, reqSize).Error()))

tempo/modules/ingester/instance.go

Line 198 in c0127c7

return status.Errorf(codes.FailedPrecondition, e.Error())

...defined here:

tempo/vendor/google.golang.org/grpc/codes/codes.go

Line 121 in c0127c7

FailedPrecondition Code = 9

A better option can be:

tempo/vendor/google.golang.org/grpc/codes/codes.go

Line 57 in c0127c7

InvalidArgument Code = 3

Surprisingly, grpC codes to http codes mapping says that both map to 400.

[ie-pham]

Hi @mghildiy ,

Thank you so much for your contribution to this issue. Unfortunately, this issue is no longer is no longer relevant due to our shift towards a "partial success" solution, as outlined in #2571. In addition, Otel states that partial successes should return a 200 instead of an error.

Your contribution to this repo is well observed by the team and is highly valued. Don't be deterred to explore other opportunities to contribute!

Closing this issue.