Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

otlptracehttp otlpmetrichttp: Handling 502 Bad Gateway and 504 Gateway Timeout #4662

Closed
pellared opened this issue Oct 23, 2023 · 0 comments · Fixed by #4670
Closed

otlptracehttp otlpmetrichttp: Handling 502 Bad Gateway and 504 Gateway Timeout #4662

pellared opened this issue Oct 23, 2023 · 0 comments · Fixed by #4670
Assignees
Labels
bug Something isn't working pkg:exporter:otlp Related to the OTLP exporter package

Comments

@pellared
Copy link
Member

pellared commented Oct 23, 2023

Description

From specification:

The requests that receive a response status code listed in following table SHOULD be retried.

HTTP response status code
429 Too Many Requests
502 Bad Gateway
503 Service Unavailable
504 Gateway Timeout

Actual behavior

return c.requestFunc(ctx, func(iCtx context.Context) error {
select {
case <-iCtx.Done():
return iCtx.Err()
default:
}
request.reset(iCtx)
resp, err := c.httpClient.Do(request.Request)
if err != nil {
return err
}
var rErr error
switch sc := resp.StatusCode; {
case sc >= 200 && sc <= 299:
// Success, do not retry.
// Read the partial success message, if any.
var respData bytes.Buffer
if _, err := io.Copy(&respData, resp.Body); err != nil {
return err
}
if respData.Len() != 0 {
var respProto colmetricpb.ExportMetricsServiceResponse
if err := proto.Unmarshal(respData.Bytes(), &respProto); err != nil {
return err
}
if respProto.PartialSuccess != nil {
msg := respProto.PartialSuccess.GetErrorMessage()
n := respProto.PartialSuccess.GetRejectedDataPoints()
if n != 0 || msg != "" {
err := internal.MetricPartialSuccessError(n, msg)
otel.Handle(err)
}
}
}
return nil
case sc == http.StatusTooManyRequests, sc == http.StatusServiceUnavailable:
// Retry-able failure.
rErr = newResponseError(resp.Header)
// Going to retry, drain the body to reuse the connection.
if _, err := io.Copy(io.Discard, resp.Body); err != nil {
_ = resp.Body.Close()
return err
}
default:
rErr = fmt.Errorf("failed to send metrics to %s: %s", request.URL, resp.Status)
}
if err := resp.Body.Close(); err != nil {
return err
}
return rErr
})

502 Bad Gateway and 504 Gateway Timeout is not be retried.

Expected behavior

Follow the specification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working pkg:exporter:otlp Related to the OTLP exporter package
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

1 participant