Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

router: add envoy-ratelimited retry policy #12201

Merged
merged 3 commits into from
Jul 22, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions docs/root/configuration/http/http_filters/router_filter.rst
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,12 @@ connect-failure
configuration <envoy_v3_api_field_config.route.v3.RouteAction.retry_policy>` or via
:ref:`virtual host retry policy <envoy_v3_api_field_config.route.v3.VirtualHost.retry_policy>`.

.. _config_http_filters_router_retry_policy-envoy-ratelimited:

envoy-ratelimited
Envoy will retry if the header :ref:`x-envoy-ratelimited<config_http_filters_router_x-envoy-ratelimited>`
is present.

retriable-4xx
Envoy will attempt a retry if the upstream server responds with a retriable 4xx response code.
Currently, the only response code in this category is 409.
Expand Down Expand Up @@ -294,9 +300,11 @@ information.
x-envoy-ratelimited
^^^^^^^^^^^^^^^^^^^

If this header is set by upstream, Envoy will not retry. Currently the value of the header is not
looked at, only its presence. This header is set by :ref:`rate limit filter<config_http_filters_rate_limit>`
when the request is rate limited.
If this header is set by upstream, Envoy will not retry unless the retry policy
:ref:`envoy-ratelimited<config_http_filters_router_retry_policy-envoy-ratelimited>`
is enabled. Currently, the value of the header is not looked at, only its
presence. This header is set by :ref:`rate limit
filter<config_http_filters_rate_limit>` when the request is rate limited.

.. _config_http_filters_router_headers_set:

Expand Down
3 changes: 3 additions & 0 deletions docs/root/version_history/current.rst
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ New Features
* http: introduced new HTTP/1 and HTTP/2 codec implementations that will remove the use of exceptions for control flow due to high risk factors and instead use error statuses. The old behavior is deprecated, but can be used during the removal period by setting the runtime feature `envoy.reloadable_features.new_codec_behavior` to false. The removal period will be one month.
* load balancer: added a :ref:`configuration<envoy_v3_api_msg_config.cluster.v3.Cluster.LeastRequestLbConfig>` option to specify the active request bias used by the least request load balancer.
* redis: added fault injection support :ref:`fault injection for redis proxy <envoy_v3_api_field_extensions.filters.network.redis_proxy.v3.RedisProxy.faults>`, described further in :ref:`configuration documentation <config_network_filters_redis_proxy>`.
* router: added new
:ref:`envoy-ratelimited<config_http_filters_router_retry_policy-envoy-ratelimited>`
retry policy, which allows retrying envoy's own rate limited responses.
* stats: added optional histograms to :ref:`cluster stats <config_cluster_manager_cluster_stats_request_response_sizes>`
that track headers and body sizes of requests and responses.
* tap: added :ref:`generic body matcher<envoy_v3_api_msg_config.tap.v3.HttpGenericBodyMatch>` to scan http requests and responses for text or hex patterns.
Expand Down
1 change: 1 addition & 0 deletions include/envoy/router/router.h
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,7 @@ class RetryPolicy {
static const uint32_t RETRY_ON_RETRIABLE_STATUS_CODES = 0x400;
static const uint32_t RETRY_ON_RESET = 0x800;
static const uint32_t RETRY_ON_RETRIABLE_HEADERS = 0x1000;
static const uint32_t RETRY_ON_ENVOY_RATE_LIMITED = 0x2000;
// clang-format on

virtual ~RetryPolicy() = default;
Expand Down
1 change: 1 addition & 0 deletions source/common/http/headers.h
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,7 @@ class HeaderValues {
const std::string _5xx{"5xx"};
const std::string GatewayError{"gateway-error"};
const std::string ConnectFailure{"connect-failure"};
const std::string EnvoyRateLimited{"envoy-ratelimited"};
const std::string RefusedStream{"refused-stream"};
const std::string Retriable4xx{"retriable-4xx"};
const std::string RetriableStatusCodes{"retriable-status-codes"};
Expand Down
8 changes: 6 additions & 2 deletions source/common/router/retry_state_impl.cc
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ namespace Router {
const uint32_t RetryPolicy::RETRY_ON_5XX;
const uint32_t RetryPolicy::RETRY_ON_GATEWAY_ERROR;
const uint32_t RetryPolicy::RETRY_ON_CONNECT_FAILURE;
const uint32_t RetryPolicy::RETRY_ON_ENVOY_RATE_LIMITED;
const uint32_t RetryPolicy::RETRY_ON_RETRIABLE_4XX;
const uint32_t RetryPolicy::RETRY_ON_RETRIABLE_HEADERS;
const uint32_t RetryPolicy::RETRY_ON_RETRIABLE_STATUS_CODES;
Expand Down Expand Up @@ -169,6 +170,8 @@ std::pair<uint32_t, bool> RetryStateImpl::parseRetryOn(absl::string_view config)
ret |= RetryPolicy::RETRY_ON_GATEWAY_ERROR;
} else if (retry_on == Http::Headers::get().EnvoyRetryOnValues.ConnectFailure) {
ret |= RetryPolicy::RETRY_ON_CONNECT_FAILURE;
} else if (retry_on == Http::Headers::get().EnvoyRetryOnValues.EnvoyRateLimited) {
ret |= RetryPolicy::RETRY_ON_ENVOY_RATE_LIMITED;
} else if (retry_on == Http::Headers::get().EnvoyRetryOnValues.Retriable4xx) {
ret |= RetryPolicy::RETRY_ON_RETRIABLE_4XX;
} else if (retry_on == Http::Headers::get().EnvoyRetryOnValues.RefusedStream) {
Expand Down Expand Up @@ -290,9 +293,10 @@ RetryStatus RetryStateImpl::shouldHedgeRetryPerTryTimeout(DoRetryCallback callba
}

bool RetryStateImpl::wouldRetryFromHeaders(const Http::ResponseHeaderMap& response_headers) {
// We never retry if the request is rate limited.
// A response that contains the x-envoy-ratelimited header comes from an upstream envoy.
// We retry these only when the envoy-ratelimited policy is in effect.
if (response_headers.EnvoyRateLimited() != nullptr) {
return false;
return retry_on_ & RetryPolicy::RETRY_ON_ENVOY_RATE_LIMITED;
}

if (retry_on_ & RetryPolicy::RETRY_ON_5XX) {
Expand Down
16 changes: 16 additions & 0 deletions test/common/router/retry_state_impl_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,22 @@ TEST_F(RouterRetryStateImplTest, PolicyResourceExhaustedRemoteRateLimited) {
EXPECT_EQ(RetryStatus::No, state_->shouldRetryHeaders(response_headers, callback_));
}

TEST_F(RouterRetryStateImplTest, PolicyEnvoyRateLimitedRemoteRateLimited) {
Http::TestRequestHeaderMapImpl request_headers{{"x-envoy-retry-on", "envoy-ratelimited"}};
setup(request_headers);
EXPECT_TRUE(state_->enabled());

expectTimerCreateAndEnable();
Http::TestResponseHeaderMapImpl response_headers{{":status", "429"},
{"x-envoy-ratelimited", "true"}};
EXPECT_EQ(RetryStatus::Yes, state_->shouldRetryHeaders(response_headers, callback_));
EXPECT_CALL(callback_ready_, ready());
retry_timer_->invokeCallback();

EXPECT_EQ(RetryStatus::NoRetryLimitExceeded,
state_->shouldRetryHeaders(response_headers, callback_));
}

TEST_F(RouterRetryStateImplTest, PolicyGatewayErrorRemote502) {
verifyPolicyWithRemoteResponse("gateway-error" /* retry_on */, "502" /* response_status */,
false /* is_grpc */);
Expand Down