Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rate-imiting seems to be inaccurate #4292

Open
gid-fieldcode opened this issue Sep 20, 2024 · 4 comments
Open

Rate-imiting seems to be inaccurate #4292

gid-fieldcode opened this issue Sep 20, 2024 · 4 comments
Labels
triage
Milestone
v1.2.0-rc1

Comments

@gid-fieldcode
Copy link

Description:
I'm imposing a "3 per minute" rate limit, yet about twice as many requests actually get accepted.

Repro steps:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy 
metadata:
  name: policy-httproute-compound
  namespace: test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: echo    
  rateLimit:
    type: Global
    global:
      rules:
      - clientSelectors:
        - headers:
          - name: x-user-id
             value: one
        limit:
          requests: 3
          unit: Minute

Environment:
Envoy Gateway v1.1.0, on AWS Kubernetes 1.29

Logs:
HTTP logs showing how the "x-ratelimit-remaining" variable is not decremented with each accepted request, as one would expect, how it suddenly increases for no apparent reason, and how some requests got accepted despite the variable being zero:

$ for i in {1..10}; do curl -I --header "x-user-id: one" https://envoy.fieldcode.club/echo ; sleep 1; done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Fri, 20 Sep 2024 14:59:27 GMT
content-length: 515
x-ratelimit-limit: 3, 3;w=60
x-ratelimit-remaining: 2
x-ratelimit-reset: 33

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Fri, 20 Sep 2024 14:59:29 GMT
content-length: 515
x-ratelimit-limit: 3, 3;w=60
x-ratelimit-remaining: 1
x-ratelimit-reset: 31

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Fri, 20 Sep 2024 14:59:30 GMT
content-length: 515
x-ratelimit-limit: 3, 3;w=60
x-ratelimit-remaining: 0
x-ratelimit-reset: 30

HTTP/1.1 200 OK <--- why?
content-type: application/json
x-content-type-options: nosniff
date: Fri, 20 Sep 2024 14:59:31 GMT
content-length: 515
x-ratelimit-limit: 3, 3;w=60
x-ratelimit-remaining: 2 <--- why?
x-ratelimit-reset: 29

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Fri, 20 Sep 2024 14:59:33 GMT
content-length: 515
x-ratelimit-limit: 3, 3;w=60
x-ratelimit-remaining: 1
x-ratelimit-reset: 27

HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
x-ratelimit-limit: 3, 3;w=60
x-ratelimit-remaining: 0
x-ratelimit-reset: 26
date: Fri, 20 Sep 2024 14:59:33 GMT
transfer-encoding: chunked

HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
x-ratelimit-limit: 3, 3;w=60
x-ratelimit-remaining: 0
x-ratelimit-reset: 25
date: Fri, 20 Sep 2024 14:59:35 GMT
transfer-encoding: chunked

HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
x-ratelimit-limit: 3, 3;w=60
x-ratelimit-remaining: 0
x-ratelimit-reset: 23
date: Fri, 20 Sep 2024 14:59:36 GMT
transfer-encoding: chunked

HTTP/1.1 200 OK <----- why?
content-type: application/json
x-content-type-options: nosniff
date: Fri, 20 Sep 2024 14:59:38 GMT
content-length: 515
x-ratelimit-limit: 3, 3;w=60
x-ratelimit-remaining: 0
x-ratelimit-reset: 22

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Fri, 20 Sep 2024 14:59:39 GMT
content-length: 515
x-ratelimit-limit: 3, 3;w=60
x-ratelimit-remaining: 2
x-ratelimit-reset: 21

Envoy Logs showing the 7 (out of 10) accepted requests, with requests getting sent a second apart.

{"start_time":"2024-09-20T14:59:27.774Z","method":"HEAD","x-envoy-origin-path":"/echo","protocol":"HTTP/1.1","response_code":"200","response_flags":"-","response_code_details":"via_upstream","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"3","x-envoy-upstream-service-time":"-","x-forwarded-for":"87.130.118.226","user-agent":"curl/7.68.0","x-request-id":"3bbccd16-d757-476f-9f8e-3a3e0c5918ff",":authority":"envoy.fieldcode.club","upstream_host":"10.65.50.172:3000","upstream_cluster":"httproute/test/echo/rule/0","upstream_local_address":"10.65.70.203:50306","downstream_local_address":"10.65.28.37:443","downstream_remote_address":"87.130.118.226:65118","requested_server_name":"-","route_name":"httproute/test/echo/rule/0/match/0/_fieldcode_club"}
{"start_time":"2024-09-20T14:59:29.087Z","method":"HEAD","x-envoy-origin-path":"/echo","protocol":"HTTP/1.1","response_code":"200","response_flags":"-","response_code_details":"via_upstream","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"4","x-envoy-upstream-service-time":"-","x-forwarded-for":"87.130.118.226","user-agent":"curl/7.68.0","x-request-id":"681b6a90-e277-4a31-bd4a-8d7ea4261959",":authority":"envoy.fieldcode.club","upstream_host":"10.65.50.172:3000","upstream_cluster":"httproute/test/echo/rule/0","upstream_local_address":"10.65.70.203:56360","downstream_local_address":"10.65.28.37:443","downstream_remote_address":"87.130.118.226:65140","requested_server_name":"-","route_name":"httproute/test/echo/rule/0/match/0/_fieldcode_club"}
{"start_time":"2024-09-20T14:59:30.416Z","method":"HEAD","x-envoy-origin-path":"/echo","protocol":"HTTP/1.1","response_code":"200","response_flags":"-","response_code_details":"via_upstream","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"2","x-envoy-upstream-service-time":"-","x-forwarded-for":"87.130.118.226","user-agent":"curl/7.68.0","x-request-id":"31072eb2-2b15-4775-98c4-2c8018b97cb2",":authority":"envoy.fieldcode.club","upstream_host":"10.65.50.172:3000","upstream_cluster":"httproute/test/echo/rule/0","upstream_local_address":"10.65.70.203:35504","downstream_local_address":"10.65.28.37:443","downstream_remote_address":"87.130.118.226:65154","requested_server_name":"-","route_name":"httproute/test/echo/rule/0/match/0/_fieldcode_club"}
{"start_time":"2024-09-20T14:59:31.736Z","method":"HEAD","x-envoy-origin-path":"/echo","protocol":"HTTP/1.1","response_code":"200","response_flags":"-","response_code_details":"via_upstream","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"3","x-envoy-upstream-service-time":"-","x-forwarded-for":"87.130.118.226","user-agent":"curl/7.68.0","x-request-id":"f0d55264-a256-4d50-8d65-d4ad3f37268f",":authority":"envoy.fieldcode.club","upstream_host":"10.65.50.172:3000","upstream_cluster":"httproute/test/echo/rule/0","upstream_local_address":"10.65.70.203:56360","downstream_local_address":"10.65.28.37:443","downstream_remote_address":"87.130.118.226:65156","requested_server_name":"-","route_name":"httproute/test/echo/rule/0/match/0/_fieldcode_club"}
{"start_time":"2024-09-20T14:59:33.074Z","method":"HEAD","x-envoy-origin-path":"/echo","protocol":"HTTP/1.1","response_code":"200","response_flags":"-","response_code_details":"via_upstream","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"2","x-envoy-upstream-service-time":"-","x-forwarded-for":"87.130.118.226","user-agent":"curl/7.68.0","x-request-id":"fc0c5adb-09cb-4a33-9bb2-42d620e3b7cc",":authority":"envoy.fieldcode.club","upstream_host":"10.65.50.172:3000","upstream_cluster":"httproute/test/echo/rule/0","upstream_local_address":"10.65.70.203:56360","downstream_local_address":"10.65.28.37:443","downstream_remote_address":"87.130.118.226:65172","requested_server_name":"-","route_name":"httproute/test/echo/rule/0/match/0/_fieldcode_club"}
{"start_time":"2024-09-20T14:59:34.384Z","method":"HEAD","x-envoy-origin-path":"/echo","protocol":"HTTP/1.1","response_code":"429","response_flags":"RL","response_code_details":"request_rate_limited","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"2","x-envoy-upstream-service-time":"-","x-forwarded-for":"87.130.118.226","user-agent":"curl/7.68.0","x-request-id":"5590a9c9-a444-402d-8a24-bbfcfb920806",":authority":"envoy.fieldcode.club","upstream_host":"-","upstream_cluster":"httproute/test/echo/rule/0","upstream_local_address":"-","downstream_local_address":"10.65.28.37:443","downstream_remote_address":"87.130.118.226:65184","requested_server_name":"-","route_name":"httproute/test/echo/rule/0/match/0/_fieldcode_club"}
{"start_time":"2024-09-20T14:59:35.705Z","method":"HEAD","x-envoy-origin-path":"/echo","protocol":"HTTP/1.1","response_code":"429","response_flags":"RL","response_code_details":"request_rate_limited","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"2","x-envoy-upstream-service-time":"-","x-forwarded-for":"87.130.118.226","user-agent":"curl/7.68.0","x-request-id":"d990c431-e25b-49fa-a49f-3a0c651cada4",":authority":"envoy.fieldcode.club","upstream_host":"-","upstream_cluster":"httproute/test/echo/rule/0","upstream_local_address":"-","downstream_local_address":"10.65.28.37:443","downstream_remote_address":"87.130.118.226:65098","requested_server_name":"-","route_name":"httproute/test/echo/rule/0/match/0/_fieldcode_club"}
{"start_time":"2024-09-20T14:59:37.023Z","method":"HEAD","x-envoy-origin-path":"/echo","protocol":"HTTP/1.1","response_code":"429","response_flags":"RL","response_code_details":"request_rate_limited","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"2","x-envoy-upstream-service-time":"-","x-forwarded-for":"87.130.118.226","user-agent":"curl/7.68.0","x-request-id":"2937d197-b937-4f04-876d-7fc099a39740",":authority":"envoy.fieldcode.club","upstream_host":"-","upstream_cluster":"httproute/test/echo/rule/0","upstream_local_address":"-","downstream_local_address":"10.65.28.37:443","downstream_remote_address":"87.130.118.226:65112","requested_server_name":"-","route_name":"httproute/test/echo/rule/0/match/0/_fieldcode_club"}
{"start_time":"2024-09-20T14:59:38.325Z","method":"HEAD","x-envoy-origin-path":"/echo","protocol":"HTTP/1.1","response_code":"200","response_flags":"-","response_code_details":"via_upstream","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"3","x-envoy-upstream-service-time":"-","x-forwarded-for":"87.130.118.226","user-agent":"curl/7.68.0","x-request-id":"85301979-d33b-4752-b6a6-6f8b640ca84f",":authority":"envoy.fieldcode.club","upstream_host":"10.65.50.172:3000","upstream_cluster":"httproute/test/echo/rule/0","upstream_local_address":"10.65.70.203:50306","downstream_local_address":"10.65.28.37:443","downstream_remote_address":"87.130.118.226:65124","requested_server_name":"-","route_name":"httproute/test/echo/rule/0/match/0/_fieldcode_club"}
{"start_time":"2024-09-20T14:59:39.670Z","method":"HEAD","x-envoy-origin-path":"/echo","protocol":"HTTP/1.1","response_code":"200","response_flags":"-","response_code_details":"via_upstream","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"2","x-envoy-upstream-service-time":"-","x-forwarded-for":"87.130.118.226","user-agent":"curl/7.68.0","x-request-id":"8146e277-7af6-4477-80ca-e704f56071df",":authority":"envoy.fieldcode.club","upstream_host":"10.65.50.172:3000","upstream_cluster":"httproute/test/echo/rule/0","upstream_local_address":"10.65.70.203:56360","downstream_local_address":"10.65.28.37:443","downstream_remote_address":"87.130.118.226:65176","requested_server_name":"-","route_name":"httproute/test/echo/rule/0/match/0/_fieldcode_club"}
@arkodg arkodg added the help wanted Extra attention is needed label Sep 20, 2024
@arkodg arkodg added this to the v1.2.0-rc1 milestone Sep 20, 2024
@shawnh2
Copy link
Contributor

shawnh2 commented Sep 21, 2024 •
edited
Loading

Hi @gid-fieldcode thanks for reporting this. Can you share the output of egctl config rl -n envoy-gateway-system?

@gid-fieldcode
Copy link
Author

gid-fieldcode commented Sep 21, 2024 •
edited
Loading

Hi @shawnh2, actually I can't, we have never used egctl, we have installed EG with Helm.

I just installed egctl v1.1.1 and executed your command, the output is "error: global rate limit feature is not enabled" but that may be because I only just installed egctl. You would know.

@gid-fieldcode
Copy link
Author

Hi @shawnh2, the problem was that we have set up a 3-node Redis cluster in anticipation of heavy traffic when we switch to EG, and it seems EG is not willing to work this way. We saw no such mention in the docs so you may want to fix that. I hope one Redis node will be able to handle 50+ requests per second.

@arkodg
Copy link
Contributor

arkodg commented Sep 25, 2024

@gid-fieldcode CLUSTER is not supported today, a workaround is to configure it by setting the ENV VAR
https://github.com/envoyproxy/ratelimit?tab=readme-ov-file#redis-type
in the rateLimitDeployment field in the EnvoyGateway startup config / ConfigMap
https://gateway.envoyproxy.io/docs/api/extension_types/#envoygatewaykubernetesprovider

@arkodg arkodg removed the help wanted Extra attention is needed label Sep 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
triage
Projects
None yet
Milestone
v1.2.0-rc1
Development

No branches or pull requests
3 participants
@arkodg @shawnh2 @gid-fieldcode