Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter api-gateway cache logging to reduce log output on server disconnect #2880

Merged
merged 2 commits into from
Sep 5, 2023

Conversation

curtbushko
Copy link
Contributor

Changes proposed in this PR:

  • I noticed this when I was working on a customer issue and it made debugging more difficult
  • I was working on an issue that was reproduced by deleting the consul-server-0 pod and the api-gateway logging spit out about 20k log lines in about 5 seconds
  • I don't think we need to log these errors out at the api-gateway level as other parts of the controller will log a similar error out but less frequently.
  • The customer will still see that the consul server is unreachable in the logs (see below)

Instead of 20k of these:

2023-08-31T14:33:54.861Z ERROR error fetching config entries for kind: http-route {"error": "Get "http://10.244.0.8:8500/v1/config/http-route?dc=dc1&index=10\": dial tcp 10.244.0.8:8500: connect: connection refused"}

They still see:

2023-08-31T14:41:43.031Z ERROR Reconciler error {"controller": "endpoints", "controllerGroup": "", "controllerKind": "Endpoints", "Endpoints": {"name":"consul-ui","namespace":"consul"}, "namespace": "consul", "name": "consul-ui", "reconcileID": "ce66b001-0173-40d1-95ab-a7a5363a6612", "error": "1 error occurred:\n\t* Get "http://:8500/v1/catalog/nodes?dc=dc1&filter=Meta%5B%22synthetic-node%22%5D+%3D%3D+%22true%22+&ns=%2A": dial tcp: lookup : no such host\n\n"}

How I've tested this PR:

  • build locally, delete consul-server-0 and look at the connect injector logs.

How I expect reviewers to test this PR:

👀

Checklist:

@curtbushko curtbushko self-assigned this Aug 31, 2023
@curtbushko curtbushko marked this pull request as ready for review August 31, 2023 14:54
Copy link
Contributor

@t-eckert t-eckert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! Thank you

@t-eckert t-eckert added the theme/api-gateway Related to Consul API Gateway label Aug 31, 2023
@t-eckert
Copy link
Contributor

I think you will want to backport this into 1.2.x

@curtbushko curtbushko added the backport/1.2.x This release branch is no longer active. label Aug 31, 2023
@curtbushko curtbushko merged commit decdca5 into main Sep 5, 2023
49 of 51 checks passed
@curtbushko curtbushko deleted the reduce-gateway-noise branch September 5, 2023 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport/1.2.x This release branch is no longer active. theme/api-gateway Related to Consul API Gateway
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants