Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v3rpc: Log expensive request in UnaryInterceptor #12086

Merged
merged 2 commits into from
Jul 1, 2020
Merged

v3rpc: Log expensive request in UnaryInterceptor #12086

merged 2 commits into from
Jul 1, 2020

Conversation

tangcong
Copy link
Contributor

@tangcong tangcong commented Jun 27, 2020 •
edited
Loading

recently, In the process of troubleshooting the etcd request timeout, I encountered the following problems:

  1. Slow query (large response packet) Although the warn log will be printed when apply takes more than 100ms, but lack of remote peer info, it is not convenient to quickly locate the source of the call.
  2. Like the Authenticate interface does not go through the apply process, there is no available log for its latency(no warn apply log,no trace log). issue Issue: etcd stops answering after ~250 concurent HTTP API clients. #11826 has the same feedback.

Adjusting the log level to debug can solve the above problems, but the production environment needs to be changed
and performance will drop sharply.

This pr will print expensive request information in grpc interceptor to solve the above problems and improve the efficiency of the troubleshooting.

@tangcong tangcong force-pushed the print-expensive-request-v3rpc branch from f2b9865 to c5ea67e Compare June 30, 2020 16:52
@tangcong tangcong force-pushed the print-expensive-request-v3rpc branch from c5ea67e to 5c6d7b4 Compare June 30, 2020 16:56
gyuho
gyuho approved these changes Jun 30, 2020
View reviewed changes
Copy link
Contributor

@gyuho gyuho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@tangcong tangcong changed the title etcdserver,v3rpc: Log expensive request in UnaryInterceptor v3rpc: Log expensive request in UnaryInterceptor Jun 30, 2020
jingyih
jingyih reviewed Jul 1, 2020
View reviewed changes
Copy link
Contributor

@jingyih jingyih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jingyih jingyih merged commit 15884e9 into etcd-io:master Jul 1, 2020
@tangcong tangcong deleted the print-expensive-request-v3rpc branch February 26, 2021 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jingyih jingyih jingyih left review comments

@gyuho gyuho gyuho approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tangcong @gyuho @jingyih