Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add aggregated logger for event and stat #229

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

yunhua-lee
Copy link

@yunhua-lee yunhua-lee commented Jan 7, 2021

[EventAggregator]
Used to log many same type events. The typical scenario is api request and response.

If we log each api request and response, the log data will be too large. But if we do not log each api request and response, when there are some errors, we can not confirm if we get the request or return the response.

EventAggregator can aggregate many same type events and log the key information within one log.
For example:
+++++++++++++++++++++++++++++++++++
API: get-user-api-resp
requestId=1053:200 | requestId=1054:200 | requestId=1055:500 | requestId=1056:200 | requestId=1057:200 |

total: 5
[CONTEXT time_window=5 number_window=20 ]
+++++++++++++++++++++++++++++++++++

[StatAggregator]
Used to log many same type performance data. The typical scenario is API process.

StatAggregator can aggregate many same type performance data and simply calc the min/max/total/avg value.
For example:
+++++++++++++++++++++++++++++++++++
API: get-user-api
min:60, max:87, total:735, count:10, avg:73.5.
[CONTEXT number_window=10 sample_rate=3 unit="ms" ]
+++++++++++++++++++++++++++++++++++

You can combine EventAggregator and StatAggregator and FluentLogger to log full information for API or other kind of event like this:

  • use EventAggregator to log key information like request id and response code.
  • use StatAggregator to log performance information
  • use FluentLogger to log detailed information for error requests or responses.

@google-cla
Copy link

google-cla bot commented Jan 7, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla google-cla bot added the cla: no label Jan 7, 2021
@yunhua-lee
Copy link
Author

@googlebot I signed it!

@google-cla google-cla bot added cla: yes and removed cla: no labels Jan 7, 2021
@hagbard
Copy link
Contributor

hagbard commented Jan 7, 2021

I'm afraid that Flogger does not accept external contributions without prior discussion of the changes being proposed. See the CONTRIBUTING.md file for more:

It's generally best to start by opening a new issue describing the bug or feature you're intending to fix. Even if you think it's relatively minor, it's helpful to know what people are working on. Mention in the initial issue that you are planning to work on that bug or feature so that it can be assigned to you.

This is a very large and complex change and would need to be preceded by a design document, and a lot of discussion regarding alternatives. There are in fact already plans to release an internal-to-Google feature which I believe covers most of the issues you are referring to.

Please raise an issue in the Flogger issue tracker and start a design document (e.g. in Google Docs) and share it, explaining the problem and listing proposed ideas (without reference to implementation). Ideally have the design document capable of receiving comments.

@yunhua-lee
Copy link
Author

commit 80b7b3b is to fix unit test error from google:master code.

@hagbard
Copy link
Contributor

hagbard commented Jan 7, 2021

Thanks for noticing the date bug (this is a locale dependent bit of testing and should be changed to not rely on exact date formats).

@netdpb netdpb added P3 type=addition A new feature labels Jan 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants