Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Audit: rewrite audit entry formatting to improve performance #27952

Merged
merged 10 commits into from
Aug 2, 2024

Conversation

peteski22
Copy link

@peteski22 peteski22 commented Aug 2, 2024

Description

This PR is a bit of a re-write/refactor of the existing audit code, intended to improve performance. Various binaries/builds have been tested internally in collaboration with our Customer Engineering team to confirm the improvement on CPU, memory usage, barrier gets etc.

The 'guts' of the PR are in audit/entry_formatter.go.

Summary of larger changes:

audit/types.go

  • combine RequestEntry and ResponseEntry into a single type Entry (as omitempty is set on fields it won't make a difference to the output).

audit/hashstructure.go:

  • change hash funcs to accept the audit types vs. the logical ones.

audit/entry_formatter.go:

  • introduce new funcs newAuth, newRequest and newResponse that can be used early to translate the incoming LogInput components into domain specific objects. These funcs will clone all the relevant parts (as opposed to the entire deep clone of the LogInput which had a significant performance impact).
  • newResponse will also handle elision of the data field at the time it's about to clone which is also an improvement.
  • Process method no longer performs a deep clone on the entire LogInput.
  • Process has also done away with having to differentiate between a request and a response.
  • We create a new eventlogger.Event as before, but we don't need to create a new AuditEvent as the only thing that matters is the (JSON/JSONx) bytes we store in the eventlogger.Event's Formatted map.
  • hashing is done after creation of the Entry which will already have elided response data if required.

NOTE: Enterprise PR: https://github.com/hashicorp/vault-enterprise/pull/6373

HashiCorp Checklist

  • Labels: If this PR is the CE portion of an ENT change, and that ENT change is
    getting backported to N-2, use the new style backport/ent/x.x.x+ent labels
    instead of the old style backport/x.x.x labels.
  • Labels: If this PR is a CE only change, it can only be backported to N, so use
    the normal backport/x.x.x label (there should be only 1).
  • ENT Breakage: If this PR either 1) removes a public function OR 2) changes the signature
    of a public function, even if that change is in a CE file, double check that
    applying the patch for this PR to the ENT repo and running tests doesn't
    break any tests. Sometimes ENT only tests rely on public functions in CE
    files.
  • Jira: If this change has an associated Jira, it's referenced either
    in the PR description, commit message, or branch name.
  • RFC: If this change has an associated RFC, please link it in the description.
  • ENT PR: If this change has an associated ENT PR, please link it in the
    description. Also, make sure the changelog is in this PR, not in your ENT PR.

@peteski22 peteski22 added core/audit hashicorp-contributed-pr If the PR is HashiCorp (i.e. not-community) contributed backport/1.17.x labels Aug 2, 2024
@peteski22 peteski22 added this to the 1.17.3 milestone Aug 2, 2024
Copy link

github-actions bot commented Aug 2, 2024

CI Results:
All Go tests succeeded! ✅

Copy link

github-actions bot commented Aug 2, 2024

Build Results:
All builds succeeded! ✅

Copy link
Contributor

@kubawi kubawi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@peteski22 peteski22 merged commit 2dbb3d4 into main Aug 2, 2024
82 of 83 checks passed
@peteski22 peteski22 deleted the peteski22/audit/zoomed-in-cloning branch August 2, 2024 18:04
peteski22 pushed a commit that referenced this pull request Aug 2, 2024
* rewrite audit entry formatting to improve performance
peteski22 pushed a commit that referenced this pull request Aug 2, 2024
…#27954)

* rewrite audit entry formatting to improve performance

Co-authored-by: Peter Wilson <[email protected]>
@peteski22 peteski22 added the backport/ent/1.16.x+ent Changes are backported to 1.16.x+ent label Aug 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport/ent/1.16.x+ent Changes are backported to 1.16.x+ent core/audit hashicorp-contributed-pr If the PR is HashiCorp (i.e. not-community) contributed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants