Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(kube-api-rewriter): add debug server #402

Merged
merged 4 commits into from
Oct 11, 2024
Merged

feat(kube-api-rewriter): add debug server #402

merged 4 commits into from
Oct 11, 2024
+2,160 −261

Conversation

yaroslavborbat
Copy link
Member

@yaroslavborbat yaroslavborbat commented Sep 25, 2024 •
edited by diafour
Loading

Description

  • Add pprof server. Enable with PPROF_BIND_ADDRESS.
  • Enable pprof when logLevel=debug.
  • Add metrics server with builtin Go and process metrics.
  • Add custom metrics to track rewriter proxy internals.

Why do we need it, and what problem does it solve?

We want to diagnose problems with kube-api-rewriter container.

Performance gains:

Rewrite PodList:

Before:
  100  time:      557ms =1
  200  time:      1.88s       1.114s   x1.69 
  400  time:     7.238s       2.228s   x3.25 
  800  time:    27.052s       4.456s   x6.07 
 1600  time:  1m31.623s       8.912s   x10.28 
 3200  time:  5m13.008s      17.825s   x17.56 
 
After:
  100  expect:       96ms  got:       96ms  x1.00
  200  expect:      193ms  got:      172ms  x0.89
  400  expect:      387ms  got:      264ms  x0.68
  800  expect:      775ms  got:      489ms  x0.63
 1600  expect:     1.551s  got:      971ms  x0.63
 3200  expect:     3.102s  got:     1.581s  x0.51

What is the expected result?

  • pprof can be enabled with logLevel=debug in mc/virtualization.
  • Prometheus scrapes kube_api_rewriter_* metrics.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

@yaroslavborbat yaroslavborbat force-pushed the feat/kube-api-rewriter/add-pprof branch from fda7f9c to 4f35e02 Compare October 2, 2024 10:45
@diafour diafour force-pushed the feat/kube-api-rewriter/add-pprof branch from 4f35e02 to a1f8ad7 Compare October 3, 2024 19:13
@diafour diafour requested a review from Isteb4k October 4, 2024 08:26
Isteb4k
Isteb4k requested changes Oct 11, 2024
View reviewed changes
images/kube-api-proxy/pkg/rewriter/list.go Outdated Show resolved Hide resolved
images/kube-api-proxy/pkg/rewriter/discovery.go Outdated Show resolved Hide resolved
images/kube-api-proxy/pkg/proxy/stream_handler.go Outdated Show resolved Hide resolved
images/kube-api-proxy/pkg/proxy/handler.go Show resolved Hide resolved
images/kube-api-proxy/pkg/proxy/handler.go Outdated Show resolved Hide resolved
yaroslavborbat and others added 3 commits October 11, 2024 16:42
@yaroslavborbat @diafour
add debug server
45bb03c 
Signed-off-by: yaroslavborbat <[email protected]>
@yaroslavborbat @diafour
fix
e0f2ed5 
Signed-off-by: yaroslavborbat <[email protected]>
@diafour
- Add builtin golang and process metrics.
824f809 
- Add proxy custom metrics:
  - Metrics to measure rewrite and request durations.
  - Counters for requests and rewrites.
  - Counters for passed bytes.

Signed-off-by: Ivan Mikheykin <[email protected]>
@diafour diafour force-pushed the feat/kube-api-rewriter/add-pprof branch from c01b252 to 824f809 Compare October 11, 2024 13:43
@diafour diafour merged commit a009052 into main Oct 11, 2024
1 check passed
@diafour diafour deleted the feat/kube-api-rewriter/add-pprof branch October 11, 2024 15:09
@deckhouse-BOaTswain deckhouse-BOaTswain mentioned this pull request Oct 11, 2024
Merged
@universal-itengineer universal-itengineer added this to the v0.14.0 milestone Oct 11, 2024
universal-itengineer pushed a commit that referenced this pull request Oct 14, 2024
@deckhouse-BOaTswain
chore(main): release 0.14.0 (#396)
a4b52fe 
Features
api: move calculation of scratch filesystem overhead to cdi controller (91d8db3)
cvi,vi,vd: replace uploadCommand field with imageUploadURLs (70ccd2e)
kube-api-rewriter: add pprof and metrics, fix rewrite performance (#402) (a009052)
vd, vmbda: add _labels and _annotations metrics (#399) (7690d2b)
vd: resizing vd in filesystem mode (#385) (7857bd3)
vm: add custom secret types for provisioning (#390) (597d551)
vm: add failure message to condition if migration is failed (78b3a42)
vm: add new VM _labels and _annotations metrics and refactor existing ones (#398) (75e57ad)
vmclass: add validation for matching virtual machine sizing policies upon virtual machine class change (#389) (029c445)
vm: disable serial console log (21d0bec)
vmop: add metrics (#370) (2436b8e)
vmop: add operation to migrate vm (#386) (d61bab4)
vmrestore: add the ability to restore virtualmachines (0c59bff)
vmsnapshot: add the ability to snapshot virtualmachines (38b84d4)

Bug Fixes
vd,vi,cvi: fix vd uploader service creating (#409) (d012007)
vd: allow to change size in spec for not ready vd (#411) (38bb0eb)
vm,vmip: improve VMIP management (#374) (d6695ef)
vmbda: reconcile from virtual disk phase changes events (7157204)
vm: fix the virtual machine matching with the virtual machine class during validation when no sizing policy is provided (#431) (4704e31)
vmip: add validating ip address for VMIP with type 'Static' (#404) (2c95be9)
vmip: fix deleting unattached vmip (#405) (56cb6cd)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@diafour diafour diafour left review comments

@Isteb4k Isteb4k Isteb4k approved these changes

Assignees

@yaroslavborbat yaroslavborbat

Labels
None yet
Projects
None yet
Milestone
v0.14.0
Development

Successfully merging this pull request may close these issues.
4 participants
@yaroslavborbat @diafour @Isteb4k @universal-itengineer