-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Collecting Filesystem metrics makes extremely high CPU #3082
Comments
@Shaye-man how did you managed to disable collecting filesystem metrics ? |
@Shaye-man, I hope that #3257 will pave way towards improved performance. EDIT: I know it looks to be abandoned. |
It causes this problem #3349 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
PR #2768 fixed docker can collect FileSystem metrics. But in the scenario of multiple devices, per seconds(
housekeeping time
) collect FileSystem metrics causes kubelet CPU usage extremely high and impacts the application responsiveness.This should be optimized or fixed in cadvisor.
CPU profiling data shows the hot code is here:
cadvisor/container/docker/handler.go
Lines 402 to 430 in dca17e6
cadvisor/fs/fs.go
Lines 451 to 472 in dca17e6
With
kubelet
(version 1.22.1) and about 202 block device, Probably consumed 40%CPUWhen I try to disable collecting FileSystem metrics, like this, only consumed 20%CPU,
kubelet
restore normal consumption.And normal pprof like this:
The text was updated successfully, but these errors were encountered: