Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix kubelet_kubelet_cgroups_cgroupfs #8500

Conversation

fungusakafungus
Copy link
Contributor

@fungusakafungus fungusakafungus commented Feb 1, 2022

If kubelet is run with systemd (as it always is when using kubespray),
it starts in systemd's /system.slice/kubelet.service cgroup.

This commit prevents a creation and usage of a second unrelated cgroup.

What type of PR is this?
/kind bug

What this PR does / why we need it:
This fixes a slight error introduced in #8123

As I have observed, kubelet starts in /system.slice/kubelet.service and then moves itself to kubeletCgroups value:

# pgrep kubelet
34006

# tail /sys/fs/cgroup/memory/system.slice/kubelet*/{cgroup.procs,memory.usage_in_bytes}
==> /sys/fs/cgroup/memory/system.slice/kubelet.service/cgroup.procs <==

==> /sys/fs/cgroup/memory/system.slice/kubelet.slice/cgroup.procs <==
34006

==> /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.usage_in_bytes <==
59301888

==> /sys/fs/cgroup/memory/system.slice/kubelet.slice/memory.usage_in_bytes <==
150597632

This results in cAdvisor kubelet metrics showing both of these cgroups.

Alternatively, we could let kubelet figure out the right cgroup for itself(by setting kubeletCgroups to "" or leaving it out of config), it seems to do that by looking at name=systemd: entry in /proc/<own pid>/cgroups(code for v1.21), which would produce the same result for me at least.

The defaulting doesn't work for kubelet_runtime_cgroups_cgroupfs on the previous line, as kubernetes currently only has special handling for docker runtime: https://github.com/kubernetes/kubernetes/blob/v1.23.0/pkg/kubelet/cm/helpers_linux.go#L343

Which issue(s) this PR fixes:
#8123

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Fix kubelet_kubelet_cgroups_cgroupfs pointing incorrectly to slice

If kubelet is run with systemd (as it always is when using kubespray),
it starts in systemd's /system.slice/kubelet.service cgroup.

This commit prevents a creation and usage of a second unrelated cgroup.
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Feb 1, 2022
@k8s-ci-robot
Copy link
Contributor

Hi @fungusakafungus. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Feb 1, 2022
@fungusakafungus
Copy link
Contributor Author

test failure for packet_debian10-calico looks like a flake:

Wait until SSH is available
fatal: [instance-1 -> localhost]: FAILED! => {"changed": false, "elapsed": 240, "msg": "Timeout when waiting for 172.30.72.147:22"}

@floryut
Copy link
Member

floryut commented Feb 1, 2022

test failure for packet_debian10-calico looks like a flake:

Wait until SSH is available
fatal: [instance-1 -> localhost]: FAILED! => {"changed": false, "elapsed": 240, "msg": "Timeout when waiting for 172.30.72.147:22"}

be patient 😛 jobs are auto retried at least once

@oomichi
Copy link
Contributor

oomichi commented Feb 1, 2022

/ok-to-test
/cc @pasqualet

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 1, 2022
@oomichi
Copy link
Contributor

oomichi commented Feb 2, 2022

@pasqualet Thank you for reviewing this pull request.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 2, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: floryut, fungusakafungus, pasqualet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 2, 2022
@k8s-ci-robot k8s-ci-robot merged commit aed187e into kubernetes-sigs:master Feb 2, 2022
sakuraiyuta pushed a commit to sakuraiyuta/kubespray that referenced this pull request Apr 16, 2022
If kubelet is run with systemd (as it always is when using kubespray),
it starts in systemd's /system.slice/kubelet.service cgroup.

This commit prevents a creation and usage of a second unrelated cgroup.
@oomichi oomichi mentioned this pull request May 28, 2022
LuckySB pushed a commit to southbridgeio/kubespray that referenced this pull request Jun 29, 2023
If kubelet is run with systemd (as it always is when using kubespray),
it starts in systemd's /system.slice/kubelet.service cgroup.

This commit prevents a creation and usage of a second unrelated cgroup.
LuckySB pushed a commit to southbridgeio/kubespray that referenced this pull request Oct 21, 2023
If kubelet is run with systemd (as it always is when using kubespray),
it starts in systemd's /system.slice/kubelet.service cgroup.

This commit prevents a creation and usage of a second unrelated cgroup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants