KEP-753: add PRR questionnaire answers for beta #4255

matthyx · 2023-10-01T12:32:35Z

One-line PR description: add PRR questionnaire answers for beta

Issue link: Sidecar Containers #753

Other comments:

Signed-off-by: Matthias Bertschy <[email protected]>

logicalhan · 2023-10-01T15:05:14Z

keps/sig-node/753-sidecar-containers/README.md

+#### Beta
+
+- Implement proper termination ordering.
+- Provide defaults for `restartPolicy` field on init containers, `nil` is not


If that's true, why did we use a pointer?

this has to merge for beta:
kubernetes/kubernetes#120176

edit: I was wrong, sorry

I think the PR is not relevant to this.

We may have something like inheritPodRestartPolicyby default?
I'd say nil is OK to indicate it is inherited from the pod's restartPolicy.

If we do that, initContainers will become sidecars for all Pods created by ReplicaSets.
We should default to Never WDYT?

It depends on the pod's restartPolicy.

If the pod has restartPolicy of 'Never', its regular init containers will never restart.

If the pod has restartPolicy of 'OnFailure', or 'Always', its regular init containers will restart only on failure.

Not sure how to name it. How about just call it 'Default' and document the detail...? (just like the 'Default' of PodDNSPolicy)

@SergeyKanzhelev WDYT?

I see, so it basically means "I'm not a sidecar". Hmm, I would still prefer nil then...

keps/sig-node/753-sidecar-containers/README.md

logicalhan · 2023-10-01T15:08:09Z

keps/sig-node/753-sidecar-containers/README.md

+and validate the declared limits?
+-->
+
+No.


Can't sidecars consume these resources?

If you take into account that before this KEP sidecars were already running as "regular" containers in the Pod, I don't consider them as being added.
Also requests and limits are respected.

keps/sig-node/753-sidecar-containers/README.md

bart0sh · 2023-10-01T19:00:28Z

/cc @SergeyKanzhelev

keps/sig-node/753-sidecar-containers/README.md

wojtek-t · 2023-10-03T11:35:39Z

keps/sig-node/753-sidecar-containers/README.md

+You can take a look at one potential example of such test in:
+https://github.com/kubernetes/kubernetes/pull/97058/files#diff-7826f7adbc1996a05ab52e3f5f02429e94b68ce6bce0dc534d1be636154fded3R246-R282
+-->
+


Can you please link to tests [in the upgrade/downgrade testing you only point to tets that you would like to create, I would like to see those tests]

Friendly ping

apparently we don't have one... I'm going to push for it now

is it fine if we add it to the code after the PRR merges but inside the k/k beta PR ?

Yes, but please:

add a sentence here that this test will be added before graduating the feature to beta in k/k

add it explicitly to beta graduation criteria

wojtek-t · 2023-10-03T11:58:18Z

keps/sig-node/753-sidecar-containers/README.md

 ###### What specific metrics should inform a rollback?

 <!--
 What signals should users be paying attention to when the feature is young
 that might indicate a serious problem?
 -->

+Pods that don't feature sidecars are not affected by the KEP.
+
+Pods with sidecars might take a long time to exit and exceed the TGPS, a new


This should be generally moved to the above question.

This question is about metrics. Can you please think through metrics that we can monitor?

wojtek-t · 2023-10-03T11:59:01Z

keps/sig-node/753-sidecar-containers/README.md

@@ -1238,12 +1504,16 @@ Longer term, we may want to require automated upgrade/rollback tests, but we
 are missing a bunch of machinery and tooling and can't do that now.
 -->

+TBD


Even if not running the test now, can you please describe the test that you will run here?
#3658 is a nice example

wojtek-t · 2023-10-03T12:00:30Z

keps/sig-node/753-sidecar-containers/README.md

- [ ] Other (treat as last resort)
-  - Details:
+- [X] Other (treat as last resort)
+  - Details: `kubectl describe pod <pod-name>` will show the new field `.spec.initContainers[i].restartPolicy` for the sidecar containers.


That doesn't mean the feature is working - that means that I wanted it to work.

What we probably want to check is if the init container with restartPolicy set is running together with the regular containers.

keps/sig-node/753-sidecar-containers/README.md

wojtek-t · 2023-10-03T12:05:50Z

keps/sig-node/753-sidecar-containers/README.md

@@ -1446,8 +1769,12 @@ For each of them, fill in the following information by copying the below templat
    - Testing: Are there any tests for failure mode? If not, describe why.
 -->

+None.


That is not something I can easily believe.
Even if we didn't observe, there are definitely failure modes that we should mention..

@SergeyKanzhelev

wojtek-t

Few more comments.

I would also like to see some resolution on #4183 before merging this one...

wojtek-t · 2023-10-04T18:53:13Z

keps/sig-node/753-sidecar-containers/README.md

 to reject such Pods when the feature gate is disabled to keep Downgrade safe.

-**Note**, For the control plane and kubelet we will implement logic to reject Pods
-with sidecar containers when feature gate got turned off.
+**Note**, We have implemented logic for the control plane and kubelet to reject Pods


Sorry for being pedantic here, but can you link to that here?

We were going back and forth on that for Alpha release and what happened was actually not what we initially agreed on, so I would like to ensure that this time [which is much more important given Beta will be on by default] we actually do that right

xref: https://github.com/kubernetes/enhancements/pull/3968/files#diff-4db34302ceb08799a81f2a75eaccc05ee8c29524d34b3a4b02f8c09dcf26247bR1103

https://github.com/kubernetes/kubernetes/blob/f19b62fc0914b38941922afefd1e34eb55f87ee7/pkg/kubelet/lifecycle/predicate.go#L78-L91

kube-apiserver drops the restartPolicy field:
https://github.com/kubernetes/kubernetes/blob/f19b62fc0914b38941922afefd1e34eb55f87ee7/pkg/api/pod/util.go#L554-L560

https://github.com/kubernetes/enhancements/pull/3968/files#diff-4db34302ceb08799a81f2a75eaccc05ee8c29524d34b3a4b02f8c09dcf26247bR1112

I think we wrote down contradictory statements at that time.

In v1.28, if the FG is disabled,

the apiserver just drops the restartPolicy field https://github.com/kubernetes/kubernetes/blob/f19b62fc0914b38941922afefd1e34eb55f87ee7/pkg/api/pod/util.go#L554-L560

the scheduler does not schedule any node for the pod with sidecars https://github.com/kubernetes/kubernetes/blob/f19b62fc0914b38941922afefd1e34eb55f87ee7/pkg/scheduler/framework/plugins/noderesources/fit.go#L256-L262

the kubelet does not admit the pod with sidecars https://github.com/kubernetes/kubernetes/blob/f19b62fc0914b38941922afefd1e34eb55f87ee7/pkg/kubelet/lifecycle/predicate.go#L78-L91

wojtek-t · 2023-10-04T18:55:17Z

keps/sig-node/753-sidecar-containers/README.md

+You can take a look at one potential example of such test in:
+https://github.com/kubernetes/kubernetes/pull/97058/files#diff-7826f7adbc1996a05ab52e3f5f02429e94b68ce6bce0dc534d1be636154fded3R246-R282
+-->
+


Friendly ping

keps/sig-node/753-sidecar-containers/README.md

wojtek-t · 2023-10-04T19:00:02Z

keps/sig-node/753-sidecar-containers/README.md

@@ -1297,18 +1604,28 @@ These goals will help you determine what you need to measure (SLIs) in the next
 question.
 -->

+- sidecar init containers are running and restarted as expected


SLOs are effectively metrics+thresholds

Can you please update this section and formulate it more in the context of SLIs below?

wojtek-t

Two small comment - other than that it LGTM (though it will still also require SIG level approval).

wojtek-t · 2023-10-05T08:08:36Z

keps/sig-node/753-sidecar-containers/README.md

-with sidecar containers when feature gate got turned off.
+**Note**, We have implemented logic for the kubelet to reject Pods
+with sidecar containers when feature gate is turned off. For the control plane
+we simply ignore the new field to maintain Pod scheduling.


Let's be more explicit what exactly is happening as mentioned by @gjkim42

For the control plane - kube-apiserver is dropping the field (if it wasn't set before) and kube-scheduler is keeping pods with the field set unschedulable.

nit: linking the code you pasted in this comment thread:
#4255 (comment)

would be extremely helpful here too

wojtek-t · 2023-10-05T08:11:10Z

keps/sig-node/753-sidecar-containers/README.md

+You can take a look at one potential example of such test in:
+https://github.com/kubernetes/kubernetes/pull/97058/files#diff-7826f7adbc1996a05ab52e3f5f02429e94b68ce6bce0dc534d1be636154fded3R246-R282
+-->
+


Yes, but please:

add a sentence here that this test will be added before graduating the feature to beta in k/k

add it explicitly to beta graduation criteria

Signed-off-by: Matthias Bertschy <[email protected]>

matthyx · 2023-10-05T10:15:09Z

@wojtek-t added the few nits. Thanks for your patience and kindness, it's not always easy to juggle between work, OSS contributions and family life.

wojtek-t · 2023-10-05T10:58:46Z

@wojtek-t added the few nits. Thanks for your patience and kindness, it's not always easy to juggle between work, OSS contributions and family life.

Sure - thanks a lot for bearing with me and for pushing this work - it's super important and I'm really happy to see a progress there.

Given the deadline I'm going to approve the PRR part without waiting for SIG-level approval - hopefully you will figure that our later today.

/lgtm
/approve PRR

For SIG-level approval
/assign @dchen1107 @mrunalp @SergeyKanzhelev

k8s-ci-robot · 2023-10-05T19:19:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: matthyx, mrunalp, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/prod-readiness/OWNERS~~ [wojtek-t]
~~keps/sig-node/OWNERS~~ [mrunalp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

KEP-753: update template milestone and history

8669330

Signed-off-by: Matthias Bertschy <[email protected]>

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 1, 2023

k8s-ci-robot requested review from dchen1107 and derekwaynecarr October 1, 2023 12:32

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/node Categorizes an issue or PR as relevant to SIG Node. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 1, 2023

matthyx force-pushed the prr branch from 11c5218 to 657e108 Compare October 1, 2023 12:41

logicalhan reviewed Oct 1, 2023

View reviewed changes

k8s-ci-robot requested a review from SergeyKanzhelev October 1, 2023 19:00

wojtek-t self-assigned this Oct 2, 2023

matthyx force-pushed the prr branch 2 times, most recently from 5368472 to 92404ba Compare October 3, 2023 06:48

wojtek-t reviewed Oct 3, 2023

View reviewed changes

matthyx force-pushed the prr branch 6 times, most recently from 0962b6d to f71978b Compare October 4, 2023 16:10

wojtek-t reviewed Oct 4, 2023

View reviewed changes

matthyx force-pushed the prr branch 3 times, most recently from 39e6869 to 7fe8891 Compare October 5, 2023 07:35

wojtek-t reviewed Oct 5, 2023

View reviewed changes

KEP-753: add PRR answers for beta

0d00fd4

Signed-off-by: Matthias Bertschy <[email protected]>

matthyx force-pushed the prr branch from 7fe8891 to 0d00fd4 Compare October 5, 2023 10:13

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 5, 2023

k8s-ci-robot assigned dchen1107, mrunalp and SergeyKanzhelev Oct 5, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 5, 2023

mrunalp approved these changes Oct 5, 2023

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 5, 2023

k8s-ci-robot merged commit 8d8dae2 into kubernetes:master Oct 5, 2023
4 checks passed

k8s-ci-robot added this to the v1.29 milestone Oct 5, 2023

Priyankasaggu11929 mentioned this pull request Oct 6, 2023

Sidecar Containers #753

Open

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-753: add PRR questionnaire answers for beta #4255

KEP-753: add PRR questionnaire answers for beta #4255

matthyx commented Oct 1, 2023

logicalhan Oct 1, 2023

matthyx Oct 1, 2023 •

edited

Loading

gjkim42 Oct 3, 2023

matthyx Oct 3, 2023

gjkim42 Oct 3, 2023 •

edited

Loading

matthyx Oct 4, 2023

logicalhan Oct 1, 2023

matthyx Oct 1, 2023

bart0sh commented Oct 1, 2023

wojtek-t Oct 3, 2023

wojtek-t Oct 4, 2023

matthyx Oct 5, 2023

matthyx Oct 5, 2023

wojtek-t Oct 5, 2023

wojtek-t Oct 3, 2023

wojtek-t Oct 3, 2023

wojtek-t Oct 3, 2023

wojtek-t Oct 3, 2023

wojtek-t left a comment

wojtek-t Oct 4, 2023

matthyx Oct 5, 2023

matthyx Oct 5, 2023

gjkim42 Oct 5, 2023 •

edited

Loading

matthyx Oct 5, 2023

wojtek-t Oct 4, 2023

wojtek-t Oct 4, 2023

wojtek-t left a comment

wojtek-t Oct 5, 2023

wojtek-t Oct 5, 2023

wojtek-t Oct 5, 2023

matthyx commented Oct 5, 2023

wojtek-t commented Oct 5, 2023

k8s-ci-robot commented Oct 5, 2023

KEP-753: add PRR questionnaire answers for beta #4255

KEP-753: add PRR questionnaire answers for beta #4255

Conversation

matthyx commented Oct 1, 2023

Choose a reason for hiding this comment

matthyx Oct 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gjkim42 Oct 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bart0sh commented Oct 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wojtek-t left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gjkim42 Oct 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wojtek-t left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthyx commented Oct 5, 2023

wojtek-t commented Oct 5, 2023

k8s-ci-robot commented Oct 5, 2023

matthyx Oct 1, 2023 •

edited

Loading

gjkim42 Oct 3, 2023 •

edited

Loading

gjkim42 Oct 5, 2023 •

edited

Loading