Topology Aware Routing

[robscott]

Enhancement Description

• One-line enhancement description (can be used as a release note): EndpointSlice controller can be configured to provide topology aware hints that will enable simple topology aware routing.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/2433-topology-aware-hints
• Discussion Link: https://docs.google.com/document/d/1ZzUoFY1SrdjVefl7gVOJZJLt1I1LHttw8pcX95nlgMY/edit
• Primary contact (assignee): @robscott
• Responsible SIGs: sig-network
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): v1.21
  • Beta release target (x.y): v1.23
  • Stable release target (x.y): v1.28
• Alpha
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s):

/sig network

[robscott]

/assign

[kendallroden]

Hi @robscott ,
Since your Enhancement is scheduled to be in 1.21, please keep in mind the important upcoming dates:

• Tuesday, March 9th: Week 9 - Code Freeze
• Tuesday, March 16th: Week 10 - Docs Placeholder PR deadline
  • If this enhancement requires new docs or modification to existing docs, please follow the steps in the Open a placeholder PR doc to open a PR against k/website repo.
• As a reminder, please link all of your k/k PR(s) and k/website PR(s) to this issue so we can track them.

Thanks!

[JornShen]

@robscott What' the develop process of this feature? I am interested in it. Is there something I can help to do ? :)

[robscott]

Hey @JornShen thanks for checking in! If you have time, I'd really appreciate any review or testing of my related PR: kubernetes/kubernetes#99522.

[kendallroden]

Hey @robscott! I know you are aware based on other enhancement work that code freeze is coming up on March 9th EOD PST and if any PRs are not merged by the deadline, you'll have to request an exception! Please also keep in mind that if this enhancement requires new docs or modification to existing docs, you'll need to follow the steps in the Open a placeholder PR doc to open a PR against k/website repo by March 16th EOD PST
Thanks!

[annajung]

Hi @robscott, with kubernetes/kubernetes#99522 merged in, we will mark this as code complete for 1.21 release.

[reylejano]

Hello @robscott , 1.21 Docs lead here.
Does this enhancement work planned for 1.21 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.21 branch in the k/website repo. This PR can be just a placeholder at this time and must be created by March 16 EOD PST
Also take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[robscott]

Hey @reylejano, thanks for checking in! This will require some significant docs additions. I'll make sure to have a docs PR ready before that deadline.

[JamesLaverack]

/milestone v1.22
/stage beta

[JamesLaverack]

Hey @robscott, 1.22 Enhancements Lead here,

I have a few questions about your enhancement, as it is currently targeted for 1.22.

• Your kep.yaml needs updating to reflect the graduation. The latest-milestone should be set to "v1.22".

I'm assuming you're targeting beta for this release (if you're not please let me know!) in which case we will also require:

• Your KEP must list graduation critera for beta.
• You will require a beta PRR review from the PRR team.

We require these merged into master by enhancements freeze at 23:59:59 on Thursday 13th May, or else require an exception after this deadline.

Please do reach out if you have any questions.

[thockin]

Moving to 1.29

[AnaMMedina21]

Hola @robscott 👋, v1.29 Enhancements team here.

I am just checking in as we approach enhancements freeze on 18:00 PDT on Thursday 6th October 2023..

This enhancement is targeting for stage stable for v1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: { v1.29}. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

For this KEP, we would just need to update the following:

• For the PRR to be updated with approval for stable

The status of this enhancement is marked as At Risk for Enhancements Freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[jeremyrickard]

The PRR could use another pass for graduating to stable, and is missing a new section in the stability section:

Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

I had a question around the SLO while reading the current version of the PRR template.

What are the reasonable SLOs (Service Level Objectives) for the above SLIs? As a starting point, it is likely reasonable for the EndpointSlice controller to experience up to a 10% sync failure rate. This is largely related to it trying to update stale EndpointSlices. When we are able to find a solution for that issue the expected sync failure rate should be significantly lower. This specific problem is most notable for large Services that have rapidly updating endpoints.

Have you been able to find a solution for the issue around updating stale EndpointSlices? A 10% sync failure rate seems somewhat high for a stable feature. Is that still a good target for understanding if this should be disabled?

[AnaMMedina21]

Hey @robscott 👋, v1.29 Enhancements team here, checking in once more as we approach the v1.29 enhancement freeze deadline on 01:00 UTC, Friday, 6th October 2023. The status of this enhancement is marked as At Risk for Enhancements Freeze. It looks like the PRR needs another pass before approval. Let me know if I missed anything. Thank you!

[robscott]

Hey @AnaMMedina21 and @jeremyrickard, sorry for losing track of this! To clarify, we're not actually trying to make any changes to the KEP this cycle. We ran out of time to complete the changes we wanted to in 1.28 so those have just carried over to 1.29. We're continuing to stay in "beta", which I think means our PRR and everything else should still be good to go:

enhancements/keps/prod-readiness/sig-network/2433.yaml

Lines 1 to 5 in 0d7e982

	kep-number: 2433
	alpha:
	approver: "@wojtek-t"
	beta:
	approver: "@wojtek-t"

Definitely let me know if I'm missing anything though.

[npolshakova]

Hello 👋, 1.29 Enhancements Lead here.
Unfortunately, this enhancement did not meet requirements for v1.29 enhancements freeze.
Feel free to file an exception to add this back to the release tracking process. Thanks!

/milestone clear

[robscott]

@npolshakova can you clarify what we were missing for this milestone? We're not making any changes to the KEP in this cycle, just trying to get the implementation caught up to the KEP... I asked for clarification on this 2 days ago in the comment above and didn't get a response.

[npolshakova]

Hi @robscott, if you have updated the alpha criteria then you do need this enhancement tracked for 1.29. If you intend to just address bug fixes in this release, this does not need to be tracked by the enhancements team for 1.29.

For the enhancement team to track this KEP we need a couple minor updates:

• latest-milestone should be updated to 1.29
• The stable milestone should be removed or bumped from 1.29

Let me know if you have any further questions. Thanks!

[robscott]

Thanks for catching that @npolshakova! I've filed #4284 to update the KEP metadata here. I don't think I can remove the stable milestone from this issue, but we did not intend to target stable in this release, sorry for the confusion.

[robscott]

@npolshakova are you able to update the labels on this issue now that #4284 has merged? Anything else I'm missing?

[salehsedghpour]

/remove-label lead-opted-in

[jonathon2nd]

Checked in to see if there has been any update to allow for previous behavior.

Based of what I see here and the various other PR's and issues, it looks like a no?

Adding image as https://kubernetes.io/docs/concepts/services-networking/service-topology/ has been deleted.

[aojea]

Checked in to see if there has been any update to allow for previous behavior.

see https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/4444-service-routing-preference

[thockin]

@robscott what's the GA pln for this?

[robscott]

@robscott what's the GA pln for this?

The outcome here is closely tied to #4444, as it's a natural evolution of part of this KEP.

This KEP (#2433) essentially bundled two things together - the EndpointSlice hints field and a Service-level annotation that could be used to automatically populate hints with a rather confusing heuristic.

#4444 continues to use the hints field, but adds a new trafficDistribution field and a simpler heuristic instead of the annotation and confusing heuristic. Maybe this table is a better summary:

	#2433	#4444
Hints	Defined in this KEP	Relied on by this KEP
Opt-In	service.kubernetes.io/topology-mode Annotation	trafficDistribution Field
Heuristic	Complicated	Simple

My recommendation would be to keep the hints field defined in #2433 and the trafficDistribution field and simpler heuristic from #4444.

If we're good with that plan, I think that when #4444 graduates to beta (likely v1.31) we should formally deprecate the annotation and corresponding heuristic defined by this KEP. Then we can work to graduate #2433 to GA as a hints-only KEP and #4444 to GA as a KEP covering the trafficDistribution field and simple heuristic.

/cc @gauravkghildiyal