Kubelet Device Plugin Registration

[saad-ali]

Feature Description

• One-line feature description (can be used as a release note): Kubelet should have a standard way to discover local plugins (CSI, GPU, etc.).
• Primary contact (assignee): @vikaschoudhary16
• Responsible SIGs: @kubernetes/sig-node-feature-requests @kubernetes/sig-storage-feature-requests
• Design proposal link (community repo):
  • Add plugin watcher design community#2369
  • Google Doc
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @RenaudWasTaken @vishh @jiayingz
• Approver (likely from SIG/area to which feature belongs): @dchen1107 @saad-ali
• Feature target (which target equals to which milestone):
  • Alpha release target (x.y): 1.11
  • Beta release target (x.y): 1.12
  • Stable release target (x.y): 1.13

[saad-ali]

Introduced as alpha in v1.11.

See PR kubernetes/kubernetes#63328
See issue: kubernetes/kubernetes#56944

Partial list of work for beta:

• Recursive detection of UDS in sub directories
• Unregistration of drivers when UDS is deleted

[justaugustus]

Thanks for the update! This has been added to the 1.12 Tracking sheet.
Who's the primary SIG on this? SIG Node or SIG Storage?

/stage beta

[RenaudWasTaken]

The umbrella issue tracker is here: kubernetes/kubernetes#65773

[vikaschoudhary16]

@saad-ali design proposal PR (migration from original google doc): kubernetes/community#2369

[zparnold]

Hey there! @saad-ali I'm the wrangler for the Docs this release. Is there any chance I could have you open up a docs PR against the release-1.12 branch as a placeholder? That gives us more confidence in the feature shipping in this release and gives me something to work with when we start doing reviews/edits. Thanks! If this feature does not require docs, could you please update the features tracking spreadsheet to reflect it?

[RenaudWasTaken]

If this feature does not require docs, could you please update the features tracking spreadsheet to reflect it?

Documentation is needed but not as part of the Kubernetes website.
Documentation will happen:

• In the kubernetes/csi repository
• In Readme's the kubernetes/pkg/kubelet/... source

[zparnold]

Sounds perfect! Thanks! I'll mark that in the feature tracking spreadsheet

[justaugustus]

@RenaudWasTaken -- please be careful using Fix[es] or Closes in a commit as it may cause us to fall into a close loop when the bots pick up a commit.

/reopen

[k8s-ci-robot]

@justaugustus: Reopening this issue.

In response to this:

@RenaudWasTaken -- please be careful using Fix[es] or Closes in a commit as it may cause us to fall into a close loop when the bots pick up a commit.

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[idealhack]

/reopen

Is there anything we can do to stop the loop?

[justaugustus]

I think it'll take effect on every branch transaction that the commit gets pulled into. I'd say that you could do it by amending the commit, but it might be less effort to just reopen this issue every time :/

[RenaudWasTaken]

Sorry I didn't see that Github converted it into a "fix". I Will pay more attention to that in the future.

[ameukam]

Hi folks,
Kubernetes 1.13 is going to be a 'stable' release since the cycle is only 10 weeks. We encourage no big alpha features and only consider adding this feature if you have a high level of confidence it will make code slush by 11/09. Are there plans for this enhancement to graduate to stable within the 1.13 release cycle? If not, can you please remove it from the 1.12 milestone or add it to 1.13?

We are also now encouraging that every new enhancement aligns with a KEP. If a KEP has been created, please link to it in the original post. Please take the opportunity to develop a KEP.

[RenaudWasTaken]

Hello @ameukam !

This feature graduated to beta in 1.12 and there are no plans to graduate it to stable in 1.13.

[saad-ali]

This feature graduated to beta in 1.12 and there are no plans to graduate it to stable in 1.13.

CSI depends on this feature. @vladimirvivien is syncing with @RenaudWasTaken to clarify. Preliminary plan is to move to stable in 1.13.

[kacole2]

@saad-ali I've added this to the tracking sheet but putting it At Risk because I feel this has not had enough time to bake. I believe there will be concern about promoting a feature so quickly. @AishSundar @spiffxp

[AishSundar]

@saad-ali what work is left for this in 1.13 to be able to go to Stable? Similarly how confident are we of CSI making it to Stable in 1.13 and if [CSI] slips what are the plans for this feature then?

[vladimirvivien]

@AishSundar
I have discussed this with author (and other interested parties). He feels confident that the code OK and stable given adoption. This feature is being used by several plugin implementers and users (NVIDIA device plugin, several CSI volume plugins) since beta. If CSI slips, this feature is adopted by other plugin providers who are Ok with v1.

[AishSundar]

@vladimirvivien thanks for the update. Could you plz point us to a list of pending PRs (code, tests and docs) for this feature?

[vladimirvivien]

@AishSundar

• PR for dev doc - Update pluginwatcher doc kubernetes#68562
• Pending issue to be handled this quarter - Plugin watcher getting notified of every Kubelet directory event kubernetes#69015
  That's all for now, I will add additional PRs as they are opened.

[AishSundar]

@vladimirvivien can I plz know the latest status on this for 1.13. I see quite a few PRs pending in kubernetes/kubernetes#69015. With Code freeze approaching this Friday 11/16, I am afraid its too little time for all the changes. Could you plz provide a list of pending PRs tracking for 1.13?

[RenaudWasTaken]

Hello @AishSundar !

There are only two PRs pending:

• Graduate Kubelet plugin registration/watcher to GA kubernetes#70559
  • Which is only missing approval, I believe we'll bring this up (again) in Tuesday's sig-node
• Ignore non socket files in the kubelet plugin watcher kubernetes#70494
  • Which only needs us to choose an option
  • Whatever option is chosen, code changes will be minimal

In both case I believe the deadline of Friday 11/16 will be respected.

Thanks for looking into this!

[vladimirvivien]

@AishSundar as @RenaudWasTaken mentioned we have an approved PR (which needs rebase) and an additional PR which should get resolved early this week. All other PR's have been resolved or merged. We are tracking this to be in by code freeze this Friday. Thank you for addressing this.

[kacole2]

closed kubernetes/kubernetes#70484 via kubernetes/kubernetes#70559

[AishSundar]

@vladimirvivien @RenaudWasTaken looks like there is still some active discussion in the PR kubernetes/kubernetes#70494. Are we still on track for Code freeze tomorrow?

[AishSundar]

Discussing with @msau42 pending PR kubernetes/kubernetes#70494 is an optimization and not really a blocker for this enhancement in 1.13. As much as we will try to get in today, this can also go as a 1.13.1 patch if merge get delayed.

[Kymb3rl33]

Stop please

[spiffxp]

What if anything remains to be done here?

[msau42]

From the CSI side, we have no additional requirements for this feature. @jiayingz @vishh is there anything more from the device plugin side that needs to be done?

[jiayingz]

We don't have additional requirements from device plugin side.

[dchen1107]

I am closing this enhancement request.