Improve the multi-platform compability

[luxas]

Kubernetes on multiple platforms

• One-line feature description (can be used as a release note): Kubernetes should work on the platforms the community expects it to work on. Automated CI e2e tests should be run for all supported architectures. It should be possible to run clusters with nodes of mixed architectures.
• Primary contact (assignee): @luxas @mkumatag @ixdy
• Responsible SIGs: no formal sig / sig-release will probably be the closest
  • Mostly this is about making our release/test tooling work on multiple platforms and providing a deployment solution that works everywhere: kubeadm
• Design proposal link (community repo): https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multi-platform.md
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @ixdy @luxas @vishh
• Approver (likely from SIG/area to which feature belongs): @ixdy @luxas @vishh @thockin @brendandburns
• Feature target (which target equals to which milestone):
  • Alpha release target (x.y): v1.3/v1.4
  • Beta release target (x.y): v1.12?
  • Stable release target (x.y): TBD

It's very hard to estimate alpha/beta/stable levels of this feature as there is no clear graduation path.
Anyway, I'd consider this feature "done" when:

• We have automated CI e2e Conformance tests running for arm, arm64, ppc64le, s390x and windows continously. Currently we're lacking the hardware support. I'm in touch with CNCF to fix this eventually. Also the e2e suite should be
• Manifest lists are pushed to the same image tag is usable on any platform without having to substitute -ARCH in order to get the right docker image downloaded.
• All Kubernetes core and Incubated projects push artifacts for all supported platforms
• QEMU inside of the multiarch containers isn't needed anymore. When building docker images for a foreign architecture, we're using QEMU for the syscall emulation, which currently requires the QEMU binary to be in the resulting image. However, with a new 4.8 kernel it can be done without any extra addition to the resulting image.

KubeCon video where I'm talking about this: https://www.youtube.com/watch?v=ZdzKQwMjg2w

[mkumatag]

Sounds good to me..

[idvoretskyi]

Labeled as "help wanted" - SIG has to be defined.

[luxas]

@idvoretskyi As soon as SIG Release is formally formed as a group, I expect the work to belong to that SIG. Also crosses some borders with the cluster lifecycle SIG.

[idvoretskyi]

@calebamiles @pwittrock is SIG-release going to work on this feature?

[luxas]

@idvoretskyi To clarify, so far I and a couple of other persons have been hacking on this with no formal SIG owning it.
This is basically a thing that touches sig-cluster-lifecycle, sig-testing, sig-release and sig-windows all at the same time.

However, I'll talk to @ixdy @vishh @thockin and sig-release where we should formally place this

[idvoretskyi]

@luxas it's amazing to see this work in progress!

At the same time, the feature has to have one or multiple SIG's, that are formally responsible for its implementation.

[idvoretskyi]

@luxas FYI - I've created a sig/release label and labeled this issue with it.

[luxas]

@idvoretskyi Perfect, waited for that label!
As you saw, I removed it from the v1.7 as it won't make it due to external dependencies (gcr.io, etc.).

[luxas]

I also removed it from Action Required as no actions has to be made for this, targeting v1.8 now

[luxas]

This depends on the GCR team adding support for manifest lists. Adding to the v1.8 milestone for now.
I'm not sure the dependency will be resolved before code freeze (I have waited a year for this to happen already), but in case it does we should track this.

[detiber]

Is GCR support for manifest list images an absolute requirement for this? Could docker hub or another public registry that supports manifest list images (if there are any) be used until GCR supports manifest list images?

[luxas]

Is GCR support for manifest list images an absolute requirement for this?

Yes, as gcr.io is the repo we're using for kubernetes release artifacts.

Could docker hub or another public registry that supports manifest list images (if there are any) be used until GCR supports manifest list images?

No unfortunately not; as that would be a huge, backwards-incompatible change and would result in lots of other complications.

I'm told the GCR team is still working on this 😄; but due to that it is not ready yet, this stays in alpha :(
On the good side; significant improvements have been made during the cycle to make sure all Conformance tests (an other requirement for beta) are now passing on the other platforms 👍

[kacole2]

@luxas @mkumatag @ixdy I'm following up from @claurence's post if there any plans for this to graduate in 1.13?

[neolit123]

given the shortened 1.13 release cycle, i would say that only some work for missing e2e tests could be done in 1.13, maybe CNI could see improvements, but that's a 3rd party dependency.

RE: incubated projects: IMHO, a separate KEP should be sent for this and we should omit the incubated project demand from this feature as outlined here.

[kacole2]

/milestone clear

[timothysc]

At this point, this really should not be owned by sig-cluster-lifecycle but should be passed onto @kubernetes/sig-release-feature-requests or the k8s-infra-team.

[dims]

@timothysc sig-testing may be another option as the main thing lacking right now is automated CI e2e Conformance tests running for arm, arm64, ppc64le, s390x and windows (well, we do have ppc64le, but others are missing)

[timothysc]

@dims I don't think it should be on sig-testings shoulders to manage things like s390x when they don't have the gear to debug. Ideally, I think the consumers should supply signal via federated testing.

[timothysc]

/unassign @timothysc

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[vielmetti]

@dims - is there a concise description of the hardware and software needs for "federated testing" as described by @timothysc above?

I see this has gone stale again, but I don't think the need has passed for it!

[dims]

@vielmetti yes!. what he meant was that interested parties should set up their own CI using whatever tools they deem fit BUT publish the results to the community test grid, so any consumers or say the release team can check if any recent changes have broken functionality in other platforms.

Here's a HOWTO post results to the community test grid:
https://github.com/kubernetes/test-infra/tree/master/testgrid/conformance

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[neolit123]

/lifecycle frozen

[neolit123]

/lifecycle frozen

[fejta-bot]

Enhancement issues opened in kubernetes/enhancements should never be marked as frozen.
Enhancement Owners can ensure that enhancements stay fresh by consistently updating their states across release cycles.

/remove-lifecycle frozen

[kacole2]

Hello @neolit123 @luxas , I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. A KEP will need to be merged before this can be included in 1.15 as well.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[dims]

/close

we have separate issues for different platforms already logged. let's close this one out

[k8s-ci-robot]

@dims: Closing this issue.

In response to this:

/close

we have separate issues for different platforms already logged. let's close this one out

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

@dims thanks for taking the action.

we have separate issues for different platforms already logged. let's close this one out

+1