Suggestion for consolidating our efforts

[mumoshu]

To be short, I'd like to find a way to merge our efforts in the area of kubernetes deployment tools.
Would "you" mind suggesting how's and/or express agreement(s) to one of the suggestions possibly in this thread?

Please read the kubernetes-dev thread for more context.

Merging our efforts would enable:

• Every kubernetes user easily get to one or two appropriate tools according to one's requirements
• We, tool developers, won't need to redo works already done for another tools

The list of tools includes:

• kube-aws
• kops
• kargo
• kubernetes-anywhere
• bootkube
• kubeadm
• etc.

---

As of today, kube-aws already plans to integrate with bootkube in the future.

However I want more; just for example, how about merging kube-aws into kops? More concretely, I imagine about kube-aws being the cloudformation provider for kops.

Of course, being one of providers for kops doesn't mean kube-aws can lose the following important features:

• Manageable with SCM like Git to support customizations of clusters (Support the use-case to manage multiple kube-aws clusters' configurations optionally inheriting organization-specific customizations with a version control system like Git #238)
• Based on CloudFormation 😃
• Integrated node pools
• H/A everywhere
• Flexibility to allow customizing via cluster.yaml, stack template templaets, stack templates, userdata
• Bring your own AWS resources to be used (where necessary, where possible)
• Various work-arounds to go beyond AWS limits(16KB in userdata, 512KB in cfn stack template's size)
• etc.

I'll keep every aspect of kube-aws for our current users, as much as possible. However, I don't believe that is something prevents us from merging our efforts.

Personally, I suggested in sig-aws about merging kube-aws into kops.
Would someone agrees and/or wants to take a leadership to make it a reality?

[redbaron]

I appreciate effort to avoid duplication, but merging with kops would be a major disappointment for us, for following few, but very important reasons:

1. kops codebase is huge, last time I checked it was ~40k LOC vs ~4k LOC in kube-aws (excluding tests and vendored libs). Kops tries to reimplement what TF/CF do well. Relative ease of adding custom features to a tool influenced our choice significantly.
2. ease of adding features without modifying a tool. Intermediate template which is then rendered into a massive CF stack is life saver. It provides endless flexibility in what one can do. Many features which started like flags in .customSettings were contributed back to kube-aws. Closest we can get to in kops would be to render terraform and then ammend it, but kops model doesn't have clean and easy to understand vars + template files, so making similar customisations won't be as easy.

[cknowles]

I have also been following developments here, mostly kops and kubeadm. kube-aws has been doing really well recently in terms of production readiness and other new features, thanks @mumoshu 🙇. Even with this, I think we could benefit from the additional community of the kops project.

Definitely agree with @redbaron about CloudFormation and customisation. It's not an obvious combination at first with kops. I suppose if the parts of kops that manage state are/were separated out and there's some good interfaces to plug in kube-aws like functionality or bypass the state management then perhaps there's some way to combine.

[rochdev]

I concur with merging with another project. It's still quite difficult to find a stable solution for deploying/managing kubernetes clusters because the community effort is so spread out at the moment.

[cknowles]

Interestingly I believe kops has some support for Container Linux and also CloudFormation now.

[mumoshu]

@rochdev Thanks for the feedback 👍 Yeah, I think so too.

Anyway, the problem is that we as the whole k8s community still don't know how it can be achieved without bloating code-base or slowing-down the development.

For example, how can we merge kops(multi os, terraform/cfn/cloudup, go), kubespray(ansible-based), kube-aws(cfn, bash, cloud-init) into one? Extracting a common provisioning scripts from these into an another project and use it all over the places? Wouldn't it slow down the development?

Similarly, for example, how can we merge kops and kube-aws without fattening code-base too much? kube-aws as a kops plugin or vice versa? How and who takes lead?

For me, integrating bootkube seems to be the only way to go ahead with consolidating our efforts without big down-side. WDYT?

[mumoshu]

@c-knowles FYI hereis a bit more context on that: https://groups.google.com/forum/#!topic/kubernetes-sig-aws/L1E2F2Easbw

[cknowles]

Thanks @mumoshu! It seems there is a reply in that thread worth looking at and some confusion over who the owner of kube-aws. I now consider that to be you rather than CoreOS since they seem to be concentrating on Tectonic primarily. As for backing and getting this done, happy to help out in any way I can.

[camilb]

@mumoshu @redbaron @danielfm @c-knowles I'm traveling this month and don't have time to investigate it but kubecorn looks very promising.

[mumoshu]

I basically agree - Not sure if re-invention of terraform or cfn is sustainable. I definitely like it being just an idiomatic go library to define a uniform interface among various tools and cloudproviders though! 2017年7月19日(水) 22:04 Camil Blanaru <[email protected]>:

…

@mumoshu <https://github.com/mumoshu> @redbaron <https://github.com/redbaron> @danielfm <https://github.com/danielfm> @c-knowles <https://github.com/c-knowles> I'm traveling this month and don't have time to investigate it but kubecorn <https://github.com/kris-nova/kubicorn/blob/master/docs/aws/walkthrough.md> looks very promising. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#464 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABV-QlvFvct1Ps_QHXkunfQL-e9y4tRks5sPf3pgaJpZM4MpoFr> .

[discordianfish]

I'm right now looking into kubernetes deployment options on AWS.
At a first glance it seems to me, never having used kops nor kube-aws before, that kube-aws was built because some people weren't happy with design decisions in kops as mentioned by @redbaron.

So I think more important than merging efforts would be to clarify the difference and maybe different believes behind the design. That would make it much easier for me to figure out which option I prefer.

[ctownsen357]

Just adding another comment to consider and this is just my opinion. I've recently tried using both kops and kube-aws to deploy to AWS GovCloud. So far I have not been able to do that with kops. It was rather trivial to do with kube-aws. The major hurdles for kops on GovCloud are:

• The AMIs being used by kops don't exist on GovCloud
• Route53 doesn't exist as a service on GovCloud
• The documentation for possible workarounds is very fragmented

kube-aws works on AWS Govcloud VERY well and very easy. I'd hate to see a merge take away any of the simplicity of kube-aws.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.