-
Notifications
You must be signed in to change notification settings - Fork 295
Destroy do not clean up everything #59
Comments
Hi @aholbreich! Have you tried creating k8s clusters with |
I'm closing this issue for the reason I've explained above but please feel free to reopen that's not correct. |
Hmm... Maybe you've right... I've tried supergiant.io as well. |
Hi @aholbreich! If you've encountered that next time, would you mind sharing error messages coming from CloudFormation? I guess those can be seen in stack events. Also note that if you've modified resources created by kube-aws's cfn stack by hand or by another script which doesn't relate to kube-aws, resource deletion would have failed. Examples:
|
@mumoshu will do. But for moment no plans to use it again. Looking forward to documentation and more configuration flexibility. |
@aholbreich Thanks for the feedback! Any specific feature/documentation request about that would be welcomed 👍 The point is that, I'm not quite sure what people don't know. If you've seen which company I belong, you'll probably notice I'm just a super active kube-aws user (just an user like you!)/a primary maintainer, not a CoreOS staff. I'm personally equipped with well-known gotcha's related to AWS, CloudFormation, etc., therefore I don't have feelings of inconvenience about kube-aws documentation yet i.e. I don't know about our users enough. So github issues with specific feature and/or documentation request is welcomed, to allow me to know more about our users 🙇 |
@mumoshu i'm not experinced with CF so far at all. But we using AWS since a couple of month. Unfortunately with ECS so far... As well i'm not an expert in kubernetes or CoreOS and at the moment i want to avoid the need for getting to know all the details of installation of that tools - my reason to use kube-aws. We need well understood, self described and kind of well documented starting point for kubernetes not only for me but as well for "duty guys" that maybe have to scale out a cluster even i've move on to the next project. Me as well do not want' go into much details on the intermediate level. The perfect CLI in my eyes is capable of:
I'm not sure if that are the goals of that project, I'm not sure if it possible with Cloudformation at all... But you've asked ;)) Maybe this helps you too: Feed back to kube-aws |
Hi @aholbreich, thanks for your detailed feedback! I agree there are still some real UX issues with the project as-is. We need to strike a balance between usability, maintainability and the need for people to make their own custom adjustments to the deployment. As far as i understand your comments here, i think the goals are aligned. One of the difficulties here is there are a lot of components to this system that take a while to grasp, especially for users who are new to both AWS and kubernetes. (me, a year ago 😆). I don't know how to make this easier for users except for really "one size fits all" defaults and maybe a few different walkthroughs for deployment scenarios. The truth is most of the kubernetes community itself is still learning how to properly deploy and maintain these complex systems. Things are improving though and i'm really looking forward to cool stuff on the horizon (self-hosted kubernetes clusters, etcd operators, better UX). As for your comments in the article:
This is possible now with the
I don't like reinventing the wheel. You can use the AWS console or the cli
The configuration options are documented inside the configuration file itself (as is standard for most tools). I can understand that some options seem a bit daunting, though i'm not sure how to fix that.
This is true. I think articles from users could really help here. We could manage a list of articles written by users so they're easier to find?
All nodes except the etcd cluster are deployed using (fixed) autoscaling groups. Scaling the cluster is a manual tasks currently (set
This only happens when you manually referenced the security groups in other security groups and is a cloudformation limitation. The other option is to forcibly remove the resources but i think it's better to fail safely in these instances.
Can you be more specific here? What kind of resources would you like to have reused? |
Hey @aholbreich, I greatly appreciate your detailed feedback including the nice blog post!
That's exactly what I need, too.
I understand we should eventually pave the way to that.
Hey, all of those are what I'd personally like to have in kube-aws 😆
It's ok because technically, all the things are achievable 👍
Yes. I appreciate this kind of feedback to shape our long-term goals and plans correctly. Our current status/actions could be taken immediately are as @pieterlange kindly explained in the above comment. I'm also guessing that if we could add something like a Anyways, thank you again for your great feedback @aholbreich 🙇 |
@pieterlange @mumoshu glad you liked my feedback and that you dealing professional with it even if parts of my feedback are not that accurate. Also sorry for not answering quite long. Thank you both for clarifying some details, i'll go back to this if get chance to spend more time on kube-aws again. At the moment i work on ansible based provisioning but maybe i have to go back and squeeze-out everything form cloudformation, so i can contribute in better quality here. |
I've been watching email theads and was planning on trying to summarize everything that I've needed to do to get to the point where we are comfortable with kube-aws working in EC2 (at least in DEV ATM). First of all thanks again to @mumoshu for all his and fellow contributors feedback. They were very helpful getting us going with kube-aws. There are a lot of disparate docs/issues, etc that are out there, but it's sometimes hard to put them all together. A limiting factor when working with kube-was for me was lack of IRC presence that I'm aware of. For example, if you run into problems with either deis or helm, I can jump on their respective IRC slack channels, and within a few minutes, I can get a developer providing crucial feedback, workarounds, known-bugs/issues, etc. I have tried to find folks on the kubernetes IRC channels (kubernetes-users, sig-aws, etc.) that have experience/knowledge of kube-aws, kops, etc., but unfortunately, as soon as I mention kube-aws, kops, etc., 'will get crickets or "I don't know anything about those. . ." It would be great to have a dedicated IRC channel for kube-aws where experts to noobs could lurk about. Regarding cloud ENV's, the important thing to keep in mind is that every user's cloud provider, network, security groups, ACL's will be different and if you think about it, how would you handle all the testing, use cases, etc? For me/us, kube-aws is the best I found, and this was after evaluating/testing: Terraform, Cloudify, kube-up, kops, etc. There are many complicated components and requirements that are involved with any tool/framework, but if you think about what you get with kube-aws, and after looking through the 1,000+ line long CloudFormation stack templates, you get a hint of and can appreciate what these developers are having to wrangle and maintain. With that said, what I found was with previous RC.4 release, kube-aws did not seem to have issues with my integrations of pre-existing security groups, etc. and was able to successfully destroy the stack. That changed for me when we went from kube-aws-v0.9.1-rc.4 to kube-aws-v0.9.1. For now I've just been using the CloudFormation API and manually deleting the stack (after kube-aws destroy fails to clear what I think was the kube policies and roles and perhaps load balancer components). This is an extra steps but only takes a few minutes max. Off the top of my head (there is more and more details but a quick summary here of what I needed to do in order to get kube-aws working in my cloud ENV)
There was also required hacks to userdata/cloud-config/etc to force the usage of AWS DNS (not internal) which is something like this replace '%H' with your etcd master IP you get from your cluster.yaml). There are issues on github which documented this and again thanks to @mumoshu for helping direct me to them
I also needed to modify my stack-template.json file to allow access to 443 for each kubernetes security group else my kube cluster etcd nodes etc never could obtain required external resources. Again, YMMV, this is what worked for me. Also AIR, EC2 SG's are stateful, so if you allow something in it will be allowed out and vice versa, but I have it specified just so I can see it and keep it straight in my head and in our SCR (git)
And Then referenced my custom SG in each of the above mentioned kube-aws security groups like this:
There are additional modifications I need to do for other features, but hopefully this might help you or others get up and working with kube-aws within your EC2 VPC ENV. I had issues when trying to get my clusters working inside an internal subnet with NAT, so for me, I am using kube-aws on an external subnet with direct access to the internet with SG modifications to only allow ssh, etc. from our internal CIDR range. This works well enough for us for now and we can refine, etc. as we go. Hope this helps others. |
Just an update/note in case anyone else runs across similar issues; the customized cloud-init hacks I did above (specifically userdata/cloud-config/etcd) obviously won't work for multi etcd/master infrastructure environments--the below noted issue documents what we needed to do for those deployments. Unfortunately now with multi-etcd masters and multiple AZ's, we're now having problems deploying into an AWS VPC that has a custom DHCP option set. The symptom/effect of the problem we're seeing is that the kube-aws provided hostnames are not resolvable in DNS, thereby creating Kubernetes (etcd) deployment problems, etc., as documented in the issue with the workaround #189. |
Just wanted to add something related to this point - |
…helm-flyte-operators to hcom-flavour * commit '218a8d037d685bc6d7d9bdba21c6fdd28f143685': Remove from install-kube-system Remove Helm and Flyte operators from kube-aws code
It looks like on destroy now all resources are cleaned up.
What i've seen as left over is:
Create New Role Role Actions
IAM > Roles > kubernetes-master
IAM > Roles > kubernetes-minion
and i believe security Groups too.. But my they are deleted meanwhile, so cannot provide names.
The text was updated successfully, but these errors were encountered: