-
Notifications
You must be signed in to change notification settings - Fork 295
Feature: Spot Fleet support for worker nodes #112
Comments
Edited several times to cover more TODOs and concerns. |
Addressed the integration tests in f46c711#diff-17c3b4ff0a8d67faed426a76a03f8430R1 |
I'm going to add support for node labels and taints in another pull request(s). The context is that If we don't need to mix up various types of nodes but just need to use Spot Fleets, labels and taints are not required. So I believe I can cut the #113 now and deliver it so that we can start supporting some use-cases. For an another use-case like "I want to mix up various types of nodes for blah-blah-blah`, I can author an another pull request addressing labels and taints. |
See updated nodepool/config/templates/cluster.yaml for the detailed guide of configuration. This is the initial implementation for kubernetes-retired#112 Beware that this feature may change in backward-incompatible ways
The initial implementation for this is now merged into master. |
Btw: I don't mean to show off, but my personal project https://github.com/mumoshu/kube-spot-termination-notice-handler would be useful for anyone wants to gracefully stop pods running on spot instances when your spot fleet lost to bids. Deploying it to your spot instances allows you to automatically run |
Though I thought it would be nice to add initially, do we really need the feature to add user-provided labels to worker nodes? |
I think this is useful; operators might want to restrict pod scheduling to certain node pools because of node capabilities or security domains. Example usecase: i have the majority of nodes in private subnets but i start a few in public subnets because they need to directly expose some service to the internet on. With node labels i restrict those pods to the public nodes. |
Do we know the difference between a node selector on a pod and a taint toleration on a pod? It seems like they could achieve similar things as far as I can see in the taint design docs. |
I'm now taking a look into an issue that worker nodes brought up from spot-fleet-enabled node pool often fail to register themselves. More specifically, if you've created 2 or more nodes backed by a spot fleet, only one of them are registered. Making TargetCapacity larger hence adding nodes seems to consistently result in spot instances successfully get launched but their corresponding nodes unregistered. kubelet does report that it successfully registered node. However immediately after that kubelet starts complaining the node it just registered can not be found. I suspect that missing Edit: Bingo! Putting a tag named I'll shortly submit a pull request to address this. Maybe I'll utilize the quay.io/coreos/awscli docker image to run a command almost the same as what @innovia described in his comment to the upstream issue.
|
Btw, just noticed that |
…leet couldn't be registered thus unable to run any pods ref kubernetes-retired#112 (comment)
@pieterlange In that case, wouldn't you like to use taints rather than labels? IMHO taints are more failproof than labels. If you've used taints to implement dedicated nodes, pods missing tolerations won't be scheduled to anywhere thus you can ensure that only the desired pods are scheduled to desired nodes. On the other hand, if you've used labels, pods missing node selectors will end up with completely useless deployment - pods get distributed over both private and public nodes. |
@c-knowles Both taints and labels could be used to select subset of nodes to schedule pods. |
IMHO it is perfectly fine to use node labels for purposes other than dedicated nodes(=reserved for specific pods). An example use-case for node labels could be that running administrative tasks on subsets of nodes. |
This complements Node Pools(kubernetes-retired#46) and Spot Fleet support(kubernetes-retired#112) The former `experimental.nodeLabel` configuration key is renamed to `experimental.awsNodeLabels` to avoid collision with newly added `experimental.nodeLabels` and consistency with `experimental.awsEnvironment`.
All the remaining TODOs are going to be addressed in v0.9.3-rc.1 |
… spot fleet to conform these nodes to the ones powered by a autoscaling group ref kubernetes-retired#112 see also http://docs.aws.amazon.com/cli/latest/reference/ec2/create-tags.html and http://stackoverflow.com/a/1250279 for implementation details
All the TODOs have been addressed. |
Closing this issue as the initial iterations to bring the feature have finished. |
…leet couldn't be registered thus unable to run any pods ref kubernetes-retired#112 (comment)
This complements Node Pools(kubernetes-retired#46) and Spot Fleet support(kubernetes-retired#112) The former `experimental.nodeLabel` configuration key is renamed to `experimental.awsNodeLabels` to avoid collision with newly added `experimental.nodeLabels` and consistency with `experimental.awsEnvironment`.
… spot fleet to conform these nodes to the ones powered by a autoscaling group ref kubernetes-retired#112 see also http://docs.aws.amazon.com/cli/latest/reference/ec2/create-tags.html and http://stackoverflow.com/a/1250279 for implementation details
Quite self explanatory but I'd like to add this to kube-aws.
Upstream issue: kubernetes/kubernetes#24472
Initial Implementation in this project: #113
Documentation: https://github.com/coreos/kube-aws/blob/master/Documentation/kubernetes-on-aws-node-pool.md#deploying-a-node-pool-powered-by-spot-fleet
Spot fleet backed worker nodes are supported since v0.9.2-rc.3:
An experimental feature to automatically taint nodes with user-provided taints is supported since v0.9.2-rc.4(not yet released) so we can ensure only pods tolerant to frequent node terminations are scheduled to spot instances/spot-fleet-powered nodes:
Utilizing Spot Fleet gives us chances to dramatically reduce cost being spent on EC2 instances powering Kubernetes worker nodes
AWS says cost reduction is up to 90%. I can confirm that in my daily used region
ap-northeast-1
it is up to 89% right now, with slightly varying cost for each instance type.I believe that on top of the recent work on Node Pools #46, it is easier than ever to implement a POC of the Spot Fleet support.
I'll send a pull request to show it shortly.
I'd appreciate your feedback(s)!
Several concerns I've came up with until now:
aws-ec2-spot-fleet-role
IAM role created in their AWS accounts automatically by accessing Spot Fleets in AWS Console at least oncekube-aws nodepool up
will fail while copy-pasting an error message from CloudFormation likeIAM role aws-ec2-spot-fleet-role doesn't exist
, which may be useless to the user as it doesn't provide any information to notify the user needed to arrive Spot Fleet in AWS console at least onceTODOs:
worker.spotFleet.unitRootVolumeSize * weightedCapacity
,worker.spotFleet.unitRootVolumeIOPS * weightedCapacity
respectivelykey=value:effect
#132--register-node=true --register-schedurable=false
followed bykubectl taint
andkubectl uncordon
would avoid any race-condition which can result in undesired pods getting scheduled to undesired nodes while adding a new node/kubelet to a cluster.Experimental.LoadBalancer.Names
is not taken into account Conform worker nodes powered by spot fleets to the ones powered by autoscaling groups #167Name
tags on a spot instance with the same value as workers in a main cluster Conform worker nodes powered by spot fleets to the ones powered by autoscaling groups #167cluster.yaml
s)The text was updated successfully, but these errors were encountered: