Add kubernetes_cluster_agent_pool #4046

titilambert · 2019-08-09T16:43:30Z

Ant59 · 2019-08-09T17:53:41Z

Does this also fix #3971 ?

djsly · 2019-08-09T17:29:43Z

azurerm/resource_arm_kubernetes_cluster_agent_pool.go

+			},
+			"resource_group_name": azure.SchemaResourceGroupName(),
+
+			"agent_pool_type": {


Agent Pools are only supported with VMSS. Should we have care about this setting ?

setting the default to AvailabilitySet doesn't seem to make sense

https://docs.microsoft.com/en-us/azure/aks/use-multiple-node-pools#register-multiple-node-pool-feature-provider

https://docs.microsoft.com/en-us/azure/aks/use-multiple-node-pools#limitations

@djsly is correct, we will not be supporting multiple agent pools with VMAS moving forward. We will still have VMAS clusters with a single pool, but if you want multiple pools you'll need to back it with VMSS.

👍 looks like we can remove/default this

titilambert · 2019-10-16T21:38:23Z

@tombuildsstuff I just rebased this patch on master
What do I have to do next ?

titilambert · 2019-10-16T21:53:47Z

@Ant59 Yes !

titilambert · 2019-10-24T14:07:51Z

@tomasaschan @jluk Tests added !

worldspawn · 2019-11-04T13:33:50Z

@titilambert needs a rebase.

Eagerly awaiting this 🤤

titilambert · 2019-11-04T17:14:45Z

@worldspawn @tombuildsstuff rebased !

tombuildsstuff · 2019-11-07T11:47:12Z

azurerm/resource_arm_kubernetes_cluster.go

@@ -107,6 +107,7 @@ func resourceArmKubernetesCluster() *schema.Resource {
 			"agent_pool_profile": {
 				Type:     schema.TypeList,
 				Required: true,
+				MaxItems: 1,


@titilambert firstly sorry for the delay reviewing this!

I've spent the last few days working through trying to reconcile the open AKS PR's (#4676, #4543, #4472, #4256 and this one) to work out how to move forward.

Spending some time on this I believe there's a few things needed to fix this/get these in:

Introduce a new default_node_pool block which is limited to a single element, which replaces the existing agent_pool_profile block (which can then be removed in 2.0) which also allows us to resolve Support planned AKS API default swap to VMSS + SLB (from VMAS + BLB) #4465

Supporting more conditional/delta updates within the AKS resource - allowing ignore_changes to properly work for the count field (amongst others) which fixes azurerm_kubernetes_cluster with multiple agent pools keeps getting re-created #4560, Autoscaling enabled AKS Cluster leads to error on terraform apply, tough no changes on aks planned #4075, Provider produced invalid plan when addon_profile in ignore_changes #3428 and Upgrading Kubernetes cluster via Azure Portal causes terraform to try to recreate it #2993

Adding a more tests covering the open bugs/use-cases

Adding support for multiple node pools (via this PR) which fixes azurerm_kubernetes_cluster: Adding node pools causes AKS cluster replacement #3971 and Terraform replacing AKS nodepool cluster when changing VM count #3835

Adding support for Standard Load Balancer via AKS: Support outbound IPs when using standard loadbalancer #4472

Whilst that's not ideal since 1-3 will supersede #4256 and #4543 (as the agent_pool_profile block is replaced) - this means users only have a breaking behavioural change once via the new block - unfortunately whatever happens it'll cause merge conflicts with #4472, but we can deal with that once the multiple node pools issue is resolved.

I've got the changes for 1-2 locally and I'm working through #3 at the moment - once that's done we should be able to get this merged :)

Since quite a few of those PR's are touching this file (azurerm/resource_arm_kubernetes_cluster.go) - I think it might be worth backing these changes out of this PR (via git reset HEAD~<numberofcommits>, rather than git revert <sha>) to avoid a series of merge conflicts with this PR - WDYT?

I figured there might be some merge conflicts with my code. I'm also happy to create a new PR after this one gets merged, if that makes things easier. 👍

Thanks for considering it.

tombuildsstuff

hey @titilambert @djsly

Thanks for this PR - as mentioned in the earlier comment I'm going to rebase this on top of #4898 now that that's shipped.

Taking a look through this is looking good - I've fixed up the issues I've commented about during the rebase. There's one thing I've intentionally removed during the rebase for the moment - which is the orchestrator_version - since this wants to be added to both the azurerm_kubernetes_cluster resource and this resource at the same time - and I figured this'd be easiest to review in a follow-up PR (since for now the split-out resource features the same fields as the built-in default_node_pool block).

As discussed offline, I hope you don't mind but I'm going to push the rebased/fixed up commits - since the azurerm_kubernetes_cluster resource has changed quite substantially in #4898 I've had to git reset HEAD~N to resolve the merge conflicts here, but I've added you as a co-author to the commit, so you'll still get the Git credit here 👍

Thanks!

tombuildsstuff · 2019-11-07T15:05:26Z

azurerm/resource_arm_kubernetes_cluster_agent_pool.go

+			},
+
+			// AKS name
+			// TODO replace by aks ID


tombuildsstuff · 2019-11-07T15:05:40Z

azurerm/resource_arm_kubernetes_cluster_agent_pool.go

+			},
+			"resource_group_name": azure.SchemaResourceGroupName(),
+
+			"agent_pool_type": {


👍 looks like we can remove/default this

tombuildsstuff · 2019-11-07T15:05:52Z

azurerm/resource_arm_kubernetes_cluster_agent_pool.go

+					string(containerservice.Linux),
+					string(containerservice.Windows),
+				}, true),
+				DiffSuppressFunc: suppress.CaseDifference,


we're making all new fields case-sensitive - as such can we remove this:

Suggested change

DiffSuppressFunc: suppress.CaseDifference,

tombuildsstuff · 2019-11-07T15:05:55Z

azurerm/resource_arm_kubernetes_cluster_agent_pool.go

+				ValidateFunc: validation.StringInSlice([]string{
+					string(containerservice.Linux),
+					string(containerservice.Windows),
+				}, true),


we're making all new fields case-sensitive - as such can we make this:

Suggested change

}, true),

}, false),

tombuildsstuff · 2019-11-07T15:07:11Z

azurerm/resource_arm_kubernetes_cluster_agent_pool.go

+				Optional:     true,
+				Computed:     true,
+				ValidateFunc: validate.NoEmptyStrings,
+			},


for the moment I'd suggest we drop this field from the split-out resource; once this is merged we can then add this to both this resource, and the default_node_pool in the main resource; adding tests for both - my main question here is around if any locking's required when provisioning/updating agents

tombuildsstuff · 2019-11-07T15:09:26Z

azurerm/resource_arm_kubernetes_cluster_agent_pool.go

+				Type:             schema.TypeString,
+				Required:         true,
+				ForceNew:         true,
+				DiffSuppressFunc: suppress.CaseDifference,


we're making all new fields case-sensitive - as such can we remove this:

Suggested change

DiffSuppressFunc: suppress.CaseDifference,

tombuildsstuff · 2019-11-07T15:10:29Z

azurerm/resource_arm_kubernetes_cluster_agent_pool.go

+			"node_count": {
+				Type:         schema.TypeInt,
+				Optional:     true,
+				Default:      1,


given users are almost always going to override this, I think it'd make more sense to make this Required and remove the default?

tombuildsstuff · 2019-11-17T10:51:47Z

@titilambert unfortunately rebasing this has broken the PR in Github - I hope you don't mind but I'm going to open another PR which includes these commits, so you'll still get the credit here :(

ghost · 2019-11-26T08:43:15Z

This has been released in version 1.37.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 1.37.0"
}
# ... other configuration ...

ghost · 2020-03-29T15:00:46Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

ghost added the size/XL label Aug 9, 2019

titilambert mentioned this pull request Aug 9, 2019

New resource - azurerm_kubernetes_cluster_agentpool #4001

Closed

djsly reviewed Aug 9, 2019

View reviewed changes

katbyte added this to the v1.33.0 milestone Aug 10, 2019

katbyte added new-resource service/kubernetes-cluster labels Aug 10, 2019

katbyte modified the milestones: v1.33.0, v1.34.0 Aug 10, 2019

tombuildsstuff modified the milestones: v1.34.0, v1.35.0 Sep 12, 2019

tombuildsstuff modified the milestones: v1.35.0, v1.36.0 Oct 2, 2019

sschne mentioned this pull request Oct 7, 2019

AKS cluster agent pool with fix for autoscaling and count. #4543

Closed

titilambert force-pushed the aksagentpool branch 2 times, most recently from 48f89ae to 3a369b6 Compare October 16, 2019 21:35

titilambert force-pushed the aksagentpool branch from 3a369b6 to 30a2afb Compare October 17, 2019 16:58

ghost added size/XXL and removed size/XL labels Oct 23, 2019

tombuildsstuff modified the milestones: v1.36.0, v1.37.0 Oct 24, 2019

tombuildsstuff self-assigned this Nov 4, 2019

titilambert force-pushed the aksagentpool branch from 7a87418 to 3796ae0 Compare November 4, 2019 17:14

tombuildsstuff reviewed Nov 7, 2019

View reviewed changes

tombuildsstuff mentioned this pull request Nov 17, 2019

r/kubernetes_cluster: supporting conditional updates / introducing default_node_pool #4898

Merged

This was referenced Nov 17, 2019

Closes #4560: Compare agent node pools from Azure API in order of terraform config #4676

Closed

r/kubernetes_cluster: autoscaling-related improvements #4256

Closed

tombuildsstuff suggested changes Nov 17, 2019

View reviewed changes

tombuildsstuff closed this Nov 17, 2019

tombuildsstuff force-pushed the aksagentpool branch from 84ee5e8 to ccb5f4f Compare November 17, 2019 10:50

ghost added size/XS and removed size/XXL labels Nov 17, 2019

tombuildsstuff mentioned this pull request Nov 17, 2019

New Resource: azurerm_kubernetes_cluster_node_pool #4899

Merged

ghost locked and limited conversation to collaborators Mar 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kubernetes_cluster_agent_pool #4046

Add kubernetes_cluster_agent_pool #4046

titilambert commented Aug 9, 2019

Ant59 commented Aug 9, 2019

djsly Aug 9, 2019

jluk Aug 9, 2019

tombuildsstuff Nov 7, 2019

titilambert commented Oct 16, 2019 •

edited

Loading

titilambert commented Oct 16, 2019

titilambert commented Oct 24, 2019

worldspawn commented Nov 4, 2019

titilambert commented Nov 4, 2019

tombuildsstuff Nov 7, 2019

artburkart Nov 11, 2019

tombuildsstuff left a comment

tombuildsstuff Nov 7, 2019

tombuildsstuff Nov 7, 2019

tombuildsstuff Nov 7, 2019

tombuildsstuff Nov 7, 2019

tombuildsstuff Nov 7, 2019

tombuildsstuff Nov 7, 2019

tombuildsstuff Nov 7, 2019

tombuildsstuff commented Nov 17, 2019

ghost commented Nov 26, 2019

ghost commented Mar 29, 2020

Add kubernetes_cluster_agent_pool #4046

Add kubernetes_cluster_agent_pool #4046

Conversation

titilambert commented Aug 9, 2019

Ant59 commented Aug 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

titilambert commented Oct 16, 2019 • edited Loading

titilambert commented Oct 16, 2019

titilambert commented Oct 24, 2019

worldspawn commented Nov 4, 2019

titilambert commented Nov 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tombuildsstuff left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tombuildsstuff commented Nov 17, 2019

ghost commented Nov 26, 2019

ghost commented Mar 29, 2020

titilambert commented Oct 16, 2019 •

edited

Loading