min_size & desired_size

[dmitry-mightydevops]

Description

eks_managed_node_groups = {
  ops-v1 = {
    min_size       = 1
    max_size       = 5
    desired_size   = 1
    ...
    instance_types = ["m5.large"]
  }

the group is created. I change the desired_size 1 -> 2, run apply and nothing. It doesn't detect the change because of the following code for the node group

lifecycle {
    create_before_destroy = true
    ignore_changes = [
      scaling_config[0].desired_size,
    ]
  }

I then try to do

eks_managed_node_groups = {
  ops-v1 = {
    min_size       = 2
    max_size       = 5
    desired_size   = 2
    ...
    instance_types = ["m5.large"]
  }

and apply fails with the following error:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.eks_cluster.module.eks_cluster.module.eks_managed_node_group["ops-v1"].aws_eks_node_group.this[0] will be updated in-place
  ~ resource "aws_eks_node_group" "this" {
        id              = "project-prod-eks:prod-ops-v1"
        tags            = {
            "Name"                                = "prod-ops-v1"
            "created"                             = "4/18/2022"
            "environment"                         = "prod"
            "updated"                             = "4/18/2022"

        }
        # (15 unchanged attributes hidden)


      ~ scaling_config {
          ~ min_size     = 1 -> 2
            # (2 unchanged attributes hidden)
        }


        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions in workspace "prod"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.eks_cluster.module.eks_cluster.module.eks_managed_node_group["ops-v1"].aws_eks_node_group.this[0]: Modifying... [id=project-prod-eks:prod-ops-v1]
╷
│ Error: error updating EKS Node Group (project-prod-eks:prod-ops-v1) config: InvalidParameterException: Minimum capacity 2 can't be greater than desired size 1
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "21006d45-9a28-40ec-8d06-0205d6abc95d"
│   },
│   ClusterName: "project-prod-eks",
│   Message_: "Minimum capacity 2 can't be greater than desired size 1",
│   NodegroupName: "prod-ops-v1"
│ }
│ 
│   with module.eks_cluster.module.eks_cluster.module.eks_managed_node_group["ops-v1"].aws_eks_node_group.this[0],
│   on .terraform/modules/eks_cluster.eks_cluster/modules/eks-managed-node-group/main.tf line 272, in resource "aws_eks_node_group" "this":
│  272: resource "aws_eks_node_group" "this" {
│ 
╵
make: *** [Makefile:38: apply] Error 1

So the only way to overcome it is

• modify ASG manually in GUI or via AWS cli
• create a new "node-group" with different basic settings.

Is this considered a bug?

• ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]:

• Terraform version: 1.1.8

• Provider version(s): 18.20.2

[bryantbiggs]

hi @dmitry-mightydevops - this is not a bug and is the desired effect. Autoscaling of nodes via cluster autoscaler or Karpenter is the preferred method by many, and unfortunately Terraform does not allow for parameterizing ignore_changes at this time so this is the default effect

[dmitry-mightydevops]

Sounds good - thanks @bryantbiggs that's what I thought.

[NickLarsenNZ]

I'm getting this too (on a cluster I scaled down manually via the console).

@bryantbiggs I think this should be reopened.

Perhaps the min_size, max_size, and desired_size values should change in the order depending on which is currently greater or less than others (to avoid breaking the constraint).

---

More info

My terraform plan shows the changes made outside of what the state knows (I had scaled it to 1):

      ~ scaling_config {
          ~ desired_size = 3 -> 1
          ~ min_size     = 3 -> 1
            # (1 unchanged attribute hidden)
        }

I then see that terraform will increase the min_size:

Note: I just noticed that the desired_size change doesn't appear below. I have checked to see if we (or the module we use) is ignoring the changes, but I can't see any occurences of that. Maybe this is a different issue

      ~ scaling_config {
          ~ min_size     = 1 -> 3
            # (2 unchanged attributes hidden)
        }

Then during apply, it fails with:

│ Error: error updating EKS Node Group (multitenant-capetown-prod:initial-20220901104640951800000008) config: InvalidParameterException: Minimum capacity 3 can't be greater than desired size 1
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "6a4f9ce0-8942-4b1b-b5db-880e95ad72c6"
│   },
│   ClusterName: "my-cluster",
│   Message_: "Minimum capacity 3 can't be greater than desired size 1",
│   NodegroupName: "initial-20220901104640951800000009"
│ }
│ 
│   with module.eks.module.eks_managed_node_group["initial"].aws_eks_node_group.this[0],
│   on .terraform/modules/eks/modules/eks-managed-node-group/main.tf line 272, in resource "aws_eks_node_group" "this":
│  272: resource "aws_eks_node_group" "this" {

[dejwsz]

Regarding the comment: #2030 (comment) I suspect it will be possible to handle this properly after this is done: hashicorp/terraform#27360, hashicorp/terraform#3116

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

[bryantbiggs]

https://github.com/bryantbiggs/eks-desired-size-hack