Autoscale does not play well if you set topologies

[kuisathaverat]

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

A terraform manifest should work after autoscaling resizes the cluster.

Current Behavior

The terraform apply will fail with different errors, but all are related to the fact that autoscale has changed the size and the plan generated by the elastic terraform provider does not have those changes and the resulting plan is incompatible with the currently deployed. Some of the issues are:

• Terraform try to downscale the cluster because you have set resources in the topology (Terraform plan wants to revert changes made by the autoscaler #462)
• Terraform fails because you reach more than 6 nodes and you need master nodes (that are already deployed by autoscale) Specifying autoscaling limits on tiers which do not support them results in an API error - JSON request does not comply with schema ([]: [Enum string expected]) #401 Unable to disable manually added cold or frozen tier: 'master' node type is missing #343
• You have set ML node in the initial deployment, in this case, you can not apply the terraform manifest anymore A plan with multiple elasticsearch.topology blocks is not idempotent #336

## Terraform definition

Case 1

terraform {
  required_version = ">= 0.12.29"

  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.4.0"
    }
  }
}

provider "ec" {
  endpoint = "https://cloud.elastic.co"
  insecure = true
  apikey = "<<REDACTED>>"

  verbose = false
}

# Create an Elastic Cloud deployment
resource "ec_deployment" "main" {
  name = "test-half-stack-oblt"

  # Mandatory fields
  region                 = "gcp-us-west2"
  version                = "8.2.0"
  deployment_template_id = "gcp-io-optimized-v2"

  elasticsearch {
    autoscale = true
    topology {
      id   = "hot_content"
      size = "2g" 
    }
    config {
      user_settings_yaml = file("./es_settings.yml")
    }
  }

  kibana {}

  integrations_server {}
}

Case 2

terraform {
  required_version = ">= 0.12.29"

  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.4.0"
    }
  }
}

provider "ec" {
  endpoint = "https://cloud.elastic.co"
  insecure = true
  apikey = "<<REDACTED>>"

  verbose = false
}

# Create an Elastic Cloud deployment
resource "ec_deployment" "main" {
  name = "test-half-stack-oblt"

  # Mandatory fields
  region                 = "gcp-us-west2"
  version                = "8.2.0"
  deployment_template_id = "gcp-io-optimized-v2"

  elasticsearch {
    autoscale = true
    topology {
      id   = "hot_cold"
    }
    topology {
      id   = "hot_content"
    }
    topology {
      id   = "hot_warm"
    }
    config {
      user_settings_yaml = file("./es_settings.yml")
    }
  }

  kibana {}

  integrations_server {}
}

Case 3

terraform {
  required_version = ">= 0.12.29"

  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.4.0"
    }
  }
}

provider "ec" {
  endpoint = "https://cloud.elastic.co"
  insecure = true
  apikey = "<<REDACTED>>"

  verbose = false
}

# Create an Elastic Cloud deployment
resource "ec_deployment" "main" {
  name = "test-half-stack-oblt"

  # Mandatory fields
  region                 = "gcp-us-west2"
  version                = "8.2.0"
  deployment_template_id = "gcp-io-optimized-v2"

  elasticsearch {
    autoscale = true
    topology {
      id   = "hot_content"
    }
    topology {
      id   = "ml"
    }
    config {
      user_settings_yaml = file("./es_settings.yml")
    }
  }

  kibana {}

  integrations_server {}
}

Steps to Reproduce

1. Apply any of the configurations above
2. ingest data into the cluster to make the autoscale size the cluster to 6 nodes
3. try to apply the terraform manifest again

Context

We change settings on elasticsearch using the settings file we pass in the terraform manifest, we also add node types sometimes, or we changed configuration on Kibana, Integrations server, or simply we changed the cluster used to monitor the cluster. Those changes do not work well with autoscaling and some terraform configurations.

Possible Solution

The workaround we use is to set the cluster without autoscale and the resources we will need, after that, we remove all the topology setting from elasticsearch and we use autoscale.

Initial deploy

terraform {
  required_version = ">= 0.12.29"

  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.4.0"
    }
  }
}

provider "ec" {
  endpoint = "https://cloud.elastic.co"
  insecure = true
  apikey = "<<REDACTED>>"

  verbose = false
}

# Create an Elastic Cloud deployment
resource "ec_deployment" "main" {
  name = "test-half-stack-oblt"

  # Mandatory fields
  region                 = "gcp-us-west2"
  version                = "8.2.0"
  deployment_template_id = "gcp-io-optimized-v2"

  elasticsearch {
    autoscale = false
    topology {
      id   = "hot_content"
      size = "16g"
      zone_count = 2
    }
    topology {
      id   = "ml"
      size = "2g"
    }
    config {
      user_settings_yaml = file("./es_settings.yml")
    }
  }

  kibana {}

  integrations_server {}
}

Autoscale update

terraform {
  required_version = ">= 0.12.29"

  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "0.4.0"
    }
  }
}

provider "ec" {
  endpoint = "https://cloud.elastic.co"
  insecure = true
  apikey = "<<REDACTED>>"

  verbose = false
}

# Create an Elastic Cloud deployment
resource "ec_deployment" "main" {
  name = "test-half-stack-oblt"

  # Mandatory fields
  region                 = "gcp-us-west2"
  version                = "8.2.0"
  deployment_template_id = "gcp-io-optimized-v2"

  elasticsearch {
    autoscale = true
    config {
      user_settings_yaml = file("./es_settings.yml")
    }
  }

  kibana {}

  integrations_server {}
}


## Your Environment
<!--- Include as many relevant details about the environment -->
<!--- you experienced the bug in. -->
* Version used: 0.4.0
* Running against Elastic Cloud SaaS or Elastic Cloud Enterprise and version: ESS
* Environment name and version (e.g. Go 1.9):
* Server type and version:
* Operating System and version: macOS 12.3.1
* Link to your project:

[dimuon]

I think the considerations from #466 (comment) can be applied here as well.

[dimuon]

Closed by #567