Specifying autoscaling limits on tiers which do not support them results in an API error - JSON request does not comply with schema ([]: [Enum string expected])

[maaydin]

Defining autoscaling attributes on an Elasticsearch topology element which does not support that attribute (e.g min_size on a data tier, any autoscaling attribute on a master tier) fails with:

JSON request does not comply with schema ([]: [Enum string expected])

The provider enriches the deployment template with information from the Terraform module definition. The provider relies on the deployment template specifying the resource type in the autoscaling_min_size and autoscaling_max_size attributes. Data tiers do not include the autoscaling_min_size attribute, similarly master tiers do not include either the autoscaling_min_size or autoscaling_max_size. This means that the final request includes a size properly without the corresponding resource, e.g:

autoscaling_min_size: {
  size: 2048
}

vs

autoscaling_min_size: {
  size: 2048
  resource: memory
}

Possible fixes:

• Error in the provider when a autoscaling dimension is configured in the provider but not supported in the template
• Fix the resource to memory (we assume as much already)

Original description

---

Expected Behavior

When a dedicated master topology added to an existing cluster, it's expected that new dedicated master nodes added to cluster.

Current Behavior

When a dedicated master topology added to an existing cluster, it returns below api error.
* api error: root.invalid_json_request: JSON request does not comply with schema ([]: [Enum string expected])

Full terraform apply log

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.main.module.elasticcloud["***************"].ec_deployment.elasticcloud will be updated in-place
  ~ resource "ec_deployment" "elasticcloud" {
        id                     = "******************"
        name                   = "**********************"
        tags                   = {}
        # (8 unchanged attributes hidden)


      ~ elasticsearch {
            # (7 unchanged attributes hidden)

          + topology {
              + id               = "master"
              + node_roles       = (known after apply)
              + node_type_master = "true"
              + size             = "2g"
              + size_resource    = "memory"
              + zone_count       = 3

              + autoscaling {
                  + max_size = "120g"
                }
            }

            # (2 unchanged blocks hidden)
        }

        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.main.module.elasticcloud["****************"].ec_deployment.elasticcloud: Modifying... [id=*********************************]

Error: failed updating deployment: 1 error occurred:
	* api error: root.invalid_json_request: JSON request does not comply with schema ([]: [Enum string expected])



  on .terraform/modules/main.elasticcloud/elastic.tf line 7, in resource "ec_deployment" "elasticcloud":
   7: resource "ec_deployment" "elasticcloud" {

## Terraform definition
Terraform module ec_deployment block

resource "random_password" "es_admin_user" {
  length           = 32
  special          = true
  override_special = "_%@"
}

resource "ec_deployment" "elasticcloud" {
  name                   = format("%s-%s-%s-%s", var.es_region_short, var.environment, var.app_name, var.es_name)
  region                 = var.es_region
  version                = var.es_version
  deployment_template_id = var.es_deployment_template_id
  traffic_filter         = var.es_traffic_filter

  elasticsearch {
    autoscale = var.es_autoscale

    dynamic "topology" {
      for_each = local.es_topology
      content {
        id               = topology.value.id
        size             = topology.value.size
        zone_count       = topology.value.zone_count
        node_type_master = topology.value.node_type_master
        autoscaling {
          max_size = topology.value.max_size
        }
      }
    }
  }

  kibana {
    topology {
      size       = "1g"
      zone_count = "2"
    }
  }

  apm {
    topology {
      size = "0g"
    }
  }
}

Topology passed to the module:

locals {
  ec_api_key = data.terraform_remote_state.az_ec_core.outputs.ec_api_key
  elastic_search_instances = {
    ********** = {
      name                   = "**********"
      version                = "6.8.19"
      deployment_template_id = "azure-compute-optimized-v2"
      topology = [
        {
          id         = "hot_content"
          size       = "60g"
          zone_count = "3"
        },
        {
          id         = "master"
          size       = "2g"
          zone_count = "3"
        }
      ]
    }
  }
}

Steps to Reproduce

1. Create cluster with "hot_content" topology only via a module that uses dynamic blocks
2. Once the cluster created with a "hot_content" topology try to add dedicated master nodes.

Your Environment

• Version used: 0.3.0 (0.2.1 also tested)
• Running against Elastic Cloud SaaS or Elastic Cloud Enterprise and version: Elastic Cloud Saas
• Environment name and version (e.g. Go 1.9): go version go1.14.10 darwin/amd64
• Operating System and version: macOS 10.15.7

[AlexP-Elastic]

(I don't understand why you get the error you do - is there any way of making terraform more verbose? - but on ESS you can only add masters once you have 6+ data nodes, and I strongly suspect that terraform current won't support letting you cross the 6 node barrier, you'd have to do that via the UI, then terraform can be used to manage it declaratively ... as long as you don't cross the 6 node boundary in the other direction, which gives a similar - more helpful! - error)

Looking at the docs, it used to be possible via node_type_master but for newer versions with node roles it won't work, so it's definitely a bug)

[maaydin]

Sorry, it's been long time and I don't have enough time to provide more detail.

As a workaround, we have created the cluster with dedicated masters to be able to scale in the future. Looks like it's not possible to start small and scale up in time, at least not possible with terraform provider.

I am not sure if it is possible through UI or API.

[scathatheworm]

I'm running into the same error message under a different scenario: When enabling autoscaling in an existing deployment, I get the same API error, see below. Am I missing something?

ec_deployment.staging: Modifying... [id=6dda67c167234bd0bf948e2[66](https://gitlab.booking.com/secplatform/infra/ess/-/jobs/102876515#L66)09c5aaf]
╷
│ Error: failed updating deployment: 1 error occurred:
│ 	* api error: root.invalid_json_request: JSON request does not comply with schema ([]: [Enum string expected])
│ 
│ 
│ 
│   with ec_deployment.staging,
│   on staging.tf line 1, in resource "ec_deployment" "staging":
│    1: resource "ec_deployment" "staging" {
│ 
╵

The terraform definition is:

resource "ec_deployment" "staging" {
  name                   = "staging"
  region                 = var.region
  version                = var.stg-version
  deployment_template_id = "aws-storage-optimized"
  tags                   = {
    env                  = "staging"
    type                 = "data"
  }
  traffic_filter         = [ec_deployment_traffic_filter.aws_vpce.id]

  elasticsearch {
    autoscale            = "true"
    topology {
      id                 = "coordinating"
      size               = "8g"
      zone_count         = 3
    }
    topology {
      id                 = "hot_content"
      zone_count         = 3
      autoscaling {
        min_size         = "174g"
        max_size         = "232g"
      }
    }
    topology {
      id                 = "master"
      size               = "17g"
      zone_count         = 3 
    }
    topology {
      id                 = "ml"
      zone_count         = 3
      autoscaling {
        min_size         = "0g"
        max_size         = "15g"
      }
    }
    topology {
      id                 = "warm"
      zone_count         = 2
      autoscaling {
        min_size         = "8g"
        max_size         = "15g"
      }
    }
    config {
      user_settings_yaml = "<redacted>"
    }
    trust_account {
      account_id         = var.account_id
      trust_all          = true
    }
  }

  kibana {
    topology {
      size               = "8g"
      zone_count         = 3
    }
    config {
      user_settings_yaml = "<redacted>"
    }
  }

  observability {
    deployment_id        = ec_deployment.monitoring.id
    logs                 = true
    metrics              = true
    ref_id               = "main-elasticsearch"
  }
}

The changes from working config, to erroring config in the above file are, (I also tried readding the size parameter without success):

@@ -10,7 +10,7 @@ resource "ec_deployment" "staging" {
   traffic_filter         = [ec_deployment_traffic_filter.aws_vpce.id]

   elasticsearch {
-    autoscale = "false"
+    autoscale            = "true"
     topology {
       id                 = "coordinating"
       size               = "8g"
@@ -18,8 +18,11 @@ resource "ec_deployment" "staging" {
     }
     topology {
       id                 = "hot_content"
-      size               = "174g"
       zone_count         = 3
+      autoscaling {
+        min_size         = "174g"
+        max_size         = "232g"
+      }
     }
     topology {
       id                 = "master"
@@ -28,13 +31,19 @@ resource "ec_deployment" "staging" {
     }
     topology {
       id                 = "ml"
-      size               = "4g"
       zone_count         = 3
+      autoscaling {
+        min_size         = "0g"
+        max_size         = "15g"
+      }
     }
     topology {
       id                 = "warm"
-      size               = "15g"
       zone_count         = 2
+      autoscaling {
+        min_size         = "8g"
+        max_size         = "15g"
+      }
     }

[andrewnazarov]

@scathatheworm did you manage to find a workaround? We are facing the same issue.

[tobio]

@scathatheworm it looks like you're defining a min_size on the data tier elements (hot_content, warm) which is triggering this issue. If you remove the min_size attribute this error should go away.

At a Cloud level, we only support autoscaling up on data tiers and defining a min_size is failing the request validation. Clearly the error message in this instance is pretty confusing and something we should improve, it's a good candidate for having some provider level validation in order to clearly identify the cause.

@andrewnazarov if you've got a different module definition triggering this error, add it here and I can take a look at what's going on.