API-Gateway Config attempts to update even when there are no changes to config.

[ricardolpd]

We have created an api-gateway module, and it fails to upload new ones, because the module is constantly being replaced even where there are no changes to the resource or the api.yaml, according to the plan the openapi_documents.document.contents is whats causing the replacement to happen, even though the base64 string is the same. I have tested this with 3.76.0 and 3.82.0 providers.

If this approach is incorrect, what best practices should we use, to avoid recreating the gateway with each deployment even when changes didnt happen?

This is the module that creates the gateway:

locals  {
  default_api_templates = {...}
  api_version  = replace(yamldecode("${data.template_file.resolve_api_yml.rendered}").info.version, ".", "-")
  version_date = formatdate("DDMMYYYY-hhmmss", timestamp())
  config_id_suffix = var.force_deployment ? "-${local.version_date}" : ""
}

resource "google_api_gateway_api" "default" {
  provider = google-beta
  api_id = var.name
  labels =  var.labels
}

resource "google_api_gateway_api_config" "default" {
  provider = google-beta
  api      = google_api_gateway_api.default.api_id
  display_name = "${var.name}-${local.api_version}${local.config_id_suffix}"
  api_config_id = "${var.name}-${local.api_version}${local.config_id_suffix}"
  labels = {
    "version" = local.api_version
  }
  openapi_documents {
    document {
      path = "spec.yaml"
      contents = base64encode(data.template_file.resolve_api_yml.rendered)
    }
  }

  #ugly as hell, but the only way to make this block optional :(
  dynamic "gateway_config" {
    for_each = var.service_account_id != null ? [1]: []
    content {
      backend_config {
        google_service_account = var.service_account_id
      }
    }
  }

  lifecycle {
    create_before_destroy = true
    ignore_changes = [
      # openapi_documents[0].document[0].contents,
      api_config_id
    ]
  }
}

resource "google_api_gateway_gateway" "default" {
  provider = google-beta
  api_config = google_api_gateway_api_config.default.id
  gateway_id = "${var.name}-${var.gcp_region}"
  display_name = "${var.name}(${var.gcp_region})" 
  region = var.gcp_region
  labels = {
    "service_name" = var.name,
    "region" = var.gcp_region
  }
}

this is the change with the plan:

  # module.api_gateway.google_api_gateway_api_config.default must be replaced
+/- resource "google_api_gateway_api_config" "default" {
      + api_config_id_prefix = (known after apply)
      ~ id                   = "projects/just-ricardo-duarte/locations/global/apis/cloud-run-with-api-gtw-example/configs/cloud-run-with-api-gtw-example-1-0-0" -> (known after apply)
      ~ name                 = "projects/509371465123/locations/global/apis/cloud-run-with-api-gtw-example/configs/cloud-run-with-api-gtw-example-1-0-0" -> (known after apply)
      ~ project              = "just-ricardo-duarte" -> (known after apply)
      ~ service_config_id    = "cloud-run-with-api-gtw-example-1-0-0-02gzx8ksc0jt9" -> (known after apply)
        # (4 unchanged attributes hidden)


      ~ openapi_documents {
          ~ document {
              ~ contents = "c3dhZ2dlcjogJzIuMCcKaW5mbzoKICB0aXRsZTogVGVzdCBhcGkgZ2VuZXJhdGlvbiBjaGFuZ2UgZGVzY3JpcHRpb24KICBkZXNjcmlwdGlvbjogQVBJIHRoYXQgZ2VuZXJhdGVzIHBkZiBmaWxlcyBmcm9tIGFueSBodG1sIGFzIGlucHV0CiAgdmVyc2lvbjogMS4wLjIKc2NoZW1lczoKICAtIGh0dHBzCnByb2R1Y2VzOgogIC0gYXBwbGljYXRpb24vcGRmCngtZ29vZ2xlLWJhY2tlbmQ6CiAgYWRkcmVzczogaHR0cHM6Ly9jbG91ZC1ydW4td2l0aC1hcGktZ3R3LWV4YW1wbGUtZWFoZ3NrdnJlYS11ZS5hLnJ1bi5hcHAKICBwcm90b2NvbDogaDIKcGF0aHM6CiAgL3N0dWI6CiAgICBnZXQ6CiAgICAgIHN1bW1hcnk6IHN0dWIgZW5kcG9pbnQKICAgICAgb3BlcmF0aW9uSWQ6ICdzdHViJwogICAgICBwcm9kdWNlczoKICAgICAgICAtICdhcHBsaWNhdGlvbi9qc29uJwogICAgICByZXNwb25zZXM6CiAgICAgICAgJzIwMCc6CiAgICAgICAgICBkZXNjcmlwdGlvbjogQSBQREYgZmlsZQogICAgICAgICAgc2NoZW1hOgogICAgICAgICAgICB0eXBlOiBmaWxlCiAgL2ltYWdlOgogICAgcG9zdDoKICAgICAgc3VtbWFyeTogVXBsb2FkIGEgcGhvdG8KICAgICAgb3BlcmF0aW9uSWQ6IHVwbG9hZFBob3RvCiAgICAgIHJlc3BvbnNlczoKICAgICAgICAnMjAwJzoKICAgICAgICAgIGRlc2NyaXB0aW9uOiBBIHN1Y2Nlc3NmdWwgcmVzcG9uc2UKICAgICAgICAgIHNjaGVtYToKICAgICAgICAgICAgdHlwZTogb2JqZWN0CiAgL2ltYWdlL3tmaWxlbmFtZX06CiAgICBnZXQ6CiAgICAgIHBhcmFtZXRlcnM6CiAgICAgICAgLSBpbjogcGF0aAogICAgICAgICAgbmFtZTogZmlsZW5hbWUKICAgICAgICAgIHR5cGU6IHN0cmluZwogICAgICAgICAgcmVxdWlyZWQ6IHRydWUKICAgICAgICAgIGRlc2NyaXB0aW9uOiBmaWxlbmFtZSBzYXZlZAogICAgICBzdW1tYXJ5OiBnZXQgYSBwaG90bwogICAgICBvcGVyYXRpb25JZDogJ2dldFBob3RvJwogICAgICBwcm9kdWNlczoKICAgICAgICAtICdhcHBsaWNhdGlvbi9qc29uJwogICAgICByZXNwb25zZXM6CiAgICAgICAgJzIwMCc6CiAgICAgICAgICBkZXNjcmlwdGlvbjogQSBwaG90byBpbWFnZQogICAgICAgICAgc2NoZW1hOgogICAgICAgICAgICB0eXBlOiBmaWxlCiAgL2hlYWx0aDoKICAgIGdldDoKICAgICAgc3VtbWFyeTogZW5kcG9pbnQgdG8gY2hlY2sgaWYgc2VydmljZSBpcyB1cCAobW9zdGx5IHRvIGNoZWNrIGxhdGVuY3kgd2hlbiBpbnN0YW5jZXMgYXJlIGRvd24gaW4gYXBpLWdhdGV3YXkgYW5kIGNsb3VkKQogICAgICBvcGVyYXRpb25JZDogJ2hlYWx0aCcKICAgICAgcHJvZHVjZXM6CiAgICAgICAgLSAnYXBwbGljYXRpb24vanNvbicKICAgICAgcmVzcG9uc2VzOgogICAgICAgICcyMDAnOgogICAgICAgICAgZGVzY3JpcHRpb246IHN0YXR1cyBvZiB0aGUgc2VydmljZQogICAgICAgICAgc2NoZW1hOgogICAgICAgICAgICB0eXBlOiBvYmplY3QK" -> (known after apply) # forces replacement
                # (1 unchanged attribute hidden)
            }
        }
        # (1 unchanged block hidden)
    }

this is the remote state:

{
      "module": "module.api_gateway",
      "mode": "managed",
      "type": "google_api_gateway_api_config",
      "name": "default",
      "provider": "provider[\"registry.terraform.io/hashicorp/google-beta\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "api": "cloud-run-with-api-gtw-example",
            "api_config_id": "cloud-run-with-api-gtw-example-1-0-0",
            "api_config_id_prefix": null,
            "display_name": "cloud-run-with-api-gtw-example-1-0-0",
            "gateway_config": [
              {
                "backend_config": [
                  {
                    "google_service_account": "cloud-run-with-api-gtw--exec@some-project-id.iam.gserviceaccount.com"
                  }
                ]
              }
            ],
            "id": "projects/some-project-id/locations/global/apis/cloud-run-with-api-gtw-example/configs/cloud-run-with-api-gtw-example-1-0-0",
            "labels": {
              "version": "1-0-0"
            },
            "name": "projects/509371465123/locations/global/apis/cloud-run-with-api-gtw-example/configs/cloud-run-with-api-gtw-example-1-0-0",
            "openapi_documents": [
              {
                "document": [
                  {
                    "contents": "c3dhZ2dlcjogJzIuMCcKaW5mbzoKICB0aXRsZTogVGVzdCBhcGkgZ2VuZXJhdGlvbiBjaGFuZ2UgZGVzY3JpcHRpb24KICBkZXNjcmlwdGlvbjogQVBJIHRoYXQgZ2VuZXJhdGVzIHBkZiBmaWxlcyBmcm9tIGFueSBodG1sIGFzIGlucHV0CiAgdmVyc2lvbjogMS4wLjIKc2NoZW1lczoKICAtIGh0dHBzCnByb2R1Y2VzOgogIC0gYXBwbGljYXRpb24vcGRmCngtZ29vZ2xlLWJhY2tlbmQ6CiAgYWRkcmVzczogaHR0cHM6Ly9jbG91ZC1ydW4td2l0aC1hcGktZ3R3LWV4YW1wbGUtZWFoZ3NrdnJlYS11ZS5hLnJ1bi5hcHAKICBwcm90b2NvbDogaDIKcGF0aHM6CiAgL3N0dWI6CiAgICBnZXQ6CiAgICAgIHN1bW1hcnk6IHN0dWIgZW5kcG9pbnQKICAgICAgb3BlcmF0aW9uSWQ6ICdzdHViJwogICAgICBwcm9kdWNlczoKICAgICAgICAtICdhcHBsaWNhdGlvbi9qc29uJwogICAgICByZXNwb25zZXM6CiAgICAgICAgJzIwMCc6CiAgICAgICAgICBkZXNjcmlwdGlvbjogQSBQREYgZmlsZQogICAgICAgICAgc2NoZW1hOgogICAgICAgICAgICB0eXBlOiBmaWxlCiAgL2ltYWdlOgogICAgcG9zdDoKICAgICAgc3VtbWFyeTogVXBsb2FkIGEgcGhvdG8KICAgICAgb3BlcmF0aW9uSWQ6IHVwbG9hZFBob3RvCiAgICAgIHJlc3BvbnNlczoKICAgICAgICAnMjAwJzoKICAgICAgICAgIGRlc2NyaXB0aW9uOiBBIHN1Y2Nlc3NmdWwgcmVzcG9uc2UKICAgICAgICAgIHNjaGVtYToKICAgICAgICAgICAgdHlwZTogb2JqZWN0CiAgL2ltYWdlL3tmaWxlbmFtZX06CiAgICBnZXQ6CiAgICAgIHBhcmFtZXRlcnM6CiAgICAgICAgLSBpbjogcGF0aAogICAgICAgICAgbmFtZTogZmlsZW5hbWUKICAgICAgICAgIHR5cGU6IHN0cmluZwogICAgICAgICAgcmVxdWlyZWQ6IHRydWUKICAgICAgICAgIGRlc2NyaXB0aW9uOiBmaWxlbmFtZSBzYXZlZAogICAgICBzdW1tYXJ5OiBnZXQgYSBwaG90bwogICAgICBvcGVyYXRpb25JZDogJ2dldFBob3RvJwogICAgICBwcm9kdWNlczoKICAgICAgICAtICdhcHBsaWNhdGlvbi9qc29uJwogICAgICByZXNwb25zZXM6CiAgICAgICAgJzIwMCc6CiAgICAgICAgICBkZXNjcmlwdGlvbjogQSBwaG90byBpbWFnZQogICAgICAgICAgc2NoZW1hOgogICAgICAgICAgICB0eXBlOiBmaWxlCiAgL2hlYWx0aDoKICAgIGdldDoKICAgICAgc3VtbWFyeTogZW5kcG9pbnQgdG8gY2hlY2sgaWYgc2VydmljZSBpcyB1cCAobW9zdGx5IHRvIGNoZWNrIGxhdGVuY3kgd2hlbiBpbnN0YW5jZXMgYXJlIGRvd24gaW4gYXBpLWdhdGV3YXkgYW5kIGNsb3VkKQogICAgICBvcGVyYXRpb25JZDogJ2hlYWx0aCcKICAgICAgcHJvZHVjZXM6CiAgICAgICAgLSAnYXBwbGljYXRpb24vanNvbicKICAgICAgcmVzcG9uc2VzOgogICAgICAgICcyMDAnOgogICAgICAgICAgZGVzY3JpcHRpb246IHN0YXR1cyBvZiB0aGUgc2VydmljZQogICAgICAgICAgc2NoZW1hOgogICAgICAgICAgICB0eXBlOiBvYmplY3QK",
                    "path": "spec.yaml"
                  }
                ]
              }
            ],
            "project": "some-project-id",
            "service_config_id": "cloud-run-with-api-gtw-example-1-0-0-02gzx8ksc0jt9",
            "timeouts": null
          },
          "sensitive_attributes": [],
          "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozNjAwMDAwMDAwMDAsImRlbGV0ZSI6MzYwMDAwMDAwMDAwLCJ1cGRhdGUiOjM2MDAwMDAwMDAwMH19",
          "dependencies": [
            "module.cloud_run_service.data.google_secret_manager_secret_version.secret",
            "module.cloud_run_service.data.google_sql_database_instance.default",
            "module.cloud_run_service.google_cloud_run_service_iam_member.allUsers",
            "module.cloud_run_service.module.service_account.null_resource.before",
            "module.cloud_storage_bucket.google_storage_bucket.bucket",
            "module.cloud_storage_bucket.google_storage_bucket.bucket_for_access_logs",
            "module.api_gateway.google_api_gateway_api.default",
            "module.cloud_run_service.data.google_project.current",
            "module.cloud_run_service.module.service_account.google_project_iam_binding.iam_policy",
            "module.cloud_run_service.module.service_account.null_resource.delay",
            "module.secret_dd_api_key.google_secret_manager_secret_version.secret_version_data",
            "module.api_gateway.data.template_file.resolve_api_yml",
            "module.cloud_run_service.google_cloud_run_service.service",
            "module.cloud_run_service.google_secret_manager_secret_iam_member.secret-access",
            "module.cloud_storage_bucket.data.google_project.current",
            "module.cloud_storage_bucket.google_storage_bucket_acl.bucket_for_access_logs_acl",
            "module.secret_dd_api_key.google_secret_manager_secret.secret",
            "module.cloud_run_service.module.gcr_image_submit.null_resource.image",
            "module.cloud_run_service.module.gcr_image_submit.random_uuid.value",
            "module.cloud_run_service.module.service_account.google_service_account.service_account",
            "module.cloud_run_service.module.service_account.google_service_account_iam_binding.iam_policy"
          ],
          "create_before_destroy": true
        }
      ]
    }

[slevenick]

I think this is caused by depending on a computed field from a data source.

When Terraform runs on a data source (the template_file in this case) the outputs of that data source may be marked as unknown (known after apply) until apply-time, which is what your plan is showing. Docs on this behavior: https://www.terraform.io/docs/language/data-sources/index.html#data-source-lifecycle

You haven't included your data source configuration, but I'm guessing that it is deferred until apply-time, which causes the .rendered attribute to be unknown until apply-time, which makes the API Gateway Config unable to know what the value of contents will be so it cannot accurately compare it, triggering the recreate.

To get around this you may want to compile the template once and save that as a file, or change how the template is compiled to make it known at plan-time (not sure if this is possible with your setup)

[ricardolpd]

you were completely right, thanks for the link about data lifecycle, i had spent most of last night, and i couldn't understand why it kept updating the resource. I had looked into locals lifecycle for something else, but end up missing looking up at the data.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.