Data sources with non-deterministic IDs show perpetual diff with Terraform 0.13

[Justin-W]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Summary

Data sources whose ID is set to a non-deterministic value (e.g. the current timestamp or a unique ID) that changes every time a refresh (the data source's Read method) runs show a perpetual diff starting with Terraform 0.13.0.

Issue #7092, and the related fix PR, show one possible way to fix data sources affected by this bug.

This has also been reported for the aws provider (hashicorp/terraform-provider-aws#14579) and other providers (see references below).

Comment posted by @jbardin in hashicorp/terraform#25805 (comment)

In general, correctly working data sources should not show up in the plan when there are no changes to the data source configuration or depends_on references. These particular data sources should be reported to the associated provider so they can be fixed.

Comment posted by @bflad in hashicorp/terraform-provider-aws#14579 (comment)

...
New tfproviderlint/awsproviderlint checks are now available for this specific issue (not enabled in our CI yet, fresh off the press):

• R015
• R016
• R017

...

Terraform Version

Terraform v0.13.3
Google Provider v2.20.3, and v3.39.0

Affected Resource(s)

• data google_compute_zones: this line appears to be the root cause of the problem (for this data source)
• plus many other data sources

To find the data sources affected, see this github repo search,
or run:

$ grep 'd.SetId' google/data_source*.go | grep Now

Terraform Configuration Files

data "google_compute_zones" "acme-web-services-guide-rails-customer-vpc-sn-0-available" {
  provider = google
  status = "UP"
}

Expected Behavior

TF v0.12 output from terraform plan (immediately after a terraform apply):

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

Actual Behavior

TF v0.13.3 output excerpt from terraform plan (immediately after a terraform apply):

...

  # data.google_compute_zones.zones-up will be read during apply
  # (config refers to values not yet known)
 <= data "google_compute_zones" "zones-up"  {
      + id      = "2020-09-23 21:11:42.856222 +0000 UTC"
      + names   = [
          + "us-west2-a",
          + "us-west2-b",
          + "us-west2-c",
        ]
      + project = "my-project"
      + region  = "us-west2"
      + status  = "UP"
    }

...

Note that the value of the id attribute in the above output is different every time that terraform plan is executed.

Steps to Reproduce

1. terraform apply
2. terraform plan > plan1.txt
3. terraform plan > plan2.txt
4. diff plan1.txt plan2.txt

Important Factoids

We use the output from terraform plan in our CI process. So if the terraform plan output is not completely idempotent, then that is a big problem for us.

References

• Data sources with non-deterministic IDs show perpetual diff with Terraform 0.13 (R015, R016, R017 linters) terraform-provider-aws#14579
• New Check: Avoid Unstable (*helper/schema.ResourceData).SetId() Call Expressions with time.Now() bflad/tfproviderlint#192
• Data source for google_secret_manager_secret_version constantly has ID changing due to timestamp issue #7092
• Data Source with Unstable id Attribute and Unconfigured Default Attribute Showing Difference Every Apply terraform-plugin-sdk#586
• terraform-providers/terraform-provider-oci#1148
• Handle semantically equivalent changes from the legacy SDK when planning data sources terraform#25812
• 0.13.0 - Is it possible to suppress or hide the "will be read during apply" messages for data resources?  terraform#25805

[edwardmedia]

I see id difference with 0.13.0, not with 0.12.18

[oscarwest]

We also see ID difference, e.g.

<= data "aws_acm_certificate" "wildcard"  {
        arn         = "arn:aws:acm:us-east-1:...:certificate/...-...-..-...-..."
        domain      = "staging.some.domain"
      ~ id          = "2020-09-24 13:57:35.730091589 +0000 UTC" -> "2020-09-24 13:59:40.610712375 +0000 UTC"
        most_recent = false
        statuses    = [
            "ISSUED",
        ]
        tags        = {}
    }

[bflad]

Drive-by comment: This problem can have a few causes -- while the data source difference will show the more obvious id attribute change, there can actually be other data source schema issues which are at play as well including:

• Ensuring that data source attributes that are optionally configurable, but always call d.Set(), should set Computed: true. Terraform can determine this as a difference internally, since it is not expecting a value in the state when it is not configured. For example in the given example above, the google_compute_zones data source region attribute is missing this currently.
• Keeping an eye on Default usage in data sources, which can unexpectedly show a difference: Data Source with Unstable id Attribute and Unconfigured Default Attribute Showing Difference Every Apply terraform-plugin-sdk#586

We have experimentally seen in the AWS Provider that sometimes these non-obvious fixes can actually hide the unexpected difference output of a data source even without stabilizing the id attribute (since in some contexts it may be considered a breaking change).

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!