google_bigquery_table shows as modified when using 'STRUCT' instead of 'RECORD'

[charleskorn]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Terraform v0.12.18
+ provider.google v3.2.0

Affected Resource(s)

• google_bigquery_table

Terraform Configuration Files

resource "google_bigquery_table" "my_table" {
  dataset_id = google_bigquery_dataset.default.dataset_id
  table_id   = "my_table"

  schema = file("${path.module}/schema.json")
}

schema.json:

[
  {
    "name": "metadata",
    "type": "STRUCT",
    "mode": "REPEATED",
    "fields": [
      {
        "name": "key",
        "type": "STRING",
        "mode": "REQUIRED"
      },
      {
        "name": "value",
        "type": "STRING",
        "mode": "REQUIRED"
      }
    ]
  }
]

Expected Behavior

After applying the configuration, running terraform plan should show no changes need to be made.

Actual Behavior

After applying the configuration, running terraform plan shows that the table schema needs to be updated:

  # module.storage.google_bigquery_table.my_table will be updated in-place
  ~ resource "google_bigquery_table" "my_table" {
        # ...snipped 
      ~ schema              = jsonencode(
          ~ [
              ~ {
                    fields = [
                        {
                            mode = "REQUIRED"
                            name = "key"
                            type = "STRING"
                        },
                        {
                            mode = "REQUIRED"
                            name = "value"
                            type = "STRING"
                        },
                    ]
                    mode   = "REPEATED"
                    name   = "metadata"
                  ~ type   = "RECORD" -> "STRUCT"
                },
            ]
        )
    }

Steps to Reproduce

1. terraform apply
2. terraform plan

References

Both of these pages refer to STRUCT instead of RECORD:

• https://cloud.google.com/bigquery/docs/schemas#standard_sql_data_types
• https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#struct-type

STRUCT is an alias for RECORD.

Workaround

Use RECORD instead of STRUCT when defining table schema.

[emilymye]

Oof! I can see the issue and how we might fix it, but honestly I'm not sure whether it is worth it if you can easily change your schema.json to use "RECORD" in your schema json. BigQuery might start returning STRUCT instead of RECORD in the future, which would eliminate the need for this, and this field is already sensitive to change, i.e. if you switched around the order of some fields so JSON is unchanged but string value changed.

The fix would involve custom diff suppress to compare the JSON values (rather than just string values), which involves comparing recursive JSON schemas.

For the immediate future, I can leave this open and add documentation about the workaround.

[emilymye]

We added documentation to schema but the actual schema diff-suppress/comparison is an issue that I'm not sure is worth fixing unless we switch client libraries. I'm going to close for now since it's a fairly old issue - if you feel this is something you strongly need, feel free to open a new issue.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!