Modifying google_bigquery_data_transfer_config.params.data_path_template fails

[charleskorn]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.12.29
+ provider.google v3.32.0

Affected Resource(s)

• google_bigquery_data_transfer_config

Terraform Configuration Files

resource "google_bigquery_data_transfer_config" "transfer" {
  display_name           = "import"
  data_source_id         = "google_cloud_storage"
  destination_dataset_id = google_bigquery_dataset.default.dataset_id
  location               = google_bigquery_dataset.default.location
  schedule               = "every 8 hours"
  service_account_name   = data.google_service_account.bigquery_transfer_service.email

  params = {
    data_path_template              = "gs://old-value/*.json"
    destination_table_name_template = "the-table"
    file_format                     = "JSON"
    max_bad_records                 = 0
    write_disposition               = "APPEND"
  }
}

Expected Behavior

Modifying the data_path_template should succeed, either updating the existing job or recreating it if needed

Actual Behavior

Fails with:

Error: Error updating Config "projects/XXXXX/locations/us/transferConfigs/XXXX": googleapi: Error 400: Immutable parameter data_path_template with value string_value: "gs://old-value/*.json"
 cannot be changed to string_value: "gs://new-value/*.json"

Steps to Reproduce

1. Create google_bigquery_data_transfer_config for transfer from GCS, and set data_path_template in params
2. terraform apply
3. Modify data_path_template
4. Re-run terraform apply

[edwardmedia]

@charleskorn this error comes from API. It seems clear data_path_template is not allowed to update. Does this make sense to you?
Immutable parameter data_path_template ......

[charleskorn]

@edwardmedia yes, I understand the error message. The reason I filed this as a bug is that normally, Terraform resources know which properties are immutable or not and therefore whether an in-place update can be performed or if the resource needs to be recreated.

For example, if I change a resource's project, Terraform knows that change can't be made in place, so it plans to recreate the resource. I'd expect the same behaviour here - the provider should plan to recreate the job when I change this property, not attempt to modify it in place.

[c2thorn]

Hey @charleskorn! It makes sense to expect the data_path_template param to recreate the resource. Unfortunately the current design of the resource treats the params block as a single field, so it doesn't have the right granularity to treat data_path_template differently. Changing this the right way would require some schema changes, and makes this more of a feature request. I'll label it as such

[charleskorn]

Makes sense, thanks @c2thorn

[c2thorn]

On second thought, it may be best to use a customize diff. Putting a note for the future implementer.

[idcmp]

With the minimum period dropping from eight hours to fifteen minutes, the transfer service becomes much more valuable and people will likely bump into this.

Are there any workarounds where I can just flag the entire resource as immutable manually?

[c2thorn]

With the minimum period dropping from eight hours to fifteen minutes, the transfer service becomes much more valuable and people will likely bump into this.

Are there any workarounds where I can just flag the entire resource as immutable manually?

The only manual force recreation option that I know of is the taint command. A less desirable workaround would be to simultaneously update a field that is set to force recreate, like display_name.

[BrettNWoods]

Has there been any more movement on this issue. I can confirm that this issue also affects changes to the destination_table_name_template field.

[roaks3]

I noticed that this documentation was clarified here to state that manually forcing recreation is the best option at the moment.

@c2thorn could you clarify how you were thinking of using a customize diff here? If it's straightforward we can have someone work on this ticket in the next batch of work.

[c2thorn]

@roaks3
In a customize diff, we could get the old and new params maps from the diff, and then compare the data_path_template and destination_table_name_template values. If either of them have changed in value, we could call diff.ForceNew("params") . Closest example I could find in a limited time: https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/templates/terraform/constants/spanner_database.go.erb#L23

[roaks3]

b/251556517

[melinath]

looks like destination_table_name_template is not immutable, reverting this change & reopening to track

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.