BigQuery Authorized Datasets gets overwritten when any new changes to BigQuery Dataset IAM are applied during tf apply #11218
Comments
|
The above bug with the provider is related to this issue here #11091. We noticed it when we tested it against the latest provider release. |
|
Looking for some clarification here. Can you provide some more logging of the terraform apply? Is the Terraform plan output stating the the resource will be replaced when adding the new user? Or does it state it will be updated in-place and then remove the resource later? |
|
Hello, We are facing the same issues. ` |
|
What is the update on this issue. |
|
We're seeing the same problem across our projects and this has been going on for quite some time. We've explored other options, including restructuring our resources to use "google_bigquery_dataset_access" or, even better (since it actually allows us to use the Google recommended BQ roles and not the BQ dataset level legacy basic roles they discourage people from using, especially in production) the "google_bigquery_dataset_iam_binding" resource. Unfortunately, neither of these will work for a variety of reasons, not the least of which is that they cannot be used together and the "google_bigquery_dataset_iam_binding" resource cannot be used with Authorized Views. |
|
Using "google_bigquery_dataset" resource worked for us. It is authoritative, so acts as a combination google_bigquery_dataset_iam_binding and google_bigquery_dataset_access. It is also more performant, as it is only a single resource per dataset + all access definitions. However, we encountered another issue, that it recreates access blocks even if they are the same sometimes. This is an unrelated bug however. |
|
I outlined the issues with all 5 BigQuery dataset access control options here: #11461 (comment) There doesn't seem to be an authoritative way to manage BigQuery dataset access that supports authorized datasets/views and does not generate horrendous plans when a single change to access is made. |
github-actions
bot
added
service/bigquery
forward/review
|
Hey, what is status of this bug? |
|
Hi @c2thorn @edwardmedia, It seems that the resource does not support VIEWS destructively if using Looking in the debug log when applying, the provider got the correct access from the dataset but when we try to apply, we observe that the As you can see, the part for configuring the Authorized Dataset is missing. This makes the terraform accidentally delete the access. |
|
I'm from the BigQuery Terraform team. Thank you all for your reports. Let's continue to use this issue to track problems related to the behavior of authorized datasets getting removed unexpectedly. For concerns about the Meanwhile I wasn't able to reproduce this issue with the |
|
Hi @wj-chen, you can try using this example resource "google_bigquery_dataset" "this" {
dataset_id = "bq_test_tf"
friendly_name = "bq_test_tf"
labels = {
"created_by" = "eka"
"environment" = "staging"
}
location = "US"
project = "my-gcp-project"
access {
dataset {
dataset {
project_id = "my-gcp-project"
dataset_id = "bq_ds_test1"
}
target_types = ["VIEWS"]
}
}
}
resource "google_bigquery_dataset_iam_member" "this" {
dataset_id = google_bigquery_dataset.this.dataset_id
project = google_bigquery_dataset.this.project
role = "roles/bigquery.dataEditor"
member = "user:[email protected]"
}The first apply will succeed but when you replan, it will re add the authorized dataset # google_bigquery_dataset.this will be updated in-place
~ resource "google_bigquery_dataset" "this" {
id = "projects/my-gcp-project/datasets/bq_test_tf"
# (13 unchanged attributes hidden)
+ access {
+ dataset {
+ target_types = [
+ "VIEWS",
]
+ dataset {
+ dataset_id = "bq_ds_test1"
+ project_id = "my-gcp-project"
}
}
}
- access {
- role = "OWNER" -> null
- special_group = "projectOwners" -> null
}
- access {
- role = "READER" -> null
- special_group = "projectReaders" -> null
}
- access {
- role = "WRITER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "WRITER" -> null
- special_group = "projectWriters" -> null
}
} |
|
Thank you @bakuljajan. I can see that in this case the |
@wj-chen it just feels unnatural to manage access using a dataset resource, access to bigquery can be granted in project, dataset, and table. For the project and table, we need to use terraform iam_member/iam_binding resources but not the dataset. Isn't that weird? To be honest, we already implemented user/service account access using iam_member/iam_binding for a long time now and we just noticed this behavior recently since we only recently explored access using authorized view/dataset. For us, the effort to migrate the whole dataset access from iam_member/iam_binding to dataset_access is really big. |
|
@bakuljajan Thank you for sharing your use cases. Dataset permissions work slightly differently from permissions for projects/tables/jobs etc. and the complication happens upstream at the API level. There is work underway to improve that but won't be available in the near future.
This appears to be a framework-level issue. I added a comment to check on the latest state in hashicorp/terraform#21901.
Acknowleged. To mitigate the issue with authorized dataset/view specifically, we will explore if it's feasible to update the behavior of Meanwhile I'd like to see if that'd also work for the other users here. If anyone is using a combination different from |
|
Might help (a bit hard to read)
|
|
This work is still pending engineering capacity from our team. Will add exploration of any potential quick fixes into our Q1 planning . |
|
This is a huge pain-point for our usage of Terraform + BigQuery where we need a lot of focus from engineers to analyse plans or risk missing out a drift in permissions or accidental removal of permissions. We train people to expect noisy plans which create perma-plan fatigue. |
|
It seems that applying any google_bigquery_dataset_iam_member to a dataset will also erase dataset authorization config that has been made outside of Terraform. EG,
Then the dataset authorization I configured in step 1 is gone! |
…ng current and new binding for google_bigquery_dataset_iam_member
…ng current and new binding for google_bigquery_dataset_iam_member
|
Draft PR-10560 aims to fix this. |
…ng current and new binding for google_bigquery_dataset_iam_member
…ng current and new binding for google_bigquery_dataset_iam_member
…ng current and new binding for google_bigquery_dataset_iam_member
…… (#11352) [upstream:649cb8b86339253af356ad48f0b523b2acfb099f] Signed-off-by: Modular Magician <[email protected]>
[upstream:649cb8b86339253af356ad48f0b523b2acfb099f] Signed-off-by: Modular Magician <[email protected]>
…… (#11352) (#7960) [upstream:649cb8b86339253af356ad48f0b523b2acfb099f] Signed-off-by: Modular Magician <[email protected]>
[upstream:649cb8b86339253af356ad48f0b523b2acfb099f] Signed-off-by: Modular Magician <[email protected]>
|
GoogleCloudPlatform/magic-modules#11352 should fix this. |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. |
Community Note
modular-magicianuser, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned tohashibot, a community member has claimed the issue already.Terraform Version
1.0.11
Affected Resource(s)
Panic Output
There is no error output, the authorized datasets are getting removed.
Expected Behavior
Authorized Datasets shouldnt have been reset.
Actual Behavior
Authorized datasets are getting reset on Dataset IAM terraform apply
Steps to Reproduce
b/300616992
The text was updated successfully, but these errors were encountered: