Need a way to migrate to google_bigquery_dataset_access

[rvandegrift]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

I have some bigquery datasets that use old-style access blocks in the dataset. I'd like to switch to the newer google_bigquery_dataset_access, but the provider doesn't correctly handle this. When the google_bigquery_dataset_access resource is created, it conflicts with the existing IAM policy elements, resulting in errors like:

google_bigquery_dataset_access.access: Creating...

Error: Unable to create DatasetAccess, existing object already found: map[role:OWNER userByEmail:[email protected]]

New or Affected Resource(s)

• google_bigquery_dataset_access

Potential Terraform Configuration

Initial config:

resource "google_bigquery_dataset" "ds" {
  dataset_id = "table"
  location   = "US"

  access {
    role          = "OWNER"
    user_by_email = "[email protected]"
  }
}

Desired config:

resource "google_bigquery_dataset" "ds" {
  dataset_id = "table"
  location   = "US"
}

resource "google_bigquery_dataset_access" "access" {
  dataset_id = google_bigquery_dataset.ds.dataset_id

  role          = "OWNER"
  user_by_email = "[email protected]"
}

Ideally, terraform would be to detect that the exact policy element already exists, and just consider it successful. If that wouldn't work, it'd still be better than nothing to have terraform import support.

References

• #0000

b/359365569

[slevenick]

Hey @rvandegrift you can try the bigquery dataset IAM policy which wraps the dataset access resource as a workaround. It should be able to support your use case

https://www.terraform.io/docs/providers/google/r/bigquery_dataset_iam.html

[rvandegrift]

I left this out of my example to keep it simple, but I need authorized view support.

[objectiveryan]

This bug is blocking my team from using google_bigquery_dataset_access; thankfully we can still use access blocks (like the OP we also have authorized views).

[CrAzE124]

We're also trying to migrate from access to google_bigquery_dataset_access and face the same problem (also in #8165 ). We'd like to move to this resource instead of bigquery_dataset_iam as we need authorized view support, but would also like the permissions to be non-authoritative, so that we can set additional permissions dynamically from client SDK's. Is there any plan to make this migration possible at the moment? Manually migrating datasets is a non-starter as we have thousands of datasets currently, and our terraform code is dynamically generated.

[alessandroberlati]

The easiest way I found was to:

• remove all access blocks in favor of google_bigquery_dataset_access
• add a single access block with your admin email as OWNER
• apply the changes
• remove the last access block that will not produce any changes
• manually remove the Data owner from UI (one per dataset)

[amerenda]

If you're having this issue with dataset VIEWS:

• Navigate to the BiqQuery GCP console page
• Expand the GCP project
• Open the source dataset for the view (click the vertical dots, open)
• Select Share Dataset
• Select Authorized Views
• Find the view that does not exist in terraform, but exists in BQ, remove it.
• Run terraform apply

[ggtisc]

Currently I did not find any argument in the terraform registry to achieve this proposal, it seems to me that it should rather be an action executed by some method

[roaks3]

Note: it seems like import would help quite a bit to solve this (#7659), but assuming import is possible by coming up with a suitable identifier, I think this use-case would also benefit from being considered for acquire-on-create behavior (ie. when adding a new google_bigquery_dataset_access, check if the exact same object already exists, and if it does use that).

[wj-chen]

I think this use-case would also benefit from being considered for acquire-on-create behavior (ie. when adding a new google_bigquery_dataset_access, check if the exact same object already exists, and if it does use that).

This sounds promising. Would it be in a pre_create? Do you know of examples we can look at as a reference? @roaks3

[roaks3]

Yea, we do it with google_project_service, and plan to do it with google_identity_platform_config as well (I think some others exist too). I'm still not sure if it's a good fit here, but presumably it would be done with a pre_create or custom_create.