Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Grants to BigQuery Materializations #212

Merged
merged 31 commits into from
Jul 11, 2022
Merged

Add Grants to BigQuery Materializations #212

merged 31 commits into from
Jul 11, 2022

Conversation

emmyoop
Copy link
Member

@emmyoop emmyoop commented Jun 29, 2022 •
edited
Loading

resolves #198

Before merge

  • Reinstate original version of dev-requirements.txt
  • Ensure merging into main rather than a custom branch

Description

This is dependent on dbt-labs/dbt-core#5263

location is required for the sql only approach but logic needs to be added around defaulting the location. Currently the working state has the location hard coded.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-bigquery next" section.

@cla-bot cla-bot bot added the cla:yes label Jun 29, 2022
jtcohen6
jtcohen6 reviewed Jun 29, 2022
View reviewed changes
Copy link
Contributor

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pushing up the draft PR !!

This also serves as a nice outline of what most adapters will need to implement to make use of the grants feature

dbt/include/bigquery/macros/materializations/copy.sql Outdated Show resolved Hide resolved
dbt/include/bigquery/macros/adapters/apply_grants.sql Outdated Show resolved Hide resolved
dbt/include/bigquery/macros/adapters/apply_grants.sql Outdated Show resolved Hide resolved
@dbeatty10 dbeatty10 mentioned this pull request Jul 7, 2022
Closed
9 tasks
jtcohen6
jtcohen6 reviewed Jul 7, 2022
View reviewed changes
dbt/adapters/bigquery/impl.py Outdated Show resolved Hide resolved
dbt/include/bigquery/macros/adapters/apply_grants.sql Outdated Show resolved Hide resolved
dbt/include/bigquery/macros/adapters/apply_grants.sql Outdated Show resolved Hide resolved
dbt/include/bigquery/macros/materializations/table.sql Outdated Show resolved Hide resolved
dev-requirements.txt Outdated Show resolved Hide resolved
@emmyoop emmyoop changed the title Er/ct 717 grant materialization Add Grants to BigQuery Materializations Jul 7, 2022
McKnight-42
McKnight-42 reviewed Jul 7, 2022
View reviewed changes
dev-requirements.txt Outdated Show resolved Hide resolved
emmyoop
emmyoop commented Jul 8, 2022
View reviewed changes
Copy link
Member Author

@emmyoop emmyoop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0/5 tests pass right now. The immediate root cause seems to be the replace logic for the privileges. I've worked through some of it but there seems to be more.

There may also be a bug in the grant/revoke logic as the version of test_model_grants in this repo fails on an assertion error of grants matching.

dbt/include/bigquery/macros/adapters/apply_grants.sql Outdated Show resolved Hide resolved
test.env.example Outdated Show resolved Hide resolved
tests/conftest.py Outdated Show resolved Hide resolved
tests/functional/adapter/test_grants.py Outdated Show resolved Hide resolved
tests/functional/adapter/test_model_grants_local.py Outdated Show resolved Hide resolved
@jtcohen6 jtcohen6 mentioned this pull request Jul 9, 2022
Merged
6 tasks
@jtcohen6 jtcohen6 changed the base branch from main to dbeatty/grantee-env-vars-take-2 July 11, 2022 07:40
@jtcohen6 jtcohen6 mentioned this pull request Jul 11, 2022
Merged
@jtcohen6 jtcohen6 marked this pull request as ready for review July 11, 2022 10:30
@dbeatty10 dbeatty10 self-requested a review July 11, 2022 14:20
dbeatty10
dbeatty10 approved these changes Jul 11, 2022
View reviewed changes
Copy link
Contributor

@dbeatty10 dbeatty10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
Looks good to me. Awesome collaboration @emmyoop and @jtcohen6 to figure out all the tricky corner cases and solve them. 🙌

Base automatically changed from dbeatty/grantee-env-vars-take-2 to main July 11, 2022 16:38
jtcohen6 and others added 5 commits July 11, 2022 11:46
@jtcohen6 @emmyoop @dbeatty10
Alt approach for grant location (#218)
27c41ed 
* Alt approach to grabbing + factoring info_schema region

* add exception for blank location

* Update dbt/adapters/bigquery/relation.py

Co-authored-by: Doug Beatty <[email protected]>

* point to existing branch

* fix pre commit errors

Co-authored-by: Emily Rockman <[email protected]>
Co-authored-by: Doug Beatty <[email protected]>
@emmyoop
Merge branch 'main' of https://github.com/dbt-labs/dbt-bigquery into …
3b9c9a4 
…er/ct-717-grant-materialization
@emmyoop
reset to point dbt-core to main instead of branch
a6ef920
@emmyoop
fix examples in test.env.example
f47322f
@emmyoop
add changelog
d94b623
@McKnight-42 McKnight-42 self-requested a review July 11, 2022 18:46
McKnight-42
McKnight-42 approved these changes Jul 11, 2022
View reviewed changes
Copy link
Contributor

@McKnight-42 McKnight-42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks awesome curious if the location thing was the issue we were hitting last week?

@emmyoop emmyoop merged commit 942d460 into main Jul 11, 2022
@emmyoop emmyoop deleted the er/ct-717-grant-materialization branch July 11, 2022 19:49
@jlarue26 jlarue26 mentioned this pull request Oct 24, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jtcohen6 jtcohen6 jtcohen6 left review comments

@dataders dataders dataders left review comments

@McKnight-42 McKnight-42 McKnight-42 approved these changes

@dbeatty10 dbeatty10 dbeatty10 approved these changes

Assignees
No one assigned
Labels
cla:yes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[CT-717] Add Grants to BigQuery Materializations
5 participants
@emmyoop @dataders @jtcohen6 @dbeatty10 @McKnight-42