BigQuery Destination: Proposal to impersonate account on bigquery #15820

marcelopio · 2022-08-20T18:20:09Z

What

Proposal to solve #15726

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

marcosmarxm · 2022-10-18T12:14:00Z

@marcelopio do you want to move this to ready to review?

marcosmarxm · 2022-11-22T20:22:59Z

Hello 👋, first thank you for this amazing contribution.

We really appreciate the effort you've made to improve the project.
We ask you patience for the code review. Last month our team was focused on Hacktoberfest event and that probably left some PR without the proper feedback. And this week, due to the Thanksgiving US Holiday, most our team is out of office with their families. Another important piece of information why code won't be merge this week is: as a safety measure the core team has decided to freeze merging code to main branch to keep the release stable. Next week we'll return to you with the proper code review and update the status of your contribution.

If you have any questions feel free to send me a message in Slack!
Thanks!

marcosmarxm · 2022-12-02T22:30:19Z

Sorry the delay here @marcelopio I'm start the process of reviewing it!

marcosmarxm · 2022-12-08T19:24:33Z

/test connector=connectors/destination-bigquery

marcosmarxm · 2022-12-08T19:30:51Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3651478687
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3651478687
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

marcosmarxm · 2022-12-15T17:08:04Z

@grishick can the database team review this contribution? CI is passing but no test was added.

grishick · 2022-12-16T23:24:53Z

Thank you for the contribution. I'll have to take some time to set up credentials and accounts to add an E2E integration test for this feature before I can merge this change.

grishick · 2022-12-16T23:30:11Z

@grishick can the database team review this contribution? CI is passing but no test was added.

this needs a new test case. To create that test case we need application credentials and a test account that allows impersonation. We'll need to add these credentials to the secrets storage, so that the test can run via CI

marcosmarxm · 2022-12-20T19:50:09Z

Hello 👋:skin-tone-2: and thank you for your contribution!

Airbyte has instituted a code freeze between 19 and 30 December, to make sure there are no disruptions during the holidays.
Because of this, reviewing and merging your contribution may take longer than usual.
We apologize for the delay, but we want everyone to have a quiet and happy holiday.

If you have any questions or need further clarification, please don't hesitate to ping via Slack.

grishick · 2022-12-20T23:07:34Z

@marcelopio could you share with me, how you tested this change?

marcelopio · 2022-12-21T17:42:35Z

You can test it in two ways:

To test without the application default credentials, just setup a service account that has permission to impersonate another service account.

To test with the default credentials, you need to test on a Google Cloud Environment. You can test on GCE adding a default service account to the VM, or on GKE, or on Cloud Shell

grishick · 2022-12-22T01:05:04Z

I've opened another PR where I added tests: #20788

grishick · 2022-12-22T22:24:54Z

@marcelopio I have been testing these changes and I from what I can see, the changes do not apply to GCS Staging. So, if a connector is configured to impersonate another account, it will not be able to use GCS Staging, unless it is also provided with non-impersonated GCS credentials. Could you please confirm this?

marcelopio · 2022-12-22T22:35:34Z

Oh, damn, that is right...

Even if the HMAC Key is generated for the impersonated account?

But the whole use of the HMAC may defeat the purpose of this for GCS. Since the idea is having just the default key for the machine and not provide a key.

marcelopio · 2022-12-22T22:37:19Z

Would Airbyte accept a change adding native use of GCS, not passing through S3 compatibility layer?

grishick · 2022-12-23T01:13:02Z

Would Airbyte accept a change adding native use of GCS, not passing through S3 compatibility layer?

I cannot say we are too attached to the S3 compatibility layer, however, the change to use GCS natively may be pretty large. Would switching to native GCS remove the need for HMAC key?

marcelopio · 2022-12-23T01:18:54Z

Yes, since we could use the application default credentials to connect to GCS also. I will see how much work is needed to do that then. It will be a fun project :)

grishick · 2022-12-23T01:25:33Z

Yes, since we could use the application default credentials to connect to GCS also. I will see how much work is needed to do that then. It will be a fun project :)

SGTM. FYI, I added test cases for impersonation in this PR: #20788 I will likely move all my test refactoring code out of that PR and close it. Once this PR is ready to impersonate both BigQuery and GCS accounts, then I'll add those new test cases back here.

sh4sh · 2023-03-07T20:07:52Z

@marcelopio @grishick checking in, is there a status update on this PR? It looks like #20851 has been merged, but it's not clear to me whether anything else needs to be done first.

marcelopio · 2023-03-07T20:10:00Z

Hey! Sorry, but life got in the way, haha. Things are getting better but I don't hope to get back at this at least until next month.

evantahler · 2023-04-01T01:44:20Z

@marcelopio I'm going to close this PR as it's been a little while since we last heard from you. Your solution looks good though, so please reopen this PR when you are ready!

Proposal to impersonate account on bigquery

dda1539

octavia-squidington-iii added community connectors/destination/bigquery labels Aug 20, 2022

github-actions bot added the area/connectors Connector related issues label Aug 20, 2022

Added default case to also be impersonated

0357a27

marcelopio marked this pull request as ready for review October 18, 2022 13:59

octavia-squidington-iv added the bounty label Oct 18, 2022

sajarin added internal and removed bounty labels Oct 20, 2022

Merge branch 'master' into feature/bigquery-impersonate

7dcf5ee

marcosmarxm requested a review from a team December 15, 2022 17:07

Merge branch 'master' into feature/bigquery-impersonate

59d662c

rodireich added the team/destinations Destinations team's backlog label Dec 15, 2022

Merge branch 'master' into feature/bigquery-impersonate

79b2311

grishick self-assigned this Dec 21, 2022

This was referenced Dec 22, 2022

Destination-bigquery: add ability to impersonate another GCP account #20788

Closed

BigQuery impersonate service account #15726

Open

subodh1810 requested review from a team and removed request for a team February 6, 2023 08:55

evantahler assigned evantahler and sh4sh and unassigned grishick Mar 30, 2023

evantahler closed this Apr 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BigQuery Destination: Proposal to impersonate account on bigquery #15820

BigQuery Destination: Proposal to impersonate account on bigquery #15820

marcelopio commented Aug 20, 2022 •

edited

Loading

marcosmarxm commented Oct 18, 2022

marcosmarxm commented Nov 22, 2022

marcosmarxm commented Dec 2, 2022

marcosmarxm commented Dec 8, 2022

marcosmarxm commented Dec 8, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Dec 15, 2022

grishick commented Dec 16, 2022

grishick commented Dec 16, 2022

marcosmarxm commented Dec 20, 2022

grishick commented Dec 20, 2022

marcelopio commented Dec 21, 2022

grishick commented Dec 22, 2022

grishick commented Dec 22, 2022 •

edited

Loading

marcelopio commented Dec 22, 2022

marcelopio commented Dec 22, 2022

grishick commented Dec 23, 2022

marcelopio commented Dec 23, 2022

grishick commented Dec 23, 2022

sh4sh commented Mar 7, 2023 •

edited

Loading

marcelopio commented Mar 7, 2023

evantahler commented Apr 1, 2023

BigQuery Destination: Proposal to impersonate account on bigquery #15820

BigQuery Destination: Proposal to impersonate account on bigquery #15820

Conversation

marcelopio commented Aug 20, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

marcosmarxm commented Oct 18, 2022

marcosmarxm commented Nov 22, 2022

marcosmarxm commented Dec 2, 2022

marcosmarxm commented Dec 8, 2022

marcosmarxm commented Dec 8, 2022 • edited by github-actions bot Loading

Build Passed

marcosmarxm commented Dec 15, 2022

grishick commented Dec 16, 2022

grishick commented Dec 16, 2022

marcosmarxm commented Dec 20, 2022

grishick commented Dec 20, 2022

marcelopio commented Dec 21, 2022

grishick commented Dec 22, 2022

grishick commented Dec 22, 2022 • edited Loading

marcelopio commented Dec 22, 2022

marcelopio commented Dec 22, 2022

grishick commented Dec 23, 2022

marcelopio commented Dec 23, 2022

grishick commented Dec 23, 2022

sh4sh commented Mar 7, 2023 • edited Loading

marcelopio commented Mar 7, 2023

evantahler commented Apr 1, 2023

marcelopio commented Aug 20, 2022 •

edited

Loading

marcosmarxm commented Dec 8, 2022 •

edited by github-actions bot

Loading

grishick commented Dec 22, 2022 •

edited

Loading

sh4sh commented Mar 7, 2023 •

edited

Loading