[AIRFLOW-4908] BigQuery Hooks/Operators for update_dataset, patch_dataset, get_dataset #5546
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pgagnon
reviewed
Jul 8, 2019
| self.delegate_to = delegate_to | ||
|
|
||
| self.log.info('Dataset id: %s', self.dataset_id) | ||
| self.log.info('Project id: %s', self.project_id) |
There was a problem hiding this comment.
I would merge this log line with the previous one.
There was a problem hiding this comment.
Make sense. I will merge them.
ryanyuan
force-pushed
the
bq-dataset-ops
branch
3 times, most recently
from
a8ea9f4
to
8c28fce
Compare
Codecov Report
@@ Coverage Diff @@
## master #5546 +/- ##
==========================================
+ Coverage 79% 79.03% +0.03%
==========================================
Files 489 489
Lines 30726 30796 +70
==========================================
+ Hits 24275 24341 +66
- Misses 6451 6455 +4
Continue to review full report at Codecov.
|
mik-laj
reviewed
Jul 27, 2019
|
Are you planning to continue working on this change? This week, I would like to deal with PR reviews related to GCP. I would be happy if you would respond to all comments |
mik-laj
reviewed
Aug 13, 2019
| def __init__(self, | ||
| dataset_id, | ||
| project_id=None, | ||
| bigquery_conn_id='google_cloud_default', |
There was a problem hiding this comment.
Suggested change
| bigquery_conn_id='google_cloud_default', | |
| gcp_conn_id='google_cloud_default', |
Reference:
#5734
mik-laj
reviewed
Aug 13, 2019
| dataset_id, | ||
| dataset_resource, | ||
| project_id=None, | ||
| bigquery_conn_id='google_cloud_default', |
There was a problem hiding this comment.
Suggested change
| bigquery_conn_id='google_cloud_default', | |
| gcp_conn_id='google_cloud_default', |
Reference:
#5734
mik-laj
reviewed
Aug 13, 2019
| dataset_id, | ||
| dataset_resource, | ||
| project_id=None, | ||
| bigquery_conn_id='google_cloud_default', |
There was a problem hiding this comment.
Suggested change
| bigquery_conn_id='google_cloud_default', | |
| gcp_conn_id='google_cloud_default', |
Reference:
#5734
mik-laj
requested changes
Aug 13, 2019
… patch_dataset and get_dataset Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset
ryanyuan
force-pushed
the
bq-dataset-ops
branch
from
8c28fce
to
c52ee64
Compare
|
@mik-laj Done. PTAL |
mik-laj
approved these changes
Aug 14, 2019
ashb
pushed a commit
to ashb/airflow
that referenced
this pull request
Oct 16, 2019
… patch_dataset and get_dataset (apache#5546) Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (cherry picked from commit 09b9610)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To create a BigQuery sink for GCP Stackdriver Logging, I have to assign
WRITERaccess to group[email protected]to access BQ dataset. However, current BigQueryHook doesn't support updating/patching dataset.Reference: https://googleapis.github.io/google-cloud-python/latest/logging/usage.html#export-to-bigquery
Implement GCP Stackdriver Logging: AIRFLOW-4779
While it is missing update_dataset and patch_dataset, BigQueryHook has get_dataset but it doesn't have operator for it.
Features to be implemented:
BigQueryBaseCursor.patch_dataset
BigQueryBaseCursor.update_dataset
BigQueryPatchDatasetOperator
BigQueryUpdateDatasetOperator
BigQueryGetDatasetOperator
Make sure you have checked all steps below.
Jira
Description
Tests
test_bigquery_operator:BigQueryPatchDatasetOperatorTest
test_bigquery_operator:BigQueryGetDatasetOperatorTest
test_bigquery_operator:BigQueryUpdateDatasetOperatorTest
test_bigquery_hook:TestDatasetsOperations.test_patch_dataset
test_bigquery_hook:TestDatasetsOperations.test_update_dataset
Commits
Documentation
Code Quality
flake8