Launch kpi_forecast tasks in derived-datasets #1387
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Since the airflow upgrade in #1377 these tasks are failing with: > ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=403, message=Required "container.clusters.get" permission(s) for "projects/moz-fx-data-shared-prod/zones/us-central1-a/clusters/bq-load-gke-1" These tasks should be writing to shared-prod, but accessing the GKE cluster using the default connection and project.
jklukas
commented
Oct 19, 2021
| dataset_id, | ||
| table_id, | ||
| gcp_conn_id="google_cloud_derived_datasets", | ||
| project_id='moz-fx-data-derived-datasets', |
There was a problem hiding this comment.
The docstring lists project_id as required, so removing default and changing position. In practice, all calls to this function are setting project_id already.
Note that for this function project_id refers to the destination table location rather than where the task is running.
jklukas
commented
Oct 19, 2021
| @@ -39,7 +39,6 @@ def simpleprophet_forecast( | |||
| return GKEPodOperator( | |||
| task_id=task_id, | |||
| gcp_conn_id=gcp_conn_id, | |||
| project_id=project_id, | |||
There was a problem hiding this comment.
We don't want to pass in project_id here, but rather use the default in GKEPodOperator
wlach
approved these changes
Oct 19, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since the airflow upgrade in
#1377
these tasks are failing with:
These tasks should be writing to shared-prod, but accessing the GKE cluster
using the default connection and project.