Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix behavior for reattach_state parameter in BigQueryInsertJobOperator #40664

Merged
merged 1 commit into from
Jul 10, 2024

Conversation

VladaZakharova
Copy link
Contributor

When trying to attach to RUNNING (but also any other state too) BQ Job using BigQueryInsertJobOperator, even though reattach_states={'PENDING', 'RUNNING'} parameter is set and Job is still running, ValueError: Job already begun is thrown.

Attaching to DONE job, which according to documentation shouldn't be possible (from airflow doc: if job with given id already exists then it tries to reattach to the job if its not done and its state is in reattach_states. If the job is done the operator will raise AirflowException.), but throws an error, as well as with any other Job type because of the validation in job._begin() call and fails if it's different than None.

This PR fixes this by adding checks of the current state before going to execute the Job in deferrable or synch mode.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Jul 9, 2024
@VladaZakharova
Copy link
Contributor Author

Hi @potiuk @Lee-W ! Can you please check changes in this PR? Thanks!

@potiuk potiuk merged commit c29eeb1 into apache:main Jul 10, 2024
51 checks passed
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants