Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: type annotation for num_of_dpus in GlueJobHook #29176

Merged
merged 1 commit into from
Jan 27, 2023

Conversation

wonhyeongseo
Copy link
Contributor

closes: #29091

Simple fix of type annotation for num_of_dpus according to AWS Glue Documentation

@Taragolis may you please review this PR?

@Taragolis
Copy link
Contributor

You also need to change in Operator (potentially in other places)

num_of_dpus: int | None = None,

I still don't know what is better annotation for this case float | None or int | float | None

@Taragolis
Copy link
Contributor

Yeah, statics check failed:

airflow/providers/amazon/aws/hooks/glue.py:97: error: Incompatible types in
assignment (expression has type "float", variable has type "int")  [assignment]
                self.num_of_dpus = num_of_dpus

@wonhyeongseo
Copy link
Contributor Author

Ok @Taragolis. Adding in float instead of replacing int for backward compatability. I checked the files for num_of_dpus, and the modified two files are all there is. May you please run the workflow again? Thanks!

Copy link
Contributor

@Taragolis Taragolis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pending approval (wait for CI)

@Taragolis
Copy link
Contributor

🤣 mypy became crazy when tried to make static check against condition in Hook

https://github.com/apache/airflow/blob/c4c14c07a99b1554346a426aa947e68eb52ce31e/airflow/providers/amazon/aws/hooks/glue.py#L95

Add annotation here, something like: self.num_of_dpus: int | float = 10

@wonhyeongseo
Copy link
Contributor Author

image
Updated as you suggested, @Taragolis! Thank you so much for guiding me. I hope to contribute more in the future. Have a great day/afternoon!

@potiuk potiuk merged commit 4402456 into apache:main Jan 27, 2023
@boring-cyborg
Copy link

boring-cyborg bot commented Jan 27, 2023

Awesome work, congrats on your first merged pull request!

@wonhyeongseo wonhyeongseo deleted the gluejobhook-type-annotation branch January 27, 2023 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:amazon-aws AWS/Amazon - related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Incorrect type annotation for num_of_dpus in GlueJobOperator/GlueJobHook
3 participants