-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Require explicit flag to remove MyPy cache when running breeze stop #29493
Require explicit flag to remove MyPy cache when running breeze stop #29493
Conversation
The change apache#29184 added cleanup of MyPy cache when `breeze stop` was called, but this is a bit too agressive. It will remove the cache always when breeze stop is run and it will prolong (on MacOS by 10s of seconds) the MyPy static checks when it happens. Since cleaning the cache is only really needed when MyPy cache gets broken (for example when we upgrade MyPy), cleaning it explicitly when you encounter such problem is a better idea. This PR: * adds flag to breeze stop to clean the mypy cache (disabled by default) * adds documentation about that feature in STATIC_CHECKS * adds hint displayed to the user when mypy check fails
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Breeze is aware of when library version changes as it asks to rebuild the image can we maybe detect mypy version changes and then promt the question to clean the cache (or just cleanit anyway under the hood)?
Well. It's not 'Always' when mypy version changes. In vast majority of cases cache worked fine when we upgraded mypy. Usually mypy can clean it up all right. It was just one case when some cache was broken few weeks ago that affected literally few people i am aware of. And also the cache could be broken for other reasons (for example when you ctrl-c while mypy doing it's job if you are unlucky/. So this is really 'easy recovery' when things get really, really wrong |
I am also not even sure if the root cause in this case was related to MyPy upgrade actually - it looked like it could be (because that was around the time when we upgraded), but I cannot be 100% sure 🤷 . It could also be because - for example - some typeshed changed or mypy information for our dependencies and some stored information in cache could be wrong and "missed" by MyPy (and that one would be next to impossible to know where such error gets triggered). So we do not really have a clear "trigger" for cleaning the cache that we are sure of. We could potentially catch the right exception and clean it up, but this kind of error might result in an exception that we won't be able to detect as "invalid cache" being the root cause. Best we can do (and I did) is revert to human-in-the-loop and in case we see an error tell the human "look - if you see strange errors, maybe this one will help". |
…29493) The change #29184 added cleanup of MyPy cache when `breeze stop` was called, but this is a bit too agressive. It will remove the cache always when breeze stop is run and it will prolong (on MacOS by 10s of seconds) the MyPy static checks when it happens. Since cleaning the cache is only really needed when MyPy cache gets broken (for example when we upgrade MyPy), cleaning it explicitly when you encounter such problem is a better idea. This PR: * adds flag to breeze stop to clean the mypy cache (disabled by default) * adds documentation about that feature in STATIC_CHECKS * adds hint displayed to the user when mypy check fails (cherry picked from commit 8841537)
…29493) The change #29184 added cleanup of MyPy cache when `breeze stop` was called, but this is a bit too agressive. It will remove the cache always when breeze stop is run and it will prolong (on MacOS by 10s of seconds) the MyPy static checks when it happens. Since cleaning the cache is only really needed when MyPy cache gets broken (for example when we upgrade MyPy), cleaning it explicitly when you encounter such problem is a better idea. This PR: * adds flag to breeze stop to clean the mypy cache (disabled by default) * adds documentation about that feature in STATIC_CHECKS * adds hint displayed to the user when mypy check fails (cherry picked from commit 8841537)
The change #29184 added cleanup of MyPy cache when
breeze stop
was called, but this is a bit too agressive. It will remove the cache always when breeze stop is run and it will prolong (on MacOS by 10s of seconds) the MyPy static checks when it happens.Since cleaning the cache is only really needed when MyPy cache gets broken (for example when we upgrade MyPy), cleaning it explicitly when you encounter such problem is a better idea.
This PR:
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.