Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Require explicit flag to remove MyPy cache when running breeze stop #29493

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions STATIC_CODE_CHECKS.rst
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,14 @@ require Breeze Docker image to be build locally.
the image by setting ``SKIP_IMAGE_PRE_COMMITS`` to "true". This will mark the tests as "green" automatically
when run locally (note that those checks will anyway run in CI).

.. note:: Mypy volume cache

MyPy uses a separate docker-volume (called ``mypy-cache-volume``) that keeps the cache of last MyPy
execution in order to speed MyPy checks up (sometimes by order of magnitude). While in most cases MyPy
will handle refreshing the cache when and if needed, there are some cases when it won't (cache invalidation
is the hard problem in computer science). This might happen for example when we upgrade MyPY. In such
cases you might need to manually remove the cache volume by running ``breeze stop --cleanup-mypy-cache``.

.. BEGIN AUTO-GENERATED STATIC CHECK LIST

+-----------------------------------------------------------+------------------------------------------------------------------+---------+
Expand Down
12 changes: 9 additions & 3 deletions dev/breeze/src/airflow_breeze/commands/developer_commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -472,20 +472,26 @@ def compile_www_assets(dev: bool):
@click.option(
"-p",
"--preserve-volumes",
help="Skip removing volumes when stopping Breeze.",
help="Skip removing database volumes when stopping Breeze.",
is_flag=True,
)
@click.option(
"-c",
"--cleanup-mypy-cache",
help="Additionally cleanup MyPy cache.",
is_flag=True,
)
@option_verbose
@option_dry_run
def stop(preserve_volumes: bool):
def stop(preserve_volumes: bool, cleanup_mypy_cache: bool):
perform_environment_checks()
command_to_execute = [*DOCKER_COMPOSE_COMMAND, "down", "--remove-orphans"]
if not preserve_volumes:
command_to_execute.append("--volumes")
shell_params = ShellParams(backend="all", include_mypy_volume=True)
env_variables = get_env_variables_for_docker_commands(shell_params)
run_command(command_to_execute, env=env_variables)
if not preserve_volumes:
if cleanup_mypy_cache:
command_to_execute = ["docker", "volume", "rm", "--force", "mypy-cache-volume"]
run_command(command_to_execute, env=env_variables)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@
"name": "Stop flags",
"options": [
"--preserve-volumes",
"--cleanup-mypy-cache",
],
},
],
Expand Down
2 changes: 1 addition & 1 deletion images/breeze/output-commands-hash.txt
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ setup:56a2ef337c354362760d247df5d05365
shell:76e0f530b7af514a2aad3032b6516c46
start-airflow:06d4aeb5f1b65f6b975f3f915558d0b3
static-checks:12e8fed2acbed0d823efc5121fd0eb58
stop:8969537ccdd799f692ccb8600a7bbed6
stop:e5aa686b4e53707ced4039d8414d5cd6
testing:docker-compose-tests:b86c044b24138af0659a05ed6331576c
testing:helm-tests:94a442e7f3f63b34c4831a84d165690a
testing:integration-tests:585da1e636f710be9c9de36a71586963
Expand Down
88 changes: 44 additions & 44 deletions images/breeze/output-commands.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
24 changes: 14 additions & 10 deletions images/breeze/output_stop.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 2 additions & 1 deletion scripts/ci/pre_commit/pre_commit_mypy.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@
if cmd_result.returncode != 0:
get_console().print(
"[warning]If you see strange stacktraces above, "
"run `breeze ci-image build --python 3.7` and try again."
"run `breeze ci-image build --python 3.7` and try again. "
"You can also run `breeze stop --cleanup-mypy-cache` to clean up the cache used."
)
sys.exit(cmd_result.returncode)