Skip to content

Commit

Permalink
Add workaround for Airflow consuming all memory in some OS'es (#29751)
Browse files Browse the repository at this point in the history
Apparently with some recent releases of some OSes where new
containerd has been released, default docker community edition
causes Airlfow to immediately consume all memory.

This happens in Breeze and Docker Compose at least.

There is a workaround described in:
ttps://github.com/moby/moby/issues/43361#issuecomment-1227617516
to use Docker Desktop instead.

The issue is tracked in containerd in this issue - proposing to
revert the change (as it impacts other applications run in docker,
not only Airlfow):
containerd/containerd#7566

GitOrigin-RevId: de2889c2e9779177363d6b87dc9020bf210fdd72
  • Loading branch information
potiuk authored and Cloud Composer Team committed Sep 19, 2024
1 parent 0fa1b71 commit 28b3ad9
Show file tree
Hide file tree
Showing 3 changed files with 37 additions and 0 deletions.
18 changes: 18 additions & 0 deletions BREEZE.rst
Original file line number Diff line number Diff line change
Expand Up @@ -670,6 +670,24 @@ Then run the failed command, copy-and-paste the output from your terminal to the
`Airflow Slack <https://s.apache.org/airflow-slack>`_ #airflow-breeze channel and
describe your problem.


.. warning::

Some operating systems (Fedora, ArchLinux, RHEL, Rocky) have recently introduced Kernel changes that result in
Airflow in Breeze consuming 100% memory when run inside the community Docker implementation maintained
by the OS teams.

This is an issue with backwards-incompatible containerd configuration that some of Airflow dependencies
have problems with and is tracked in a few issues:

* `Moby issue <https://github.com/moby/moby/issues/43361>`_
* `Containerd issue <https://github.com/containerd/containerd/pull/7566>`_

There is no solution yet from the containerd team, but seems that installing
`Docker Desktop on Linux <https://docs.docker.com/desktop/install/linux-install/>`_ solves the problem as
stated in `This comment <https://github.com/moby/moby/issues/43361#issuecomment-1227617516>`_ and allows to
run Breeze with no problems.

Advanced commands
=================

Expand Down
18 changes: 18 additions & 0 deletions docs/apache-airflow/howto/docker-compose/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,24 @@ Older versions of ``docker-compose`` do not support all the features required by
docker run --rm "debian:bullseye-slim" bash -c 'numfmt --to iec $(echo $(($(getconf _PHYS_PAGES) * $(getconf PAGE_SIZE))))'
.. warning::

Some operating systems (Fedora, ArchLinux, RHEL, Rocky) have recently introduced Kernel changes that result in
Airflow in Docker Compose consuming 100% memory when run inside the community Docker implementation maintained
by the OS teams.

This is an issue with backwards-incompatible containerd configuration that some of Airflow dependencies
have problems with and is tracked in a few issues:

* `Moby issue <https://github.com/moby/moby/issues/43361>`_
* `Containerd issue <https://github.com/containerd/containerd/>`_

There is no solution yet from the containerd team, but seems that installing
`Docker Desktop on Linux <https://docs.docker.com/desktop/install/linux-install/>`_ solves the problem as
stated in `This comment <https://github.com/moby/moby/issues/43361#issuecomment-1227617516>`_ and allows to
run Breeze with no problems.



Fetching ``docker-compose.yaml``
================================
Expand Down
1 change: 1 addition & 0 deletions docs/spelling_wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,7 @@ conn
connectTimeoutMS
connexion
containerConfiguration
containerd
ContainerGroup
containerinstance
ContainerPort
Expand Down

0 comments on commit 28b3ad9

Please sign in to comment.