high memory leak, cannot start even webserver

[antonio-antuan]

Apache Airflow version

2.5.1

What happened

I'd used airflow 2.3.1 and everything was fine.
Then I decided to move to airflow 2.5.1.
I can't start even webserver, airflow on my laptop consumes the entire memory (32Gb) and OOM killer comes.

I investigated a bit. So it starts with airflow 2.3.4. Only using official docker image (apache/airflow:2.3.4) and only on linux laptop, mac is ok.

Memory leak starts when source code tries to import for example airflow.cli.commands.webserver_command module using airflow.utils.module_loading.import_string.
I dived deeply and found that it happens when "import daemon" is performed.
You can reproduce it with this command: docker run --rm --entrypoint="" apache/airflow:2.3.4 /bin/bash -c "python -c 'import daemon'". Once again, reproducec only on linux (my kernel is 6.1.12).
That's weird considering daemon hasn't been changed since 2018.

What you think should happen instead

No response

How to reproduce

docker run --rm --entrypoint="" apache/airflow:2.3.4 /bin/bash -c "python -c 'import daemon'"

Operating System

Arch Linux (kernel 6.1.12)

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[antonio-antuan]

ok, clarified

this is the place where things happens: site-packages/daemon/daemon.py:868

def get_maximum_file_descriptors():
    """ Get the maximum number of open file descriptors for this process.

        :return: The number (integer) to use as the maximum number of open
            files for this process.

        The maximum is the process hard resource limit of maximum number of
        open file descriptors. If the limit is “infinity”, a default value
        of ``MAXFD`` is returned.
        """
    (__, hard_limit) = resource.getrlimit(resource.RLIMIT_NOFILE)

    result = hard_limit
    if hard_limit == resource.RLIM_INFINITY:
        result = MAXFD

    return result


_total_file_descriptor_range = (0, get_maximum_file_descriptors())
_total_file_descriptor_set = set(range(*_total_file_descriptor_range))

apache/airflow:2.3.1 (and above I think) doesn't have this version of library so nothing happens during initialization.
I'd like to suggest better ulimits for image

upd: forgot the most important thing, please see screenshot

[antonio-antuan]

created an issue for python-daemon: https://pagure.io/python-daemon/issue/72

[potiuk]

Thanks for looking into it and finding out the root cause being python-daemon. So far we missed the root cause of it, but we knew the new containerd has the problem due to their changed settings.

This is a known issue with container.d changing their default setting - we had been discussing it recenty in #29731 and the discussion contains also some workarounds that you can use.

The fact that we know it's python-daemon opens up a possibility that we can likely patch it somehow while waiting for either containerd reverting the change or Python daemon fixing their behaviour

[antonio-antuan]

Glad to help :)

[potiuk]

The python-daemon==3.0.0 has been releassed with the fix - you can upgrade it and it should fix the problem.