BashOperator Windows compatibility

[SledgeHammer01]

Apache Airflow version

2.9.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When attempting to run the simple tutorial from:

https://airflow.apache.org/docs/apache-airflow/stable/tutorial/fundamentals.html

on a Windows machine, there are two minor issues preventing this from working on Windows (related to #10388).

I resolved these issues locally and it seems to work on Windows now.

in airflow/hooks/subprocess.py line 77 in run_command() method:

            self.sub_process = Popen(
                command,
                stdout=PIPE,
                stderr=STDOUT,
                cwd=cwd,
                env=env if env or env == {} else os.environ,
                preexec_fn=pre_exec,
            )

fix #1: if running under Windows, set preexec_fn=None.

fix #2: for Popen call, we also need to pass in close_fds=True. On Windows, the code as is, will result in a race condition where the sub_process has not fully released the directory created in cwd = stack.enter_context(TemporaryDirectory(prefix="airflowtmp")), and we'll get an exception:

  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.1264.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 637, in _rmtree_unsafe
    os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\xxx\\AppData\\Local\\Temp\\airflowtmpngs54_vn'

What you think should happen instead?

BashOperator should work on Windows. Airflow 2.10 addresses a few other blocking issues.

How to reproduce

Running the simple tutorial at https://airflow.apache.org/docs/apache-airflow/stable/tutorial/fundamentals.html on a Windows box in Pycharm a few times will result in the exception.

Operating System

Windows 11

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

Race condition exception occurs 95% of the time when attempting to run the tutorial.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[shahar1]

Welcome to Apache Airflow!
As it currently seems, Windows support is not highly prioritized within the community - so chances are low that other people will resolve this issue for you anytime soon. However, as it seems that you have the knowledge to resolve it - you're welcome to create a PR, while ensuring compatibility with current supported OS (it means that you'll need to run Airflow on Linux distribution to check that it works - you could do it via WSL2).

Small note - when you write #1 or #2 as regular text, it automatically links to the correponding issues/PRs, which might be confusing. To avoid confusion - please use other phrasing instead, like "no. 1" or "no. 2", or just make it a list :)