Confusing paragraph in Airflow Bestpractice documentaion (Airflow Variables)

[KadeRyu]

What do you see as an issue?

In the Airflow Best Practices article, I found a paragraph that was confusing to readers:

---

Bad example:

from airflow.models import Variable

foo_var = Variable.get("foo")  # DON'T DO THAT
bash_use_variable_bad_1 = BashOperator(
    task_id="bash_use_variable_bad_1", bash_command="echo variable foo=${foo_env}", env={"foo_env": foo_var}
)

bash_use_variable_bad_2 = BashOperator(
    task_id="bash_use_variable_bad_2",
    bash_command=f"echo variable foo=${Variable.get('foo')}",  # DON'T DO THAT
)

bash_use_variable_bad_3 = BashOperator(
    task_id="bash_use_variable_bad_3",
    bash_command="echo variable foo=${foo_env}",
    env={"foo_env": Variable.get("foo")},  # DON'T DO THAT
)

Good example:

bash_use_variable_good = BashOperator(
    task_id="bash_use_variable_good",
    bash_command="echo variable foo=${foo_env}",
    env={"foo_env": "{{ var.value.get('foo') }}"},
)

@task
def my_task():
    var = Variable.get("foo")  # this is fine, because func my_task called only run task, not scan DAGs.
    print(var)

---

In the "Bad Example" section, it states that Variable.get("foo") should not be specified in an Operator. Instead, it recommends calling it using a Jinja template, as described in the "Good Example" section.
However, it explains that it is possible to call it in the @task operator. This might make it seem like users who want to call the Variable.get() method shouldn't do so in a traditional Operator.

Secondly, to simplify the code, the Operators are not in the DAG and are not indented, so there is room for interpretation that the BashOperators in the bad example are outside the DAG (top-level code), and therefore the Variable.get() method is also top-level.

Solving the problem

I think some sentences should be added to explain why Variable.get() methods called from inside an Operator are a bad example.

I don't know enough about the logic that the scheduler is parsing to make a precise suggestion, but I think there are cases like this

• If there is no case where you need to use the Variable.get() method inside an Operator, and using a Jinja template is absolutely fine: add to the documentation why calling Variable.get() inside an Operator is recognized as top-level code, and calling Variable.get() inside a @task operator is fine.
• When it's okay to use the Variable.get() method inside an Operator: add appropriate examples to the documentation

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[eladkal]

Thank you for trying to improve the docs!
However I am not sure what is missing in the explnation?

I think some sentences should be added to explain why Variable.get() methods called from inside an Operator are a bad example.

But it's OK to use Variable.get() in operator executed code like python callable of PythonOperator.
Users should avoid using it in parts that are parsed by the scheduler.

Could you check https://stackoverflow.com/a/71758614/14624409 if that clarify the issue we can work on modifying the docs to reflect it.

[KadeRyu]

Hi @eladkal! thanks for your information.
As you said it's okay to use Variable.get() method in any operator, I wonder why below examples are bad:

bash_use_variable_bad_2 = BashOperator(
    task_id="bash_use_variable_bad_2",
    bash_command=f"echo variable foo=${Variable.get('foo')}",  # DON'T DO THAT
)

bash_use_variable_bad_3 = BashOperator(
    task_id="bash_use_variable_bad_3",
    bash_command="echo variable foo=${foo_env}",
    env={"foo_env": Variable.get("foo")},  # DON'T DO THAT
)

[potiuk]

@KadeRyu. It's simple, but you either need to understand how Python parsing works or experiment with it.

Run python <your_dag>.py. This runs python parsing. if your Variable.get() is called during that operation it's bad. Because it is executed during python parsing. Using Variable.get() is fine when you use it in execute() method of the operator (This method is NOT executed during parsing)..

[potiuk]

Experiment with that and you will see.

[potiuk]

#34006