Bug: operator_args are modified in place in the Airflow converter

[jbandoro]

A user reported this in the Slack thread here. Even though we deprecated operator_args vars and env in favor of ProjectConfig.dbt_vars and ProjectConfig.env_vars, there is now a problem where if a user supplies operator_args to multiple DbtDag or DbtTaskGroup's with vars on env, they will only be used in the first dag/task group object.

Example problem (copied from the user report)

OSMOS_COMMON_RENDER_CONF = dict(
    emit_datasets=False, test_behavior=TestBehavior.AFTER_EACH, load_method=LoadMode.CUSTOM, dbt_deps=False
)


with DAG(
        dag_id="my-amazing-dag",
        default_args=default_args,
        schedule="50 4 * * *",
) as dag:
    dbt_task_group_base_config = {
        "project_config": MY_WONDERFUL_PROJECT_COSMOS_PROJ_CONF,
        "profile_config": MY_WONDERFUL_PROJECT_COSMOS_PROFILE_CONF,
        "execution_config": BI_PLATFORM_COSMOS_EXEC_CONF,
        "operator_args": {
            "vars": {"logical_date": "{{ execution_date }}"},
        },
    }

    dbt_tasks_google_data = DbtTaskGroup(
        group_id="dbt_tasks_google_data",
        render_config=RenderConfig(
            **COSMOS_COMMON_RENDER_CONF,
            select=[
                "config.tags:gam",
                "config.tags:dfp_perf",
                "path:models/staging/google",
                "path:models/marts/google",
                "path:models/reporting",
            ]
        ),
        **dbt_task_group_base_config,
    )

    dbt_tasks_facebook_data = DbtTaskGroup(
        group_id="dbt_tasks_facebook_data",
        render_config=RenderConfig(
            **COSMOS_COMMON_RENDER_CONF,
            select=[
                "path:models/staging/facebook",
                "path:models/marts/facebook",
            ]
        ),
        **dbt_task_group_base_config,  # logical_date doesn't get passed
    )

    dbt_tasks_customer_portal = DbtTaskGroup(
        group_id="dbt_tasks_customer_portal",
        render_config=RenderConfig(
            **COSMOS_COMMON_RENDER_CONF,
            select=[
                "path:seeds/customer_portal",
                "path:models/staging/customer_portal",
                "path:models/marts/customer_portal",
            ]
        ),
        **dbt_task_group_base_config, # logical_date doesn't get passed
    )

The responsible lines for this bug are in the converter is below:

astronomer-cosmos/cosmos/converter.py

Lines 224 to 225 in 714ecbd

	env_vars = project_config.env_vars or operator_args.pop("env", None)
	dbt_vars = project_config.dbt_vars or operator_args.pop("vars", None)

The fix is really straightforward, to use get instead of pop in the lines above, this is ok because Cosmos will already raise an exception if both operator_args and ProjectConfig with env vars or dbt vars are both set here.

[dosubot]

🤖

Thank you for bringing this issue to our attention! It's great to see that you've identified a straightforward fix for the problem. Your contribution to the repository would be highly valuable. We encourage you to go ahead and create a pull request with your proposed fix. Your effort will help improve the overall quality of the project. Keep up the good work!

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.