on DbtTaskGroup creation dag parameter is ignored.

[tboutaour]

Currently, there is an issue when attempting to create a DbtTaskGroup using the dag parameter directly. However, this limitation can be bypassed using the with statement, where there is no issue.

Functional example:

with DAG(...):
    DbtTaskGroup(...) 

Problematic example:

dag = DAG(...):
DbtTaskGroup(dag=dag...) 

This problem stems from the airflow_kwargs function within converter.py. Although the dag argument is assumed to be present in DbtToAirflowConverter, this parameter is not being passed to the creation of TaskGroup.

I suggest two potential solutions to the problem:

1. Adjusting the DbtTaskGroup operator to explicitly pass the dag parameter when creating the task group.
2. Adjusting the behavior of the airflow_kwargs function to ensure that the dag parameter is properly handled when creating the TaskGroup.

[dosubot]

To address the issue where the dag parameter is ignored when creating a DbtTaskGroup directly, but works within a with statement, consider the following adjustments:

1. Modify DbtToAirflowConverter: Ensure the dag parameter is explicitly passed to the TaskGroup during its creation. This might involve revising the logic in the build_airflow_graph function within the DbtToAirflowConverter class to correctly associate the dag with the TaskGroup.

2. Adjust airflow_kwargs Function: Update the airflow_kwargs function to correctly include the dag parameter in its output. This function should filter kwargs for those specific to Airflow's DAG or TaskGroup initialization methods, ensuring dag is recognized as a valid keyword argument and not removed by the filtering logic.

def airflow_kwargs(**kwargs):
    """
    Ensure 'dag' parameter is correctly handled.
    """
    new_kwargs = {}
    for arg_key, arg_value in kwargs.items():
        if arg_key == 'dag':  # Explicitly include 'dag'
            new_kwargs[arg_key] = arg_value
    return new_kwargs

3. Review TaskGroup Creation Logic: In the DbtToAirflowConverter, where TaskGroups are created, make sure the dag parameter is explicitly passed if available. This ensures that the TaskGroup is correctly linked to the DAG.

These adjustments should help in resolving the issue with the dag parameter being ignored when passed directly to the DbtTaskGroup. This approach aligns with ensuring that all necessary parameters are correctly passed and handled within the Airflow DAG and TaskGroup creation process.

Sources

• DbtTaskGroup should not need dag=dag
• cosmos/converter.py
• cosmos/airflow/task_group.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.