Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add install_deps in the template fields for supported operators #1227

Open
1 task done
SoheilSalmani opened this issue Sep 25, 2024 · 1 comment
Open
1 task done
Labels
area:dependencies Related to dependencies, like Python packages, library versions, etc dbt:deps Primarily related to dbt deps command or functionality enhancement New feature or request execution:virtualenv Related to Virtualenv execution environment triage-needed Items need to be reviewed / assigned to milestone

Comments

@SoheilSalmani
Copy link

SoheilSalmani commented Sep 25, 2024

Description

It would be great to specify the install_deps field as an Airflow template field so that we can use Jinja templating it its value.

Use case/motivation

I would like to be able to manually decide when I want to run dbt deps via Airflow (typically just for the first run or when package dependencies change) by using an Airflow Param, while having the default value to False.

I see something like this:

dag = DbtDag(
    dag_id=dag_id,
    project_config=ProjectConfig(dbt_project_path=dbt_project_path),
    render_config=RenderConfig(emit_datasets=False),
    profile_config=profile_config,
    execution_config=venv_execution_config,
    schedule=None,
    operator_args={"install_deps": "{{ params.install_deps }}"},
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=["dbt"],
    params={
        "install_deps": Param(
            False, type="boolean", description="Install dbt dependencies?"
        ),
    },
    render_template_as_native_obj=True,
)

On Airflow when triggering a DAG:

image

Related issues

No response

Are you willing to submit a PR?

  • Yes, I am willing to submit a PR!
@SoheilSalmani SoheilSalmani added enhancement New feature or request triage-needed Items need to be reviewed / assigned to milestone labels Sep 25, 2024
@dosubot dosubot bot added area:dependencies Related to dependencies, like Python packages, library versions, etc dbt:deps Primarily related to dbt deps command or functionality execution:virtualenv Related to Virtualenv execution environment labels Sep 25, 2024
@tatiana
Copy link
Collaborator

tatiana commented Sep 30, 2024

@SoheilSalmani it seems a fair use-case, please, feel free to open a PR on this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:dependencies Related to dependencies, like Python packages, library versions, etc dbt:deps Primarily related to dbt deps command or functionality enhancement New feature or request execution:virtualenv Related to Virtualenv execution environment triage-needed Items need to be reviewed / assigned to milestone
Projects
None yet
Development

No branches or pull requests

2 participants