Convert cli commands to use dbt executable path

[chrishronek]

Currently, we are sourcing the venv here. Instead, we should reference the dbt executable so that implementing the dbt-ol wrapper is easier.

Before:

dbt_path = [bash_path, "-c", f"source {self.python_venv} && dbt {self.base_cmd} "]

After:

{self.python_venv}/bin/dbt {self.base_cmd}

[chrishronek]

@jlaneve & @iancmoritz, after merging this, we should probably tag a new minor release - this is going to break some shit

[jlaneve]

hmm I was thinking this would look more along the lines of:

DbtTaskGroup(
  ...,
  dbt_executable="/usr/local/airflow/dbt_venv/bin/dbt"
)

This would abstract away the venv, lineage, etc management from our operators and let the user handle it, so you could do something like:

DbtTaskGroup(
  ...,
  dbt_executable="/usr/local/airflow/dbt_venv/bin/dbt-ol"
)

And I believe we could also default it to just dbt (since, after pip installing it, it'll be in the Docker image's PATH).

@pohek321 what do you think?

[chrishronek]

hmm I was thinking this would look more along the lines of:

DbtTaskGroup(
  ...,
  dbt_executable="/usr/local/airflow/dbt_venv/bin/dbt"
)

This would abstract away the venv, lineage, etc management from our operators and let the user handle it, so you could do something like:

DbtTaskGroup(
  ...,
  dbt_executable="/usr/local/airflow/dbt_venv/bin/dbt-ol"
)

And I believe we could also default it to just dbt (since, after pip installing it, it'll be in the Docker image's PATH).

@pohek321 what do you think?

@jlaneve Ahhh, yeah that makes way more sense - that's a pretty easy change anyway. I'll get it done.

[chrishronek]

Did some successful testing:

dbt-ol

Running dbt-ol (with venv)

Dockerfile:

FROM quay.io/astronomer/astro-runtime:7.1.0

# install python virtualenv to run dbt
WORKDIR /usr/local/airflow
COPY dbt-requirements.txt ./
RUN python -m virtualenv dbt_venv && source dbt_venv/bin/activate && \
    pip install --no-cache-dir -r dbt-requirements.txt && deactivate

dbt-requirements:

openlineage-dbt

Operator:

DbtDepsOperator(
    ...
    python_venv='/usr/local/airflow/dbt_venv/bin/dbt-ol'
)

Result:

['/usr/local/airflow/dbt_venv/bin/dbt-ol', 'deps', '--project-dir', '/usr/local/airflow/dbt/jaffle_shop', '--profile', '***_profile']

---

Running dbt-ol (no venv)

requirements.txt

openlineage-dbt

Operator:

DbtDepsOperator(
    ...
)

Result:

['/usr/local/bin/dbt-ol', 'deps', '--project-dir', '/usr/local/airflow/dbt/jaffle_shop', '--profile', '***_profile']

---

dbt

Running dbt (with venv)

Dockerfile:

FROM quay.io/astronomer/astro-runtime:7.1.0

# install python virtualenv to run dbt
WORKDIR /usr/local/airflow
COPY dbt-requirements.txt ./
RUN python -m virtualenv dbt_venv && source dbt_venv/bin/activate && \
    pip install --no-cache-dir -r dbt-requirements.txt && deactivate

dbt-requirements:

dbt-core

Operator:

DbtDepsOperator(
    ...
    python_venv='/usr/local/airflow/dbt_venv/bin/dbt'
)

Result:

['/usr/local/airflow/dbt_venv/bin/dbt', 'deps', '--project-dir', '/usr/local/airflow/dbt/jaffle_shop', '--profile', '***_profile']

---

Running dbt (no venv)

requirements.txt

dbt-core

Operator:

DbtDepsOperator(
    ...
)

Result:

['/usr/local/bin/dbt', 'deps', '--project-dir', '/usr/local/airflow/dbt/jaffle_shop', '--profile', '***_profile']

[chrishronek]

@jlaneve sorry for the long comment ☝🏻

[jlaneve]

nice! that looks great, can we rename python_venv to something like dbt_executable_path?

[chrishronek]

nice! that looks great, can we rename python_venv to something like dbt_executable_path?

@jlaneve just committed it up -- thoughts?

[jlaneve]

apologies for the big review here, but tried to be overly verbose to turn this into a checklist 😬

overall looking really good! just a handful of minor things

[jlaneve]

this looks great!