You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Use customised adapter, provide jinja with further python methods
When you want to add more methods to existing adapter and make these methods available to jinja.
you can use dbt_custom_adapter variable and use your adapter class with dbt.
Step-1: Extend existing adapter, add new methods to it.
Using customized adapter and a custom materialization we can extend dbt to run local python code.
this is useful for the scenarios where data is imported from external API.
Step-1: Extend existing adapter, Add new adapter method which runs given python code.
Here we are extending DuckDBAdapter and adding new method submit_local_python_job to it. This method executes given
python code as a subprocess
Step-2: Create materialization named executepython, In this materialization (from the jonja) we call this new(
above) adapter method to run compiled python code
fromopendbt.airflowimportOpenDbtAirflowProject# create dbt build tasks for models with given tagp=OpenDbtAirflowProject(resource_type='model', project_dir="/dbt/project_dir", profiles_dir="/dbt/profiles_dir",
target='dev', tag="MY_TAG")
p.load_dbt_tasks(dag=dag, start_node=start, end_node=end)
Create dag to run tests
fromopendbt.airflowimportOpenDbtAirflowProject# create dbt test tasks with given model tagp=OpenDbtAirflowProject(resource_type='test', project_dir="/dbt/project_dir", profiles_dir="/dbt/profiles_dir",
target='dev', tag="MY_TAG")
p.load_dbt_tasks(dag=dag, start_node=start, end_node=end)
Create page on Airflow Server to serve DBT docs
While its very practical to use airflow for dbt executions, it could also be used to server dbt docs.
here is how:
Step-1: Create python file under airflow /{airflow}/plugins directory, with following code.
Adjust the given path to the folder where dbt docs are published.