Question: testing dbt through this package

[samLozier]

Feel free to disregard if this is out of scope, but I'm curious how the developer or other users of this package are testing their mwaa/dbt projects so that they can deploy with confidence. My current process is basically:

• develop/run dbt locally
• create a dag for the project
• push to CI
• ci runs dbt from the cli with a config that should more closely match production
• deploy dbt project to s3
• mwaa picks up dag/code
• user manually clicks through mwaa dashboard to trigger a run

The interface with mwaa, dbt and this package is where I'm having the hardest time testing code since errors only come up when I'm deep into the process. Is there a way that I could be running dbt locally through this package, or in CI (basically without getting all the way to mwaa) that would allow for faster development?

I've looked through the tests for this package and am thinking about going that route, but it seems to miss some of the airflow issues that could come up. I was also thinking of installing airflow and running the dags from the airflow cli in CI, but again not sure if I'm missing something.

Any advice, even just high level, would be much appreciated.

[tomasfarias]

We are using a combination of unit tests and integration tests:

Unit tests

Our unit tests follow Airflow's recommendations in Best Practices and are mainly concerned with testing whether our DAGs can load, have a correct structure (we do much more besides just running Dbt* operators), and are parametrized correctly.

Integration tests

For a complete integration testing, ideally I'd want a second MWAA environment, where to deploy my DAGs first, and I'd run my DAGs in a testing cluster that contains a subset of your production data. You can then make assertions on your data, for example using DbtTestOperator, or regular dbt test.

Of course, this has a significant added cost, as we would need to duplicate our entire production environment. To save some costs, instead of an entirely different cluster, one could use a different database or schema in the same production cluster. Moreover, the second MWAA environment may also be skipped, with some name-mangling of DAGs and usage of variables to parametrize the dbt target to the testing environment, or even triggering the DAGs in CI/CD (for example, our DAGs test is running a sample DAG).

Naturally, using the same systems as in production will stress production resources, so good tracking of production performance is required, especially if your SLAs are tight.

All of our DAGs are constantly running DbtTestOperator to ensure correctness in both testing and production environments. We do not deploy to production until our entire pipeline has successfully run in our testing environment. This does take quite a lot of time though and has slowed down our release cadence, so we are actually looking into fast tracking some changes by scoping down some test runs.

About this package's tests

airflow-dbt-python is built on top of dbt and Airflow, so we do not test dbt or Airflow themselves. Moreover, covering all possible testing scenarios would be impossible due to how complex a dbt project can be. We do try to cover basic dbt features with some mock models, and I do want to cover all the extra features airflow-dbt-python provides.

For this reason, I'd be interested to know which Airflow issues that could come up are you referencing? It may be valuable to add some testing coverage here, depending on what they are.

Thank you.

[tomasfarias]

I'll be converting this one to a discussion for the time being, as there are no issues to tackle and fits the Q&A format better.

If we arrive at the need to increase test coverage, we can create specific issues for that task.