This repo demonstrates how you can build your first CI pipeline with GitHub Actions for a simple dbt project. By the end of this project, you will see a CI pipeline that will, upon a new PR being opened in your dbt project, accomplish three things:
- Ensure your dbt models compile and build properly
- Test your dbt models with the test you’ve established for them
- Run a SQL linter against your code changes
The article goes into more detail on each step here.
I also walk through the same tutorial in this 5 minute Loom:
It uses seeds that includes fake raw data from a fictional app, via dbt lab's jaffle shop test project. You can also download the data directly from here.
The best way to learn how to create your first GitHub Actions workflow is to fork this repository and follow our tutorial (link to be added upon publication).
You can take a look at what's in our super simple workflow here called Our first dbt PR job
By the end of the tutorial, you will have run your first workflow!
Imagine you would like to only analyze orders made in March 2018 (the full dataset, which you can see in the orders.csv file, contains orders between January to April 2018).
Here's what a CI workflow should look like
- We'll create a new branch to make our change in. In your terminal:
git checkout -b "update-fct-orders"
- Then, update the
fct_orders.sqlfile to add a filter:
orders as (
-- Filtering orders to March 2018
select * from {{ ref('stg_orders')}}
where order_date >= '2018-03-01' and order_date <= '2018-03-31'
),
-
Commit the change to your respository and open a new PR. Here's the open PR from this repository.
-
Wait for our GitHub Actions workflow, that was automatically triggered with the opened PR, to finish running. Success! You can now merge to main with the confidence that our modified dbt model did not break anything and our code underwent linting.
Learn more about:
- SQLFluff's resource for GitHub Actions
- data-diff and Datafold Cloud
- dbt