Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery incremental tables and hooks #802

Closed

Conversation

lewish
Copy link
Contributor

@lewish lewish commented Jun 19, 2018

  • Uses delete statements for unique_key incremental tables for now, should probably be changed to merge commands as discussed.
  • Tests are... basic, have attempted to do most testing using actual dbt data tests because BQ complicated things otherwise. If I can find a way to run arbitrary SQL for BQ in tests, these can be a lot more complete.
  • We have been using this in production for a couple of weeks now and seems to be working well (without using unique_key).

Cheers!

@drewbanin
Copy link
Contributor

drewbanin commented Jul 19, 2018

@lewish thanks for the PR! We ended up implementing Incremental models in #856 (using the merge statement) and hooks in #836

Just as a note: BigQuery automatically creates a temporary table for every query. We're using this mechanism to create "temp" tables, though I do maintain that your timeout approach was super clever :)

Going to close this PR as the above changes are shipping in our 0.10.2 release!

@drewbanin drewbanin closed this Jul 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants