Create a simple plugin system for writing data to external destinations #173
Conversation
…irst implementation
Bumps [dbt-tests-adapter](https://github.com/dbt-labs/dbt-core) from 1.5.0 to 1.5.1. - [Release notes](https://github.com/dbt-labs/dbt-core/releases) - [Changelog](https://github.com/dbt-labs/dbt-core/blob/v1.5.1/CHANGELOG.md) - [Commits](dbt-labs/dbt-core@v1.5.0...v1.5.1) --- updated-dependencies: - dependency-name: dbt-tests-adapter dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]>
|
Going to merge this and then start in on revamping the source-side plugins to move them into |
|
Awsome work @jwills! Do have or know of any examples of this functionality? |
|
Hey @JCZuurmond, thanks so much! So I rewrote the existing glue stuff to use the new plugin framework to do its thing (though it’s backwards compatible with the old way of doing it), but I haven’t created any new ones yet— very much open to suggestion here! maybe sqlalchemy? I still don’t feel like I grok delta-rs to understand how folks actually use it for stuff like this. Pyiceberg doesn’t support general writes yet afaict. What do you think? |
|
sqlachemy would be nice! I want to start with a reference use case, maybe as a blog post or talk, to show the Excel capabilities and to inspire other plugin ideas. |
Fixes #143
This implements a write-side plugin system that can run after a relation is materialized (either in DuckDB or externally) and can be used to take some action on the resulting data. I'm starting out by porting the Glue database stuff (which can persist an output parquet file as a relation in the AWS Glue catalog) to guide the implementation of the plugins to act as a dual to the source-side plugins.
One rub here is that I don't have a good functional test of the glue stuff, so I'm going to create one in a separate PR that runs against the existing glue impl, merge it, and then port the impl over to confirm that it will work against my new impl here (and likely fix the bugs I find in the process.)