Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tentatively disallowed normalization for DuckDB #22528

Merged
merged 1 commit into from
Feb 8, 2023
Merged

Tentatively disallowed normalization for DuckDB #22528

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions airbyte-config/init/src/main/resources/seed/destination_definitions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -449,9 +449,4 @@
dockerImageTag: 0.1.0
documentationUrl: https://docs.airbyte.io/integrations/destinations/duckdb
icon: duckdb.svg
normalizationConfig:
normalizationRepository: airbyte/normalization-duckdb
normalizationTag: 0.2.25
normalizationIntegrationType: duckdb
supportsDbt: true
releaseStage: alpha
2 changes: 0 additions & 2 deletions airbyte-integrations/bases/base-normalization/integration_tests/dbt_integration_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,6 @@ def setup_db(self, destinations_to_test: List[str]):
self.setup_clickhouse_db()
if DestinationType.TIDB.value in destinations_to_test:
self.setup_tidb_db()
if DestinationType.DUCKDB.value in destinations_to_test:
self.setup_duckdb_db()

def setup_postgres_db(self):
start_db = True
Expand Down
2 changes: 1 addition & 1 deletion docs/integrations/destinations/duckdb.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ This destination writes data to a file on the _local_ filesystem on the host run

If you set [Normalization](https://docs.airbyte.com/understanding-airbyte/basic-normalization/), source data will be normalized to a tabular form. Let's say you have a source such as GitHub with nested JSONs; the Normalization ensures you end up with tables and columns. Suppose you have a many-to-many relationship between the users and commits. Normalization will create separate tables for it. The end state is the [third normal form](https://en.wikipedia.org/wiki/Third_normal_form) (3NF).

If turn off the Normalization, each stream will be output into its own table `_airbyte_raw_{stream_name}`. Each table will contain 3 columns:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section is still true, so let's keep it. I.e. just remove the "if you turn off normalization" bit

Each table will contain 3 columns:

* `_airbyte_ab_id`: a uuid assigned by Airbyte to each event that is processed.
* `_airbyte_emitted_at`: a timestamp representing when the event was pulled from the data source.
Expand Down
Loading