🎉 Destination-gcs handling v1 data protocol #20635

suhomud · 2022-12-19T09:42:41Z

What

Destination gcs protocol V1 support

How

The main change is done in Destination s3 protocol update (#20088) plus json-avro-converter (airbytehq/json-avro-converter#17)
This PR has next change:

integration tests are using V1 dataset
minor changes for gcs test's comparator
- base64 decoding for test comparison.
- time comparison.

🚨 User Impact 🚨

No impact

suhomud · 2022-12-19T09:50:10Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/3730271587
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/3730271587
No Python unittests run

Build Passed

Test summary info:

All Passed

suhomud · 2022-12-19T09:50:57Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3730276998
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3730276998
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

suhomud · 2022-12-19T09:52:22Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3730287260
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3730287260
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

suhomud · 2022-12-19T11:24:41Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/3730964775
❌ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/3730964775
🐛 https://gradle.com/s/w36scx4jlpq6a

Build Failed

Test summary info:

Could not find result summary

edgao

glad to see this didn't require any changes to runtime code - code reuse 💯

ryankfu · 2022-12-19T19:42:08Z

...on/java/io/airbyte/integrations/destination/gcs/GcsAvroParquetDestinationAcceptanceTest.java

@@ -85,7 +91,7 @@ private JsonNode getJsonNode(final AirbyteStream stream, final String name) {
  }

  private Set<Type> getExpectedSchemaType(final JsonNode fieldDefinition) {
-    final JsonNode typeProperty = fieldDefinition.get("type");
+    final JsonNode typeProperty = fieldDefinition.get("type") == null ? fieldDefinition.get("$ref") : fieldDefinition.get("type");


Nice work, suggestion to maybe considering a comment here that notes that $ref is utilized as a result of the version migration and that type is the original standard since tests are a great way to maintain knowledge of business context

The type is still valid since we have object and arrays (for example: "type": ["object"]) in schemas but it will be worth to mentioned the $ref in the comments since it's not obvious. Thanks for suggesting!

Awesome, both changes look great, and thanks for making this code re-useable

ryankfu · 2022-12-19T19:55:42Z

...test-integration/java/io/airbyte/integrations/destination/gcs/GcsAvroTestDataComparator.java

@@ -33,4 +36,23 @@ protected boolean compareDateTimeValues(String airbyteMessageValue, String desti
    return super.compareDateTimeValues(airbyteMessageValue, format.format(dateTime));
  }

+  @Override
+  protected boolean compareTime(final String airbyteMessageValue, final String destinationValue) {
+    var destinationDate = LocalTime.ofInstant(getInstantFromEpoch(destinationValue), ZoneOffset.UTC);


Minor nit: instead of var have this be LocalTime so it's clear that the difference between this super.compareTime and this child method is that this is doing a LocalTime comparison vs String comparison

ryankfu

LGTM, minor comments but nothing blocking, great usage of code reuse

suhomud · 2022-12-20T09:02:43Z

The fixed tests for bigquery-denormalized #20640
Will re-run after code freeze is finished

…rotocol_v1

suhomud · 2023-01-18T03:53:53Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/3945556774
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/3945556774
No Python unittests run

Build Passed

Test summary info:

All Passed

Update destination-gcs to handle v1 data protocol

58c411b

octavia-squidington-iv added area/connectors Connector related issues connectors/destination/gcs labels Dec 19, 2022

suhomud linked an issue Dec 19, 2022 that may be closed by this pull request

Update destination-gcs to handle the new data types protocol #17646

Closed

suhomud marked this pull request as ready for review December 19, 2022 14:35

suhomud requested a review from a team as a code owner December 19, 2022 14:35

edgao approved these changes Dec 19, 2022

View reviewed changes

ryankfu reviewed Dec 19, 2022

View reviewed changes

ryankfu approved these changes Dec 19, 2022

View reviewed changes

grishick approved these changes Dec 19, 2022

View reviewed changes

suhomud added 2 commits December 20, 2022 16:23

Review changes

d008dda

Merge branch 'master' into suhomud/17646_destination_gcs_data_types_p…

6208ad3

…rotocol_v1

suhomud merged commit 036c46d into master Jan 18, 2023

suhomud deleted the suhomud/17646_destination_gcs_data_types_protocol_v1 branch January 18, 2023 06:29

octavia-squidington-iii mentioned this pull request Jan 24, 2023

Bump Airbyte version from 0.40.28 to 0.40.29 #21767

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Destination-gcs handling v1 data protocol #20635

🎉 Destination-gcs handling v1 data protocol #20635

suhomud commented Dec 19, 2022 •

edited

Loading

suhomud commented Dec 19, 2022 •

edited by github-actions bot

Loading

suhomud commented Dec 19, 2022 •

edited by github-actions bot

Loading

suhomud commented Dec 19, 2022 •

edited by github-actions bot

Loading

suhomud commented Dec 19, 2022 •

edited by github-actions bot

Loading

edgao left a comment

ryankfu Dec 19, 2022

suhomud Dec 20, 2022

ryankfu Dec 20, 2022

ryankfu Dec 19, 2022

ryankfu left a comment

suhomud commented Dec 20, 2022 •

edited

Loading

suhomud commented Jan 18, 2023 •

edited by github-actions bot

Loading

🎉 Destination-gcs handling v1 data protocol #20635

🎉 Destination-gcs handling v1 data protocol #20635

Conversation

suhomud commented Dec 19, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

suhomud commented Dec 19, 2022 • edited by github-actions bot Loading

Build Passed

suhomud commented Dec 19, 2022 • edited by github-actions bot Loading

Build Passed

suhomud commented Dec 19, 2022 • edited by github-actions bot Loading

Build Passed

suhomud commented Dec 19, 2022 • edited by github-actions bot Loading

Build Failed

edgao left a comment

Choose a reason for hiding this comment

ryankfu Dec 19, 2022

Choose a reason for hiding this comment

suhomud Dec 20, 2022

Choose a reason for hiding this comment

ryankfu Dec 20, 2022

Choose a reason for hiding this comment

ryankfu Dec 19, 2022

Choose a reason for hiding this comment

ryankfu left a comment

Choose a reason for hiding this comment

suhomud commented Dec 20, 2022 • edited Loading

suhomud commented Jan 18, 2023 • edited by github-actions bot Loading

Build Passed

suhomud commented Dec 19, 2022 •

edited

Loading

suhomud commented Dec 19, 2022 •

edited by github-actions bot

Loading

suhomud commented Dec 19, 2022 •

edited by github-actions bot

Loading

suhomud commented Dec 19, 2022 •

edited by github-actions bot

Loading

suhomud commented Dec 19, 2022 •

edited by github-actions bot

Loading

suhomud commented Dec 20, 2022 •

edited

Loading

suhomud commented Jan 18, 2023 •

edited by github-actions bot

Loading