Normalization: handle non-object top-level schemas; treat binary data as string #22165

edgao · 2023-01-31T19:40:36Z

old schema

{type: [object, string], properties: {...}}

new schema

{oneOf: [{type: object, properties: {...}}, {ref: string}]}

normalization doesn't like this.

also, treat binary data as string for now - this is failing in a mysql->bigquery and mssql->snowflake sync.

github-actions · 2023-01-31T19:43:04Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (16)

Connector	Version	Changelog	Publish
`destination-bigquery`	`1.2.13`	✅	✅
`destination-bigquery-denormalized`	`1.2.12`	✅	❌ (diff seed version)
`destination-clickhouse`	`0.2.2`	❌ (changelog missing)	✅
`destination-clickhouse-strict-encrypt`	`0.2.2`	🔵 (ignored)	🔵 (ignored)
`destination-jdbc`	`0.3.14`	🔵 (ignored)	🔵 (ignored)
`destination-mssql`	`0.1.22`	✅	✅
`destination-mssql-strict-encrypt`	`0.1.22`	🔵 (ignored)	🔵 (ignored)
`destination-mysql`	`0.1.20`	✅	✅
`destination-mysql-strict-encrypt`	❌ `0.1.21` (mismatch: `0.1.20`)	🔵 (ignored)	🔵 (ignored)
`destination-oracle`	`0.1.19`	✅	✅
`destination-oracle-strict-encrypt`	`0.1.19`	🔵 (ignored)	🔵 (ignored)
`destination-postgres`	`0.3.26`	✅	✅
`destination-postgres-strict-encrypt`	`0.3.26`	🔵 (ignored)	🔵 (ignored)
`destination-redshift`	`0.3.56`	✅	✅
`destination-snowflake`	`0.4.47`	✅	✅
`destination-tidb`	`0.1.0`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)

base-normalization

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

...e-integrations/bases/base-normalization/normalization/transform_catalog/catalog_processor.py

github-actions · 2023-01-31T20:09:00Z

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage	24.51%	❌

gosusnp

The change about binary looks reasonable to me. About the oneOf, I'd defer to Greg or someone with more experience with json schemas / data types.

edgao · 2023-01-31T20:15:40Z

I'm going to publish this, I think the worst case is no worse than current behavior.

edgao · 2023-01-31T20:15:53Z

/publish connector=bases/base-normalization run-tests=false

edgao · 2023-01-31T20:33:05Z

/publish connector=bases/base-normalization run-tests=false

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

edgao · 2023-01-31T20:48:19Z

/publish connector=bases/base-normalization run-tests=false

🕑 Publishing the following connectors:
bases/base-normalization
https://github.com/airbytehq/airbyte/actions/runs/4058215908

Connector	Did it publish?	Were definitions generated?
bases/base-normalization	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2023-01-31T20:59:54Z

Platform Test Results

  243 files +  5   243 suites +5 23m 55s ⏱️ + 8m 52s
1 661 tests +40 1 650 ✔️ +36 11 💤 +5 0 ❌ - 1
1 680 runs +44 1 669 ✔️ +40 11 💤 +5 0 ❌ - 1

Results for commit cf2a30d. ± Comparison against base commit 1ebd913.

♻️ This comment has been updated with latest results.

github-actions · 2023-01-31T21:18:59Z

Kube Test Results

  47 files ±0   47 suites ±0 20m 57s ⏱️ +12s
208 tests ±0 202 ✔️ ±0 6 💤 ±0 0 ❌ ±0
226 runs ±0 220 ✔️ ±0 6 💤 ±0 0 ❌ ±0

Results for commit cf2a30d. ± Comparison against base commit 1ebd913.

This pull request removes 4 and adds 4 tests. Note that renamed tests count towards both.

io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowTest$FailedActivityWorkflow ‑ [1] Thread[Thread-8,5,main]
io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowTest$FailedActivityWorkflow ‑ [2] Thread[Thread-9,5,main]
io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowTest$FailedActivityWorkflow ‑ [3] Thread[Thread-10,5,main]
io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowTest$FailedActivityWorkflow ‑ [4] Thread[Thread-11,5,main]

io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowTest$FailedActivityWorkflow ‑ [1] Thread[Thread-6,5,main]
io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowTest$FailedActivityWorkflow ‑ [2] Thread[Thread-7,5,main]
io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowTest$FailedActivityWorkflow ‑ [3] Thread[Thread-8,5,main]
io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowTest$FailedActivityWorkflow ‑ [4] Thread[Thread-9,5,main]

♻️ This comment has been updated with latest results.

edgao · 2023-01-31T21:25:14Z

/publish connector=bases/base-normalization run-tests=false

🕑 Publishing the following connectors:
bases/base-normalization
https://github.com/airbytehq/airbyte/actions/runs/4058487740

Connector	Did it publish?	Were definitions generated?
bases/base-normalization	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

edgao · 2023-01-31T21:58:21Z

/approve-and-merge reason="OC change, ran a subset of normalizaton integration tests locally"

octavia-approvington · 2023-01-31T21:59:01Z

What are we doing again?

…ary data as string (#22165)" This reverts commit 8276d03.

* Revert "Normalization: handle non-object top-level schemas; treat binary data as string (#22165)" This reverts commit 8276d03. * Revert "Normalization: check for ref type existence (#22161)" This reverts commit dbe56d6. * Revert "🎉Updated normalization to handle new datatypes (#19721)" This reverts commit c1d7736. * revert dest definitions * also dockerfile * re-add to changelog * add comment in dockerfile

handle dumb top-level schemas

96c4e2f

edgao requested a review from a team as a code owner January 31, 2023 19:40

octavia-squidington-iv added the normalization label Jan 31, 2023

grishick reviewed Jan 31, 2023

View reviewed changes

...e-integrations/bases/base-normalization/normalization/transform_catalog/catalog_processor.py Show resolved Hide resolved

grishick reviewed Jan 31, 2023

View reviewed changes

...e-integrations/bases/base-normalization/normalization/transform_catalog/catalog_processor.py Show resolved Hide resolved

edgao added 2 commits January 31, 2023 11:58

version bump

9986fb1

also definitions

2b9a651

octavia-squidington-iv added the area/documentation Improvements or additions to documentation label Jan 31, 2023

edgao temporarily deployed to more-secrets January 31, 2023 20:00 — with GitHub Actions Inactive

edgao changed the title ~~Normalization: handle dumb top-level schemas~~ Normalization: handle non-object top-level schemas Jan 31, 2023

edgao changed the title ~~Normalization: handle non-object top-level schemas~~ Normalization: handle non-object top-level schemas; treat binary data as string Jan 31, 2023

treat binary as string

01c0da1

edgao temporarily deployed to more-secrets January 31, 2023 20:09 — with GitHub Actions Inactive

fallback case

bb73a1f

edgao temporarily deployed to more-secrets January 31, 2023 20:14 — with GitHub Actions Inactive

gosusnp approved these changes Jan 31, 2023

View reviewed changes

edgao temporarily deployed to more-secrets January 31, 2023 20:14 — with GitHub Actions Inactive

format

255280a

edgao temporarily deployed to more-secrets January 31, 2023 20:28 — with GitHub Actions Inactive

edgao temporarily deployed to more-secrets January 31, 2023 20:29 — with GitHub Actions Inactive

Merge branch 'master' into edgao/fixes

431f8c9

edgao temporarily deployed to more-secrets January 31, 2023 20:44 — with GitHub Actions Inactive

new variable

cf2a30d

edgao temporarily deployed to more-secrets January 31, 2023 21:27 — with GitHub Actions Inactive

grishick approved these changes Jan 31, 2023

View reviewed changes

edgao enabled auto-merge (squash) January 31, 2023 21:40

octavia-approvington approved these changes Jan 31, 2023

View reviewed changes

octavia-approvington merged commit 8276d03 into master Jan 31, 2023

octavia-approvington deleted the edgao/fixes branch January 31, 2023 21:59

edgao added a commit that referenced this pull request Feb 2, 2023

Revert "Normalization: handle non-object top-level schemas; treat bin…

7fae32d

…ary data as string (#22165)" This reverts commit 8276d03.

octavia-squidington-iii mentioned this pull request Feb 2, 2023

Bump Airbyte version from 0.40.32 to 0.40.33 #22335

Closed

github-actions bot mentioned this pull request Feb 7, 2023

Bump helm chart version reference to 0.43.28 #22489

Closed

This was referenced Feb 8, 2023

Bump Airbyte version from 0.40.32 to 0.40.33 #22578

Closed

Bump Airbyte version from 0.40.32 to 0.40.33 #22689

Closed

Bump Airbyte version from 0.40.32 to 0.40.33 #22839

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Normalization: handle non-object top-level schemas; treat binary data as string #22165

Normalization: handle non-object top-level schemas; treat binary data as string #22165

edgao commented Jan 31, 2023 •

edited

Loading

github-actions bot commented Jan 31, 2023 •

edited

Loading

✅ Sources (0)

❌ Destinations (16)

👀 Other Modules (1)

Actionable Items

github-actions bot commented Jan 31, 2023

gosusnp left a comment

edgao commented Jan 31, 2023

edgao commented Jan 31, 2023

edgao commented Jan 31, 2023 •

edited by github-actions bot

Loading

edgao commented Jan 31, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jan 31, 2023 •

edited

Loading

github-actions bot commented Jan 31, 2023 •

edited

Loading

edgao commented Jan 31, 2023 •

edited by github-actions bot

Loading

edgao commented Jan 31, 2023

octavia-approvington commented Jan 31, 2023

Normalization: handle non-object top-level schemas; treat binary data as string #22165

Normalization: handle non-object top-level schemas; treat binary data as string #22165

Conversation

edgao commented Jan 31, 2023 • edited Loading

github-actions bot commented Jan 31, 2023 • edited Loading

Affected Connector Report

✅ Sources (0)

❌ Destinations (16)

👀 Other Modules (1)

Actionable Items

github-actions bot commented Jan 31, 2023

Airbyte Code Coverage

gosusnp left a comment

Choose a reason for hiding this comment

edgao commented Jan 31, 2023

edgao commented Jan 31, 2023

edgao commented Jan 31, 2023 • edited by github-actions bot Loading

edgao commented Jan 31, 2023 • edited by github-actions bot Loading

github-actions bot commented Jan 31, 2023 • edited Loading

Platform Test Results

github-actions bot commented Jan 31, 2023 • edited Loading

Kube Test Results

edgao commented Jan 31, 2023 • edited by github-actions bot Loading

edgao commented Jan 31, 2023

octavia-approvington commented Jan 31, 2023

edgao commented Jan 31, 2023 •

edited

Loading

github-actions bot commented Jan 31, 2023 •

edited

Loading

edgao commented Jan 31, 2023 •

edited by github-actions bot

Loading

edgao commented Jan 31, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jan 31, 2023 •

edited

Loading

github-actions bot commented Jan 31, 2023 •

edited

Loading

edgao commented Jan 31, 2023 •

edited by github-actions bot

Loading