Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update normalization to handle the new $ref data types (including binary data) #17642

Closed
Tracked by #17637
edgao opened this issue Oct 5, 2022 · 1 comment · Fixed by #19721
Closed
Tracked by #17637

Update normalization to handle the new $ref data types (including binary data) #17642

edgao opened this issue Oct 5, 2022 · 1 comment · Fixed by #19721
Assignees
Labels
team/destinations Destinations team's backlog

Comments

@edgao
Copy link
Contributor

edgao commented Oct 5, 2022

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/bases/base-normalization/normalization/transform_catalog/utils.py needs to be updated to check for $ref values instead of type/airbyte_type/format. This updated version of normalization will never see old-style catalogs, because it will be released after the platform protocol versioning migrations, and normalization is released as part of platform (i.e. it's impossible for someone to upgrade normalization without also upgrading platform).

Also, add handling for oneOf declarations (which will replace type: [...] decls).

Normalization also should add handling for BinaryData. A minimal implementation is to just treat this as string; a less-minimal implementation is to use destination-specific BINARY columns.

https://github.com/airbytehq/airbyte/blob/master/airbyte-protocol/protocol-models/src/main/resources/airbyte_protocol/well_known_types.yaml

Useful links:
https://docs.google.com/document/d/1aJdnNgpqm6rVV711HaABEpCz6wdl3deCA7A41Rl4CRI/edit?pli=1#heading=h.a98kcaptkze3

https://docs.google.com/document/d/1dZOEbCkW2uDHdvIEdQeGL8gigSwGSKCGX5bESFicE9I/edit#heading=h.oeobczvupt7v

https://docs.google.com/document/d/17DvoWuT88TI5FEMYTiq19IamLOKHfn84jikdEiic3Wk/edit#heading=h.42w4y05moqk2

@grishick
Copy link
Contributor

This is merged into the feature branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team/destinations Destinations team's backlog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants