Update normalization to handle the new $ref
data types (including binary data)
#17642
Labels
team/destinations
Destinations team's backlog
https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/bases/base-normalization/normalization/transform_catalog/utils.py needs to be updated to check for
$ref
values instead oftype
/airbyte_type
/format
. This updated version of normalization will never see old-style catalogs, because it will be released after the platform protocol versioning migrations, and normalization is released as part of platform (i.e. it's impossible for someone to upgrade normalization without also upgrading platform).Also, add handling for
oneOf
declarations (which will replacetype: [...]
decls).Normalization also should add handling for BinaryData. A minimal implementation is to just treat this as string; a less-minimal implementation is to use destination-specific
BINARY
columns.https://github.com/airbytehq/airbyte/blob/master/airbyte-protocol/protocol-models/src/main/resources/airbyte_protocol/well_known_types.yaml
Useful links:
https://docs.google.com/document/d/1aJdnNgpqm6rVV711HaABEpCz6wdl3deCA7A41Rl4CRI/edit?pli=1#heading=h.a98kcaptkze3
https://docs.google.com/document/d/1dZOEbCkW2uDHdvIEdQeGL8gigSwGSKCGX5bESFicE9I/edit#heading=h.oeobczvupt7v
https://docs.google.com/document/d/17DvoWuT88TI5FEMYTiq19IamLOKHfn84jikdEiic3Wk/edit#heading=h.42w4y05moqk2
The text was updated successfully, but these errors were encountered: