Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle APIs which emit numbers as strings #4984

Closed
sherifnada opened this issue Jul 26, 2021 · 2 comments
Closed

Handle APIs which emit numbers as strings #4984

sherifnada opened this issue Jul 26, 2021 · 2 comments
Labels
area/connectors Connector related issues type/enhancement New feature or request

Comments

@sherifnada
Copy link
Contributor

Tell us about the problem you're trying to solve

Some APIs like Facebook or Shopify output numbers as strings e.g: #4827 and #4841 This probably happens in more APIs as well.

Given that this is probably a ubiquitous issue we should have a simple and easy way to handle it.

There are some options here:

  1. In the catalog type definition for such fields, add an annotation e.g: airbyte_type_hint: number indicating that a particular string is actually a number and then have Airbyte or the destination worker. This is nice because the source doesn't have to worry about transformation. On the other hand it is not ideal because we'd be relying on each destination to support this custom type hint.
  2. Do the transformation in the source directly (e.g: hardcode which fields have this issue and transform in the code), making sure the JSON Schema type is output as number in the catalog. if we go this route we might want to add this capability into the CDK.
  3. Something else I'm not thinking of

Describe the solution you’d like

  1. Research and decide on which option to use
  2. create follow up tickets for implementation
@sherifnada sherifnada added type/enhancement New feature or request area/connectors Connector related issues labels Jul 26, 2021
@keu
Copy link
Contributor

keu commented Jul 26, 2021

  1. we can just use existing json_schema field airbyte_type_hint -> format

@sherifnada
Copy link
Contributor Author

redundant now that we have type transformer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues type/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants