Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: adds client side filtering to export #149

Merged
merged 1 commit into from
Sep 1, 2023

Conversation

a-rampalli
Copy link

@a-rampalli a-rampalli commented Sep 1, 2023

What

We wanted to fix the param being sent to export api so that we get data from the timestamp we want. Turns out if we do that ordering of records will not hold in the api response we get. Not having ordering will create problems in us storing state at regular intervals. This is a bigger problem because of the unexpected end of json errors
So we are doing client side filtering of records fetched from export api.

Testing

The records now are ordered by time and also have time value more than state {"export": {"time": "2023-08-09T10:59:59Z"}}
Screenshot 2023-09-01 at 3 54 44 PM
If we set state value greater than end date we get 0 records as intended
Screenshot 2023-09-01 at 4 54 23 PM

@github-actions
Copy link

github-actions bot commented Sep 1, 2023

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • The connector tests are passing in CI
  • You've updated the connector's metadata.yaml file (new!)
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

@a-rampalli a-rampalli merged commit f5448be into main Sep 1, 2023
2 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants