Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add coverage report #6045

Merged
merged 1 commit into from
Sep 15, 2021
Merged

Add coverage report #6045

merged 1 commit into from
Sep 15, 2021

Conversation

avida
Copy link
Contributor

@avida avida commented Sep 14, 2021

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Describe the solution

Recommended reading order

  1. x.java
  2. y.python

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/SUMMARY.md
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions
  • Connector added to connector index like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions
  • Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

  • Issue acceptance criteria met
  • PR name follows PR naming conventions
  • If adding a new generator, add it to the list of scaffold modules being tested
  • The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
  • Documentation which references the generator is updated as needed.

@jrhizor jrhizor temporarily deployed to more-secrets September 14, 2021 08:26 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets September 14, 2021 08:35 Inactive
@avida avida linked an issue Sep 14, 2021 that may be closed by this pull request
@avida
Copy link
Contributor Author

avida commented Sep 14, 2021

/test connector=connectors/source-braintree

🕑 connectors/source-braintree https://github.com/airbytehq/airbyte/actions/runs/1232888132
❌ connectors/source-braintree https://github.com/airbytehq/airbyte/actions/runs/1232888132
🐛 https://gradle.com/s/yamrrmdv6gnni

@jrhizor jrhizor temporarily deployed to more-secrets September 14, 2021 08:40 Inactive
@avida
Copy link
Contributor Author

avida commented Sep 14, 2021

/test connector=connectors/source-braintree

🕑 connectors/source-braintree https://github.com/airbytehq/airbyte/actions/runs/1232911645
✅ connectors/source-braintree https://github.com/airbytehq/airbyte/actions/runs/1232911645
""

@jrhizor jrhizor temporarily deployed to more-secrets September 14, 2021 08:48 Inactive
@avida avida force-pushed the drezchykov/code-coverage branch 2 times, most recently from 5d1c0ee to 565b93f Compare September 14, 2021 08:58
@jrhizor jrhizor temporarily deployed to more-secrets September 14, 2021 09:01 Inactive
@avida avida force-pushed the drezchykov/code-coverage branch 3 times, most recently from 4c8b9fc to 38e30ee Compare September 14, 2021 09:24
@jrhizor jrhizor temporarily deployed to more-secrets September 14, 2021 09:26 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets September 14, 2021 09:33 Inactive
@avida avida force-pushed the drezchykov/code-coverage branch 2 times, most recently from dfd142d to 1526474 Compare September 14, 2021 09:47
@jrhizor jrhizor temporarily deployed to more-secrets September 14, 2021 09:49 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets September 14, 2021 09:53 Inactive
@avida avida force-pushed the drezchykov/code-coverage branch 2 times, most recently from 6edd443 to dbcf0e2 Compare September 14, 2021 10:02
@avida
Copy link
Contributor Author

avida commented Sep 14, 2021

/test connector=connectors/source-braintree

🕑 connectors/source-braintree https://github.com/airbytehq/airbyte/actions/runs/1233164159
✅ connectors/source-braintree https://github.com/airbytehq/airbyte/actions/runs/1233164159
  ---------- coverage: platform linux, python 3.8.10-final-0 -----------
  Name                                           Stmts   Miss  Cover
  ------------------------------------------------------------------
  source_braintree/init.py                       2      0   100%
  source_braintree/schemas/init.py               8      0   100%
  source_braintree/schemas/cards.py                 86      0   100%
  source_braintree/schemas/common.py                42      0   100%
  source_braintree/schemas/customer.py              28      0   100%
  source_braintree/schemas/discount.py              13      0   100%
  source_braintree/schemas/dispute.py               37      0   100%
  source_braintree/schemas/merchant_account.py      29      0   100%
  source_braintree/schemas/plan.py                  21      0   100%
  source_braintree/schemas/subscription.py          35      0   100%
  source_braintree/schemas/transaction.py           99      0   100%
  source_braintree/source.py                        18      0   100%
  source_braintree/spec.py                          23      0   100%
  source_braintree/streams.py                      100      0   100%
  ------------------------------------------------------------------
  TOTAL                                            541      0   100%

@jrhizor jrhizor temporarily deployed to more-secrets September 14, 2021 10:05 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets September 14, 2021 10:05 Inactive
@avida
Copy link
Contributor Author

avida commented Sep 14, 2021

/test connector=connectors/source-braintree

🕑 connectors/source-braintree https://github.com/airbytehq/airbyte/actions/runs/1233208330
✅ connectors/source-braintree https://github.com/airbytehq/airbyte/actions/runs/1233208330

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                           Stmts   Miss  Cover
	 ------------------------------------------------------------------
	 source_braintree/__init__.py                       2      0   100%
	 source_braintree/schemas/__init__.py               8      0   100%
	 source_braintree/schemas/cards.py                 86      0   100%
	 source_braintree/schemas/common.py                42      0   100%
	 source_braintree/schemas/customer.py              28      0   100%
	 source_braintree/schemas/discount.py              13      0   100%
	 source_braintree/schemas/dispute.py               37      0   100%
	 source_braintree/schemas/merchant_account.py      29      0   100%
	 source_braintree/schemas/plan.py                  21      0   100%
	 source_braintree/schemas/subscription.py          35      0   100%
	 source_braintree/schemas/transaction.py           99      0   100%
	 source_braintree/source.py                        18      0   100%
	 source_braintree/spec.py                          23      0   100%
	 source_braintree/streams.py                      100      0   100%
	 ------------------------------------------------------------------
	 TOTAL                                            541      0   100%

Copy link
Contributor

@sherifnada sherifnada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! Well done @avida

How does this handle multiple pytest runs? For example in a single /test command there can be unit tests, custom integration tests, and acceptance tests. Ideally we output coverage from unit tests and custom integration tests (integration_tests directory) combined but I understand it might be difficult.

If that's not possible/easy to do, maybe let's add a header to the coverage report printed to make it clear that the coverage comes from unit tests?

Also we should be prepared that a connector was not written in python and will not have a coverage output

Copy link
Contributor

@sherifnada sherifnada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to merge once you've addressed the above comment

@sherifnada
Copy link
Contributor

sherifnada commented Sep 14, 2021

/test connector=source-mysql

🕑 source-mysql https://github.com/airbytehq/airbyte/actions/runs/1235492434
✅ source-mysql https://github.com/airbytehq/airbyte/actions/runs/1235492434


@sherifnada
Copy link
Contributor

sherifnada commented Sep 14, 2021

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1235514058
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1235514058


@jrhizor jrhizor temporarily deployed to more-secrets September 14, 2021 22:51 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets September 14, 2021 22:51 Inactive
@avida
Copy link
Contributor Author

avida commented Sep 15, 2021

/test connector=source-mysql

🕑 source-mysql https://github.com/airbytehq/airbyte/actions/runs/1236530098
✅ source-mysql https://github.com/airbytehq/airbyte/actions/runs/1236530098
Python unittest coverage:


@avida
Copy link
Contributor Author

avida commented Sep 15, 2021

/test connector=source-braintree

🕑 source-braintree https://github.com/airbytehq/airbyte/actions/runs/1236535816
✅ source-braintree https://github.com/airbytehq/airbyte/actions/runs/1236535816

@avida
Copy link
Contributor Author

avida commented Sep 15, 2021

/test connector=source-mysql

🕑 source-mysql https://github.com/airbytehq/airbyte/actions/runs/1236539888
✅ source-mysql https://github.com/airbytehq/airbyte/actions/runs/1236539888

@jrhizor jrhizor temporarily deployed to more-secrets September 15, 2021 06:41 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets September 15, 2021 06:42 Inactive
@avida
Copy link
Contributor Author

avida commented Sep 15, 2021

How does this handle multiple pytest runs? For example in a single /test command there can be unit tests, custom integration tests, and acceptance tests. Ideally we output coverage from unit tests and custom integration tests (integration_tests directory) combined but I understand it might be difficult.

It only captures output for python unittest. What do you mean by integration/acceptance test coverge? If it code under "airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests" directory it would be not so difficult.
But if it actual connector's code run by SAT it might be a tricky task and require some additional investigation.

If that's not possible/easy to do, maybe let's add a header to the coverage report printed to make it clear that the coverage comes from unit tests?

Done

Also we should be prepared that a connector was not written in python and will not have a coverage output

Updated, in case that would be no python unittest output it update comment with "No python unittest run" line.

@avida
Copy link
Contributor Author

avida commented Sep 15, 2021

/test connector=source-braintree

🕑 source-braintree https://github.com/airbytehq/airbyte/actions/runs/1236574814
✅ source-braintree https://github.com/airbytehq/airbyte/actions/runs/1236574814
Python unittest coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        45     45     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  776    430    45%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                           Stmts   Miss  Cover
	 ------------------------------------------------------------------
	 source_braintree/__init__.py                       2      0   100%
	 source_braintree/schemas/__init__.py               8      0   100%
	 source_braintree/schemas/cards.py                 86      0   100%
	 source_braintree/schemas/common.py                42      0   100%
	 source_braintree/schemas/customer.py              28      0   100%
	 source_braintree/schemas/discount.py              13      0   100%
	 source_braintree/schemas/dispute.py               37      0   100%
	 source_braintree/schemas/merchant_account.py      29      0   100%
	 source_braintree/schemas/plan.py                  21      0   100%
	 source_braintree/schemas/subscription.py          35      0   100%
	 source_braintree/schemas/transaction.py           99      0   100%
	 source_braintree/source.py                        18      0   100%
	 source_braintree/spec.py                          23      0   100%
	 source_braintree/streams.py                      100      0   100%
	 ------------------------------------------------------------------
	 TOTAL                                            541      0   100%

@avida
Copy link
Contributor Author

avida commented Sep 15, 2021

/test connector=source-mysql

🕑 source-mysql https://github.com/airbytehq/airbyte/actions/runs/1236575198
✅ source-mysql https://github.com/airbytehq/airbyte/actions/runs/1236575198
No python unittest run

@avida
Copy link
Contributor Author

avida commented Sep 15, 2021

/test connector=source-braintree

🕑 source-braintree https://github.com/airbytehq/airbyte/actions/runs/1236660432
✅ source-braintree https://github.com/airbytehq/airbyte/actions/runs/1236660432
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        45     45     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  776    430    45%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                           Stmts   Miss  Cover
	 ------------------------------------------------------------------
	 source_braintree/__init__.py                       2      0   100%
	 source_braintree/schemas/__init__.py               8      0   100%
	 source_braintree/schemas/cards.py                 86      0   100%
	 source_braintree/schemas/common.py                42      0   100%
	 source_braintree/schemas/customer.py              28      0   100%
	 source_braintree/schemas/discount.py              13      0   100%
	 source_braintree/schemas/dispute.py               37      0   100%
	 source_braintree/schemas/merchant_account.py      29      0   100%
	 source_braintree/schemas/plan.py                  21      0   100%
	 source_braintree/schemas/subscription.py          35      0   100%
	 source_braintree/schemas/transaction.py           99      0   100%
	 source_braintree/source.py                        18      0   100%
	 source_braintree/spec.py                          23      0   100%
	 source_braintree/streams.py                      100      0   100%
	 ------------------------------------------------------------------
	 TOTAL                                            541      0   100%

@jrhizor jrhizor temporarily deployed to more-secrets September 15, 2021 07:27 Inactive
@avida avida merged commit 7a6da86 into master Sep 15, 2021
@avida avida deleted the drezchykov/code-coverage branch September 15, 2021 07:50
htrueman added a commit that referenced this pull request Sep 17, 2021
* Update check connection method

* #5796 silence printing full config when config validation fails (#5879)

* - #5796 silence printing full config when config validation fails

* fix unit tests after config validation check changes

Co-authored-by: Marcos Eliziario Santos <[email protected]>

* Format google-search-console schemas (#6047)

* Update ads_insights.json (#5946)

fix ads_insights schema according to [facebook docs](https://developers.facebook.com/docs/marketing-api/reference/adgroup/insights/) and my own data

* Bump connectors version + update docs (#6060)

* 🐛 Source Facebook Marketing: Convert values' types according to schema types (#4978)

* Convert values' types according to schema types

* Put streams back to `configured_catalog.json`

Put back `ads_insights` and `ads_insights_age_and_gender` streams.

* Pickup changes from #5946

* Implement change request + fix previous PR

* Update schema

* Remove items_type from convert_to_schema_types()

* Bump connectors version

* add oauth to connector_base dependencies (#6064)

* use spec when persisting source configs (#6036)

* switch most usages of writing sources to using specs

* fix other usages

* fix test

* only wait on the server in the scheduler, not the worker

* fix

* rephrase sanity check and remove stdout

* 🎉 Source Stripe: Add `PaymentIntents` stream (#6004)

* Add `PaymentIntents` stream

* Update docs

* Implement change request + few updates

Split `source.py` file into `source.py` and `streams.py` files.
Update `payment_intents.json` file.

* Bump connectors version + update docs

* Add skeleton for databricks destination (#5629)

Co-authored-by: Liren Tu <[email protected]>
Co-authored-by: LiRen Tu <[email protected]>

* Revert "Add skeleton for databricks destination (#5629)" (#6066)

This reverts commit 79256c4.

* 🎉 New Destination: Databricks (#5998)

Implement new destination connector for databricks delta lake.
Resolves #2075.

Co-authored-by: George Claireaux <[email protected]>
Co-authored-by: Sherif A. Nada <[email protected]>

* Source PostHog: add support for self-hosted instances (#6058)

* publish #6058 (#6059)

* Destination Kafka: correct spec json and data types in config (#6040)

* correct spec json and data types in config

* bump version

* correct tests

* correct config parser NPE

* format files

Co-authored-by: Marcos Marx <[email protected]>

* Fix or delete broken links (#6069)

* Fix more doc issues (#6072)

* 🎉 Added optional platform flag for build image script (#6000)

* Fix dependabot security alert. (#6073)

* Pin set value to greater than 4.0.1 to fix security warning.

* Format the rest of the connectors.

* add coverage report (#6045)

Co-authored-by: Dmytro Rezchykov <[email protected]>

* Fix the format of the data returned by Google Ads oauth to match the config accepted by the connector (#6032)

* update salesforce docs (#6081)

* 🎉 Source Github: add caching for all streams (#5949)

* Source Github: add checking for all streams

* bump version, update changelogs

* Disable automatic migration acceptance test (#5988)

- The automatic migration acceptance test no longer works because of the new Flyway migration system.
- The file-based migration system is being deprecated.

* 🎉 CDK: Add requests native authenticator support (#5731)

* Add requests native auth class

* Update init file.
Update type annotations.
Bump version.

* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.

* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md

* Add requests native authenticator tests

* Add CDK requests native __call__ method tests.
Update CHANGELOG.md

* Add outdated auth deprication messages

* Update requests native auth __call__ method tests

* Bump CDK version to 0.1.20

* Interface changes to support separating secrets from the config (#6065)

* Interface changes to support separating secrets from the config
* Cleanup from PR comments and whitespace

* Update log message for empty env variable (#6115)

Co-authored-by: Jared Rhizor <[email protected]>

* Bump Airbyte version from 0.29.17-alpha to 0.29.18-alpha (#6125)

Co-authored-by: davinchia <[email protected]>

* return auth spec in the API when getting definition specification (#6121)

* Ignore python test coverage files (#6144)

* CDK: support nested refs resolving (#6044)

Co-authored-by: Dmytro Rezchykov <[email protected]>

* feat: path for nested fields (#6130)

* feat: path for nested fields

* fix: clipRule error

* fix: remove field name

* Fix request middleware for ConnectionService (#6148)

* Jamakase/update onboarding flow (#5656)

* Doc explains normalization full-refresh implications (#6097)

* update docs

* add info in quickstart connection page

* update abhi comments

Co-authored-by: Marcos Marx <[email protected]>

* Fix migration validation issue (#6154)

Resolves #6151.

* Bump Airbyte version from 0.29.18-alpha to 0.29.19-alpha (#6156)

Co-authored-by: tuliren <[email protected]>

* Add information on which destinations support Incremental - Deduped History in their docs (#6031)

Co-authored-by: Abhi Vaidyanatha <[email protected]>

* Update Airbyte Spec acknowledgements. (#6155)

Co-authored-by: Abhi Vaidyanatha <[email protected]>

* Update new integration request

* Add back the migration acceptance test (#6163)

* 🎉 Create a Helm Chart For Airbyte (#5891)

See number #1868. This creates an initial helm chart for installing Airbyte in Kubernetes to make it easier for users who are more familiar with helm. It also includes GitHub actions to help continually test that the chart works in the most basic case.
All of the templates are based off of the kustomize folder, but minio and postgres have been removed in favor of adding the bitnami helm charts as dependencies since they have an active community and allow easily tweaking their install.

* Fix OAuth Summary strings (#6143)

Co-authored-by: Marcos Eliziario Santos <[email protected]>
Co-authored-by: Marcos Eliziario Santos <[email protected]>
Co-authored-by: oleh.zorenko <[email protected]>
Co-authored-by: Mauro <[email protected]>
Co-authored-by: Sherif A. Nada <[email protected]>
Co-authored-by: Jared Rhizor <[email protected]>
Co-authored-by: George Claireaux <[email protected]>
Co-authored-by: Liren Tu <[email protected]>
Co-authored-by: LiRen Tu <[email protected]>
Co-authored-by: coeurdestenebres <[email protected]>
Co-authored-by: Marcos Marx <[email protected]>
Co-authored-by: Marcos Marx <[email protected]>
Co-authored-by: Harsha Teja Kanna <[email protected]>
Co-authored-by: Davin Chia <[email protected]>
Co-authored-by: Dmytro <[email protected]>
Co-authored-by: Dmytro Rezchykov <[email protected]>
Co-authored-by: Yevhenii <[email protected]>
Co-authored-by: Jenny Brown <[email protected]>
Co-authored-by: davinchia <[email protected]>
Co-authored-by: Iakov Salikov <[email protected]>
Co-authored-by: Artem Astapenko <[email protected]>
Co-authored-by: tuliren <[email protected]>
Co-authored-by: Abhi Vaidyanatha <[email protected]>
Co-authored-by: Abhi Vaidyanatha <[email protected]>
Co-authored-by: Jonathan Stacks <[email protected]>
Co-authored-by: Christophe Duong <[email protected]>
htrueman added a commit that referenced this pull request Sep 17, 2021
* Add GET_FBA_INVENTORY_AGED_DATA data

* Add GET_MERCHANT_LISTINGS_ALL_DATA stream support

* Update schemas

* Update configured_catalog.json

* Update connector to airbyte-cdk

* Add amazon seller partner test creds

* Update state sample files

* Apply code format

* Update acceptance-test-config.yml

* Add dummy integration test

* Refactor auth signature.
Update streams.py

* Remove print_function import from auth.py

* Refactor source class.
Add pydantic spec.
PR fixes.

* Add dummy integration test

* Typing added.
Add _create_prepared_request docstring.

* Add extra streams and schemas

* Update docs and spec

* Post merge code fixes

* Fix test setup

* Fix test setup

* Add sample_state.json

* Update reports streams logics.
Update test and config files.

* Update tests config.
Small code style fixes.

* Add reports stream slices.
Update check_connection method.

* Post review fixes.

* Streams update

* Add reports document retrieval and decrypting.
Update schemas and configs.

* Add CVS parsing into result rows

* Update ReportsAmazonSPStream class to be the child of Stream class.
Update GET_FLAT_FILE_OPEN_LISTINGS_DATA and GET_MERCHANT_LISTINGS_ALL_DATA schemas.

* Schema updates

* Source check method updated

* Update ReportsAmazonSPStream retry report logics

* Update check_connection source method

* Update reports read_records method.
Update report schemas.

* Update streams.py

* Update acceptance tests config.
Add small code fixes.

* Update report read_records logics

* Add reports streams rate limit handling logics.
Add rate limit unit tests.

* Source Amazon SP: Update reports streams logics. (#5311)

* Update check connection method

* #5796 silence printing full config when config validation fails (#5879)

* - #5796 silence printing full config when config validation fails

* fix unit tests after config validation check changes

Co-authored-by: Marcos Eliziario Santos <[email protected]>

* Format google-search-console schemas (#6047)

* Update ads_insights.json (#5946)

fix ads_insights schema according to [facebook docs](https://developers.facebook.com/docs/marketing-api/reference/adgroup/insights/) and my own data

* Bump connectors version + update docs (#6060)

* 🐛 Source Facebook Marketing: Convert values' types according to schema types (#4978)

* Convert values' types according to schema types

* Put streams back to `configured_catalog.json`

Put back `ads_insights` and `ads_insights_age_and_gender` streams.

* Pickup changes from #5946

* Implement change request + fix previous PR

* Update schema

* Remove items_type from convert_to_schema_types()

* Bump connectors version

* add oauth to connector_base dependencies (#6064)

* use spec when persisting source configs (#6036)

* switch most usages of writing sources to using specs

* fix other usages

* fix test

* only wait on the server in the scheduler, not the worker

* fix

* rephrase sanity check and remove stdout

* 🎉 Source Stripe: Add `PaymentIntents` stream (#6004)

* Add `PaymentIntents` stream

* Update docs

* Implement change request + few updates

Split `source.py` file into `source.py` and `streams.py` files.
Update `payment_intents.json` file.

* Bump connectors version + update docs

* Add skeleton for databricks destination (#5629)

Co-authored-by: Liren Tu <[email protected]>
Co-authored-by: LiRen Tu <[email protected]>

* Revert "Add skeleton for databricks destination (#5629)" (#6066)

This reverts commit 79256c4.

* 🎉 New Destination: Databricks (#5998)

Implement new destination connector for databricks delta lake.
Resolves #2075.

Co-authored-by: George Claireaux <[email protected]>
Co-authored-by: Sherif A. Nada <[email protected]>

* Source PostHog: add support for self-hosted instances (#6058)

* publish #6058 (#6059)

* Destination Kafka: correct spec json and data types in config (#6040)

* correct spec json and data types in config

* bump version

* correct tests

* correct config parser NPE

* format files

Co-authored-by: Marcos Marx <[email protected]>

* Fix or delete broken links (#6069)

* Fix more doc issues (#6072)

* 🎉 Added optional platform flag for build image script (#6000)

* Fix dependabot security alert. (#6073)

* Pin set value to greater than 4.0.1 to fix security warning.

* Format the rest of the connectors.

* add coverage report (#6045)

Co-authored-by: Dmytro Rezchykov <[email protected]>

* Fix the format of the data returned by Google Ads oauth to match the config accepted by the connector (#6032)

* update salesforce docs (#6081)

* 🎉 Source Github: add caching for all streams (#5949)

* Source Github: add checking for all streams

* bump version, update changelogs

* Disable automatic migration acceptance test (#5988)

- The automatic migration acceptance test no longer works because of the new Flyway migration system.
- The file-based migration system is being deprecated.

* 🎉 CDK: Add requests native authenticator support (#5731)

* Add requests native auth class

* Update init file.
Update type annotations.
Bump version.

* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.

* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md

* Add requests native authenticator tests

* Add CDK requests native __call__ method tests.
Update CHANGELOG.md

* Add outdated auth deprication messages

* Update requests native auth __call__ method tests

* Bump CDK version to 0.1.20

* Interface changes to support separating secrets from the config (#6065)

* Interface changes to support separating secrets from the config
* Cleanup from PR comments and whitespace

* Update log message for empty env variable (#6115)

Co-authored-by: Jared Rhizor <[email protected]>

* Bump Airbyte version from 0.29.17-alpha to 0.29.18-alpha (#6125)

Co-authored-by: davinchia <[email protected]>

* return auth spec in the API when getting definition specification (#6121)

* Ignore python test coverage files (#6144)

* CDK: support nested refs resolving (#6044)

Co-authored-by: Dmytro Rezchykov <[email protected]>

* feat: path for nested fields (#6130)

* feat: path for nested fields

* fix: clipRule error

* fix: remove field name

* Fix request middleware for ConnectionService (#6148)

* Jamakase/update onboarding flow (#5656)

* Doc explains normalization full-refresh implications (#6097)

* update docs

* add info in quickstart connection page

* update abhi comments

Co-authored-by: Marcos Marx <[email protected]>

* Fix migration validation issue (#6154)

Resolves #6151.

* Bump Airbyte version from 0.29.18-alpha to 0.29.19-alpha (#6156)

Co-authored-by: tuliren <[email protected]>

* Add information on which destinations support Incremental - Deduped History in their docs (#6031)

Co-authored-by: Abhi Vaidyanatha <[email protected]>

* Update Airbyte Spec acknowledgements. (#6155)

Co-authored-by: Abhi Vaidyanatha <[email protected]>

* Update new integration request

* Add back the migration acceptance test (#6163)

* 🎉 Create a Helm Chart For Airbyte (#5891)

See number #1868. This creates an initial helm chart for installing Airbyte in Kubernetes to make it easier for users who are more familiar with helm. It also includes GitHub actions to help continually test that the chart works in the most basic case.
All of the templates are based off of the kustomize folder, but minio and postgres have been removed in favor of adding the bitnami helm charts as dependencies since they have an active community and allow easily tweaking their install.

* Fix OAuth Summary strings (#6143)

Co-authored-by: Marcos Eliziario Santos <[email protected]>
Co-authored-by: Marcos Eliziario Santos <[email protected]>
Co-authored-by: oleh.zorenko <[email protected]>
Co-authored-by: Mauro <[email protected]>
Co-authored-by: Sherif A. Nada <[email protected]>
Co-authored-by: Jared Rhizor <[email protected]>
Co-authored-by: George Claireaux <[email protected]>
Co-authored-by: Liren Tu <[email protected]>
Co-authored-by: LiRen Tu <[email protected]>
Co-authored-by: coeurdestenebres <[email protected]>
Co-authored-by: Marcos Marx <[email protected]>
Co-authored-by: Marcos Marx <[email protected]>
Co-authored-by: Harsha Teja Kanna <[email protected]>
Co-authored-by: Davin Chia <[email protected]>
Co-authored-by: Dmytro <[email protected]>
Co-authored-by: Dmytro Rezchykov <[email protected]>
Co-authored-by: Yevhenii <[email protected]>
Co-authored-by: Jenny Brown <[email protected]>
Co-authored-by: davinchia <[email protected]>
Co-authored-by: Iakov Salikov <[email protected]>
Co-authored-by: Artem Astapenko <[email protected]>
Co-authored-by: tuliren <[email protected]>
Co-authored-by: Abhi Vaidyanatha <[email protected]>
Co-authored-by: Abhi Vaidyanatha <[email protected]>
Co-authored-by: Jonathan Stacks <[email protected]>
Co-authored-by: Christophe Duong <[email protected]>

* Bump source version.
Update source docs.

* Mock time.sleep in test_reports_stream_send_request_backoff_exception test

* Acceptance test basic_read test disabled

Co-authored-by: Marcos Eliziario Santos <[email protected]>
Co-authored-by: Marcos Eliziario Santos <[email protected]>
Co-authored-by: oleh.zorenko <[email protected]>
Co-authored-by: Mauro <[email protected]>
Co-authored-by: Sherif A. Nada <[email protected]>
Co-authored-by: Jared Rhizor <[email protected]>
Co-authored-by: George Claireaux <[email protected]>
Co-authored-by: Liren Tu <[email protected]>
Co-authored-by: LiRen Tu <[email protected]>
Co-authored-by: coeurdestenebres <[email protected]>
Co-authored-by: Marcos Marx <[email protected]>
Co-authored-by: Marcos Marx <[email protected]>
Co-authored-by: Harsha Teja Kanna <[email protected]>
Co-authored-by: Davin Chia <[email protected]>
Co-authored-by: Dmytro <[email protected]>
Co-authored-by: Dmytro Rezchykov <[email protected]>
Co-authored-by: Yevhenii <[email protected]>
Co-authored-by: Jenny Brown <[email protected]>
Co-authored-by: davinchia <[email protected]>
Co-authored-by: Iakov Salikov <[email protected]>
Co-authored-by: Artem Astapenko <[email protected]>
Co-authored-by: tuliren <[email protected]>
Co-authored-by: Abhi Vaidyanatha <[email protected]>
Co-authored-by: Abhi Vaidyanatha <[email protected]>
Co-authored-by: Jonathan Stacks <[email protected]>
Co-authored-by: Christophe Duong <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unittest code coverage for python connectors
5 participants