Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Source Github: add custom pagination size for large streams #9664

Merged

Conversation

yevhenii-ldv
Copy link
Contributor

@yevhenii-ldv yevhenii-ldv commented Jan 20, 2022

Bug-fix Request

Fixes https://github.com/airbytehq/oncall/issues/75*


Reason

Reduced pagination size for heavily loaded streams, as this can be a problem for reading these streams using the API (the case was in the PR #4708 where the page size values were already reduced for large streams).
I also added a parameter to the specification that allows the user to configure this parameter depending on the degree of load of his repositories

Confirmation

  • Was reproduced locally

Described above

How does the code change in the PR fix the issue?

Described above


Recommended reading order

  1. x.java
  2. y.python

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here
  • After the new connector version is published, connector version bumped in the seed directory as described here
  • Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

@yevhenii-ldv
Copy link
Contributor Author

yevhenii-ldv commented Jan 20, 2022

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1725753957
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1725753957
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                        Stmts   Miss  Cover
	 -----------------------------------------------
	 source_github/__init__.py       2      0   100%
	 source_github/source.py        78     35    55%
	 source_github/streams.py      369    162    56%
	 -----------------------------------------------
	 TOTAL                         449    197    56%

@github-actions github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Jan 20, 2022
@yevhenii-ldv yevhenii-ldv temporarily deployed to more-secrets January 20, 2022 21:51 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets January 20, 2022 21:52 Inactive
@yevhenii-ldv yevhenii-ldv temporarily deployed to more-secrets January 21, 2022 12:57 Inactive
@yevhenii-ldv yevhenii-ldv temporarily deployed to more-secrets January 21, 2022 13:06 Inactive
@yevhenii-ldv
Copy link
Contributor Author

yevhenii-ldv commented Jan 21, 2022

/publish connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1728816467
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1728816467

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets January 21, 2022 13:11 Inactive
@yevhenii-ldv yevhenii-ldv merged commit 2ddf0bc into master Jan 21, 2022
@yevhenii-ldv yevhenii-ldv deleted the ykurochkin/add-custom-pagination-size-for-large-streams branch January 21, 2022 13:22
@yevhenii-ldv yevhenii-ldv temporarily deployed to more-secrets January 21, 2022 13:23 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants