[ISSUE #20771] limiting the number of requests performed to the backe… #21525
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nd without flag
maxi297
force-pushed
the
issue-20771_limit-number-of-requests-to-endpoint
branch
from
57bd846
to
00a3558
Compare
maxi297
commented
Jan 18, 2023
airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/default_paginator.py
Show resolved
Hide resolved
girarda
approved these changes
Jan 19, 2023
airbyte-cdk/python/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
Outdated
Show resolved
Hide resolved
brianjlai
approved these changes
Jan 20, 2023
airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/default_paginator.py
Outdated
Show resolved
Hide resolved
airbyte-cdk/python/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
Outdated
Show resolved
Hide resolved
* [ISSUE #20771] adding slices to connector builder read request * [ISSUE #20771] formatting * [ISSUE #20771] set flag when limit requests reached (#21619) * [ISSUE #20771] set flag when limit requests reached * [ISSUE #20771] assert proper value on test read objects __init__ * [ISSUE #20771] code review and fix edge case
girarda
approved these changes
Jan 23, 2023
airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/default_paginator.py
Outdated
Show resolved
Hide resolved
airbyte-cdk/python/unit_tests/sources/declarative/test_manifest_declarative_source.py
Show resolved
Hide resolved
airbyteio
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
airbyteio
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
Airbyte Code Coverage
|
This was referenced Jan 27, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…nd without flag
What
This addresses the first part of #20771 where we want to limit the number of requests done to the back. As for this PR, the flag is excluded and another PR will follow.
How
Change the behaviour at runtime of the paginator and the simple retriever.
Recommended reading order
airbyte-cdk/python/airbyte_cdk/sources/declarative/retrievers/simple_retriever.pyairbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/default_paginator.py🚨 User Impact 🚨
Users of the connector builder should see a great improvement in the response time when reading a stream. However, the response can now be partial if there is more than 5 slices of 5 pages